Interesting write up, but what it really needs is a nice task vs tools table at the end as a "conclusion".

Someone reading the blog the first time might well be willing to read through the large amount of text.

But returning to it at a later date ? A quick reference TL;DR table would not go amiss IMHO.

Well, I find this post looks good, but a like lot of 'data for developers' posts it's just a list of tools. As if a collection of tools banded together actually makes your customer successful. What's missing? 1. There's nothing about deployment. How do I take this collection of tools and code and actually deploy it into production, or actually regression test it functionally? How do I make a small change in a database table and not have a massive regression? How do you do that automatically? How do you do it quickly? 2. It's cursory on testing. One of the biggest differences from a software developer to a data engineer is that your data providers give you crap data all the time. It could break. How do you test data? How do you get adequate test coverage? These things are essential for software developers and are actually doubly essential for data engineers and building analytics systems. 3. It's what success looks like. It's not just about a collection of tech; it's about making your customers successful. What does it mean to deliver good insight? How do you do it? How do you measure customer success, and measure your success? As a team, you wouldn't talk about software engineering without mentioning DevOps or DORA metrics. There's nothing here about that.

sigh

excluding Denodo from the list sows this is more of a non-enterprise guide to data management and tools. There is only one real semantic layer that can cover operational and historical data and thats Denodo. If you use snowflake horizon or unity, all the data needs to be loaded first and not real time.

Good to finally know the difference between a lake and a warehouse

This was great, thanks for writing it up. Even as someone in the data space for a long time, I learned quite a bit.

Apache Avro has 2 encodings: binary AND json.

What about PI?

Now I'll be thinking of "L" in ETL as "Land" and not "Load". Although the article doesn't propose that but uses a lot of "Land" terminology. "Load" => "load where? or FROM where?" - ambiguous "Land" => "land where?" - clear

Data is the new oil