Top
Best
New

Posted by OlegWock 5 hours ago

Guide to data tools landscape for developers(sinja.io)
91 points | 28 comments
aleda145 1 hour ago|
As a data engineering person I can say that this is a great write up!

Some thoughts:

A "bubbling" topic right now is conversational analytics (i.e. talk to your data). There has been an explosion of tools in the last 6 months. YC is backing one too: https://getnao.io/

I feel like pandas is also somewhat frowned upon, the industry has moved on from that. Most SQL tools can now do everything that we could only do with pandas.

In my network everyone is talking about DuckDB. As long as you are under a 1TB it will have everything you need. I think most people should start with that vs locking themselves into something like Snowflake

estetlinus 1 hour ago||
I love what I am hearing. I still see a lot of engineers using pandas, but it is such a horrible tools. You usually find an abandoned notebook with 100s of ”df_final_2” with sequential wrangling, making it impossible to understand what’s happening. Notebooks are also horrible for the same reason IMO.

With chat-your-data you have Hex, Claude + MCP, snowflake, Databricks etc… everyone’s in on it.

sdpy 1 hour ago||
Just to add, people in my network have been talking about polars (as an alternative to pandas) and other dataframe libraries. They're much easier to use now thanks to the Narwhals compatibility layer (for example, Narwhals was recently added as a dependency to scikit-learn).
jbonatakis 4 hours ago||
> A data warehouse on the other hand is an OLAP database and is optimized to work on columns

A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.

datadrivenangel 2 hours ago|
You are pedantically correct but technically wrong, as even optimized postgres is going to suffer on analytical patterns without extensions. With extensions (DuckDB or Citus) you can do large aggregations, but regular postgres at medium/large scale (billions of rows, 100s of GB) starts having a lot of foot guns and complex babying to do analytics. A bunch of indexes and you'll be fine though.
otter-in-a-suit 56 minutes ago||
This is a fantastic primer.

A few suggestions:

- Add a "last updated" note, since this space changes often (see the prefect/dagster situation that just happened)

- Add a note about MCPs and other LLM-driven tools and features are becoming more and more important (e.g. hex.ai or the various MCPs shipped with some of the tools you mention, such as OpenMetadata)

- Maybe organize the various tools you mention briefly by their license/model (lots of them can be self hosted, some are SaaS only), since a fully self-hosted data platform is (at least for now...) very much feasible

I also wish more people would talk more about the "engineering" part of "data engineering". I've seen way too many people who claim a title like "data engineer" but lack the fundamentals of building software and are really just copy-paste scripts together.

What I'd love more DEs to think about are things like {unit,integration,e2e,performance} tests, deployments, infrastructure, networking, monitoring (you do touch on that), and all the other things a regular SWE is expected to have at least basic competency in at a certain level. For instance, tools like dbt natively support tests, but people need to write them. Or how you don't have to click-ops Airbyte, there's a terraform provider etc.

hermitcrab 35 minutes ago||
Interesting summary. I'm surprised you didn't mention any desktop ETL tools (such as Alteryx, Knime, Easy Data Transform).
flarco 2 hours ago||
Nice guide. One addition in the ingestion section: sling (https://slingdata.io/). it's a single go binary, cli-first.
ozhero 1 hour ago||
This is an excellent well written article with just the information you need to get a basic understanding of the field.
stronglikedan 9 minutes ago|
beep boop
datadrivenangel 1 hour ago||
"Popular metadata catalog solutions are Hive Metastore, AWS Glue Data Catalog, and Unity Catalog from Databricks.

For query engine you can use, for example, Apache Spark, Trino, or Amazon Athena."

DuckDB is eating the query engines and catalogs. Really could use more coverage on how DuckDB is changing the data tools landscape.

estetlinus 1 hour ago|
Can you suggest any good resources for your statement? I know DuckDB is hyped, but didn’t know they’re eating engines and catalogs.
chrisweekly 3 hours ago||
Great post! Also, I dig your site -- it's attractive and highly usable, and the "personal" toggle in the footer is a clever affordance I haven't seen before for separating professional content.
MNeverOff 4 hours ago||
It's a good all-round primer, well written. Would love to hear more about larger-than-memory tasks and running local Dask clusters. I processed many-a-dataset that way that would normally make pandas choke.
jrhizor 1 hour ago|
Really a great layout of the space! I like how it doesn’t try to enumerate all options.
More comments...