While I usually think that at least having some standard that people agree on I think OpenTelemtry should be dropped.
A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards and a lot better both in terms of less added complexity and the results you get.
I think OpenTelemetry turned metrics into a farce. In many situations even self-rolled telemetry works better even with the added stuff. The annoying thing is that OpenTelemtry is that big standard now one kind of has to to add compatibility. So please, if you write software, make sure you don't lock yourself into OTel.
> A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards
By all metrics (hah), Prometheus is the more popular solution and is the de-facto standard, as far as I know.
If you get rid of that, and just pass dependencies around, create some appropriate local abstraction around them.. the tooling, be it datadog or honeycomb does a great job making it useful. Can't really say the same for grafana, but ymmv - depending on budget
It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:
1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.
2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).
My life of working with it got easier when I started just looking at the actual code, using network level tools like nc/tcpdump, making extensive use of the debug exporter, and almost ignoring the docs entirely except as a basic summary of what a thing does.
I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects
Every time I share your blog (and I share it a lot) I tell people:
"This guy started a blog in 2024. Wrote three posts and all three of them would still make my top ten list of 'greatest posts on observability' today".
'A practitioner's guide to wide events' especially is still my number 1.
The net of this is, otel is a very flexible system you can use and adapt in all kinds of ways and while the spec is important, using the toolkit to FAFO yourself, ahead of any beaten path, should really be encouraged. That's the message I'd want to see being radiated out about otel.
The only choices you get is full auto instrumentation, which breaks most non-trivial apps, or zero assistance/documentation.
There is no in-between where I can inject the functionality required in a way that is compatible with the application.
- Running an old version of Golang (older than 1.18 if memory serves), or
- has libraries that the eBPF probes don't like.
And while I like OTel, I agree with the OP that you are absolutely going deep-sea diving if you're going to do anything beyond the examples provided (which is very easy to do!)
If I recall the primary issue was the forced loading of the django settings file by otel.
I get that fully automated instrumentation should be turn-key and the current approach kinda works on basic applications.
But most production django applications are monoliths and generally larger apps. They have non-trivial configuration processes which are often multi step and source settings from multiple places.
Otel should not assume it can just randomly load a the django settings at an arbitrary time point in the startup process.
In one of our apps the MIDDLEWARE setting specifically is dynamically generated and re-ordered based on enabled features. That application's startup process also has multiple stages and the initialisation of django occurs much later, after dependant config loaders etc have been initialised.
What would allow us to integrate with opentelemetry-instrumentation-django much more easily is a set of smaller primitives that we can configure and call at the appropriate time.
opentelemetry-instrumentation-django has (had?) a lot of logic hidden inside a large "inject" function which could not easily be extracted into the constituent parts and applied in a compatible manner.
https://github.com/open-telemetry/opentelemetry-python-contr...
At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!
Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.
Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated, either, and in fact the fact that they're not complicated is their greatest strength. You can and should understand every facet of what they entail. You should build the (very small) shims that they need for your company's framework every time. It's not hard. It's not hard because it's not complicated. The fact that it's not complicated seems to break people's brains. They are accurate because they're simple and they're easy to work with because they're simple, and OTel is neither.
Node exporter runs on my Prometheus server next to Blackbox Exporter. Blackbox Exporter handles TLS expiry metrics.
Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.
Kinda like people hating Obamacare but loving the ACA.
Jaeger does not implement all the OTEL features, though. It's specifically focused on traces rather than metrics.