Every number in Google Analytics is wrong

Posted by hricha_shandily 10/28/2024

Every number in Google Analytics is wrong(plausible.io)

10 points | 10 comments

Arnt 10/28/2024|

A lot of text there, but more deflecting than substantive. The central claim seems to be this: «In fact, according to the Orbit Media study shared above, the accuracy of GA4 as compared to Plausible –– due to the cookie consent banner’s presence –– was exactly at 55.6%. More than half of the data was missing, making Google Analytics stats not reliable!»

I think that tries to say that if Plausible generates graphs of 1000 pageviews, GA's graphs are based on 556 of the 1000, and that this makes GA's graphs half as accurate.

Say again? Basing graphs on a sample of half the users doesn't reduce accuracy very much in my experience (which is old). The per-day graphs and reports I generated looked very much like their per-week equivalent, despite being based on 15% of the data.

There's more blah, about how GA is late for example. I didn't really notice any difference between yesterday's data and that from a few days ago. Who cares about such details, really.

Some sentences I didn't notice in the article: "This lack of accuracy is a problem in practice, because …" or "Plausible detects problems with your funnels that GA4 doesn't, because …" or anything like that. They just brag about accuracy and promptness, without any argument that the accuracy or promptness makes a difference.

Sigh.

viraptor 10/28/2024|

> Basing graphs on a sample of half the users doesn't reduce accuracy very much

On a random sample - shouldn't reduce it too much. On a sample with the selection biased in almost every category - it's going to be entirely different data as a result.

Arnt 10/28/2024||

Yes, indeed. Now, does GA's sample actually have significant bias? If it does, then showing the practical effects should be easy for experts at a competing company, don't you too think so?

viraptor 10/28/2024||

> Now, does GA's sample actually have significant bias?

Yes it does. Both for the adblock https://backlinko.com/ad-blockers-users and the consent banner https://www.advance-metrics.com/en/blog/cookie-behaviour-stu...

Arnt 10/29/2024||

I looked at the page page's numbers and arrived at the opposite conclusion.

Take the top graph on the first page: countries. I've worked at two places where I had insight into web access and logs and such. I shouldn't go into too much detail so I'll pick a different pair of neighbour countries: if France and Germany were target markets, the difference in ad blocker usage would not change any decisions. Germany has a higher share ad blockers, and because of that a graph comparing the two countries might be 3% away from reality, but 3% wouldn't affect any decision. 3% difference in flows of money might, but 3% on something that doesn't reflect money? Not a chance.

viraptor 10/29/2024||

Sure, if you have a list of countries that spans 10-40%, but choose two entries that are 1% apart... then I guess you wouldn't see the difference. But I'm not sure you're providing a great example here.

Arnt 10/29/2024||

Thinking back on the discussions and decisions I remember, they all concerned countries that were similar to each other, and in a sense they were discussed precisely because of that. In theory you might have Indonesia as your home base and pick Ghana as your second market (the most different countries in that table), but in my experience people pick a country near/similar to the one they already know, where they already have some success. So yes, I think France/Germany a great example.

Further, I suspect that because they're similar in many other ways, they're also similar on that graph. It's not just a coincidence.

chrisjj 10/28/2024||

> Plausible will reduce your page weight

I believe it.

> and will prevent your site from loading slow.

I don't! :)

hricha_shandily 10/28/2024|

A site won't get slow, from Plausible's side (that's the claim). Other factors can definitely affect the overall performance.

chrisjj 10/28/2024||

> that's the claim

That should be the claim.

dzarasovlloyd 10/28/2024||

[dead]