It is a naive question but I do wonder how hard it is to scrape the web and build web indexes that are shared via bit torrent and self host a tfidf traditional engine using a distributed effort. How big the web really is?

Is there a lot of additional secret sauce that made google work well in its prime?

I don't think it hurts thinking (at least I haven't noticed anything on myself yet), but it hurts the internet a lot, and may even destroy it in parallel to LLM. Online search became completely unusable, and it started well before chatgpt arrived.

Now when I am looking for information on this or that I'm obliged to append reddit, Wikipedia or YouTube in my search to hope to have something with real content (and I hate watching videos to be kept informed and I feel there is more and more LLM generated content there also).

I even have a subscription to chatgpt now that I use more and more as a replacement for a Google search, or at least as a supplement.

I don't like it very much because in the best case I have only a handful of sources that it selected instead of being able to look by myself at all the possible sources, but it is still better than spending hours finding something that is not SEO-slop.

I'm convinced it will never improve because it helps google and the others push their AI stuff, and they desperately need it to justify the billions poured in it, but also because it gave them a much deeper control on what is provided to the users.

Yeah, I agree with what you are saying, people seem to take the AI response at face value. I answer lies in the users recognizing that HAL and others can, and right now often do, make mistakes.

I‘m outsourcing all tedious searches to ChatGPT now. Anytime I try searching anything non-obvious, it seems to be buried in the Nth page. Both Google and DDG have this problem. Chat spares me the brain power of sifting through the endless slop and grift, and I can easily click through and read the actual content.

[dead]

[dead]