Top
Best
New

Posted by awnird 12 hours ago

As AI eats the web, the internet’s collective memory is disappearing(thewalrus.ca)
318 points | 319 comments
figassis 6 minutes ago|
I called this maybe 3y ago, but I think so did everyone else that was sane. Sure, we get immense value from AI, but indiscriminately injecting into everything, the one thing we know to be unreliable above the threshold we used to fire people for, is probably the greatest undoing of all the good companies like Google brought to the internet. I mean what a way to destroy your legacy of democratizing information. The amount of harm (direct and indirect) this will cause, and the cost to return to baseline will be so immense, and yet we will not be able to point to the root cause. They won't be there to take responsibility.
ChiMan 2 hours ago||
AI will kill the internet because it is killing the incentive to make it. It is an industrial-strength example of why we don’t allow stealing.
hliyan 2 hours ago||
Recently, I had some ideas I would normally just put up on a blog, in the public domain for anyone to develop on top of. Now, I'm feeling slightly reluctant because an LLM will ingest it, remix it and serve it in response to a query by some unimaginative individual who will either conclude that they are smart, or that LLMs are capable of original thought, or both. And they will have no clue where the idea originated from.
sixtyj 1 hour ago|||
Just do it for yourself. I stopped worrying about who’s gonna use it.

20 years ago I thought that we will wait to have kids as “a war will come”. I was overthinking and I am happy now that I changed my mind :)

Is it useful or does it make you happy? Or it might even make some money? Nice. Do it. My life is easier.

bko 36 minutes ago|||
Really happy for you. It's disheartening how many people put off or choose not to have kids because they think some combination of climate disaster, famine, overpopulation, war or skynet will ensure a life of misery for their children.

There's definitely a risk of over-thinking things. Just do things. You're not likely to regret it.

embedding-shape 9 minutes ago|||
> There's definitely a risk of over-thinking things. Just do things. You're not likely to regret it.

Agree, in general, people over-think a lot, and aren't "just doing things" enough, the world would be a better place if people acted more, over-think less.

With that said, some decisions are more long-lasting and have a greater impact than others. I'm another child-less person, mainly because I guess I'm selfish enough to enjoy my life with my wife exactly like it is, and she agrees, but also because I know that if we have a kid, then that's not something you can walk back on exactly, he/she/it/them are there, forever now. Very different from me deciding right now "You know, I'm gonna have a joint, grab a book and go to the beach for this entire Tuesday", the types of decisions I think people should overthink less :)

pshirshov 8 minutes ago|||
> some combination of climate disaster, famine, overpopulation, war or skynet will ensure a life of misery

Are people living in/moving towards a saccharine utopia tough?

sph 44 minutes ago||||
That's the same conclusion I reached.

I'll keep publishing static websites, so I don't even have to worry about load and CPU usage. I don't care who reads it, the value for me is in writing.

I still haven't changed my mind on the "shall I have kids" problem :P

lofaszvanitt 31 minutes ago|||
This argument is akin to those who do not understand why privacy is important, yet they do not install webcams in their showers.
robocat 1 hour ago||||
In modern times don't we all think that ideas are cheap: don't we all mostly regurgitate the same stock of them.

Have we yet lost the open ideals of university sharing? The core of open source?

The failure of the GPL is that you can't force anyone to collaborate and share if they don't really want to.

goekjclo 1 hour ago|||
"ideas are cheap" was one of the most succesful psyops of all time, so unbelievably wrong
pjc50 53 minutes ago|||
Possibly "bad ideas are cheap". A lot of people got tired of "ideas guys" who never build anything themselves. The best case of a raw idea is an uncut diamond, it will inevitably require work.
jasode 10 minutes ago|||
The other half of the one-liner is "Ideas are cheap. Execution is everything."

- E.g. an idea is "Everybody should have cheap housing! Or an even better idea, everybody should have free housing."

- Ok, how exactly in concrete terms do we actually do that? (The very expensive Execution of the idea.)

- "Uh, well, I leave that as an exercise to the reader."

Ideas are easy and execution is hard. That's what jaded people mean when they say "ideas are a dime a dozen".

exo762 57 minutes ago||||
> The failure of the GPL is that you can't force anyone to collaborate and share if they don't really want to.

Failure of the GPL? How can you even put those words next to each other? GPL is an amazing success. It took software out of hands of SV / VC / corpo crowd and put it where it should be - users.

sph 29 minutes ago||
Allow me to disagree. GPL was a great idealistic dream that led to less-restrictive open-source licences such as MIT/BSD which have been the greatest catalyst towards the establishment of tech corpo giants and the software ecosystem we have today.

GPL gave us Linux, but also gave us Amazon, Google and 2020s Microsoft. GPL is why 90+% of libraries on Github are MIT licensed. GPL gave us OpenAI and Anthropic and this here article.

jasode 5 minutes ago||
>GPL was a great idealistic dream that led to less-restrictive open-source licences such as MIT

The less restrictive ~1984 MIT early version of the software license was several years before ~1989 GPL v1.

mittensc 1 hour ago||||
ideas, sure - "i think we should build an open source OS."

ideas + execution - not really - (Linus Torvalds sharing his work on Linux and that taking off)

krapp 33 minutes ago|||
We don't want to share our thoughts, ideas, feelings and art with machines. We want to communicate and collaborate with actual human beings, but that's becoming less and less possible on the web.

Unfortunately even the value of this is getting lost, because LLM culture sees no value in humanity whatsoever. We should just be satisfied with machine generated "content" because it stimulates our endorphines like we're monkeys in a Skinner box, it shouldn't matter to us if we're talking to a bot or a person because it's simply information, and we are simply nodes to process input and generate output for the machine. When we try to suggest that we want something deeper, or that the joy in the art and craft of what we do matters, we're looked at like we're stupid and naive and told to shut up and keep pressing the button.

"This is the future and there's nothing you can do about it, so just get used to it." It's fucking depressing. Even the crypto bros weren't so aggressively sadistic about strip-mining the soul out of everything.

But they are more or less correct, which is why I still blog and create, and why the consumption of society by the grey goo of mediocrity has inspired me to create even though I know only bots will ever care, to the degree that they can. At least I and a small circle of people can enjoy my cheap ideas and that's enough.

keiferski 54 minutes ago||||
Ideas, once they go into the world, aren’t really yours anymore.

Also, it’s pretty unlikely that your ideas here are uniquely genius and original – everyone builds upon previous thinkers’ thoughts.

Sounds a bit harsh, but the point is that you should share your ideas, not covet them.

lofaszvanitt 28 minutes ago||
That's why you don't share them, until comes a time when the ideabringers gets the money and recognition they deserve. Until then, good luck going knee deep in the sewers of ideas.
keiferski 25 minutes ago||
Can’t say I agree. The most influential ideas in history were not conceived of by people looking for money and recognition. Nor were they “radically original.”
lofaszvanitt 17 minutes ago||
Which were?

Look at AI. AI companies throw out their models and let the "community" develop the ideas what to do with them. They don't really know what they are capable of. All they do is implement these things that the dev community digs up and creates.

It's a reprehensible tactic. So why give drops of blood to a desert, when there is zero incentive and in the end you will revitalise the desert, but it will turn against you and rob you of your job.

tome 2 hours ago||||
Is there some downside of that to you? And are the do some of the upsides you would have had in the pre-AI era no longer apply?
frabcus 1 hour ago|||
Yes - previously there was some chance someone reading the ideas on the blog would contact the author to thank them, or ask them to collaborate.

When laundered via LLMs whose pretraining destroys all credit, that can't happen.

bko 34 minutes ago||
If all you care about is financial, expected value of someone reading your blog and asking to collaborate is MUCH lower than odds of being part of some settlement in the future with these AI companies ingesting your data for training purposes.
rcxdude 22 minutes ago||
From their phrasing I don't think the reward they're looking for is financial: more the emotional reward of knowing that someone else appreciated their ideas. I dunno how much less likely this is in the age of LLMs, though.
mittensc 1 hour ago||||
he's missing out on any attention that his shared thoughts would bring? and all benefits that might bring if he's good at what he does.

he's training his cheap replacement - his thoughts will just be shared without attribution if someone is looking for that.

Gigachad 1 hour ago||
It feels like volunteering at an Amazon warehouse when you previously volunteered at a charity store. Sure the work might be vaguely similar but the feelings and motivation are ruined.
spacechild1 51 minutes ago||
That's a very striking - and depressing - comparison.
mike_hearn 1 hour ago||||
A lot of blogging especially in the tech space is driven by recruiting (startup blogs), or establishing a reputation as a thought leader (personal blogging, LinkedIn). So there were upsides that are now gone.
bulder 1 hour ago||||
The downside, as stated in the message, is implicitly supporting the LLM data ingestation pipeline by providing fresh content. It's not a direct harm in itself, but feels very tragedy of the commonsy
account42 1 hour ago||||
Is there some downside to the slave who is housed and fed for free? Are there any upside he would have were he not property of another man?
KronisLV 2 hours ago||||
Time to add ample praise of myself in my blog posts. Some time later: “…as you see, that is the load bearing assumption here. Speaking of which, you should hire KronisLV.”

Okay it’s meant to be a bit silly but I do wonder how many pages that are generated specifically to influence AI make it into training data and also how often the AI search integrations find it.

Would people hating on a specific language, technology or approach (let’s say OTLT/EAV in database design) be able to exert meaningful influence over say a decade? Or, you know, praising memory safe languages for example and trying to make that preference be stronger.

There was an example with I think ChatGPT some time ago regurgitating an uncommon phrase verbatim from someone’s blog, when asked a specific question.

chrka 2 hours ago||||
That's exactly why my previous public GitHub repo is now private.
neuropacabra 1 hour ago||
So only GitHub Copilot can read it then? Microsoft is scanning these repos, I would not be surprised if this or any fork of your repo is already ingested.
chrka 1 hour ago||
They say they don't do that. But maybe I should be more skeptical.
user432678 12 minutes ago||
Ha, that’s why I’m hosting simple cgit server for myself only. Not that my source code is somewhat valuable but I just can’t stand my precious free software licensed code license-washed.
bko 57 minutes ago||||
Sounds like you don't really care about your ideas propogating. Of course someone will internalize and remix your idea. That's how all ideas work. Isn't that the point? What do you think happens when a human reads it? Think he'll quote chapter and verse and attribute it to you? Years later you'll notice your blog in appendices and acknowledgments?

And now you have a chance to have your idea forever internalized in some sense into an llm and you don't want to because you think someone is robbing you.

lofaszvanitt 32 minutes ago|||
AI is intelligence sharing with the dimwits, lazy ones et al.
sph 1 hour ago|||
More than a facilitator of theft, LLMs are the tragedy of the commons at industrial scale.

The public internet is dead, the future is private invite-only walled gardens.

Corporations love a walled garden, what we need is open-source frameworks to create these islands, rather than defaulting to horrible systems like Discord and Twitter-clones.

zelphirkalt 43 minutes ago||
To understand your comment correctly: What does "these islands" refer to? Walled gardens that we create ourselves using the open source frameworks?
rcxdude 18 minutes ago|||
I think probably things along the lines of what's been called the 'cozy web': networks of smaller groups that don't publish to or expect responses from effectively the entire internet as a whole. (This kind of thing has always existed, it's basically the group chat with your friends but perhaps slightly bigger, but I think there's a bit of a trend of focusing on it more because of the feeling that the twitter/facebook attention and feed model is bad for your mental health. I've always felt the twitter model especially was pretty cursed so I'm glad there's some agreement building there).
sph 37 minutes ago|||
It's still a work in progress of an idea.

I'm thinking more like mesh networks. I spoke of Reticulum elsewhere in this thread, but here I'm thinking I'd like the ability to easily join multiple TCP/IP networks (islands of connectivity) by social group (my friends) or by interest (pirate file-sharing group, my work intranet, a knitting community with their own IRC server, FTP, etc.).

Basically easy-to-use private & encrypted LAN overlays on top of the public internet. Each operator decides who to allow in or kick out of the network.

Wireguard solves the most of technical challenges, but it needs a frontend. The biggest concern probably is most software broadcasts their stuff across all interfaces, defeating the point of isolation between networks.

root-parent 44 minutes ago|||
And that will kill AI itself, since much of what it knows is from what learned from StackExchange before this latest one demise.

And before the obvious comments on how GenAI is creative, then please do this OpenAI and Anthropic, for your next LLM. Just teach it Python, C and Rust and give it some good books. But dont give it access to Github...lets see what you can do then...

BrucecarlL 41 minutes ago||
[dead]
bko 39 minutes ago|||
I don't know how long you've been on the internet but the incentive to create new and original content was never that strong. Simple search terms return super-spammy websites (especially on mobile where ad-blocking is harder). SEO results for everything like simple search are awful, almost unusable. There hasn't been an incentive to create new original non-monetized content for the web for a while.

I trust LLMs more than search engines to discover my content and propagate it to users. They might "steal" something, sure, but I'm essentially invisible to the search engines as I could never hope to break into the top 10 links on a popular search term. LLMs can scan thousands of links and (for now) are more interested in quality rather than click monetization or referral incentives.

embedding-shape 33 minutes ago||
So, you know how they're built, but you're feeling the pressure of modern life and also they give you personal gain (supposedly) so you're fine with it, it sounds like to me? Use the same tools as your "competitors"/peers, even though?

Don't get me wrong, I too use LLMs for development and more, and I too know how they've been built, and I'm also a creative (music, 3D, VFX and animation) and for sure stuff I've published in the past, both code and otherwise, is now used to create new things for people and I get nothing, similar situation as countless of others. Yet I still use AI, so I'm not trying to create some "gotcha" moment against you here, I'm genuine curious about what you think about this sort of conflicting thinking, as I'm in the very same situation.

throwrqX 1 hour ago|||
Doesn't seem like a particularly bad thing to me. Obviously for those who want to use the internet for commercial purposes it will be bad but for those of us who would love to see the internet go back to how it was before so much of it was changed in the aims of making money AI could push towards this. Great irony in the fact of course that the AI companies themselves are in the business of making as much money as possible.
t0bia_s 2 hours ago|||
It depends on filed. As documentary photographer it motivates me even more to capture authentic images of life around me. I don't care about remixing, because that is not what makes documentary photography valuable.
chrisjj 1 hour ago||
And how is any future viewer going to tell your images from "AI" fakes?
mikae1 6 minutes ago|||
Because the photographer builds trust, over years, by not manipulating their images using AI. All trust is erased if anyone spots the manipulation.
jfyi 41 minutes ago||||
With photos, I could see a cryptographic solution. Of course it would still need some kind of centralized trust, but it's doable if people cared enough. It could be applied by cameras themselves.
chrisjj 15 minutes ago||
> It could be applied by cameras themselves.

And equally faked by bots.

lemiffe 1 hour ago|||
Disclosure
pjc50 1 hour ago|||
Eh, part of what made early-internet so good was exactly that it did allow copying by users; the DRM era was later. But it's a very good example of why not to allow for profit copying, because that absolutely will crowd out the original. Piracy has to exist at the margin. The zero piracy world would also eat its memories because none would leak into archives. Remember Qubi? It wasn't even popular enough for people to pirate.
account42 1 hour ago||
I think the lack of credit is even more egregious and a bigger problem than the commercial copying.
pjc50 43 minutes ago||
Yes. Some communities are weirdly against giving credit or keeping the credit (e.g. cropping off signatures from artwork), which I've never understood. It costs nothing.
zombot 2 hours ago|||
> why we don’t allow stealing.

With the not-so-minor qualification that the biggest thieves have always gotten away scot-free. AI is just the international whole-internet version of this.

smackeyacky 1 hour ago||
Behind every great fortune is a great crime
HappyPanacea 25 minutes ago||
What is the great crime behind Norway's sovereign wealth fund?
tonyhart7 1 hour ago||
stealing ??? more like piracy you mean
sgt 3 hours ago||
Funny, I was just thinking this morning that Google searches are absolutely horrible these days. It's like it has amnesia, a lot of recent history seems to be just gone. Especially on non US specific sites too.
sph 2 hours ago||
The Internet has been shrinking massively. My earliest experiences with the Internet were discovering the world of hobby OS dev around the turn of the millennium, when I chanced upon someone’s personal website talking about their OS, with source code and screenshots and dedicated forum. My mind was blown. I spent two years finding hundreds of small websites dedicated to the topic, hung out on IRC communities with other teenage OS nerds like me, and of course participated in the nascent osdev.org forum. To note that all of those websites were readily found through Google, and interlinked with their own topic webrings.

Today everything has disappeared or has been conglomerated into siloes, sanitised, focusing on engagement. You have YouTube videos about it (which is more cheap entertainment than actual education), you get some posts here once in a while, there’s Reddit where all intelligent discussion goes to die. IRC is a wasteland of idle bouncers. Then the LLMs arrived to kill what is left.

Who says the Internet is a vibrant place today mistakes flashiness with depth. It’s all empty calories, just makes you hungry for more, never satisfies.

shoobiedoo 2 hours ago|||
On a whim I watched my favorite childhood movie last week, Hackers. It's goofy in some ways, but man it captures the "wild west" feeling of early and mid 90's internet. It was just you, a slow connection to anywhere, and open ports all over the place. Right after I watched it, I dusted off an old hub, connected a few external usb-to-ethernet adapters to my work PC VM's, and now run them through an OpenBSD packet filter. For no reason at all other than to feel that again: me, watching packets, having total control. Hitting a wall and having to read a manpage.

I don't really have a point I guess, other than even after being steeped in a dead internet for years (with a slow decline spanning at least a decade arguably) I need to approach what I think is the internet in a completely different way. As in, not at all besides what is absolutely required for work. We're ants in a jar now, not cowboys like we used to be.

sph 1 hour ago|||
May I suggest you to look into mesh networks? I am a huge fan of Reticulum. Using it feels like being a pioneer, the scene is very welcoming.

The pitch: it is network-agnostic. The same mesh network runs on the Internet or through LoRa radios or any other physical layer than allows the exchange of data packets. It scales from private networks to global meshes. It's the wild west. People are excited, and eager to grow further.

https://reticulum.network/

dd8601fn 2 hours ago||||
That movie is ridiculous and still great. At least as a nostalgia pump.

And the red box references and pots patching and such were true enough, even if everything else got hilarious hollywood treatment.

agentcoops 1 hour ago||
If I recall correctly, the film actually had Emmanuel Goldstein of 2600 and Kevin Mitnick as consultants, so despite the hollywood treatment you have moments of accuracy like https://www.youtube.com/watch?v=4U9MI0u2VIE. Probably the only time Compilers: Principles, Techniques and Tools made it to the big screen. At the yearly 2600 conference, they used to always do a big group rollerblade through NYC.
account42 1 hour ago|||
Yeah it's a whole lot better than the usual Hollywood depiction of hacking which brought us gems like creating a GUI in visual basic to trace and IP address. And more importantly it doesn't take itself too seriously which I think makes the ridiculous parts work just like much sci-fi takes liberties with the science part when required for the story.
dd8601fn 1 hour ago|||
Cereal Killers name was Emmanuel Goldstein, too.

That movie still gives me wardialing and 2600 meetups and “voice bridging from a Dennys payphone bank at silly hours” memories.

farzd 2 hours ago|||
"Let's echo 23, see what's up"
TFNA 1 hour ago||||
I, too, look back fondly on the early web. Lately, though, I have been thinking that one reason for its decline wasn't just corporate interests like we often talk about here. A significant number of early bloggers were middle-aged and elderly people. After decades, they simply aged out. The younger generation that replaced them (albeit not so much among OS nerds like yourself) was less likely to use a real computer and keyboard as their interface to the internet, just a smartphone. Hence long-form text died.
mike_hearn 36 minutes ago||
While we're here exchanging old man stories, I remember the web before blogs existed! I guess we can organize the internet into these eras, each of which was in some sense harder and more expensive to find available info than in the previous:

1. Pre-web. Internet is mostly about messages sent to individuals or groups. USENET organizes group discussion into browseable topic-oriented hierarchies, IRC does the same but with lists in fragmented networks. If the discussion exists at all, finding it is easy.

2. Early web. Dominated by topic focused websites, early online shops and personal home pages. Search engines suck and face strong competition from manually maintained topic-oriented directories (did anyone else here contribute to DMoz?), content discovery is mutual and webmasters help each other out by joining "web rings". DoubleClick and AdSense start to funnel small amounts of money to creators, but it's enough to offset hosting costs and in many cases can make web hosting effectively free or even yield a small profit. This encourages an explosion of website creation. Discussion moves off USENET onto phpBB forums. Every organization decides it's a cultural imperative to have a presence on the information superhighway. Finding information is easy as long as you can figure out what topic it belongs to.

3. Blogging and centralization era. The internet starts to rebuild itself around people as the primary object, not the topic or category. Directories die because websites can no longer be categorized by content. Web rings die for the same reason. IRC is replaced by instant messengers that are about connecting people with pre-existing friends, not mutual interest groups. Outside of institutional websites that exist to promote the organization, things become hard to find without highly centralized search engines because nobody is putting any effort into organizing or indexing what they write anymore: maybe you get a few tags if you're lucky. Spam, hacking and lack of SSO causes forums to centralize onto Reddit. This is the peak of the search engine era because you are forced to use Google to find anything. The power eventually corrupts the tech firms and they begin political censorship to benefit the left in 2015 [1]. Enormous amounts of information is deliberately made unfindable as part of a large-scale programme of social control.

4. Social media era. All the same problems as blogging except now the bulk of the content goes behind login walls that stop search engines from surfacing them. Video and podcasts start to matter more, both of which are unsearchable by default. Eventually video completely dominates, as few younger people want to read when they could watch instead. Firefox starts to replace IE6, and then Chrome. They bring ad blockers in their wake which starts to choke off ad revenues, so many websites from the web's first era go unmaintained and eventually offline. This is somewhat but not entirely compensated by the falling cost of web hosting. Social media remains because it puts people's faces next to everything, allowing clout farming and viral notoriety that can sometimes be monetized by becoming an influencer. The only part of the web's first era that really survives into this era is Wikipedia and Reddit, which by this time substitute monetary rewards for power tripping by a small group of ideologically driven moderators.

5. AI era. Information is so heavily scattered over so many tiny sourcelets and search engines have become sufficiently useless that full neural integration of knowledge is required, with LLMs issuing massively parallel and complex search engine queries as a backstop.

What can we predict for the AI era? Institutional websites will remain because institutions still have an interest in getting their agenda into LLMs, but visual redesign efforts will largely cease as traffic stats seen by executives show visits completely dominated by AI. There will be lots of conversations of the form, "why redesign our website to look more modern when 99% of traffic is AI which won't care?" Blogs will go the same way as the thematic websites they killed, disappearing as the authors age out. A lot of effort will be put into finding ways to block AI crawlers to create 'human only' spaces, especially by social media firms, but these will fail because AI will just be integrated directly into browsers and become unblockable - and anyway, the incentives to create will be ignored. ChatGPT style text oriented interfaces will last until inferencing capacity catches up, being eventually replaced by voice interaction and on the fly video generation for nearly all users.

Where we go from here is hard to say. Content creation was most pure in the web's first era, where people with knowledge were incentivized to share it with the world by the promise of a bit of fame combined with ad clicks to offset hosting costs. Ad blockers, social media and AI killed that world. You could however bring it back by producing a new platform that isn't like the web, one where AI and search engines are blocked via technological means (e.g. confidential computing). How much anyone would actually enjoy such a web is unclear.

[1] https://arctotherium.substack.com/p/the-closure-of-the-inter...

Retr0id 24 minutes ago||||
Most hobby communities in my spheres of interest have been subsumed into Discord. It is yet another silo, but it does feel lively. I don't think I'm contradicting you here, I just feel less negatively about it.
Razengan 2 hours ago||||
Well, just as the "internet" supplanted newspapers, magazines (gosh those classic gaming mags), and broadcast television for many people,

and how the newspapers replaced the town criers before them,

why shouldn't the "internet" be supplanted by a more accessible medium?

Why should I have to suffer through Fandom raping me with screen-obscuring banners and "PLEASE ALLOW ADS" just to make some sense of fucking Warhammer 40K lore (written by unpaid volunteers anyway)? instead of just asking ChatGPT what the fuck Globriznaroks is/are.

Why should we support shady companies by sitting through their ads on YouTube videos for minute topics instead of just asking AI for the shit I want to know about?

Why should we submit to the whims of 3 mods on a subreddit deciding what thousands should get to see (fuck /r/AskScience) and then getting low-effort answers or outright trolling anyway? instead of just asking AI?

Bury me, I am ready.

pjc50 1 hour ago|||
This is better for you as an individual, but worse for society as a whole. As you've noticed, monetization is the weak spot. AI will not escape being ruined by monetization, but it might be harder to notice when it arrives.
Tarq0n 2 hours ago|||
Where is the information going to come from though? LLMs aren't primary sources, they consume and regurgitate.
Razengan 1 hour ago||
I guess/hope that's where Yann LeCun and their proposed "world models" etc are going to fill in :)
Melatonic 2 hours ago|||
Checkout Kagi SmallWeb !
sph 57 minutes ago||
I love Kagi, I am a paid user since day 1, but SmallWeb is a collection of English-speaking tech blogs, it cannot even begin to compete in diversity with the GeoCities era of the web. (to be fair: I've had Kagi employees telling me their dataset is growing larger and more diverse every day, so worth keeping an eye on it)

Sometimes I use Marginalia's "Vintage Web" search for niche topics; most results are dead blogs and old .edu personal websites that someone forgot to delete, still a vanishing minority of anything one could find in 2001.

spiderfarmer 2 hours ago|||
That's 100% correct. Since Google's "helpful content update" Webmasters get massive amounts of "Crawled, not indexed" reports for anything Google considers "more of the same" or "thin content". If you're not an authority on a subject, simply meaning: you already rank for similar content, or if you don't get links from more popular domains, your content is in the abyss.

It's all under the guise of "We're fighting SPAM", but the algorithm (or model) they use is heavily skewed towards intents (actions) and brands (because they 'trust' big names).

And it's not working.

A simple, short informative blog about a tool you used that could be of interest to max. 100 people on this planet is no longer getting ranked, if it gets indexed at all.

Those posts tick all boxes: no incoming links, no authority, thin content.

It changes somewhat between "Google Updates", but it's pretty clear that it's no longer working.

Multiply the 100 people not finding that post by millions of queries and it's now a big problem for Google.

My dad asked me the other day if Google got worse because smaller companies are not paying Google enough money.

He didn't see any difference between ads and content, because all results are now big brands only.

"Helpful content" is such a misnomer. It removes all helpful content in favour of AI overviews and only shows intent-driven, commercial content.

We're watching the end of Google's hegemony for sure.

agentcoops 1 hour ago|||
This is fascinating. I haven't really kept up with SEO for years, but was recently helping an academic publisher setup their web presence and ran into exactly this issue: Google was/is refusing to index the actual journal articles on the new site and they were moved into this liminal "Crawled, not indexed" state before vanishing entirely. Comically, it's now easier to get your content indexed and served with a visible link by ChatGPT than it is Google...
mschuster91 1 hour ago|||
> We're watching the end of Google's hegemony for sure.

Who is standing by to replace them though? OpenAI and Anthropic certainly not, they are burning money in a fire pit to stay alive. There is no way in hell they can afford the compute necessary to replace Google.

spiderfarmer 1 hour ago||
If ChatGPT goes bust, they'll find another ChatGPT, not another Google. I wouldn't be surprised if we'll see a popular competitor from China in the years to come. TikTok already took social media by storm, something nobody thought that was possible either.
wickedsight 2 hours ago|||
About 15 years ago, I won a phone in a contest. Last week I tried to find information about it, but I couldn't. No AI, nor Google could find anything the contest I won it in. When people say "the internet is forever" that can certainly be true, but it isn't for everything.
sgt 1 hour ago||
That reminds me - my wife had a friend who was killed by a shark (no joke). There were newspaper articles, but Google has completely forgotten about this and the newspapers are often behind a paywall, changed their URL's or simply the article vanished from those sites as well, which certainly doesn't help.
account42 54 minutes ago||
I wonder how libraries, who have traditionally been the ones to archive the news, have kept up with everything moving online and now being subscription-holed. Hopefully it's not just the Internet Archive doing this, which has its own problems.
armchairhacker 2 hours ago|||
Kagi
neysofu 1 hour ago|||
Kagi is not immune to this (it's the substrate itself that's losing quality, not just search) but it's a decent antidote.
Gigachad 2 hours ago||||
Kagi can’t solve the fact that the internet itself has gone to shit. The independent forums are largely gone, the mainstream social media’s have all locked down and been spammed with AI slop.
Seattle3503 2 hours ago||||
A lot of the time I use Kagi to find the Reddit post or Wikipedia page I need. There isn't much else.
matkoniecz 1 hour ago|||
They only repackage what Google et all returns.

Or have they started own indexing?

gspr 3 hours ago|||
They're so horrible that I've started defaulting to their AI summaries. And I hate those summaries. It's just that the regular results are so terrible now, and seemingly getting worse at a noticeable pace.

I used to not worry. I was sure that a competitor would come along and fix search. But the longer that's not happening, the more nervous I'm getting that we'll actually lose search. If a few more years pass in the current state, I'm afraid the majority of people will forget what search was like and default to AI summaries.

I've tried alternatives, including Kagi (not actually relevant because there's no way I'm – directly or indirectly – buying Russian products) and Uruky, but they're not good enough.

(Edit: Added "directly or indirectly" about Kagi to point out that I'm not claiming that Kagi itself is Russian.)

ehnto 3 hours ago|||
Unsurprisingly all search engines seem to be struggling with AI content sites as well. It's rare to get human articles, sometimes rare to even get authorative websites. It's frequently a bot site with a plausible enough name like, potterspainterly.com or medhealthdirect or something with oddly specific articles written in the last year.
Gigachad 2 hours ago||
This is the root problem. The idea that Google is deliberately sabotaging search seems far less likely than the idea that the internet is mostly garbage and SEO slop.
mittensc 2 hours ago||
perhaps this would not be a problem if google did not directly monetize AI Slop... for that matter why even include AI crap in search.
Gigachad 2 hours ago||
The Google conspiracy has been getting shared for long before AI slop came in to the picture.

The fact is that SEO people got too good at their jobs and filled the search results with junk.

mittensc 1 hour ago||
Google can properly classify email spam.

They should be able to fight this as well.

Problem is they are the ones funding the poor quality spam and they're in turn profiting by taking money from advertisers.

germandiago 2 hours ago||||
Probably what is happening here is that in the race for AI, which, whether we like it or not, means power and control, Google crafted things in a way that it does not do "self-competition" by their old search engine. Idk, just throwing ideas aloud here.
gostsamo 2 hours ago||||
Kagi is not russian. The founder is from Serbia and the company is american.

PS: I hope not many people are the type to see a slavic name and conclude Russia.

gspr 2 hours ago||
Kagi pays Yandex for data. Yandex is definitely Russian.
gostsamo 1 hour ago|||
Yandex also moved outside of Russia and there only one vendor.
szundi 2 hours ago|||
[dead]
onli 3 hours ago||||
Brave search works really well, I haven't switched to Google search for months.
spiderfarmer 2 hours ago|||
If I recall correctly Brave scrapes the web via their users, cannot be individually disallowed in robots.txt and Brandon Eich is conversing in a pretty hostile manner in every thread about him or his company.
onli 1 hour ago|||
I'm not in the brave ecosystem otherwise nor a big fan. But the search engine was competitive with google when it was still cliqz, before it was shut down there and the leftovers bought by brave. And it still works really well.

Even if brave were problematic it would be the lesser evil to me.

GreenVulpine 2 hours ago|||
[dead]
gspr 2 hours ago|||
Thank you for your suggestion. I think I've discounted Brave automatically because my brain is numb to the dime-a-dozen chromium browsers out there. I'll definitely give the search a try!
apothegm 1 hour ago||||
How is Kagi indirectly Russian?
matkoniecz 1 hour ago||
They are repackaging bunch of search engine results, including Yandex (and paying for it).

Yandex is Russian.

I would not describe it as Kagi being indirectly Russian.

casey2 2 hours ago||||
That's certainly the goal, to drive more users to AI by making the other option worse. The real question is when do you put a stop to it. When brain implants are 2x as productive are you going to say nah? It's times like these were you are supposed to take a step back and consider what you are actually producing and why.
drysine 2 hours ago||||
> there's no way I'm – directly or indirectly – buying Russian products

Why not?

matkoniecz 1 hour ago||
In my case: Russian state is my direct enemy and most likely to invade my country. And would do it if they would consider success likely.

Previous wars with Russia were obnoxious with very bad consequences, so I dislike idea of even very indirectly funding them.

And I support actions that are harmful to Russian economy, also when they are harmful to me - as long as it is not too badly balanced. As this is much cheaper than directly participating in war.

(I am from Poland)

PS

Yes, I understand that at some point there are some indirect effects that you cannot avoid.

I also understand if for some people paying Kagi that pays tiny fraction of that to Yandex that is paying taxes in Russia which funds their wars is too tenuous connection to care.

roysting 2 hours ago|||
[dead]
roysting 2 hours ago||
[dead]
ilamont 9 hours ago||
After publishers successfully sued the Internet Archive over its digital lending program, calling it unauthorized copying

No. The court specifically determined that the Internet Archive was guilty of unauthorized copying. It was not simply an unfounded or unproven allegation. The Authors Guild, the National Writers Union, the European Writers Council, and the Society of Authors in the UK all came out against the Internet Archive, and supported the suit.

Each new restriction limits the archive’s ability to act as a comprehensive backstop.

This self-inflicted damage to the wayback machine is the real tragedy of this entire affair. When IA was asked to stop CDL - many times - founder Brewster Kahle continued. The National Writers Union tried to open a dialogue as early as 2010 but was ignored:

The Internet Archive says it would rather talk with writers individually than talk to the NWU or other writers’ organizations. But requests by NWU members to talk to or meet with the Internet Archive have been ignored or rebuffed.

https://nwu.org/nwu-denounces-cdl/

When the requests to abandon CDL turned into demands, Kahle dug in his heels. When the inevitable lawsuits followed, and IA lost, he insisted that he was still in the right and plowed ahead with appeals. And here we are today.

arjie 3 hours ago|
Seems inevitable, doesn’t it? Expecting otherwise would have been hoping that notorious atheist Richard Dawkins somehow spared one specific god. Making websites accessible with history ignoring copyright is sort of what it does. That he would do it with books seems entirely in keeping with the philosophy.
rmunn 2 hours ago||
I was initially confused what Dawkins was doing with books, until I realized that the "he" in your last sentence was Kahle, not Dawkins. Might want to edit your comment to put his name in, because otherwise you have a pronoun referring to a person named in a different comment (rather than the person named in your comment), which could get quite confusing if more people comment on the parent and their comments push yours down the page.
comrade1234 11 hours ago||
I spent the last three days (off and on) using Gemini to configure my edge router 4 with my iOS devices on a vpn and it's been awesome. In the past I'd do a google search and read a few sources of documentation, do another google search and read another set of documentation. Now, Gemini aggregates multiple pages together so all of the work of reading source docs from multiple locations is now n a single step.

Oh, I should mention though. There was no advertising at all. They didn't make any money off me. It was 100% Gemini which I recognize as not long-term feasible.

paxys 10 hours ago||
All the information Gemini surfaced was created with human effort and published on the internet with the expectation that humans would visit the website and the creator would get some reward - advertising dollars, bragging rights, popularity, subscribers or whatever else.

If the only visitors to websites are now LLM training bots then what incentive is there to publish anything new? For how long can we continue to rely on pre-2024 non-AI generated content?

neuralkoi 9 hours ago|||
I've seen websites put up some draconian measures to try and get a grip on the scraping. So much for the sub-second loading experience when you have Cloudflare, Google, Anubis, and all these other captcha services trying to see if you're a human. It's made the web browsing experience so much worse.

Some of the proposals to address this include charging bots for access to web resources, but they will also have repercussions for regular users. I don't see how you solve this cleanly.

kees99 45 minutes ago|||
> Cloudflare, Google, Anubis, and all these other captcha services trying to see if you're a human.

Yep. IMO, this is so far the biggest AI-inflicted damage to the web. A bit of anecdata - wikipedia (and all other wikimedia sites) are blocking my Firefox since about a week, with a "please respect our bot policy" message. Outright block, not even a captcha.

It took me a while to figure out they don't like me disabling some SSL ciphers, so now "JA4 browser fingerprint" is not matching user-agent. Funnily enough curl (what I would imagine a bot would use) pulls exact same URLs from exact same client IP, just fine.

robinsonb5 3 hours ago||||
Sure - it sucks, unfortunately the alternative is the sites going away entirely. When the load from scraper bots is constantly knocking the site offline the choices are literally to allow it to remain inaccessible for much of the time, put up a layer of defenses with all the user-annoyance compromises that entails, or just give up and unpublish the site.
Borg3 3 hours ago||
The alternative is simple.. Go dark. VPN tech is known from like 30 years. Pretty much everyone can use it (VPN providers). But instead using it to browse net, build VPN overlay networks of interest for people. Gaming networks, R&D networks, Retro Networks. People will peer to PoP and use resources. Bad actor? BAN it from network. You have control. This could be done in Internet, but big corpos and big money won the battle. Just wake F*ing up...
kukkeliskuu 1 hour ago||
Continuing on your suggestion.

There could be open source tooling to create custom private "closednets", with

- trust ring mechanism to allow invitations, flagging, banning, and banning those that invite people who were banned

- the rules of the closednet

- search engine with opt-in scraping

- portal (remember the 80s?) with all the registered nodes, perhaps by service category such as public git repo hosts, web sites etc.

etc.

The first closednet could be Hacker News.

Grimburger 3 hours ago||||
Cloudflare specifically has a block for LLM and AI training bots now.

Not sure of the effectiveness but it's there.

matherial 3 hours ago|||
Minimal. I'm behind Cloudflare and 90% of the traffic is still scrapers. I don't think they're serious about the long tail.

I think the main thing Cloudflare is trying to do is block direct traffic from frontier labs and then start charging them for access. They might end up shooting themselves in the foot, as this simply empowers sketchy residential-proxy outfits to undercut Cloudflare and sell the data to labs for less.

bakugo 2 hours ago|||
It still only blocks "well-behaved" bots that have proper User-Agents and respect robots.txt, so it's largely pointless.

The problematic bots are all disguising themselves as Chrome and sending requests from millions of residential proxy IPs, and the only real solution to those is some sort of captcha or PoW page on first visit.

JKCalhoun 9 hours ago|||
Me, I'm just scraping the parts of the internet I like, toying with local LLMs… ready really to just shove off.
littlecranky67 3 hours ago||||
> "humans would visit the website and the creator would get some reward"

That expectation is a problem, has always been a problem, and Tim Berners Lee never mentioned anything about a reward structure when coming up with the WWW.

pferde 3 hours ago|||
Your thinking too narrowly about the reward. Sometimes, it's just about the getting the knowledge out there that's motivating the creator, not anything tangible for themselves.
boredhedgehog 2 hours ago|||
> it's just about the getting the knowledge out there that's motivating the creator

In that case the creator should welcome AIs with open arms; a human reader will forget eventually, but the AI will preserve the knowledge forever.

matkoniecz 1 hour ago||
"the AI will preserve the knowledge forever"

no, only some mangled form of it

esposm03 3 hours ago|||
> Your thinking

Should be "You're thinking".

cryptoz 1 hour ago||
> Should be "You're thinking".

This should be "This should be 'You're thinking'." don't you think? Why bother correcting someone's grammar with a sentence fragment? You're just trading one mistake for another. I'm hoping someone finds a grammar error in my post, because continuing this would be hilarious.

danhite 50 minutes ago||
> This should be "This should be 'You're thinking'." don't you think?

Reflexively, I think it should be more like ...

  javascript: `This should be "You're thinking".` ; 
  // to preserve the original character use and to avoid '...'...' parse foos
  // however `"...".` also possibly deserves a [sic] to critique the original 
  // i.e. ~grammar police say the period belongs within the quote marks, no?
... but then that's just me, in [my] quirks mode.
gspr 3 hours ago|||
Did the operators of HN mention anything about a reward structure for posting comments here? I'm sure you can see how that's still attractive to some.
jefftk 9 hours ago||||
> If the only visitors to websites are now LLM training bots then what incentive is there to publish anything new?

I write because I have ideas I want to share, and whether that happens with LLMs as an intermediary isn't important to me.

exmadscientist 9 hours ago|||
The danger that's concerning people (rightly or wrongly) isn't that LLMs are going to be an intermediary to your website. It's that they'll be the only thing reading it. No one will ever read your post or know what you wrote. The only consumers will be LLMs, they'll train on a version that strips out you as the author (probably more due to expedience than any sort of malice; it's not like you're famous, are you?), and your idea might get embedded into a set of model weights somewhere. No human will see a byte of it.

Are you actually saying you'd be OK with that?

jefftk 8 hours ago|||
Yes, that would be fine. I write primarily communicate ideas, not for credit or fame.

Empirically, however, LLMs don't strip out the author: the big models know a lot about what I've written even with search disabled. Ex: https://claude.ai/share/8cbcdf88-a360-421a-8c06-ae7b7992e866

frabcus 1 hour ago||
Sure they know about you if you ask, but generally they won't credit you if they cite an idea from their latent space that came from you.
JKCalhoun 9 hours ago|||
Not the OP, but I suspect no one goes to my website anyway. (I write nonetheless.)
altern8 2 hours ago||
I do! You invented the crayon picker in MacOS..?
mittensc 3 hours ago||||
I mainly shared my projects for learning, discussion and bragging rights.

LLMs just use everything, generate similar code with no attribution and keep users from visiting, so no bragging rights or attention.

Worse, there are some PRs that seem fully generated ...

So i mostly stopped sharing and started pulling my old repos offline.

At this pace, i don't want to compete with a clone of myself in the future that will do my work for much cheaper.

shuwix 5 hours ago||||
Your level of thinking is defined by ICD-10.

Yeah, authors don't want to be recognized as authors, they don't want any reward for their work, they don't want to amass pool of loyal readers, interact with them, etc.

All they want is for halucinating AI to take excerpts of their work and compile it with random sh!t.

GENIUS

wartywhoa23 3 hours ago||
Thanatic drive masquerading as transhumanist virtue signalling.
laughing_man 8 hours ago||||
There are different types of writing. If we depend on people writing because it's enjoyable at some level, we're going to lose writing that's important but also a bit tedious.
jefftk 8 hours ago||
Of course! I do think we'd lose a lot of great writing if it went amateur-only. But my parent seemed to be saying the incentive would entirely disappear, so I wanted to give my perspective.
gspr 3 hours ago|||
> I write because I have ideas I want to share, and whether that happens with LLMs as an intermediary isn't important to me.

Sure. But you can see that for some people (myself included), writing for peers is part of the joy? And that if instead a megacorp places an opaque computer program between the author and the readers, that joy might be ruined?

dwayne_dibley 1 hour ago||||
Easy to fix a well documented router now. Difficult to fix a non documented router in five years time because no one has been contributing to the web about its bug fixes.
swingandamiss 9 hours ago||||
Gemini can just consume the device documents. There's an incentive for device makers to publish this content.
dspillett 9 hours ago||
There had always been some incentive for manufactures to publish device documentation, and yet it has often been quite lacking either in quality or overall existence. I doubt LLM/agents being the readers will change that at all. What I expect AI scraping and using without credit will impact is people publishing their own unofficial help and guidance, and the affect there is likely to be negative. It won't stop all of them, but enough to be noticeable. Another possible negative is the manufactures documentation being AI generated without sufficient review, so possibly more erroneous than before, or intentionally not producing full documentation at all and expecting AI to fill the gap (MS seems to be heading this way: pushing "ask copilot" all over Azure instead of links direct to good reference material). All this would add up to a situation that is somewhere between "a little worse than pre-AI" and "an absolute shit show".
no-name-here 7 hours ago||||
At least LLMs almost always transform the original - it usually isn’t as straightforward as “Here’s the original but without the ads that pay for it”.

But we already have the latter case that exists - ad blockers. Ad blockers literally serve up the word-for-word original content minus the ads.

penneyd 9 hours ago||||
Well in the example above the manufacturer still has incentive to provide the manual's and guides that describe how to use their products, and if that is subsequently served by an LLM that's totally fine. The only sites that LLM's would have a negative effect on are those that are only hosting content for the ad views.
Eddy_Viscosity2 9 hours ago||
Manuals don't always well explain how to use their products with everybody else's products because there are too many to do that. But there are lots of people trying things out and might figure out the fine details on how to make various things work. They then publish these how-to pieces (which exist no where else) to the internet, or at least they used to when there were incentives to do so.
latexr 3 hours ago||||
Exactly. What’s problematic about comments like your parent is the absence of mid-to-long term thinking.

It’s like bragging about a new highly addictive psychedelic drug that a dealer gave you a taste of for free. The effects are awesome today, you feel so fun and free! Never mind that it’s destroying your body and that the dealer will eventually charge you or demand you pay in other ways, that’s a problem for another day. Weeee!

karmakaze 7 hours ago||||
Funny, I published information in the hopes that humans would benefit from it. If it happens to be through collective intelligence of LLMs I'm ok with that--even more so if through open models.
wartywhoa23 3 hours ago||
> Funny, I published information in the hopes that humans would benefit from it.

Sure, humans would benefit.

It took them searching, reading themselves, maybe even understanding something in the process, to complete a 360° revolution of their squirrel cages in time T.

Now they can omit searching, skip reading to the regurgitated answer, throw away understanding, and complete a full revolution in T/N, where N is a heuristic value directly proportional to the amount of skin in the AI hype.

But the catch is that the squirrel cage must run non-stop still.

_dark_matter_ 10 hours ago||||
I wonder the same thing. I only imagine that what comes next is worse: AI companies using vast resources to develop new training data, in house, locked down. They are already doing this with developers and code at Meta. Information will become locked away behind AI paywalls and chatbots.
satvikpendem 9 hours ago||||
People write and create regardless of profit motive, it has been that way for thousands of years.
watwut 4 hours ago|||
People are way less likely to write if there is no one to read it. And blog were also monkey see monley do - people seen other peoples blogs and got inspired.

When people wont see others blogs, they wont start writing own. When there will bw no ome to actually read it, they will go to do something else.

_DeadFred_ 8 hours ago|||
Actually no. The original copyright laws were created in part because of the realities of needed profit motive to have high value writing done, time consuming compilation work done. It was even titled "An Act for the Encouragement of Learning". The thousands of years writing you are talking about was often funded by patrons, who kept the output in their private libraries to show off (and maybe lend out) for prestige. It was a horrible limitation of knowledge and ideas. Much worse than the profit motive, copyright based system that came after that spawned a new age of knowledge in which everyone had cheap access, and those that didn't had access to the (no longer just private) libraries.

I'm sure the billionaire class would love a return to patronage based libraries, NDAs on authors of books, and the elitism they would feel with a return to private libraries locking away all kinds of knowledge that would happen if patronage become the only way authors could make money (such as with AI just regurgitating their works, or if the stupid 'do away with copyright' people got their way).

graemep 2 hours ago|||
Actually it grew out of censorship and monopolies: https://en.wikipedia.org/wiki/Statute_of_Anne#Background

Cheap access came from the invention of cheap printing . The laws were passed to restrict it.

nairboon 4 hours ago|||
It's not copyright that caused cheap access. The printing machine allowed for cheaper publications, that's what spawned the new age of knowledge. The raw materials and the duplication of knowledge was the bottleneck. With digital systems this cost is minuscule, but still there.
Havoc 9 hours ago||||
That is next earnings quarters problem is the approach being taken
ekianjo 3 hours ago||||
> incentive is there to publish anything new? For how long can we continue to rely on pre-2024 non-AI generated content?

obviously new content still has value because it remains the source layer for LLM agents. it just wont be ads giving you revenues thats all.

dudefeliciano 3 hours ago|||
Well companies are starting to put hidden ads in text content if the user agent is an AI crawler
lnrd 1 hour ago|||
That works now because there are human made sources that the AI can find and summarize for you. But now there are no incentives at all for humans to write anything on the internet and if they do the content will be buried by hallucinated content someone else posted at a larger scale.

I had the similar experience to yours yesterday and it lead nowhere. Funnily enough I was also trying to configure a vpn on a router, google didn't return anything useful (besides a blog post clearly written by AI and with absolutely no information in it). Claude managed to give some interesting pointers, but its suggestions were not working and I also noticed that it started to hallucinate badly about ipv6 and gave me some suggestions that were just plain untrue. Claude Opus is smart, usually when it gets so convinced about something is after researching the internet and not just based on its training data. I wonder where it got so convinced about it. Maybe reading some other hallucinated blog post like the one I stumbled upon?

lopis 1 hour ago|||
That only works when there's an abundance of documentation for your specific device. The moment you're on a more recent version of something and have a weird issue, all hope is lots. You get stuck in loops because all LLMs keep recycling old advice that no longer applies. This problem will only get worse and worse as people no longer as questions on public forums, so answers are not publicly available either.
JKCalhoun 9 hours ago|||
Yes. When my kids wonder why I don't hate AI the way they do I tell them it's because I hate Google even more.

To be more precise, I hate the SEO shithole the internet has become, that Google serves up, that Google facilitated, indirectly created.

(I really don't have any tears to shed if there is a death of the Corporate Internet™.)

jononor 3 hours ago|||
AI seems to be on the same trajectory? Search was very useful in the start also, until it became entrenched. Then search placement became a target, and they are just focusing on extracting rents. All way paying the content providers zero or near-zero. And with years of that dynamic, we end up where we are now. It was the same with "social media". The same will happen with AI. AI is a power for more enshittification - being currently less shit than Google is (mosy likely) temporary.
zombot 1 hour ago||
It's definitely temporary. Remember there are no ad blockers for LLMs.
quantummagic 1 hour ago||
You're right, this is the best it will ever be. But there likely will be ad blockers for them -- local LLMs that filter for any brand placement, etc.
dualvariable 8 hours ago||||
They're probably going to figure out how to aggressively monetize and enshittify LLMs at some point.

We're likely at the "golden age" of LLM-assisted web searching and summarization.

Hopefully open models keep it cracked open, but expecting enshittification is always the safe bet these days.

badestrand 7 hours ago||
For the free to use LLMs, for sure. But if I pay $20 monthly, why should they enshittify it?
Gigachad 2 hours ago|||
$20 a month is not nearly enough. The current finances require AI companies to make far more money than that to not implode.
archagon 6 hours ago||||
Because that’s what always happens? Because their raison d’etre is to squeeze you dry?
0x262d 5 hours ago||||
look at every paid streaming service...
Eisenstein 4 hours ago|||
The amount of money you pay is irrelevant if you don't leave the platform once they starting adding revenue streams that degrade your experience.
no-name-here 7 hours ago|||
Is there some search engine that you think could have become popular and not ended up with SEO optimization?
Melatonic 5 hours ago|||
One you pay for yourself !

Very happy with Kagi personally

no-name-here 4 hours ago||
>> Is there some search engine that you think could have become popular and not ended up with SEO optimization?

> One you pay for yourself !

SEO is the practice done by webmasters of optimizing a website to improve its visibility and ranking in search engine results.

If you are paying to use your search engine, does that mean webmasters are no longer incentivized to/will not try to improve their visibility/ranking in your search results?

csnover 3 hours ago|||
I don’t agree with the parent that the solution for SEO spam is paying for the search engine (I think there are other good reasons to do this though), especially since people have been doing things like naming their company “AAA Auto Repair” to be first in the phone book since before computers ever existed. But the person you originally replied to does have a point in blaming Google for the problem. Most SEO spam sites make their money from ads, and Google are the ones who run the ad network, which means Google are the ones funding them and creating an incentive for them to exist.
Eisenstein 4 hours ago|||
> SEO is the practice done by webmasters of optimizing a website to improve its visibility and ranking in search engine results.

They do this because they benefit from their site being visited or the information they are providing being noticed.

> If you are paying to use your search engine, does that mean webmasters are no longer incentivized to/will not try to improve their visibility/ranking in your search results?

I see no reason it would have that effect. It does, however, create different incentives for the search provider to improve the signals indicating page relevance since the user is the priority instead of advertisers.

no-name-here 4 hours ago||
Ah, the argument is that Google intentionally avoids showing you the most relevant results, or at least avoids "solving" the problem of webmasters who attempt to 'game' the system.

>>>> Is there some search engine that you think could have become popular and not ended up with SEO optimization?

>>> One you pay for yourself !

>> SEO is the practice done by webmasters of optimizing a website to improve its visibility and ranking in search engine results.

> They do this because they benefit from their site being visited or the information they are providing being noticed.

>> If you are paying to use your search engine, does that mean webmasters are no longer incentivized to/will not try to improve their visibility/ranking in your search results?

> I see no reason it would have that effect.

Agreed.

watwut 4 hours ago|||
Google did not really lost to optimization, they enshittified to make you spwnd more time on search (and see more ads)
no-name-here 4 hours ago||
Ah, the argument is that Google intentionally avoids showing you the best results, so that you spend more time searching (and being exposed to ads)?
robinsonb5 3 hours ago||
I'd frame it more that Google is incentivised to show you the results which are most lucrative for them to display, rather than the ones which are most beneficial for you to see.
oblio 2 hours ago||
Incentives. Which is why platforms (such as search) should never be allowed to be in the same company with things built on top of them (such as ads), if the combined company is significant for an important market.

We used to know better, Standard Oil vertical integration was dismantled.

mvanbaak 45 minutes ago|||
While I understand that you feel good as you got the device configured, you would have been better off without gemini. The 'old' way as you call it would have resulted in you knowing the backgrounds and the inner workings of your router, making maintaining it a breeze and helps you actually understand your setup. Besides that, it would probably trigger you to rethink some of the things you now blindly have implemented because gemini did not show alternatives nor reasoning behind it. (something that most definitely would have been documented on the source pages)
DrScientist 1 hour ago|||
Yep - it's awesome, the problem is that Google isn't sharing the revenue with the context creators anymore - over time, unless fixed, this will decimate the knowledge base it feeds on.

> Oh, I should mention though. There was no advertising at all. They didn't make any money off me

Are you sure about that? Even if you didn't see ads ( remember people pay even if you don't click - just like a billboard ) - they are still profiling you to better sell you ads in the future, and using your interaction as free training data.

laughing_man 8 hours ago|||
I've run into major problems with LLMs as I maintain my home Linux systems. If I just copied commands they list, I would have a near 100% failure rate, as most of the information they have has been gleaned from forum posts that are years out of date.

They're a good jumping off point, but I need to delve into the original sources just like I did when I used Google.

suslik 1 hour ago|||
I'm not sure. I run a homelab tailscale/k3s setup with gitops, dozens of services, VMs, backups, etc, all vibed by claude, and works just fine. Didn't write a single line of code for this. I don't know kubernetes and never will.
iLoveOncall 1 hour ago||
> I don't know kubernetes and never will.

He said, proud of his own ignorance.

suslik 1 hour ago||
Proud? No, but I am not ashamed of it also.

I would be the first to admit my ignorance on the absolute majority of topics. There is a limited number of things I can learn in life, and kubernetes won't be one of them - I'm just not interested in it (and all the other infra stuff, to be honest), as long as it works.

AussieWog93 3 hours ago||||
I've had the opposite experience giving claude code SSH/ADB access to my devices. Not sure if it's the model itself or the harness, but I haven't had to manually do a sysadmin task in months.
arcfour 7 hours ago||||
What on earth LLM are you using and do you tell it which distro/version of Linux they are supposed to be working with + give them access to web search or man pages? I haven't had this issue in like a year and a half.
laughing_man 6 hours ago||
By default I use Leo, and I do specify the distro. I tried to qualify by version, but when I do that I lose out on a lot of correct information for things that haven't changed in awhile.
Randor 4 hours ago|||
Have a look at https://github.com/ThorOdinson246/whatisit-nl2sh

It seems to get shell scripts right most of the time.

snapplebobapple 10 hours ago|||
its pretty good with logs too. i mostly paste a few pages i suspect hace a problem in them and let it go to town. its almost always correct and is way faster than me
queenkjuul 10 hours ago|||
I've been mostly enjoying Gemini, but it also clearly and definitively told me something i was trying to do was not possible with the library im using, so i wrote a different implementation, an hour later to discover that the library does in fact do precisely what i wanted in exactly the way i wanted with less headache. If I'd just gone right to the documentation instead, it actually would have saved me time.
matt4711 10 hours ago|||
There will be new ways and incentives for content creators to be compensated. Many AI search startups are already talking about this or have created programs that help incentivize content creation.
renegat0x0 3 hours ago|||
I hate to break it to you, but we are at the start of enshittification circle. Once Gemini has monopoly you will reminisce with joy google search results.
jjulius 10 hours ago|||
And I've spent the last few days irritated that everything I ask Gemini is answered with something that's blatantly wrong and I'm not even a subject matter expert. A quick Google search for the same questions gives plenty of results that counter what the LLM gave me.

YMMV.

-0_0- 10 hours ago||
Confidently wrong summaries are the bane of Google search, and unfortunately I'm finding the AI seems to bleed into the actual search results too now, often turning up pages that back up what the summary is (wrongly) suggesting instead of surfacing actually relevant results for what I'm asking.
indigodaddy 9 hours ago|||
Believe me they're making money off you one way or the other
moralestapia 6 hours ago|||
You deserve what's coming for you.
ls612 11 hours ago|||
I built a simple Claude Code container on my homelab to do the same. I can just SSH in and get tech support when I need it.
keybored 3 hours ago|||
AI is on the whole bad but it’s impossible to forget that the ostensibly human-curated Web as presented by modern search engines is bad too. Invasive ads, popups, and worst of all the substantive content has a nine inch frame of SEO filler and a three inch picture (what you were after). And some things are not even high-tech slop stolen. It is just old-school verbatim copied from another website and repackaged with another frame.

But yeah, text remix machines are not a long-term solution to that problem.

nsonha 4 hours ago|||
Imagine someone making an indie router and sell it too you then the AI not be able to surface their doc
stringfood 10 hours ago|||
no, no, no. you are supposed to romanticize the hunt for correct information /s
mrweasel 1 hour ago||
Even with the /s I think you misunderstand. If you just want to configure your router, then AI is neat. If you want to learn and understand what's going on in the router as you set it up, then being given the answers basically teaches you nothing.

It's the same reason why we don't give students the answers to things, we teach them to find the answers.

nixonpjoshua 10 hours ago|||
Are you sure the lack of advertising isn't long term feasible? It seems to me that AI models have proven themselves to be something consumers ARE willing to pay a subscription - or even pay per use/token for.
jjulius 10 hours ago|||
Are they profiting from these subscriptions yet?
queenkjuul 10 hours ago|||
I've been using Gemini free tier exclusively for all my AI "needs" and would not be willing to pay for it
theplumber 10 hours ago||
It took you 4 days because you used Gemini. Gemini is the worst AI model I ever used. It is way behind even open models. It looks like Google just reached its AOL moment.
noosphr 10 hours ago|||
Gemini isn't great as a model, googles search and ability to cite textbooks down to the paragraph make it better than every other model for human in the loop tasks.

I end up using the gemini api for with search enabled for the cases that I don't have access to good grounding data even in agentic tasks.

queenkjuul 9 hours ago|||
Come now, copilot is worse in every way
JKCalhoun 9 hours ago|||
Guessing this is the relevant part: "worst AI model I ever used"
satvikpendem 9 hours ago|||
Copilot can use any model like Sol or Opus and is just a harness so not sure why people say this.
queenkjuul 5 hours ago||
Personally it's because i used it when it was brand new and work paid for it and i had no idea what model it was using, my boss just turned it on for automatic PR summaries and code reviews and it was universally dogshit, and the auto complete in my IDE was awful as well.
bsammon 3 hours ago||
> While the web has always been organized around intermediaries that shape what survives online and who sees it,

This statement, from the sixth paragraph of the article, is something that I would have liked to see addressed more in the article. The article implies that this is something that must always be true, or cannot be changed, and simply focuses on how we could have better/better funded/better protected intermediaries (AKA gatekeepers), and doesn't discuss the possibility of an internet (or part of the internet) without gatekeepers (and doesn't ask if it has ever existed/does exist/should exist)

novafunc 11 hours ago||
I occasionally use Google Search when DuckDuckGo fails to give me relevant. Almost always, Google has better results.

Though I can find its AI answers annoying aggressive. I'll look up like two search terms and the AI will bullshit multiple paragraphs out of despite having zero context of what I am looking for.

DuckDuckGo seems to have detection of whether it should give an AI answer. And it allows you to have more granular control of when you want to get an AI answer. And is overall less distracting than Google's.

8organicbits 10 hours ago||
Interesting, I've seen much better results on DDG. Most recently was the search: `site:feeds.bbci.co.uk inurl:rss.xml` which works on DDG but gives zero results on Google. As far as I can tell, Google just decided not to index these.
jamesfinlayson 10 hours ago|||
Yeah I mostly still use Google out of habit but there have been a few times where Google has decided something isn't worth indexing (too niche, doesn't use SSL).

I miss when Google was like a grep for the entire visible Internet. Now it tries to second-guess my search and direct me to a bunch of sites which all have identical information that isn't what I'm looking for.

ehnto 2 hours ago|||
I find DDG is struggling to, or chosen not to, filter or derank obvious AI generated content farm sites. Of which there are an insane amount of already.

We thought blogspam was bad, at least it was easy to ignore. It's hard to find authoritative sources for a number of topics, worryingly health advice is one of them.

crnkofe 2 hours ago|||
The problem and what the article is pointing out is that original content is slowly and progressively being replaced by AI content. And since AI gets trained on this content as well it will eventually train itself on previous gen. content that was also AI generated. It is slowly eating the web. Eventually you won't even be able to evade it because it'll be everywhere. Before AI became this expressive I could at least expect someone writing articles, FAQs, blog posts to have some backbone. Now I frequently run into content that obviously was never even checked by a human.
dijksterhuis 10 hours ago|||
https://noai.duckduckgo.com is a thing fyi

i don’t agree with the google has better results thing. sometimes it does. most of the time it’s just that google has the site i want higher in the ordering than DDG. personally i’m fine scrolling down a little bit more. it’s rare i need to go to google for something that DDG doesn’t have at all in their results, but it does happen.

i do have to go to google for maps/directions/planning travel. a lot that’s annoying.

NooneAtAll3 9 hours ago||
> https://noai.duckduckgo.com is a thing fyi

or you can press the gear button -> "Ai features: Manage" -> Search assist

a2ff6eeb0 10 hours ago|||
That was true until a few months ago. Now, it almost never has relevant results, and I've given up on it.
brikym 6 hours ago||
It seemed like Google was good ten+ years ago and then gradually trended towards being rotten around 2020-2022. In those days blogspam was king. I want a cooking recipe and it would show me a life story. Or I wanted a OG web game but a link farm would come up top. I think there are leaked internal comms where they discuss nerfing search results to pump engagement and ads impressions. No doubt it would boost ad bids too if businesses couldn't be found organically.

Around that time Bing and DDG were actually better. Then LLM's came along and they started to take things seriously again. Maybe they think the OpenAI threat has abated enough to begin enshitification cycle 2.0.

arcade79 3 hours ago||
I know Google started deteriorating. It was in 2008 or 2009. Up until then Google would return 0 results if it couldn't find a document with all the words you specified.

Then it started serving synonyms, attempted to correct spelling, and so forth. Instead of serving up that there was 0 results, it attempted to be "helpful".

For a while, you could enable "verbatim" search, but even that has gotten corrupted.

Their search quality has deteriorated ever since that change. It is sad.

verdverm 9 hours ago|||
that control (so I can turn it off completely) is why I picked DDG to replace Google, who force feeds us the hallucinations

I've stopped using DDG now because of result quality. I now use a "meta" search backed by EXA, Tavily, and SearXNG in parallel. It can be agentically de-dupped or summarized as needed. Search as we knew it is done, largely because clicking through to evaluate result relevance before diving deeper sucks. Now we have agents that can do that portion and perform multiple searches, building on information in the last batch, to collect good results

amatecha 10 hours ago||
Yeah, there are times where DDG has like, literally three results. Yet, I know for an absolute fact, there are hundreds of pages on the web that contain the terms I specified. Web search is becoming utter garbage.
slig 9 hours ago||
Try brave search.
CommieBobDole 9 hours ago||
The article touches on something that I've been thinking about with regards to Google's AI strategy; the automatically-generated AI search summaries are not great. They very frequently confidently misinterpret what the user is searching for and generate half a page of useless information that pushes actual results down the page, and they are occasionally hilariously incorrect, with hallucinated facts.

This is probably a difficult-to-solve problem; given that they generate billions of these a day, not even Google can afford to devote enough compute to each query to reliably generate quality results. You can see this by selecting the "AI mode" from the search interface after getting the mediocre summary - the results are much better and generally perfectly usable. Though even that is probably a special minimal-compute version of the lowest tier of Gemini, it's still maybe an order of magnitude more capable than whatever generates the search summaries.

The bigger problem is that these search summaries are the default and by far the most common interaction that the general public has with "AI", and because this experience sucks, they just assume that all LLMs are similarly stupid and mostly useless. In non-technical spaces I frequently see the argument that "AI" is not useful for anything, all it generates is garbage hallucinations, and almost invariably they cite some actual terrible experience with the Google AI search summary. I would argue that the strategy of adding LLM summaries to every search is the worst of both worlds - it makes classic search worse while poisoning users against the idea of actual LLM-assisted search.

oblio 2 hours ago|
> not even Google can afford to devote enough compute to each query to reliably generate quality results

https://www.dw.com/en/german-court-holds-google-liable-for-f...

umvi 11 hours ago||
I feel like collecting, curating, and protecting high quality corpuses of "truth" is going to become increasingly important for high quality AI.

There will come a day (and probably soon) when "training on the public internet" (Reddit, etc) will taint your model with metric tons of corporate contamination, political poison, and other adversarial content intentionally crafted to bias AIs for various reasons (corporate gain, geopolitical information warfare, etc). Basically the AI-equivalent of SEO.

jessetemp 11 hours ago||
All of that already existed for the purpose of biasing people and now it biases ai for free. A company would have to make an effort to remove or change the bias
marcd35 2 hours ago|||
Reddit has already begun the effort to start poising the well - https://www.reddit.com/r/poisonai/
Good4boothee 1 hour ago||
Isn't that effort totally redundant? As in - plenty of people are already filling entire internet with slop for SEO purposes? And LLM slop by default is a mix of facts with few plausible but made up facts - it might be harder to craft such perfect poison on purpose.
hadlock 11 hours ago|||
I think most everyone already has a curated training library; Web scraping exists but I don't think anyone is still using it as a primary information vector
dylan604 10 hours ago||
Otherwise they'd be slurping in their own slop
altairprime 2 hours ago|||
If I ever curate again it will certainly not be for the public. That led to PageRank which kickstarted this whole dystopian nightmare that Google has been planning since as early as 2003. No thank you.
satvikpendem 9 hours ago|||
This already exists, there are archives of Reddit or other sites, and Anna's Archive for papers and books.
asawfofor 9 hours ago|||
Isn’t this what the paper-bound encyclopedia companies do, albeit shallowly
latexr 3 hours ago||
> There will come a day (and probably soon)

That day has already arrived, it is already happening.

smackeyacky 6 hours ago|
Gemini has been a hilarious companion to my while I fixed the balance shaft chain guides in my old Mitsubishi triton (mighty max for US readers).

First it told me I could just remove said balance shaft chain as an emergency repair. Sorry Gemini, it also drives the oil pump.

Then it told me I could remove the water contaminated oil caused by removing the timing case by filling the crankcase with hot, soapy water and running the engine. Lord no.

Then it gave the wrong instructions for putting new gears on the balance shafts which meant the chain guides didn’t align with the chain. I’ll do it my way thanks Gemini.

The rest of the mistakes are too trivial to recount and sure it’s a pretty obscure subject but if I trusted it with a topic I’m not familiar with there is a huge potential for damage if you blindly follow it’s overconfidence. I miss normal searching.

ElFitz 5 hours ago|
Meanwhile, ChatGPT correctly diagnosed what was wrong with my plant from a single photo, identified which leaves I should cut, and annotated the picture showing where to cut and what not to touch.

I honestly expected a made-up useless generated image that matched the idea but not the actual thing.

Guess I’m still living in 2024.

chrisjj 1 hour ago||
What makes you think the chatbot's diagnosis is correct?
ElFitz 37 minutes ago||
[dead]
More comments...