https://news.ycombinator.com/item?id=46624740 has the earliest writeup that I know of. It was running it via a script and intentionally using cache busting techniques to try to increase load on the hosted wordpress infrastructure.

they silently ran the DDoS script on their captcha page (which is frequently shown to visitors, even when simply viewing and not archiving a new page)

Wikimedia could pay, they have an endowment of ~$144M [1] (as of June 30, 2024). Perma.cc has Archive.org and Cloudflare as supporting partners, and their mission is aligned with Wikimedia [2]. It is a natural complementary fit in the preservation ecosystem. You have to pay for DOIs too, for comparison [3] (starting at $275/year and $1/identifier [4] [5]).

With all of this context shared, the Internet Archive is likely meeting this need without issue, to the best of my knowledge.

[1] https://meta.wikimedia.org/wiki/Wikimedia_Endowment

[2] https://perma.cc/about ("Perma.cc was built by Harvard’s Library Innovation Lab and is backed by the power of libraries. We’re both in the forever business: libraries already look after physical and digital materials — now we can do the same for links.")

[3] https://community.crossref.org/t/how-to-get-doi-for-our-jour...

[4] https://www.crossref.org/fees/#annual-membership-fees

[5] https://www.crossref.org/fees/#content-registration-fees

(no affiliation with any entity in scope for this thread)

Archive.org is the archiver, rotted links are replaced by Archive.org links with a bot.

https://meta.wikimedia.org/wiki/InternetArchiveBot

https://github.com/internetarchive/internetarchivebot

There are ways to work around this. I've just tested this: I've used the URL inspection tool of Google Search Console to fetch a URL from my website, which I've configured to redirect to a paywalled news article. Turns out the crawler follows that redirect and gives me the full source code of the redirected web site, without any paywall.

That's maybe a bit insane to automate at the scale of archive.today, but I figure they do something along the lines of this. It's a perfect imitation of Googlebot because it is literally Googlebot.

Exactly. If I was an admin of a popular news website I would try to archive some articles and look at the access logs in the backend. This cannot be too hard to figure out.

Because it works too reliably. Imagine what that would entail. Managing thousands of accounts. You would need to ensure to strip the account details form archived peages perfectly. Every time the website changes its code even slightly you are at risk of losing one of your accounts. It would constantly break and would be an absolute nightmare to maintain. I've personally never encountered such a failure on a paywalled news article. archive.today managed to give me a non-paywalled clean version every single time.

Maybe they use accounts for some special sites. But there is definetly some automated generic magic happening that manages to bypass paywalls of news outlets. Probably something Googlebot related, because those websites usually give Google their news pages without a paywall, probably for SEO reasons.

.. but what about subscription only, paywalled sources?

But it is reliable in the sense that if it works for a site, then it usually never fails.

This will always be done unless the original url is marked as dead or similar.

Archive.today has just about everything the archived site doesn't want archived. Archive.org doesn't, because it lets sites delete archives.

Wayback machine removes archives upon request, so there’s definitely stuff they don’t make publicly available (they may still have it).

Trying to search the Wayback machine almost always gives me their made-up 498 error, and when I do get a result the interface for scrolling through dates is janky at best.

Accounts to bypass paywalls? The audacity to do it?

Well, that's an odd idea of "can be replaced".

> editors can remove Archive.today links when the original source is still online and has identical content

Hopeless. Just begs for alteration.

> a different archive site, like the Internet Archive,

Hopeless. It allows archive tampering by the page's own JS and archive deletion by the domain owner.

> Ghostarchive, or Megalodon

Hopeless. Coverage is insignificant.