Posted by bryanbraun 10/25/2024
I purchased a valuable premium domain to host a personal art collection (of anime cels). For some bizarre reason, the site was inaccessible from my work computer and it was de-listed from Google even if I typed the url itself into search.
I hired a square space specialist to figure out why, to no avail. I then begged our company’s CISO to investigate and it turns out we had some firewall setting on UniFi that blocked the domain because it appeared on a list. Once I checked way back, it turns out that it was as an anime porn aggregator years back. I personally reached out to all the web filters out there (Google, Symantec, bing) and one by one filed tickets for them to mark it as art instead of pornography and it worked. I am now properly crawled on Google but still MIA on Bing, search console is giving me some BS error that’s incomprehensible, typical of MSFT.
I have a +100 cel backlog that I need to catalog and photograph. Was planning to do it this holiday season so check back in.
We're talking like 20 years back. Holy shit, my brain is getting jostled by this sudden tsunami of forgotten memories.
EDIT: Digging around on Wayback Machine (obviously NSFW, for the curious), apparently it was actually still around until somewhere between 2018 and '19 when it finally died. The snapshots from around 2007 are peak Web 1.5 design with stuff like affiliate buttons and table layouts. Man I miss that era.
Also the resources->galleries was useful, found some new but actually old sites to check out.
Yahoo Auctions is more popular over there and proxy services (I use Buyee) make it pretty simple bid/buy and not too much more expensive if you wait for their (Buyee) coupons.
I had no idea such a thing existed.
If you can set up your own domain why would you need someone that specializes in a super limited non technical frontend for customizing prebuilt web templates?
Employees are not robots. They are human beings. Sometimes human beings have human problems that need the assistance of other humans. This makes humans happier and more productive.
It's depressing to think that there are people who actually believe that optimal use of work resources is even worth calling out as an issue. In 2024.
Setting aside moral arguments, if it raises to the level of embezzlement, it’s a crime.
Using a small amount of that slack to keep another employee happy can be a good investment. In addition, it's good for someone like the CISO to poke around the innards of your network (etc) configs from time to time, just to stay up to date with what's going on in the company and to perhaps flag anything that smells suspicious.
You can do these kinds of exploration exercises completely free form, or you can take a little task like 'figure out exactly why this specific site is blocked' as a token of motivation.
I agree that all of this mostly only makes sense, if it doesn't take too much time.
Though if this specific task would take a lot of time, that would also indicate that either the CISO needs to upskill, or the network config is too complicated. In either case, that would be a valuable insight.
Sticking to your strict productivity line of thought, this kind of ask would:
1) be a great small teaching task for an intern, and
2) build goodwill elsewhere in a company, something good CISOs won't pass up an opportunity to do when the cost is relatively cheap.
But it's also likely that the CISO just wanted to help.
It's not that the smooth path you can get via nepotism is the base way things work which people who don't "know a guy" are excluded from. Rather, everything is falling apart and shitty, and if you're lucky, you occasionally get to circumvent that shittyness.
Well, obviously it isn't if you're not in the 1%. If you're in the 1% then that's the way the world has always worked and you don't know anything differently.
I don't believe that human society can, practically, get particularly close to the ideal. I question the choice of fatty meat as a substrate for minds.
For my money, I'd suggest that merit will get you further today than in the days of letters of recommendation, but that failures of meritocracy are more visible.
Where I am there is no forced disclosure, no costs costs assigned, and it is $150 to file.
And while a lawyer can represent a large firm, an employee has to be present, and the lawyer cannot use excessive legalise, the court is carried on in plain language... with the judge expaining things to you if you don't userstand.
That's pretty accessible.
The biggest risk is not knowing about no required discovery, and costs. Lawyers for big corp will ask for things, and hope you work your tail off. I just say no.
They will also elude to how expensive this will be, to which I typically snort.
Said large companies typically spend 50k to 100k on lawyers, and I spend $150 and a dozen or two hours of my personal time.
All very amusing.
Anyhow, a good equalizer.
Sadly, I think this would be instantly gamed by abusers. They would release the domain name and attempt to register as a new owner or start repeatedly doing handoffs. It's difficult to tell who the owner is changing between and whether or not the new one is a better actor than the former.
This doesn't seem like that hard of a problem to solve, because these are domains with negative reputation, i.e. worse than zero.
So if a) the domain is no longer hosting any of the stuff previously complained about and b) is no longer receiving new complaints over a period of a year, it costs you nothing to reset the domain to zero. Because the bad actors don't have to behave for a year to get back to zero, they can just register a new domain.
All you're doing is giving the new owner the same fresh start that anybody can get by buying a never before registered domain for the same price as a year's renewal on the existing one.
A google rank at zero and lots of 2 hop routes to your site that google can either penalize for being an accurate historical record or not is better than a rank of zero and a domain that has never been in historical artifacts.
If they were going to consistently use the same domain for links while they churn through hundreds/thousands a year for Google, the extra cost for one extra renewal for the persistent domain would be entirely negligible. And on top of that would make it trivial for Reddit/Facebook/etc. to disable all the historical links because they all go to the same scam site.
- Any empty domain starts with the same reputation
- Registering a new domain is a 0 cost action
- The eng effort to reset domain reputation is 0
Certain domains are used by abusers more often, usually due to them being cheaper. Forcing them to move domains is extra friction to the abusers which "haunted" domains force more than the proposed new system.
For the last point, I think it's simplifying a complex system change. Even if the new system was marginally better, it could be a large eng effort and not worth pursuing.
edit: styling
What basis would you have to do otherwise, and if there is something (like TLD), why wouldn't "resetting to zero" in terms of past content just mean resetting to that zero?
> Registering a new domain is a 0 cost action
No, that registering a new domain has a similar cost to renewing an existing domain, which is a valid assumption. In fact, the new domains are often cheaper because registrars often discount the initial registration as a loss leader with the expectation that people will make future renewals at a higher price.
> The eng effort to reset domain reputation is 0
It is the job of the party operating that system to make it operate as correctly as feasible. Needlessly causing collateral damage purely out of laziness and unaccountability is how you get people showing up at government offices demanding for you to be regulated or broken up, if not showing up at your offices with a disposition to cause bodily harm.
> Certain domains are used by abusers more often, usually due to them being cheaper.
Running out of domain names is physically impossible. There are more possible domain names in any given TLD than there are atoms in the observable universe. So the low price is going to be the price set by the registry for that TLD.
Whether the TLD itself has some reputation is orthogonal to the reputation of one domain in that TLD relative to another one in the same TLD. Moreover, you would presumably do the same thing for the TLD -- if one TLD is doing promotion and has $1 registrations this year and then gets used for a lot of scams, and then next year it costs $15 and so do the renewals so the scammers move to a different TLD, the reputation of the TLD should be reset just the same as the individual domains.
> Even if the new system was marginally better, it could be a large eng effort and not worth pursuing.
If the primary goal is to reduce engineering effort then the obvious solution is to delete the entire reputation system so it doesn't have to be maintained anymore. If the primary goal is to make it work well then you have to, well, you know.
Fair enough, but I'm not sure it resolves "haunted" domains as a TLD which is often abused could have a lower "0" reputation and thus by default is "haunted". Perhaps it lessens the impact though by how much is quite opaque to us.
> Whether the TLD itself has some reputation is orthogonal to the reputation of one domain in that TLD relative to another one in the same TLD.
I think this depends on how reputation works and is not so clear. Registrars for these TLD also have a responsibility but have no incentives to stop abusers. If TLD domain reputation is not orthogonal to reputation individual domains on that TLD then that could be an incentive for them to also crack down on abuse as their domains have bad SEO etc.
> If the primary goal is to reduce engineering effort then the obvious solution is to delete the entire reputation system so it doesn't have to be maintained anymore. If the primary goal is to make it work well then you have to, well, you know.
I think this is the most uncharitable interpretation. The eng effort could go to features that improves other customer experiences affecting more people.
A good technique to evaluate ideas though is to try and view it from different perspectives.
In this case from the owner of a non-haunted domain. Can you see any potential problem with your idea when viewed from that perspective?
Now, if there are potential problems, consider the relative sizes of the two groups. Do the benefits to one outweigh harm to the other?
This technique can be used every day with pretty much any idea.
I would want to experiment judging them based on what they’ve been seen to do in the past month.
If you remove the blacklist, they’d just stop doing that and it would be even easier for them.
Yes, the spammers can sit on their domains once blacklisted, renew them, and redeploy their spam on them 12 months later, but they'd have nothing to gain from the reuse, since the names of their domains are just nonsense anyway.
I’m guessing that would complicate blacklist maintenance quite a bit, which is why we aren’t seeing it work that way.
Most of these blacklists (at least initially) were emergency type measures - ‘block these spammers’, then move on with life.
Blacklist maintainers would need to maintain date first seen/date last seen info, and purge/re-add correctly.
Technically, seems like an ‘append only’ type thing is what they’ve been doing for the most part.
As this evolves and the idea that these do need some kind of expiration or we end up with more maintenance headaches becomes more widely known, maybe eh?
Or if there is some kind of legal rules around it.
I'm not up to date with SEO so unsure whether Google would (or is able to) reset the domain's backlink profile, I'd guess it would be possible. A lot of the value of using expired domains is for backlinks (or at least was)
So I checked the Bing Webmaster Tools. URL Inspection says "Discovered but not crawled - The inspected URL is known to Bing but has some issues which are preventing indexation. We recommend you to follow Bing Webmaster Guidelines to increase your chances of indexation."
That's quite unhelpful. What's more, when I open the "Live URL" tab, it says, in green: "URL can be indexed by Bing."
It's a simple static Hugo site hosted on Cloudflare R2 (DNS mapped directly to bucket). https://pagespeed.web.dev gives it a score of 100 in every category.
Anyone else had something like this happen?
It's a handwritten HTML website, enhanced with JS but not reliant on it, hosted on Cloudflare. Not quite a 100 in every PageSpeed category, but just about.
I've seen a few sites become de-indexed and the 'give away' is the type of results that first appear when the penalty is eventually lifted. For example, just a dozen or so urls with really weird query strings that never existed before. The real stuff does come back after time though and, in my limited experience, it's a one-off incident.
Just to add, not many sites are insignificant enough not to attract negative seo - especially this type of low-level, zero cost malarkey.
HSTS (which forces browsers to validate HTTPS when connecting) asks browsers to cache the configuration for a set "max-age". Some sites set huge values here, like Twitter's 20 year max-age[1]. There's also the preload lists [2] to consider. This creates a problem if you want to serve non-HTTPS/unencrypted HTTP on your new domain and the previous owner didn't.
MTA-STS [3] is another variant that's becoming more popular. It limits which mail servers your domain uses and enforces TLS certificate verification. "max_age" is capped to a year by the RFC. If you don't set your own policy, then the previous domain owners policy would impact any senders who previously cached the policy.
Thankfully HPKP (key pinning) is obsolete, otherwise you'd also need to worry about old pinned keys too. That RFC recommended, but did not enforce, a 60 day max-age limit.
These are especially tricky as the old security policy only lives in the caches of any end-user devices that previously connected to the domain. Double haunted.
[1] https://alexsci.com/blog/hsts-adoption/
[3] https://alexsci.com/blog/smtp-downgrade-attacks-and-mta-sts/
So the sender is supposed to obey the normal DNS TTL caching period, and re-query the assertion record if TTL expired. It should re-fetch the MTA-STS policy if the 'id' value in the DNS assertion changed, or the max_age in the previously fetched policy has expired.
> RFC 8461 section 3.3: Conversely, if no "live" policy can be [...] fetched via HTTPS, but a valid (non-expired) policy exists in the sender's cache, the sender MUST apply that cached policy.
You'll also need to host a "none" policy doc. Full instructions are here: https://www.rfc-editor.org/rfc/rfc8461.html#section-8.3
Look at the milka.fr problems... Milka is also a female name over here, and that already proved to be a problem in france. But so are Mirka and Minka so yeah... no domain for them? Also Micka. Oh and mivka is (beach) sand. Want to sell beach sand? It's just one letter away from milka, so no domain for you either.
Still seems better to raise the issue as early as possible so they can find a solution (appeal or chose a different domain) before investing into the unusable domain name. It would also mean that the dispute is at a layer (ICANN) where you at least theoretically have some rights instead of at the hands of a megacorporation that thinks the best way to reduce customer support costs is to make it impossible to get support.
Just one more place where the web gets screwed by a company too big to have to do basic customer service.
- knowing all the complexities of every local, state, federal, international jurisdiction that might interfere with the whitelist
- awareness of the content in question which could be millions of subpages
- a customer support team that is definitely not incentivized based on tickets triaged per day, but is somehow incentivized to spend hours on “whale” tickets.
- going through ticket history and solving the problem for everyone now that its policy to solve this
- dealing with the inevitable rush of fraud that follows every tiny change in google systems