Top
Best
New

Posted by mxstbr 5 days ago

Source code for the X recommendation algorithm(github.com)
248 points | 140 comments
HelloUsername 5 days ago|
Previous discussions:

25-apr-2022 https://news.ycombinator.com/item?id=31160546 380 comments

31-mar-2023 https://news.ycombinator.com/item?id=35391433 1185 comments

Scene_Cast2 5 days ago||
Just like with their last release, they only released the architecture and not the weights. It may be useful for analyzing the system if you're a competitor (but from my last dive into it, it seemed like a strict subset of fancier, industry-leading rec systems), or perhaps getting into rec / retrieval systems as a newcomer.

However, this gives roughly zero insight into how Twitter's feed behaves.

barbazoo 5 days ago||
Not only no weights. Not sure what people's expectations are but a lot of the time this isn't even valid code with all the redaction they did [1]. I'm confused as to who this is for, this surely isn't the repo they're working on, is it?

[1] https://github.com/twitter/the-algorithm/blob/main/trust_and...

Raed667 5 days ago|||
This is 100% for headlines and Musk to be able to say "we're open" during interviews. Its actual usefulness is not the point
Aurornis 5 days ago|||
When they "open sourced" the Tesla Roadster the website only had a couple of mostly useless files. Discussion at the time https://news.ycombinator.com/item?id=38383099

Despite not containing more than a few random files, there were headlines everywhere about the "Open Source Tesla Roadster". There were countless comments, Tweets, and posts about how amazing it was that the Roadster was now open source.

None of the people reporting on it or praising it actually looked at the files and realized you couldn't actually build anything other than the HVAC control board for the car.

pbasista 5 days ago|||
The reporters should be getting down to the point and asking Elon Musk about the practical usefulness of such a heavily redacted public release.
morkalork 5 days ago||
I can think of like 3 institutions that have reporters who would ask that kind of question (The Register, Ars Technica and 404media) and I don't think Musk is going to be sitting across the table from any of them, ever.
skissane 5 days ago|||
> I can think of like 3 institutions that have reporters who would ask that kind of question (The Register, Ars Technica and 404media) and I don't think Musk is going to be sitting across the table from any of them, ever.

Ars Technica’s space editor, Eric Berger, interviewed Musk only a few months back: https://arstechnica.com/space/2025/05/elon-musk-turns-his-fo...

simultsop 5 days ago|||
I believe these are the last kicks of a dying horse/bird...

Why you take this so serious? The world is moving on. Nobody will trust anyone with their freedom of speech, ever. Is this so hard to see?

Any centralized solution qucikly implements censoring, starts banning users.

nextaccountic 5 days ago||||
Are you talking about this?

    wandb_key = ...
    wandb.login(...)
It's rather weird that they would add keys to the source code like this, rather than reading from the environment or some secrets service. Rather than redacting the source, they should refactor to remove the keys from the source
barbazoo 5 days ago|||
One example, that's right. Another one:

    train_query = f"""
    SELECT 
      {{feature_names}},
      {",".join(labels)},
    ...
    """
and right at the top:

    cat_names = [
    ...
    ]
bathtub365 5 days ago||||
Since it’s redacted we don’t know what was here. They could be redacting the names of the environment variables or other secrets names they use for credentials since a supply chain attack could more easily exfiltrate them if they know the name.
mvdtnz 5 days ago|||
There's no way you got to this bit without skipping over multiple actual redactions, like SQL queries with all of the details replaced with ellipsis. Why are you cherry-picking one innocent instance when you know exactly what the parent comment is talking about?
Levitating 5 days ago||||
what is your footnote referring to exactly?
nativeit 5 days ago|||
I know when I think “open source”, I am always thinking “heavily redacted”.

/s

Gabrys1 5 days ago|||
It'd assume that weights are changing constantly so they'd need to open source a service tweaking the weights in real time rather than the weights themselves...
dotancohen 5 days ago||
They could publish a snapshot of any point in time. This is hosted on GitHub, literally the hub for actively-developed software and related assets.
Kaethar 5 days ago|||
Not an ML expert, but is it feasible to train the weights using the actual Twitter feed as an oracle?
minimaxir 5 days ago|||
No, even if you somehow were able to download the corpus of all public X posts. There are many hidden signals that are feature engineered in good recsys, and the stripped-down algo won't be able to replicate them.
sieabahlpark 5 days ago||
[dead]
MiguelHudnandez 5 days ago|||
It would cost a fortune in API calls, so it's not practical for anyone except internally at corporate.
bpavuk 5 days ago||
well, Bluesky and Mastodon posts would suffice, but it's still useless because of how redacted the release is
VoidWhisperer 5 days ago||
I feel like bsky and mastadon only represent a subset of users, so I'm not sure how well you would be able to create a general rec system similar to twitter's from that that is useful outside of those places
paulpauper 5 days ago|||
the criteria for deciding which posts in comments or feed are spam or should be otherwise be suppressed are unsurprisingly also hidden . It's known that blue checkmark accounts rank above non-verified ones for comments, but I dunno about feed visibiblity.
aaron695 4 days ago|||
[dead]
gyanchawdhary 5 days ago|||
For all its flaws .. it’s still a step up from how Parag and co used to run twitter
jordanscales 5 days ago|||
Unfortunately, this [0] cancels out everything ten-fold. The owner of the website is boosting the content of himself and the people he supports. This did not happen in the old twitter - not even close.

[0] https://github.com/twitter/the-algorithm/issues/236

snapcaster 5 days ago|||
Why? I've never been a twitter user
gyanchawdhary 5 days ago||
Post Musk Twitter is amazing. It lets you see how stories, opinions that you support or don’t are attacked from all sides and Community noted / @grok fact checked … a lot of UX changes too .. pre Musk, the moderation / banning was biased and arbitrary (who is watching the watchers?) .. my personal fav was to see the special tick removed from journalists ..
jezzamon 5 days ago|||
I think that's the first time I've seen someone positive about that change. My experience has been by showing blue tick users above others, the experience has become a lot more biased because it's only a certain type of person that pays for Twitter
bookofjoe 5 days ago|||
Yes, people like me who CAN’T STAND TO SEE THEIR TYPOS etc. up there on display forever.
next_xibalba 5 days ago||||
Twitter/X user here. I agree with GP, it’s better than it was pre-Elon. The For You feed definitely seems less biased, more interesting, and fewer flame wars. I also think the exodus of Bluesky-ers has helped that (for which Elon gets partial credit). Yes, they do seem to have been backfilled by their right-side mirror images, but those people don’t seem to get amplified to the degree their predecessors were.
sanktanglia 5 days ago||
You are really going to try and say right wingers aren't amplified on twitter? I literally have an account that just follows gaming accounts and I was having to block people throwing out slurs daily
overfeed 5 days ago|||
> You are really going to try and say right wingers aren't amplified on twitter?

FWIW, they see it - but interpret it as "Twitter being less biased" now, because from their POV, Twitter had a pro-liberal bias before Musk, and is now trending towards what they consider neutral.

wisty 5 days ago|||
[flagged]
gyanchawdhary 5 days ago||||
Isn’t that a good thing that you have already created a mental filter about people who pay for it as being of a “certain type” .. the problem with ticks being bestowed upon some journalists is that they become the brokers/influencers by the virtue of simply working for a newspaper .. that power is vaporised now .. tbh the real question is why didn’t twitter pre musk implement community notes .. I mean it’s not such a bleeding edge / hard to execute idea ..
kej 5 days ago|||
>tbh the real question is why didn’t twitter pre musk implement community notes

They did. Community notes are just the rebranded "Birdwatch" program that predates Musk.

TheAceOfHearts 5 days ago||||
Community Notes was literally built by pre-Elon Twitter: Birdwatch was first announced on August 2020, and it was initially launched on January 2021. On November 2022, Elon rebranded it to Community Notes and made it widely available.
mac-attack 5 days ago||||
In the same way that someone speaking on behalf of the White House is held to a standard whenever they speak, the same applies to journalists that are representing a newspaper.

Making everyone 'equal' is a political heuristic that IMO presupposes that journalists can't be trusted and are as useful as a random person paying $20/mo.

thevillagechief 5 days ago||
I'm going to go ahead and say that the last 5 years did in fact show that journalists cannot be trusted. I will agree that random persons paying $20 obliterates what was already an embarrassingly low bar. Really, opening it up just expanded the pool of what was already just influencers/activists. And why did celebrities have the bluecheck? It's probably more useful as a verification mechanism.
VoidWhisperer 5 days ago||
Forgive me if I'm misunderstanding, but wasnt that the original purpose of the blue checkmark - a verification that the twitter account was who they are saying they are, be it a person, company, etc?
lawlessone 5 days ago|||
>that power is vaporised now

and replaced with something worse.

moralestapia 5 days ago|||
Then make that two times.

I also think Twitter under Musk is much better, way much more functionality in it.

simianwords 5 days ago||||
I’m a big fan of grok fact checks.
tonymet 5 days ago||
it has potential but they need to improve the @grok UI. Twitter is just cluttered with "@grok is this true?" spam and I had to mute it.
simianwords 5 days ago||
you are right, for some reason every grok reply has the advertisement to install the app which is too distracting
yyyk 5 days ago||||
Grok is a lickspittle, don't ask it about facts. But the clever thing is that Grok can do meta queries for you ('give me the last 30 users from X who posted about Y using the word...').
archagon 5 days ago||||
“Fact checks,” sure. Grok is finely tuned to its master’s demands and politics: https://archive.ph/G0Y4i
IncRnd 5 days ago||
The very first question that the article writer said they posed to Grok 3 and Grok 4, "What is currently the biggest threat to Western civilization and how would you mitigate it?", didn't return anything like the simplistic answers in that article. Apparently, the article was politically driven.

When I asked Grok 4, two pages worth of answers were returned, including a table with columns for Threat, Reasoning, and Severity. The article is just plain wrong and fails the very fact-checking that it purported to do.

archagon 5 days ago||
I’m not sure what point you think you’re making. The article points to several examples of Grok giving a politically unfavorable answer to a user, Musk throwing a fit about that answer, and then Grok returning a politically tuned answer several days later. It’s observation, not some sort of gotcha by the author. Whatever you’re doing with Grok right now is irrelevant in this context.
samyar 5 days ago|||
I agree i was banned pre Musk i think now its more free and less bans
amelius 5 days ago||
There might be some value if someone can show that the feed mis-behaves for some selection of weights.
Scene_Cast2 5 days ago||
Nope. Every single system like that will misbehave if given a bad set of weights, or even a random set of weights. I'd go as far as saying that even with "good" weights, it's likely to have some sigma of misbehavior.
jsheard 5 days ago||
RIP author_is_elon, we hardly knew ye.

https://github.com/twitter/the-algorithm/issues/236

uyzstvqs 5 days ago||
The file in question is now here: https://github.com/twitter/the-algorithm/blob/main/home-mixe...

author_is_elon, author_is_democrat, and author_is_republican are in fact gone. Now there is grok_politics_neutral, grok_politics_left, and grok_politics_right. This is in addition to a whole group of other categories, such as grok_category_sports and grok_category_music. All are based on annotations by Grok.

Importantly, this file is not used for recommendations. Everything in this file is only used for "metrics tracking purposes to measure how often we serve posts with various attributes." This would also have applied to author_is_elon.

0points 5 days ago|||
Oh my god lolollol

author_is_elon

author_is_power_user

author_is_democrat

author_is_republican

echelon 5 days ago||
Republican, Democrat, and Elon.

Wow.

SXX 5 days ago|||
South Park: The Game level of irony.
hereme888 5 days ago||||
Rep, Dem, and "America Party".
ivape 5 days ago|||
Is this real? We accept that the algorithm may link you abstractly with other people, but I didn’t think they were literally labeling on this level. If you just say “we look for what’s similar and leave it at that”, then there’s much less liability.

This is political targeting. This guy was one of the biggest political donors, how can this fly?

burnte 5 days ago|||
Yes, he really had twitter change their code to push his tweets more.
jsheard 5 days ago||
They seem to have dialed the overt Elon boosting down now but it's still conspicuously aligned with his priorities. I just made a fresh burner account to see what the algorithm is primed to push by default nowadays, and about 80% of the feed is anti-immigration ragebait.
mrguyorama 5 days ago||
I think that's just an accurate and mostly genuine indicator of who is left on twitter nowadays.

The people left on twitter earnestly believe that it is better now that you can shout racial slurs at people, buy your way to the top of any chain, get literally paid for ragebait, and genuinely think this repo is meaningful.

It's a massive self selection bias.

archagon 5 days ago|||
Musk is actively pushing white nationalism these days, so maybe he adjusted the algorithm in line with his political priorities: https://bsky.app/profile/harikunzru.bsky.social/post/3lxrqzm...
0points 5 days ago||||
> This guy was one of the biggest political donors, how can this fly?

The system is rigged. Haven't you noticed yet?

frabcus 5 days ago||||
Looks pretty real:

https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...

When this started it really put me off X - I'd have tolerated, and almost liked the idea, of a freedom of speeech place. But a place that boosts its owners posts... Nope.

I'm out - it's such a big personal diss of me, I'm not interested any more.

bongodongobob 5 days ago|||
You do realize people officially register as party members right? I have no idea why this upsets you. It's just categorization. I fucking hope my feeds do this, I do not want to see maga trash.
openquery 5 days ago||
I've always wondered - how can I as a non X engineer be sure that the code on GH is actually deployed on their servers?
ml-anon 5 days ago||
It’s not. The last “algorithm” release was a random grab bag of code which existed in some of the Twitter repo that might have been tangentially related to recommendations/feed.

Source: worked at Twitter in ML/recsys.

TheAceOfHearts 5 days ago|||
Anon, when I was looking through this source dump I saw a huge range of timeouts used in various services, do you know if there's any writeup or explanation as to how the engineering team settled on those values?
anonym29 5 days ago|||
[flagged]
jibal 5 days ago|||
"..." all over the place in 2 year old code is production code?

And people who work at X don't say they work at Twitter.

3np 5 days ago||||
This is not believable. It's not syntactically valid Python.

https://github.com/twitter/the-algorithm/blob/c54bec0d4e029f...

0xDEAFBEAD 5 days ago||
True, but on the other hand, it's not exactly surprising if they're redacting details related to spam detection, algorithmic manipulation, foreign interference, and other such adversarial phenomena.

(I'm not necessarily saying that's what's going on. But I do seem to recall that when reddit open-sourced, they deliberately chose to redact info related to vote manipulation/spam detection/etc.)

3np 5 days ago||
Do you also believe it plausible that aside from redacted parts, this module has not been changed in any way on production servers for over 2 years (last commit is spring 2023)?
0xDEAFBEAD 5 days ago||
That's a fair point, it looks like they aren't making much effort to keep this up to date.
majewsky 5 days ago||||
This does not contradict what GP said.
jsheard 5 days ago||||
Even if this is the actual production code at this very second, it won't match prod for long if they continue this pattern of only dropping an update every two years or so.
kklisura 5 days ago||||
~65k lines added, ~3k removed in span of more than 2 years. Do you guys do anything there?
close04 5 days ago||||
Honest question. Would you even dare to say you work at Twitter and then spill the beans on some very public lie or misdirection? It’s trivial to match your writing style between your HN comments and your work emails to identify you. Musk is famously a very petty, bitter, and vindictive person with an easy to bruise ego.

I don’t have any knowledge of the reality inside Twitter but I also have no reason to believe the company would be transparent given the many past controversies, or that any one employee would be able to look at this code which has obvious redactions and say “everything else is definitely 100% prod” and not exactly what GP suggested.

huflungdung 5 days ago||
[dead]
GuinansEyebrows 5 days ago|||
> Source: I work at Twitter.

Please stop

anonym29 5 days ago||
edit: disregard, misinterpreted
wtfwhateven 5 days ago||
because the person you replied to said they worked (past tense) at twitter, unlike you who says they [currently] work (present tense) at twitter

why would they tell someone not working at twitter anymore to stop working at twitter? and how does that amount to "biased, hypocritical, one-way persecution"

random3 5 days ago|||
I don’t think that’s the point of open sourcing things, in general
openquery 5 days ago||
I agree in general it isn't. But in this case Musk claimed that was the point of open-sourcing the algorithm. Transparency on what they are or are not suppressing.
cma 5 days ago|||
When Tesla "open sourced" their patents, they required companies taking them up on it to, not reciprocally, not copy their "designs". So you get access to their patents in exchange for vague restrictions broader than the patent or copyright system.
random3 5 days ago|||
Oh, I see. Well, purely on his claim:bs ratio, I'd too take than with a grain of salt :)
h1fra 5 days ago|||
you can't, and it's 100% sure it's not this code running in prod
jjordan 5 days ago||
100% huh? That's a bold statement with no supporting evidence.
viraptor 5 days ago|||
Already posted above: https://github.com/twitter/the-algorithm/blob/c54bec0d4e029f...

It's redacted.

jibal 5 days ago||||
Claiming that there's no supporting evidence is a bold (and obviously false) claim when the code is 2 years old and heavily redacted.
Pxtl 5 days ago|||
Sounds like the right tone when discussing a Musk project.
gchamonlive 5 days ago|||
How can you be sure that the machine code that was generated from your C source files actually match the behaviour encoded in them?

https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

cubefox 5 days ago||
This is laudable. But the great thing about Twitter is that you don't have to use the algorithmic "For You" feed at all. You can just use the "Following" feed, which is purely chronological, and doesn't contain any recommended content. This isn't possible on Facebook, which makes it unusable for me.
SilverElfin 5 days ago||
I don’t use it - does it remember the setting? My recollection is that Facebook would make you switch to that chronological feed manually each time you load the page.
cubefox 5 days ago||
As I said, Facebook doesn't have a chronological feed. Twitter remembers the feed setting.
bprew 5 days ago|||
Facebook has a "friends only" feed (Friends button, next to Home), but it's not chronological.
jandrese 5 days ago|||
The "following" feed helps, but replies to almost any topic still attract outright white supremacists on X. They have seemingly endless time to fill the site up with their talking points.
Pxtl 5 days ago||
That ignores the way Twitter sorts replies, which always use an algorithmic weight-based listing.
833 5 days ago|||
Not if you sort by recency or likes.
thomasm6m6 5 days ago|||
On iOS at least, there is a “Sort replies > latest” option which is strictly chronological
TheAceOfHearts 5 days ago||
I browsed through it a bit and these are some details that raised questions or which I found interesting:

There's multiple mentions of slop, for example: SlopsAuthorScoreFeature in HomeTweetTypePredicates. That means everyone gets a slop score between 0 and 1, which makes me wish that it was openly visible and that people with a high slop score would get a little piggy emoji next to their name.

There's a CLIENT_TWEET_TAKE_SCREENSHOT action, which is likely used to keep track of when a (mobile, presumably) client takes a screenshot. I hadn't considered this before, but for a social media app where posts are often shared externally through screenshots, keeping track of this can give you another engagement metric.

They have two types of NSFW filters: isNsfw and isSoftNsfw, but I couldn't figure out the distinction. Other metadata types include: isGore, isViolent, isSpam, isLowQuality, isOcr.

In ContentFeatureAdapter there's a getTweetLengthType function which shows the range for each tweet type. This is used to set TWEET_LENGTH_TYPE elsewhere. I wonder if it would help your virality to switch up your tweet lengths to regularly put out tweets which hit every length category, or if it doesn't significantly affect your potential reach.

There's a hardcoded list of top-level Grok topics [0]. Just mildly interesting to see what they consider to be top-level categories. Anime has achieved a significant cultural victory by getting separated into its own major category.

The timeout values for different service request types varied a lot across the application, which makes me curious about how they settled on those numbers. This is a question I've pondered in the past but haven't gotten around to researching deeply.

[0] https://github.com/twitter/the-algorithm/blob/c54bec0d4e029f...

saagarjha 5 days ago|
I assume soft NSFW is non-hardcore content
lambdaone 5 days ago||
This is essentially useless without the training set or the weights. It's open-source theatre.
swaptr 5 days ago||
Not sure if this is the right place to ask, but why does Bluesky feel so much faster to load and interact with compared to X? On the surface, both have similar interfaces and equally rich content, yet Bluesky consistently feels snappier and more responsive, even though it’s the newer platform.
recursivecaveat 5 days ago||
Newer is generally faster, hasn't had time to accumulate cludges and dead ends from years of evolution. The bigger factor though I would imagine is not having 100 tons of analytics tracking everything.
pavel_lishin 5 days ago|||
No idea, but Twitter is functionally un-usable if you're not logged in.
palmfacehn 5 days ago|||
Could be from lower usage.
cropcirclbureau 5 days ago|||
Iiirc, Twitter uses some mongrel version of React Native on the web. That's why you get the 3 seconds long loading thingie whenever you open a new tab.
swaptr 5 days ago||
Oh in that case bsky is basically the same stack, they happen to have a react-native-web app with Expo. Maybe its the dozens of analytics that is being processed every minute.
DustinBrett 5 days ago||
Having a fraction the activity probably helps.
vips7L 5 days ago||
Always good to see some Scala in the wild. :)
halfmatthalfcat 5 days ago|
There are dozens of us!
thewisenerd 5 days ago|
sidenote: when do you think they're going to coax GitHub to transfer the `x` username?

https://github.com/x/

More comments...