When you open the For You tab, X has a few hundred milliseconds to reduce everything posted in the last 48 hours down to the 35 posts in front of you. The code that does it is open source. This article walks that code end to end: the half that runs while you sleep, the half that runs when you ask, and the places where what the system rewards is a deliberate, legible choice.
[reach]why your post travelled or didn’t[shadowban]how content gets hidden[eng]implementation detail[recsys]for people who build recommenders[policy]a value judgment encoded in codeThe X engineering team open-sourced a much larger portion of their recommendation system earlier this month, the third update since the original release in January, and by far the biggest. Both of us were invited to review the code with the team before it went public.
We’ve spent the last few weeks reading it, running it, and arguing about it, and we had more fun than we expected to. We think you will too, and not only if you build recommender systems. If you just use the platform and have ever wondered why one post got 10,000 views and a near-identical one got 200, the answer is in here somewhere.
Two notes before we start. We assume you have some technical background and the patience for something longer than a thread; in exchange, we’ll try to make it more interesting than a lecture. And everything here we read, traced, and where we could, ran ourselves.
As for why you’d read this instead of just cloning the repo: the code is organized the way software is organized, by service, by module, by team. It is not organized the way the system actually runs. So that’s what we’ve done here instead. One post, from the moment it’s published to the moment it lands in someone’s feed, in the order the machine actually touches it.
2Every stage you are about to walk through is an instance of the same small set: go fetch things (source), fill things in (hydrator), throw things out (filter), put a number on things (scorer), sort and cut (selector), do something afterward (side effect).
3In Section B, we’ll go through the For You page. When a user opens a request, when the user opens a For You page, within hundreds of milliseconds, this is what happens
3.1This is where the whole For You page gets initialized when a user enters the app by signaling the home-mixer api for the timeline curation / generation for a given validated viewer_id. A For You gRPC request hits get_for_you_feed_urt. Before any posts or ads are fetched, QueryBuilder::build turns that request into a ScoredPostsQuery.
3.2So then, the section decides what appears on the timeline besides posts. These seven sources listed below run in parallel, each contributing to a different type of item.
We look you up before looking at any post. There are 17 parallel lookups, all about the user:
Zero posts are involved yet. The system builds a picture of the reader before it fetches anything to read. This is what we meant: it really reads the user action sequence, the graph, and the state.
Once we get enough understanding about the user, we try to go into several sources, including:
This results in a wide, personalized pool of around 3,000 to 4,000 posts that will be hybrid-filtered and scored at a later stage. Right now, however, they are just ID numbers of posts we think you will be interested in.
If (3) found ≥ 500 cached posts, has_cached_posts = true and the request takes a genuinely different code path: only the cached-posts source enables, all six live recall sources disable themselves, and many downstream hydrators plus PhoenixScorer itself skip too (cached posts carry their stored scores). It’s a reuse mechanism for pagination/refresh, not a seventh recall algorithm.
3.5in-network flag · mutual-follow flag · core post data · quoted post · media · subscription status · author data · blocks in both directions · filtered topics · language · engagement counts · semantic ID
Now you get a huge list of IDs, but what are they? What really are those posts? It’s like stock; it’s just paper money. The IDs are really useless on their own.
Twelve more parallel lookups fill each in: the text, the media, the person who posted it, the language, basically the content. This process is what we call the “hydrator.”
3.6Filters here for the obvious notes: there are 18 checks in order.
The actual execution order is:
There are a few interesting filters at this stage:
First is the 48-hour ceiling: the max post age is 48 hours, which matches the retention system. The two independent systems basically agree on the same number, meaning the feed supply window is two days. After that, a post is not just suppressed, it completely ages out of the candidate pool. One caveat worth being precise about, because it is easy to overstate: AgeFilter ignores the query entirely, so within this pipeline the cutoff is unconditional — but this pipeline is only one of the seven sources feeding the timeline. Jetfuel frames, push-to-home, ads and who-to-follow never pass through it, and a conversation module can hang an older reply off a recent post. The 48 hours binds out-of-network post recommendations, not everything you will ever see in the app.
Then there is the hardcoded legal filter on the Brazil 2026 election, which includes a fixed list of accounts excluded from recommendations, with the Brazilian electoral court’s statutory language quoted directly in the source comment, which I think is pretty interesting.
3.7Read user hash embedding
Now we are finally onto AI.
Basically, in one pass, it is fed: this is this person, this is their last 1,024 actions, and here are 3,000 posts. For each post, it returns a set of probabilities: that the user may like it (12% chance), reply (1%), share (2%), or report it (0.01%). So the AI here, which is the Phoenix architecture, predicts the behavior. Again, nothing is ranked; just the behavior is predicted.
This stage makes a judgment about the user. Surviving candidates go to Phoenix in a single request carrying your action sequence and the post. It looks like: you, your history, candidate 1, candidate 2, through candidate N. An attention mask lets each candidate attend to you and your history, but never to another candidate, enforced inside the attention kernel rather than masked afterward.
This constraint buys three things:
There are two small details to reinforce:
Your history is right-anchored (the newest event always sits at the same position), and every candidate is pinned to the same position encoding. The candidates have no order among themselves, not even implicitly.
The cost is real: the model cannot reason about a set of posts. It cannot notice that two candidates are near-duplicates or that the top 10 are all from one person, which is why those problems must be filtered downstream: by the value model, the diversity re-rank, and visibility filtering.
There is really no vocabulary anywhere; every ID is hashed twice into a shared table rather than looked up in an index. Memory is fixed no matter how large the platform grows. The price is that collisions are permanent and invisible: there is no row that belongs strictly to your post, only a bucket shared with others.
Your learned user embedding is read as a row in the same hash table fitted during training. Nothing about it is interpretable, no axis means anything, and it is not uniquely yours. It is a small piece of personalization; that 1,024-event history carries far more signal than a single user vector.
3.8The weights are global, personalization lives within the model.
Phoenix takes user action sequence [read at query hydration] (up to 1024 recent engagements) and learned user embeddings [read at the ranking model] and predict P(action) - how likely would a user react (reply/ like/ share/ report) certain posts
User graph & state [read at query hydration] also used
Wi is the same set of global rules
Weights here, or we call params, are the things here.
System is steered away from attention maxing (but prompted with several policy question)
Alongside the 26: BidirectionalFollowReplyWeightBoost 15.0 · BidirectionalFollowDwellWeightBoost 0.0 OonWeightFactor 0.75 · TopicOonWeightFactor 0.5 · AuthorDiversityDecay 0.5 · AuthorDiversityFloor 0.25
Three of these changed during the August release window (13 → 28 Aug): Vqv 0.05 → 0.0, VideoOpen 0.05 → 0.07, and Dwell 0.0 → 0.05 in commit 0d3cdd8 on 25 Aug.
The scored candidates then go through a separate service running greedy MAP-DPP (that is, the Determinantal Point Process), which picks a subset that is simultaneously high-quality and mutually dissimilar. Quality comes from the score just computed; dissimilarity comes from a cosine distance between post embeddings.
Two things to focus on here:
Unselected candidates get a score of zero. Because this runs before the top-50 selector, zeroing is functionally just dropping, and diversity is implemented as score suppression (the codebase’s general idiom for gating).
It overrides everything upstream. VMRanker returns its own score for the posts that are selected, so the 26 weights are subject to replacement by a model whose parameters are not in the repository. That is an honest caveat on the entire scoring disclosure: you can read exactly how the linear value model works, yet a separate service gets the last word.
There is another really interesting detail that we noticed when reading the codebase: when a post embedding is missing, it is replaced by a random unit vector. To the diversity algorithm, this looks absolutely maximally diverse, so missing data is not really penalized here.
3.11Sort by score and keep 50
3.12This is the only stage that can make a post cease to exist for you. For each survivor, the safety system decides whether to show it, blur it behind a tap, or not show it at all. Everything before this only reorders and filters out.
For each surviving post, HomeMixer asks visibility filtering a question that gets one of three answers: allow, interstitial, or drop. The rules evaluate in order: the first drop wins, and the name of the deciding rule becomes the telemetry.
Very importantly, HomeMixer asks two different questions: one for in-network posts and one for recommendations. The difference between the two is really the mechanism people mean when they ask, “Am I getting shadowbanned?”
The two rule sets share a base, but recommendations get an extra block of drop-only rules appended. Several signals appear with different verdicts: interstitial in-network, drop out-of-network. If you follow the author, the post is blurred and you can tap through; if you don’t, it simply isn’t seen. Same post, same label, same moment, but two different realities decided by whether you had already chosen to see the person.
It fails open inside and fails closed on the edge. With visibility filtering, an outage in an upstream store resolves to no blocks and no mutes, so content flows. At the boundary, the default flips: an unresolved author is dropped, and any post ID missing from the response is backfilled by the client as a drop.
Then, posts whose quoted posts or thread ancestors were vetoed are dropped too, and only the highest-scoring branch of any conversation survives.
3.13Blending basically spaces things out. Ads get spaced out, each with a safe neighbor:
Structurally, ad placement is a separate, deterministic step applied after ranking finishes. So nothing in the 26 weights is about revenue, which is, I guess, really good.
3.14Then, after you already have your feed, it writes down what it did: a sample of the top 50 with the exact weights used, so tomorrow’s model can learn from today’s feed. And it caches the leftovers for your next scroll.
The result is marshalled into a Thrift URT timeline, cursors, the new-posts pill, “not interested” feedback actions, conversation modules, and capped at 47 items (RESULT_SIZE(35) + FEED_MODULE_SLOTS(4) + MAX_JETFUEL_FRAMES_PER_RESPONSE(8); 38 is RANKED_FOLLOWING_MAX_RESULT_SIZE, a different surface).
Then, without blocking the response, 17 side effects fire. Three matter enough to name in the feedback loops.
4A post is created and immediately becomes an event on Kafka: tweet_events. From here, it takes two entirely separate journeys that never rejoin until someone’s feed is being built.
The first journey is making it available (Thunder) . It goes into a memory store so that people who follow the author can be served it within milliseconds.
The second makes it understood (Grox). A battery of models and rules decide what the post is and whether it’s allowed to travel, and publishes its own set of content-understanding topics.
Neither journey knows about the other. This separation is the single most important structural fact in the system. It is why we get questions like “Why was I ranked so low?” or “Why did my post disappear?” All of those questions are actually about these two different machines.
4.2Thunder is written in Rust. It holds every post on the platform under 48 hours old entirely in RAM, indexed by author.
The architecture is straightforward:
It is a very narrow job. The system is fed with the list of accounts that you follow and returns recent posts really fast.
Key specifications:
This confirms something people often assume isn’t true: the in-network half of the feed involves no ML at the sourcing stage. It is simply a time-ordered list of everything the people you follow have posted in the last two days.
4.3Grox is used to understand content. The posts you post get read, with the text and images rendered into a single interleaved document, handled by a vision-language model, Grok in most modes, though a 26B Gemma model is paired with Grok for three of five modes (STANDARD, RECOVERY, LIVE_CLUSTER_ANCHORS), with Grok alone for DELUXE and BACKFILL.
The safety classification here runs in two stages:
The first is a call named “decide,” which determines which broad policy categories are violated.
A focused call per violated category that pins down a specific leaf policy.
The model routing varies by category and by how much traction the post already gets. This is the only component in the entire system that reads content (the ranking model never does).
On the redaction, fairly stated, the prompt templates are deliberately withheld to (according to our conversation with the X engineering team) prevent people from reverse-engineering the safety guards and to prevent people from A/B testing their way around it. But you can see the policy taxonomy being public, even though the wording isn’t.
4.4Following Grok, there are a few more media models specifically trained by the team that handles media and trust and safety:
A stateless proxy sits in front of the fleet, fetches each piece of media once, and pushes it out to every model. The media score is then fused with an account-level health score before a verdict.
This means the picture is judged by all the policies that go into media model policies, as well as who posted it.
4.5Now you know that a post is judged by both the media model policies and the account-level scoring. In this section, we will dive more into the account-level scoring.
There are practically three systems that score the account using very different methods:
AGATHA computes how other people react to an account (block, report, spam report) and normalizes it per out-of-network favorite. Using the favorite as a denominator is a very neat idea: it controls for audience reach, so a large account is not penalized simply for being seen more.
BDSM (not the BDSM you are thinking of) is a transformer over the account’s chronological action sequence, with positional encoding derived from real timestamps rather than token index. It tracks the chronological history of the user to see the rhythm and timing of what you have done, scoring a set of inauthentic behavior heads. Enforcement requires two thresholds to be crossed at once, and an account near the boundary gets a liveness challenge instead of a suspension.
user-cred-v2 (user credibility) uses PageRank over the follow and engagement graphs to produce a reputation score.
4.6Model scores are not labels: something has to decide that a score of 0.83 on some head means a post gets marked.
There are three services around the rule engines that turn model scores into labels of some sort: Bot Maker, Scarecrow, and Abuse Enforcement Service.
Bot Maker: An in-house domain-specific language for rules of the form “if this condition, take that action,” compiled and hot-reloaded on a short interval. Despite the name, it has nothing to do with detecting bots; the “bots” here are the little automated rules themselves (in my understanding, like the little soldiers of Trust & Safety).
Scarecrow: A deployed instance running on the rule package that ships in botmaker-rules.
Abuse Enforcement Service: The service that works on things at the account level, evaluating rules over model score streams and issuing labels, challenges, and suspensions (with credibility, exemptions, deduplication, and daily caps).
The account pipeline we are talking about here defaults to suspend, while the post pipeline we talked about earlier defaults to skip. The default posture is obviously stricter for people than for posts.
4.7safety-label-user-agg (aggregation) closes the loop. It consumes the stream of label events that we were mentioning from content understanding to the rule engines, rescans the author’s recent posts, and writes in account-level labels based on windowed counts, which the visibility system then reads from the next request.
So, labels on your posts get added to your account (not to the posts themselves). This changes how your future posts are treated, which then produces more labels. The labeling process there becomes a closed loop.
All the above that we described is stored in three tiers:
Cache lifetimes are derived from the post creation timestamp, so recently published posts recheck their labels more often than old ones: a post under five minutes old gets a 30-second TTL, anything older gets 60 seconds. A fresh post that gets classified sees the consequences within half a minute.
These stores are the entire interface between the two halves of the system. The labeling pass never calls the request pass. It just writes the labels to the store.
4.9Before a post can be recommended to strangers, it has to enter the retrieval index, which is what Phoenix-RankAll builds. Collections of eligible posts are windowed and materialized into snapshots that the retrieval model loads.
The critical detail here is in the layer that decides membership: before a post joins any index, that layer asks visibility filtering whether the post should be dropped at the recommendation safety level (you can see the call at event processing configuration).
So safety is really applied twice, at two different times and in two different senses:
Once here at index admission, globally and in advance. A post that would be vetoed for recommendations never becomes a retrieval candidate for anyone.
Once again at request time (via the visibility check), per viewer, with the answer depending on who is asking.
This matters for the transparency question more than almost anything else in this repository. “Was my post filtered?” has two possible answers at two different times, and only the second one is personal. The first one happens before any particular reader exists.
4.10This is the most novel thing in this release, and the part a recommender system reader will find the most directly useful. If you are an engineer working in recommenders, you might also find this quite interesting.
Worth saying first where those embeddings come from, because the article has not so far: a post’s text and images are encoded by Qwen/Qwen3-VL-Embedding-8B (phoenix/reference/mm_encoder.py). That is a different model from the Grok and Gemma pair doing safety classification, and it is not X’s own — they ship the embedding algorithm verbatim and leave you to supply the stock public weights. So the vectors being quantised below are reproducible by anyone.
This is exactly the process of how a post becomes six tokens: rather than represent a post by an opaque embedding, the Phoenix retrieval path quantizes embeddings into semantic IDs (six levels of 256 centroids each, produced by a residual quantized k-means). The training loop is about 200 readable lines: fit centroids at level one, subtract to get the residual, refit the next level’s centroids on that residual, and repeat six times (in phoenix/reference/sid_codebook.py).
There are three reasons why this is really important:
It uses k-means, not a learned VAE. Semantic IDs are usually associated with the RQ-VAE approach, which requires auxiliary training and careful handling of codebook collapse. This does the same job with a clustering algorithm and no extra model. If you have been putting off adding semantic IDs because of the training overhead, that objection just got weaker.
The address space is enormous, and the representation is tiny. 256 raised to the power of 6 is roughly 2.8 × 10^14 addressable items, in just 6 tokens per post.
They are not just an index, but are truly in the user history. Semantic IDs are hydrated into the user history at request time through a dedicated lookup service, and the retrieval tower is then trained with them. Remove them and the checkpoint no longer loads; that is a far deeper architectural commitment than using semantic IDs for indexing only.
4.11SimCluster is a candidate generation method dating back to 2020, still carried forward into the 2026 release. It is definitely one of the methods from the 2020 era, sitting alongside a transformer, and is still here because it surfaces something that the newer models don’t: a community index.
SimCluster factorizes the follow graph into roughly 145,000 communities, then propagates community members to users, posts, and topics. A streaming layer maintains decayed community-to-post indexes, and an ANN (approximate nearest neighbor) server answers queries under a tight deadline.
5The ranking model has to learn somewhere: it learns from logs of feeds that have already been served (here is what we showed someone, here is what they did, adjust, repeat). Training reads either the live event stream or the Parquet dumps, converts them into batches, and splits them into in-batch negatives: candidates borrowed from other users in the same batch, used as examples of items not engaged with. The loss combines cross-entropy across the action heads with a regression term for dwell time.
You can actually run this, which is the most reusable and coolest part of this release. They ship a synthetic data generator, so the entire loop runs with no access to anything of theirs.
The loop has five main steps, all inside the reference folder:
What you cannot do at all here is train on real data: while the loop is real and complete, the data is entirely synthetic.
There are also no reference metrics. We fed this back to the Twitter team, but it is very difficult for them to provide them. Nothing tells you whether your run is correct: no expected loss curve, no held-out numbers, and no baselines. There is only a measured timing section, which measures wall-clock performance, not quality. If you want to know whether your training actually worked, it is essentially just a vibe check.
5.2So there’s actually very little cross-service communication per se. All services write rows into the shared database, and everyone just reads from the common knowledge of the cross-model relationship in the system. They’re all store-mediated. Except for home mixers fanning out in scoring and filtering services, all else travel through a durable store in a Kafka topic.
This is very much a system-level insight of the whole release: the architecture is not really a call graph, but a set of servers agreeing on a shared schema. That is why the two paths (A and B) can develop independently, why there is really no inter-service communication, and honestly, how we should describe a change in which the store’s content is changed.
5.3The training log. A sampled Kafka record of each served top-50, carrying the exact applied weight map, so offline analysis can reproduce any score that was actually served. The cached model request is its join key. Today’s feed is tomorrow’s training data.
The warm-cache fast path. This request’s top-scoring posts are cached and become the next request’s candidate source (the candidate sources), bypassing recall and scoring entirely.
The label loop. Post labels aggregate into account labels, which change how future posts are treated (the label aggregator).
Each loop is short and each is invisible from inside a single request.
Idhant Gulati and Yushan Li read, traced, and ran the open-source pipeline, and both reviewed portions of the code with the X engineering team before public release.
Reviewers, mentors, and friends we learnt a great deal from: Vaibhav Kumar, b1f6c1c4 Tu, Matvei Popov, and Jonathan Stray.
All references are to xai-org/x-algorithm at commit
5e40600 (upstream bc8e5f0), 28 August 2026, unless noted
otherwise. The one exception is the value-model figure: it reads
home-mixer/params/param.rs from main each time the page
loads, so those weights are whatever X is shipping now rather than what they were
when this was written. It says which it is showing. Three of them changed during the
fortnight we spent writing.
For attribution in academic contexts, please cite this work as
Li & Gulati, "Inside the Machine: A Technical Analysis of X's 2026 Recommendation Algorithm", ysli.dev, 2026.
BibTeX citation
@article{li2026inside,
author = {Li, Yushan and Gulati, Idhant},
title = {Inside the Machine: A Technical Analysis of X's 2026 Recommendation Algorithm},
journal = {ysli.dev},
year = {2026},
note = {https://ysli.dev/writing/x-recsys/}
}