Posted by nomaxx117 1/23/2026
Basically, my understanding (simplified) is:
- they originally had a Miami router advertise Bogota prefixes (=subnets) to Cloudflare's peers. Essentially, Miami was handling Bogota's subnets. This is not an issue.
- because you don't normally advertise arbitrary prefixes via BGP, policies were used. These policies are essentially if/then statements, carrying out certain actions (advertise or not, add some tags or remove them,...) if some conditions are matched. This is completely normal.
- Juniper router configuration for this kind of policy is (simplifying):
set <BGP POLICY NAME> from <CONDITION1>
set <BGP POLICY NAME> from <CONDITION2>
set <BGP POLICY NAME> then <ACTION1>
set <BGP POLICY NAME> then <ACTION2>
...
- prior to the incident, CF changed its network so that Miami didn't have to handle Bogota subnets (maybe Bogota does it on its own, maybe there's another router somewhere else)
- the change aimed at removing the configurations on Miami which were advertising Bogota subnets
- the change implementation essentially removed all lines from all policies containing "from IP in the list of Bogota prefixes". This is somewhat reasonable, because you could have the same policy handling both Bogota and, say, Quito prefixes, so you just want to remove the Bogota part.
HOWEVER, there was at least one policy like this:
(Before)
set <BGP POLICY NAME> from is_internal(prefix) == True
set <BGP POLICY NAME> from prefix in bogota_prefix_list
set <BGP POLICY NAME> then advertise
(After)
set <BGP POLICY NAME> from is_internal(prefix) == True
set <BGP POLICY NAME> then advertise
Which basically means: if you have an internal prefix advertise it
- an "internal prefix" is any prefix that was not received by another BGP entity (autonomous system)
- BGP routers in Cloudflare exchange routes to one another. This is again pretty normal.
- As a result of this change, all routes received by Miami through some other Cloudflare router were readvertised by Miami
- the result is CF telling the Internet (more accurately, its peers) "hey, you know that subnet? Go ask my Miami router!"
- obviously, this increases bandwidth utilization and latency for traffic crossing the Miami router.
This didn’t catch the fact that removing that line essentially removed all conditions, allowing received routes to be re-advertised by the Miami router.
Communities are useful in this case, but this kind of thing could have happened with any kind of configuration.
Example:
(Before)
set firewall family inet filter FILTER NAME term TERM1 from source-address 10.10.10.1
set firewall family inet filter FILTER NAME term TERM1 from destination-port ssh
set firewall family inet filter FILTER NAME term TERM1 then discard
What happens when you remove references to 10.10.10.1, maybe because that IP is not blacklisted anymore? You’re simply removing one condition, leaving all ssh traffic to be discarded. That’s essentially what happened with the BGP outage, only here you have no BGP communities to save you.
That’s why I re-read the RCA, because this kind of incident is way more general than BGP-specific misconfigurations.
Why even bother to write an article about it then haha