5:16 in the afternoon, Manila time. I was on the third floor of a co-working space in Makati, halfway through an argument with a colleague about whether the peso's slide against the dollar was a liquidity story or a current account story โ the kind of argument we have every week and never win.
Then my phone started buzzing. Not one message. Eleven.
The first came from a founder I had introduced to a Singapore fund a month earlier. Two words: GitHub down. The second came from a developer at a regional exchange. Can't merge. Can't deploy. Can't ship. The third was a screenshot of a status page โ the same three timestamps half of us ended up staring at that evening: 17:16, 17:25, 17:36. API. Issues. Pages. Pull requests. Actions. Every one of them listed as degraded, in a slow crawl of updates that never quite said what was actually broken.
Here is the thing that hit me before any of the technical detail did: nobody in that group chat asked how long it would last. Everybody asked whether it was time to start mirroring their repositories somewhere else.
That is a different question. That is the question of trust.
We didn't panic about the code. The code was fine. Every line of it sat exactly where we left it. What broke was the layer that lets humans touch the code โ the replication layer, the auth layer, the shared state sitting underneath the green buttons. And if you have spent any real time tracing the dependency graphs of modern financial infrastructure, you already know that the basement is where the story always lives.
This is not an article about an outage. It is an article about the basement.
WHAT ACTUALLY HAPPENED, STRIPPED OF THE MARKETING
Start with what this was not. It was an availability incident. Not a breach. Not an exfiltration event. Not a compliance story. Nobody's private keys walked out the door. That distinction matters, because the reflex in our industry is to read every infrastructure headline through a security lens, and this one does not belong there.
Now the causal chain, as stated: a replication delay in the database backing collaboration systems, which produced elevated errors on an authorization interface, which drove up the overall system error rate.
Read that chain again. The whole article is in three clauses. Everything I am about to say for the next three thousand words is an unpacking of those three clauses, because that sequence โ stale data, then broken permission checks, then a global error rate spike โ is the single most predictable failure mode in modern software, and almost nobody outside of infrastructure teams can see it coming.
Here is the architecture, and none of this is a leak, it is all publicly documented engineering. GitHub's data layer is understood to be sharded MySQL with an in-house replication system sitting on top of the Vitess lineage. Replication, by default, is asynchronous. The primary commits. The replica applies the change later. On a good day, that gap is milliseconds. On a bad day, it is seconds. On a very bad day, the replica serves reads that are minutes behind, and every service that trusted it serves garbage with a completely straight face.
Why does authorization break first? Because an authorization interface is a read-heavy, latency-sensitive, correctness-critical path. It answers a simple question: does this token belong to this user, does this user hold this permission, is this organization allowed to do the thing it just asked to do. If the replica is stale, the answer is wrong. And a wrong authorization answer does not fail quietly. It fails loudly, everywhere, at once. It fails at the API boundary. It fails in the IDE plugin. It fails in the bot that opens the pull request. It fails in the security scanner running on a schedule nobody remembers setting up.
The blast radius covered five surfaces and each one matters for a different reason. The API is the largest, because every third-party integration speaks it โ bots, IDEs, CI providers, dependency graph jobs, security tooling, dashboards. Actions is the CI/CD pipeline: build, test, deploy. Pages is static hosting, which is where an embarrassing amount of documentation, status pages, and grant-program microsites live. Issues and pull requests are the collaboration surface, the place where work is actually negotiated.
So this was not the website being slow. This was the shop floor going dark.
And we have seen this shape before. In October 2018, a network partition between East Coast and West Coast data centers triggered an orchestrator-driven MySQL failover that went sideways, and the platform spent roughly twenty-four hours in various states of degradation. The postmortem that followed is still one of the best pieces of engineering writing that company has ever published, and it is still required reading for anyone who operates stateful systems at scale. Same shape. Data layer anomaly. Cascade. Elevated global error rate.
If the shape rhymes six-plus years later, you are not looking at bad luck. You are looking at long-tail technical debt living in the hardest possible place to retire debt, which is the stateful core.
Now the part that frustrates me as an analyst. The available record gives us three timestamps nine minutes apart and no date, no duration, no ETA, no quantified impact, no root-cause conclusion. The status page is a genre with its own conventions โ passive voice, present continuous, the words investigating, monitoring, resolved. It is written to be legally defensible, not to be informative. So I cannot tell you whether this was a nine-minute blip or a nine-hour slog, and neither can you. Any confident claim about severity is a claim made about a document that was never designed to carry that information.
Why does the Manila angle matter here? Because this city's developer economy runs on that platform. Grant programs, hackathon submissions, DAO contributor bounties, the entire open-source contribution ladder that a thousand engineers in Southeast Asia are climbing โ all of it is mediated by pull requests and CI runs. When pull requests stall, payouts stall. When CI stalls, the underfunded team that deploys on Friday afternoon just does not deploy, and the release slips a week. And I say this as someone who spent the 2022 bear market organizing meetups in BGC rather than reading postmortems. I know exactly how easy it is to let this slide.
THE DEPENDENCY GRAPH NOBODY DRAWS
Let me trace it out loud, because I do not think most people in this industry have ever actually written it down.
Take a mid-sized crypto exchange, or a DeFi protocol, or a wallet, or an indexer. Anything real. What does it actually run on?
Cloudflare for DNS and edge. AWS us-east-1 for compute, because everyone else is there and the latency budget assumes it. GitHub for code and CI. Discord and Telegram for operations communications. X for announcements. A handful of RPC providers โ Infura, Alchemy, QuickNode, and maybe one self-hosted node that has been promising to be the backup since 2021. A handful of oracle networks. A handful of analytics vendors.
Count the names. You get to about eight, and three of them are load-bearing. If you removed any one of those three for a week, most of this industry would stop functioning, and it would stop functioning in ways that would not be visible on any dashboard until it was too late.
GitHub is not a nice-to-have for this sector. It is the factory floor. Smart contract repositories, client implementations, node software, indexers, bridge contracts, monitoring configurations, the deployment scripts that push to mainnet at 2 a.m. because the gas is cheap. All of it. When Actions degrades, deploys stop. Not because the chain stopped. The chain never stops. Bitcoin kept producing blocks the entire time. Ethereum kept finalizing. Finality did not care.
The chain is always up. The layer that humans touch is not. That sentence is the whole thesis, and it applies to almost every infrastructure conversation we have.
Here is the crypto-specific sting, and it cuts at something I care about. The tooling around Bitcoin is exactly this kind of centralized. Ordinals indexers, the ord client, community explorers, wallet backends โ these are repositories, maintained by small teams, hosted in the same place as everything else. The inscription wave gave Bitcoin something it badly needed: a fee market and a narrative, at a moment when the security-budget conversation was getting genuinely uncomfortable for anyone who was paying attention. I have been arguing for a while that without that wave, Bitcoin's fee-revenue story would look considerably worse than the price chart suggests. But the tooling layer that made that wave possible is a stack of hosted software and a handful of volunteer-maintained indexers.
Consensus did not blink. Everything around consensus did.
The same sentence applies to every NFT marketplace indexer, every DeFi dashboard, every liquidation bot that runs on a VPS in Frankfurt and a scheduled job in a public repository. The chain is the part that cannot break. The part that touches it, all of it, is the part that can.
REPLICATION LAG IS THE UNIVERSE REMINDING YOU ABOUT PHYSICS
I want to sit with the actual failure mode for a minute, because I think the industry treats it as a boring implementation detail when it is in fact a law of nature with a configuration file.
Replication lag is not a bug. It is a tradeoff. You choose asynchronous replication because synchronous replication costs you latency on every single write, and latency on every single write is a business problem, while replication lag is a theoretical problem that only becomes real on the worst day of the quarter. So you take the trade. Then you spend the rest of your engineering life managing the consequences of having taken it.
The consequences have names. Read-your-writes violations, where a user writes a record and then does not see it. Stale reads, where the system confidently reports a state that has already been superseded. The infamous case where a user exists on the primary and does not exist on the replica, so a login succeeds and then immediately fails, and support gets a ticket that reads like a ghost story.
Underneath all of it is the CAP tradeoff. Partition tolerance is not optional in a system whose data centers sit on opposite sides of a continent. You are choosing consistency or availability on every single request, whether you say it out loud in an architecture review or not.
Every system that promises you a live number is quietly promising you a slightly old one.
Now translate that into crypto and watch how uncomfortable it gets.
Oracle feeds. Look at how they actually work. A set of professionally operated nodes, a deviation threshold, a heartbeat interval, an aggregation rule, and an on-chain push. It is genuinely clever engineering. It is also, structurally, a replication problem with a staleness window dressed in decentralized clothing. A price is committed by nodes, aggregated, and written to the chain on an interval. If the market moves faster than that interval, the chain prices the past โ and it prices the past with total confidence, because there is no mechanism on-chain for doubt.
I have said this before and I will keep saying it: oracle feed latency is DeFi's Achilles heel, and solving decentralization by operating a permissioned set of nodes is a compromise that the marketing quietly buried. That is not a knock on the engineers, who are solving a genuinely hard problem. It is a knock on the accounting. Protocols underwrite liquidations on top of a number that carries a latency budget, and then describe the resulting risk as trust-minimized.
Recall March 2020. Recall the hours when the feed did not move and the liquidation engine did. Recall every incident where a lending market executed at a price that existed on-chain two minutes after it existed in the world. Those were not anomalies. They were the latency budget coming due.
And this pattern is not crypto-only, which is why I keep dragging macro into these conversations. Proof-of-reserves attestations are snapshots. ETF custody reporting is a snapshot. NAV calculations are snapshots. Stablecoin reserve reports are snapshots.
A snapshot is just replication lag with better public relations.
Here is the generalization I want you to carry out of this section. The outage you can see on a status page and the staleness you cannot see inside an oracle feed are the same class of failure. Both are asynchronous systems pretending to be synchronous. One gets a Hacker News thread. The other gets a risk parameter, and then a liquidation cascade, and then a governance forum post asking what happened.
THE SENTIMENT LEDGER
Now let me do what I actually do, which is read the room.
The social response was, from a sentiment standpoint, textbook. The Hacker News thread appeared within twenty minutes. X filled up with screenshots โ red X marks across CI pipelines, unmerged pull requests posted like war photographs, the status page itself captured and reposted as if it were evidence in a trial. The jokes arrived on schedule. Global productivity down forty percent and nobody is complaining. Somebody made a graph. Somebody made a graph of the graph.
The emotional arc ran the way it always runs. Denial, then jokes, then anger, then the migration question, then amnesia. If you want a number for the half-life of an infrastructure outage in the collective memory, call it seventy-two hours. After that, the next shiny thing arrives and the thread goes quiet.
But do not dismiss the sentiment, because there is real information in it. Complaining about infrastructure status is in-group signaling. It is a way of saying I am a developer without saying I am a developer. For about three hours, this outage was one of the last genuinely shared cultural moments on a fragmented internet. Everyone was in the same room. That is rare now, and it is worth noticing when it happens, because shared rooms are where narratives get made.
And here is my actual point, the one that makes this a sentiment story rather than a technology story.
The sentiment was completely unpriceable. There is no futures contract on GitHub uptime. There is no insurance product for a halted deploy pipeline. There is no instrument that lets you express a view on whether the next twelve months contain more data-layer cascades than the last twelve. The crowd generated an enormous quantity of emotional energy and exactly zero price signal, and I sat there watching it happen in real time with a phone in my hand.
That is the recurring theme in everything I write. Sentiment runs ahead of fundamentals โ that is precisely why sentiment is usually the better leading indicator. But sentiment without an instrument is just noise with a timestamp attached to it. It tells you how people feel. It does not tell you what anything costs.
Meanwhile, the actual pricing happened somewhere much quieter. It happened three weeks later in a platform team's architecture review, when somebody put up a slide titled single vendor exposure, and everyone in the room nodded, and then somebody asked what the mitigation would cost, and the meeting moved on to the next agenda item.
THE QUIET TAX: SERVICE CREDITS AND THE RENEWAL CONVERSATION
Enterprise availability commitments in this category tend to sit around 99.9% monthly. Run the arithmetic. That is roughly forty-three minutes of permitted downtime per month. It sounds generous until you remember that the entire industry is built on the assumption that things stay up, and that forty-three minutes of a CI/CD pipeline is not forty-three minutes of a website being slow. It is forty-three minutes of every deploy in the world being queued.
Enterprise agreements carry service credits. The credits are small by design. They exist to acknowledge the event, not to compensate for it. Anyone who has ever tried to build a business case on service credits knows how thin that compensation is against even a single lost release window.
The real cost is not in the credit line. The real cost is in the renewal conversation. It is in the seats that do not get added next year. It is in the platform lead who now has a budget line item for a mirror, and a mandate, and a mandate is much harder to kill than a wish.
Net revenue retention does not erode from a single incident. It erodes from frequency. One outage is a story. Two outages are a pattern. Three outages are a line item in an architecture review, with a slide, and a slide has a way of becoming a decision.
And the behavior that follows an outage is almost never migration. Migration is expensive and terrifying and everybody knows it. What follows is divergence. Nobody leaves. Everybody hedges. A GitLab mirror for the team that wants control of its own runners. A Gitee mirror for the contributors who need a compliant path in. A self-hosted Forgejo instance for the security engineer who has been asking since 2019 and who now, finally, has ammunition.
The most expensive thing about an outage is not the outage. It is the architecture decisions made in the seventy-two hours that follow it. Those decisions persist for years, and they all point the same direction, which is away from concentration.
For a vendor, that is a slow leak in a moat that otherwise looks bottomless.
THE LAZY TAKE: DECENTRALIZATION WOULD HAVE FIXED THIS
You know what came next. I knew what came next before I opened the app.
This is why we need decentralized infrastructure. Put your code on IPFS. Move to Radicle. Self-host everything. Own your stack.
It is a satisfying take. It is also answering the wrong question, and I want to push on it, because this is the argument my own industry defaults to and defaults are where the blind spots live.
Counter-evidence, four items, all of them ours.
Solana. Decentralized in design, and repeatedly halted by congestion, client monoculture, and the sheer cost of running a validator. Decentralization did not immunize it against fragility. It changed the shape of the fragility. In some cases it made things worse, because when a decentralized network has a bad day, there is nobody to be angry at and nobody with an incentive to publish a postmortem.
Bridges. Decentralized custody of hundreds of millions of dollars, with the keys ultimately held by a small multisig and a signing ceremony. We have watched that movie roughly a dozen times now, and the ending is always the same.
The sequencer question. Most rollups have exactly one sequencer. Everyone in the building knows. Almost nobody prices it. The system is decentralized in the parts that are cheap to decentralize and centralized in the part that would be expensive, and we describe that arrangement with a straight face as a scaling solution.
The uncomfortable conclusion: reliability comes from redundancy, isolation, and practiced failure. It has nothing to do with the centralization score on a website.
A well-run centralized system with properly isolated failure domains beats a badly-run decentralized system every single time, and I will die on that hill. AWS has outages. Nobody seriously suggests replacing AWS with a federated mesh of Raspberry Pis, because everyone understands that the alternative is not more reliable, it is merely more distributed.
Now let me go one layer deeper, because there is something in the emotional reaction that tells us more than the technical reaction does.
Why did the status page make people so angry? Not because the service went down. Services go down. The anger was aimed at the writing. Passive voice. No estimated time. No names. No numbers. No one to be annoyed at.
We do not actually want decentralization. We want accountability. We want a name to attach the frustration to. We want a company that can be embarrassed. And here is the genuinely contrarian part, the part I will take the hit for: the crypto industry is far more exposed to shared infrastructure than the traditional financial institutions that openly laugh at us.
A bank has its own data centers, its own mainframes, its own compliance stack, and a change-management process designed by people who are personally afraid of regulators. A crypto exchange runs on Cloudflare, AWS, GitHub, Discord, and a handful of RPC providers. We built the most self-consciously decentralized industry in the history of finance on top of about six companies.
And the reason is not stupidity. The reason is incentives. Nobody pays for redundancy. Capital flows toward yield, not insurance. The same crowd that will pile into a four-hundred-percent APY without blinking will not pay twenty percent more for a multi-region deployment, because the multi-region deployment does not show up anywhere on a P&L until the day everything is on fire, and by then the next cycle is already running and nobody is looking backward.
I will be honest about my own contribution to that pattern. In 2022, when the industry finally had downtime โ when nobody was shipping, when the charts were red, when the social fabric was genuinely frayed โ that was the moment to fund the boring fixes. That was the window. Instead I was in BGC, every month, organizing meetups. Drinks, macro talk, a room full of people who needed to be in a room together. It helped. It genuinely helped, and I would do it again. And every single one of those evenings was an evening not spent on a postmortem or a redundancy project or a chart about failure domains that nobody wanted to look at.
I do not regret the meetups. I regret thinking that was the whole answer.
Sentiment prices narrative. It has never once priced architecture.
WHY THE NEXT CYCLE WILL REPRICE THE BORING STUFF
So where does that leave us, sitting in a bull market with green candles and a hot narrative chasing everybody around the room?
What I am watching first is recurrence. If a data-layer cascade shows up again inside twelve months, that is not bad luck, that is a pattern, and patterns eventually get priced. Second, duration โ the record here gives us no date and no recovery time, which means the severity question is still open, and open questions in infrastructure have a way of being answered by the next incident. Third, whether a real postmortem ships, because a real postmortem is the cheapest possible way to convert an outage into credibility, and most organizations are too frightened to do it. Fourth, how loudly the alternatives talk. GitLab, Bitbucket, Gitee โ they will use this, and honestly they should. Self-hosted control is a much easier pitch after an outage than before one.
My positioning thought, if you want it in one line: infrastructure reliability is one of the most systematically underpriced factors heading into the back half of this cycle. Valuation frameworks price growth, narrative, and liquidity. Almost nobody prices the probability that the factory floor goes dark for four hours on a Tuesday afternoon. The teams that price it will look conservative right up until the moment they look prescient, and that moment always arrives faster than anyone expects.
And the question I would put to anyone reading this with a hot narrative in their feed and a green candle on their screen: if that platform went dark for a week, do you know where your repository actually is? Do you know whether your deploy pipeline has a second path, or whether the whole thing terminates in a single workflow file that lives in one account? Do you know who holds the keys to your RPC provider, and whether they answer messages on a Sunday?
We didn't ask those questions in 2017. We didn't ask them in 2021. The music is loud enough right now that we probably will not ask them in 2026 either.
Right up until the moment we have to.
The chain keeps producing blocks. It always will. It was the basement underneath everything else that had to hold, and last Tuesday, at 17:16 Manila time, it briefly did not.