Tail Risk
A note on shipping fast but also doing it right, a note on engineering
At Coinbase, I was a senior engineer and tech lead on the team that moved all of the money. Every onchain transfer, every buy and sell, every institutional trade, went through a system called the ledger. It’s a single system of record for the entire company. This didn’t just move billions of dollars, it moves trillions of dollars a year and acts as the virtual balances for hundreds of billions in fiat and crypto. It’s one of the most important systems in crypto.
As a result, I’ve seen pretty much everything that can go wrong when you move money, a lot of it.
I am convinced that the default death state of many of these large-scale crypto projects is one thing: tail risk. It exists for every project, and what’s important is how you manage it. It comes down to engineering.
The highest risk areas are:
People risk (founder or employee goes off the rails or commits fraud)
Engineering quality risk (engineer ships bug that fucks things up)
Security risk
A story. If you look at the FTX collapse, a lot of the story is with the fact that Alameda was special-cased in FTX’s systems. They couldn’t be liquidated. Why though? It traces back to an engineering incident (tail risk). There was an incident where Alemeda was almost accidentally liquidated. That’s why in their systems they put in the flag to make it that they were special. It wasn’t malicious, they just wanted to avoid destroying Alameda accidentally and nuking the market. After everything I’ve seen in crypto, I 100% believe if you had a subpar eng org running that kind of company, they would do this, and I guess they did.
Now, yes, SBF was a fraudster, just in a different way than I think most people realize. My (guess) is that he wasn’t paying attention to the accounting and a lot more of it was negligence than anything. In fact (insider knowledge) is that I know at the time of the FTX collapse, they didn’t really know where the books sat fully and there was a small team working on figuring it out. They had been moving so fast and thought they were making so much money they didn’t need to worry, then when the market moved, they were so underwater and the didn’t even have the full scope. Just complete incompetence. The fraud happened once they realized, which was likely way too late - this is actually where he’s a criminal. In that situation, you just need to come clean and stop spending customer funds.
Tail risk there was two-fold: human and engineering.
This is just one story, but if you look at a lot of the large, explosive deaths in crypto, they mostly look like this. On the decentralized side there’s a lot of protocol risk especially security-wise. It’s how most things die, they rarely fizzle.
Here’s the truth - every crypto project from the smallest protocols to the largest exchanges has this tail risk, and it’s all about how you manage it so it doesn’t kill you. Some builders are better at it than others. I think about this a lot while I’m building.
How this applies to Higherrrrrrr?
While we are shipping very fast, I’m also trying to do it right. Having done this for years, I kind of have good eyes on where the risk areas are. I’m working 16-18 hours a day 7 days a week. The assumption is that some of these systems are going to both move and store a lot of user funds and it’s important to be thoughtful about that. My intuition is that base memes are going to get a lot more popular and we’re soon going to be talking 9 figures of money movement. I joke a lot on twitter, but when it comes to this, it’s extremely fucking serious.
Shipping them in a matter of days or weeks is insane, and takes some mastery over the whole system. I care a lot more about getting it right than narrative shaping. My default assumption is we have the momentum to be successful, and it’s actually more important to make sure we don’t die a firey death.
Human tail risk is mostly in me, and I’m actually being pretty intentional about not giving myself keys. I don’t have keys to the protocol, which is a huge start, and I’ll slowly move to not really having keys to any of tech (although we still need to host a frontend).
Engineering risk is where I’m spending a lot of my time. We move features quite fast, but for certain things with money we move slower. The protocol being one example, including audits etc. The agents are another one of those special cases. I don’t think anyone has built a generic agent creator, and managing the hot wallets correctly is very important.
Hope you found this interesting! Always open to helping / advising other projects who want to chat about this stuff and doing it right.