Ethereum Devs Grapple With Worst-Case Scenarios

This week, I’ll discuss the fallout from last Wednesday’s test network issues that revealed a bug in Ethereum’s majority software client, Geth. Though a patched version of Geth software has since been released for London, some users, developers and mining pools are calling for further testing of the upgrade, which is scheduled to go live next week.

The following is an overview of network activity on the Ethereum 2.0 Beacon Chain over the past week. For more information about the metrics featured in this section, check out our 101 explainer on Eth 2.0 metrics.

Disclaimer: All profits made from CoinDesk’s Eth 2.0 staking venture will be donated to a charity of the company’s choosing once transfers are enabled on the network.

As Ethereum prepares for the activation of its 11th backward-incompatible upgrade, also called a “hard fork,” on Wednesday, Aug. 4, some developers are concerned that the upgrade could use more testing before deployment. 

Shortly following the biweekly Ethereum core developers meeting on Friday, July 23, Ethereum Foundation’s Tim Beiko wrote in the All Core Developers Discord chatroom, “A couple people have reached out or tweeted about not necessarily being happy with not delaying [the hard fork] … I asked about this [in the meeting] and no one seemed to have a strong opinion, but some folks mentioned this maybe wasn’t the right approach.”  

In response to Beiko’s comment, Ethereum software client developer Alexey Akhunov said he agreed it was “strange” there wasn’t more discussion in the biweekly meeting about potentially delaying the hard fork, dubbed “London,” in light of recent events. 

“I suppose I know why,” wrote Akhunov. “Delaying [London] is a sensitive topic and no one wants to take the heat, understandable.” 

Others in the chatroom chimed in imploring Ethereum developers to give serious consideration to delaying London a few more weeks for further testing. 

Concerns over the risks of the London upgrade – which includes a controversial code change impacting the fee market of Ethereum known as Ethereum Improvement Proposal (EIP) 1559 – grew after a bug was discovered in Ethereum software client Geth. 

For background, Geth is the most popular software used to connect to Ethereum. According to Ethernodes.org, an estimated 86% of all computers, also called nodes, synced to the Ethereum network run Geth client software. 

On Wednesday, July 21, the Ethereum test network Ropsten, which activated the London hard fork a month ago, suddenly experienced a chain split after an invalid transaction was mined into a block by nodes running Geth while being rejected by nodes running minority clients Besu and Open Ethereum. 

Within a few hours, a hot fix was released by the Geth team and all users were encouraged to update their software to the latest version number, Terra Nova 1.10.6. 

While no developers argued that the bug should delay main network activation of London during Friday’s call, some developers did discuss the appropriate course of action if such a bug were discovered on Ethereum rather than on a test network. 

“What would we do if something like this happened on mainnet, especially in a spot where say Geth, the majority client, they’re producing blocks? It obviously takes several hours to have a fix out,” said Beiko during the meeting.

Ethereum Foundation’s Martin Holst Swende stressed that these bugs are not unprecedented occurrences on Ropsten, and while they’re “a pain in the ass” to resolve, there are two ways to address them. 

First, if a user’s node is following the wrong version of the blockchain, the user will need to internally “rewind the chain” back to the block before the chain split and sync to the new chain using patched Geth software. Second, if a user’s node isn’t already synced to a version of the blockchain but is trying to connect to the network to gather data about recent transactions or execute transactions, the user may end up connecting to the wrong version of the chain. To avoid this, these users will need to “whitelist” certain nodes on Ethereum that are following the correct chain and isolate from others that are stuck on the wrong chain. 

Both rewinding and whitelisting Ethereum nodes can be done through Geth. Miners on Ropsten were able to resolve the chain split that took place last Wednesday using these tactics, though one miner noted during Friday’s meeting that instructions for repairing chain splits were not effectively communicated before Wednesday’s incident and accordingly left many miners confused as to how to restart their nodes correctly. 

User “AlexSSD7” wrote in the Discord chatroom that as the representative of an Ethereum mining pool, they were “worried” about the bug in Geth, noting, “A single minute [of network] downtime costs us a lot. One hour of downtime is $20,000 to us.” 

Unexpected bugs in client software would indeed be disruptive to exchanges and businesses operating on the main network, which is why developers stressed the need for a robust monitoring system that could quickly alert node operators of chain splits and encourage them to pause operations until further investigation. 

“This seems like a pretty low-hanging fruit that provides a tone of value to the ecosystem. If you’re not sure how to get started, just ask in the Discord,” said Beiko in Friday’s meeting. 

While these solutions would certainly be helpful if a bug similar to what occurred on Wednesday were to happen again after deployment of London on mainnet, they wouldn’t necessarily be the same solutions used to address larger-scale issues, such as the event of a hacker magically printing 100 million ETH. 

In the event of something so catastrophic, Ethereum Foundation’s Danny Ryan said in Friday’s meeting that it would be difficult to know in advance how developers would proceed. 

“I think there are just many options for the many types of bugs and many types of peculiarities that will emerge,” said Ryan. 

The more severe the impacts of a network bug are, the more intrusive the solution to resolving the bug will likely be – and the more damaging to Ethereum’s reputation as a secure blockchain. 

With increasingly ambitious hard forks in the near-term horizon on Ethereum’s development roadmap, figuring out potential solutions to a worst-case scenario and plans for damage control with network stakeholders may soon become an imperative for developers to consider. 

The following is a special edition of Validated Takes highlighting a handful of panel discussions and keynote presentations from last week’s Ethereum community conference in Paris, France. For the full conference agenda, check out the official EthCC website. 

“DeFi for Traditional Markets: When Security Tokens,” Talk by Fountain co-founder Mathieu Chanson. Highlights: Fountain is a decentralized exchange on Ethereum that enables users to buy and sell security tokens. Chanson highlighted the liquidity and accessibility that blockchain technology offers, being accessible 24 hours a day and allowing for immediate settlement. There are several other benefits to tokenizing securities, including transparency and fractionalization of assets that further increase accessibility. However, there are plenty of challenges with creating a fully decentralized securities exchange. Onboarding customers and new securities will both require following international regulations, including Know Your Customer laws and custody licenses.

“The Power of Credit Delegation,” Talk by Aave founder Stani Kulechov. Highlights: Aave is a decentralized lending protocol built on Ethereum. The team behind the protocol has built a product that can provide zero-collateral loans. Kulechov believes this is a step forward in bringing DeFi liquidity into the real economy and driving up borrowing demand on Aave.

“Things that Matter Outside of DeFi,” Talk by Ethereum creator Vitalik Buterin. Highlights: Beyond financial services, social media and public goods funding are two activities that have yet to take off on Ethereum. Buterin argues the network’s token economy and censorship resistance are two reasons why these activities could benefit from being built atop a decentralized blockchain.

“Uniswap, DeFi & the Future of Consumer Finance,” Talk by Uniswap growth lead Ashleigh Schap. Highlights: Uniswap Labs is trying to create partnerships with blockchain infrastructure companies such as Talos, Paxos and Fireblocks to connect DeFi solutions into the backend of prominent fintech companies such as PayPal and E*Trade.

“Why DEXs Are Eating the World,” Talk by Curve protocol developer Julien Bouteloup. Highlights: At its best, [decentralized finance] allows citizens of the world to have equal access to all currencies, equities, and financial platforms. As the space advances, decentralization will be on a spectrum. Regulators will watch over protocols that are used by the traditional financial world and users will still have access to the “Wild West” testing ground that DeFi is today.