Block 11234873 and the Geth chain split

What happened and why it didn’t affect Chainstack

In brief

On November 11, 2020, a transaction included in block 11234873 of Ethereum mainnet caused a chain split.

The chain split was caused by the Geth clients starting from v1.9.17 and above being incompatible with the older Geth versions after block 11234873.

Timeline

A brief timeline of how the chain split happened:

• November 7, 2019 — Geth v1.9.7 is released. The release accidentally breaks the EIP-211 implementation and introduces a bug to which no one is alerted because the bug stays undiscovered at that point.
• July 15, 2020 — John Youngseok Yang from Software Platform Lab reports the bug to the Ethereum bounty program.
• July 20, 2020 — The bug is silently fixed and released with Geth v1.9.17. The release set the stage for the potential chain split between Ethereum nodes running Geth versions v1.9.16 and older and those running v.1.9.17 and newer.
• November 11, 2020 — A transaction in block 11234873 triggers the chain split. All Ethereum clients, including Geth v1.9.17 and higher, stay on the longer chain version. Geth v1.9.16 and below stay on the shorter chain version.

Impact

Chain splits happen often and are a part of the normal blockchain network operation.

Two factors affected why this chain split made the news:

• Backwards incompatibility between Geth v1.9.17+ and older Geth versions.
• Some of the major products and services staying on older Geth versions.

Why stay on the older Geth version?

Why did some of the major services running on Ethereum stay on older Geth versions?

As Ethereum evolves as the network, the idea, and the software implementing it, it’s become so much less of a garage project and so much more of the enterprise software realm.

Staying on a working older version and not upgrading unless there is an absolute necessity to is something that is very common in industries like banking software and so on.

As Ethereum crosses the territory from being a project run by enthusiasts to the fully-fledged industry with many many services and products around it and depending on it, staying on the stable software version is direct evidence of this space getting more mature.

And bugs will always happen.

Resolution

The resolution is to upgrade to the latest Geth version.

Why did the chain split not affect Chainstack users?

Chainstack has the practice of ensuring the nodes of our users are on Geth versions that are at most two versions behind.

At the time of the issue, the Chainstack dedicated Ethereum nodes were using at least v1.9.21, which is not affected by the bug.

Our upgrade strategy

Before being released to Chainstack users, any software upgrade goes through a set of standard tests in our sandbox environment internally.

What happens when a new Geth version is released?

1. We are informed of the release via emails and other notifications.
2. We look through the changelog to identify any critical changes. Most releases are maintenance and light bug fixes. We act on the changelog items accordingly.
3. If the release is for an impending hard fork or for critical security fixes, we will spend additional time to research them and understand them.

Sandbox environment

1. We push some of our sandbox nodes up with the latest release and observe any anomality within them.
2. We monitor GitHub Geth issue pages for any latest issues created due to the latest version regularly.
3. If everything is good for a period of days, we internally mark this Geth version to be stable for production.

Production environment

Once we deem the sandbox Geth version to be stable for production, dedicated Ethereum node customers will be informed of a scheduled node maintenance for an upgrade to either latest (if critical) or be upgraded to a version that is one or two releases behind latest.

What if things go wrong?

We have snapshots containing the last sync with the latest version and two previous versions, and we plan to use them in case of reverts.

Other things we note on changelogs

• Is the release backward compatible?
  • Geth v1.9.20 introduced a database change. This means, we cannot revert to Geth v1.9.19.
• Any deprecated commands or new features?
  • We will have to change our deployment commands if there are deprecated commands.
  • If there are new features, we see how we can apply them to make the customer’s experience better.
• If this release is to support a hard fork, which block does it start from?
  • We will ensure all nodes are upgraded by that block number.

Benefits of using Chainstack

• We always provide native JSON-RPC and WebSocket connectivity. This means, you are connected to the node directly through our high-performance nginx server instead of going through any middleman caching mechanisms.
• Your node’s Geth client version is always visible in the Chainstack UI.
• We support GraphQL within Geth which enables users to do quick and efficient queries on our nodes. See GraphQL on Ethereum: Availability on Chainstack and a quick rundown.

Join our community of innovators

• To learn more about Chainstack, visit our Knowledge Center or join our Discord server and Telegram group.
• Sign up for a free Developer account, or explore the options offered by Growth or Business plans.
• Take a look at our pricing tiers using a handy calculator to estimate usage and number of nodes.

Have you already explored what you can achieve with Chainstack? Get started for free today.