Candidate Mainnet Upgrade 2?

Governance
1

Hi all,

Based on discussions in the last two community calls:

there’s initial community consensus about a set of small tweaks to the protocol, and about a path forward on working around the Noble/Circle USDC middleware bug introduced by the Noble upgrade in November.

These changes could be deployed in a second mainnet upgrade, subject to community consensus, governance, and validator coordination!

The small tweaks are:

  • UIP-4: Spend Backreferences
    • Rendered UIP
    • Forum Thread
    • Adds 48 bytes of data to each Spend action that allows clients to traverse their transaction graph, opening the door for much faster syncing
  • UIP-5: Outbound PFM
    • Rendered UIP
    • Forum Thread
    • Allows clients to include packet forwarding metadata in outbound IBC transfers, to allow, e.g., withdrawing ATOM directly from Penumbra to Osmosis (by forwarding through the originating chain, the Cosmos Hub)
  • UIP-6: App Version Safeguard
    • Rendered UIP
    • Forum Thread
    • Adds version checking for chain upgrades, hopefully preventing operator mistakes during an upgrade process
    • The initial implementation of UIP-6, released in v0.80.9, caused problems, which validators need to be aware of: see below.

The workaround for Noble’s USDC middleware bug is:

  • UIP-8: Transparent Addresses
    • Rendered UIP
    • Forum Thread
    • Creates a new encoding format that can encode a single one of a user’s many Penumbra addresses as a 32-byte string that looks exactly the same as a Cosmos address, and allows this “transparent address” to be used in IBC deposits.
    • While funds sent to a “transparent address” are actually private (there are no transparent accounts – this is just a different and more restricted address encoding), there’s only one transparent address, so using it for multiple IBC transfers creates linkability. Choosing the name “transparent address” gives users a safer conceptual frame for this tradeoff, even if t-addrs on Penumbra are much more private than t-addrs on Zcash or Namada.

PL has tested an upgrade flow using a candidate branch, to be released shortly, and verified that:

  • an upgrade to a testnet worked correctly
  • the upgraded fullnode software was able to process transactions created by pcli that made USDC withdrawals to the Noble testnet, grand-1
  • clients were able to successfully resume syncing their private state post-upgrade

If this upgrade were successfully deployed by the community:

  • pcli would gain support for USDC transfers to Noble immediately
  • Prax Wallet would continue to work with the upgraded network, but would not immediately be able to use the new features
    • A new Prax Wallet release (let’s say ~1w latency, potentially interrupted by holidays) would enable USDC transfers via Prax
  • In the meantime, USDC transfers with pcli would allow the creation of USDC/USDY stableswap positions that would allow Prax users to trade USDC for USDY and withdraw USDY (which is not affected by Noble’s USDC bug).

v0.80.9 issues

Unfortunately, the initial implementation of UIP-6 in pd v0.80.9 introduced a state corruption bug that did not affect the operation of the network, but does interfere with the upgrade mechanism. Essentially, writing the app version into the state (so the node could check that it was correct during an upgrade) was done in a way that incremented the Jellyfish Merkle Tree version, which is supposed to correspond 1:1 to the block height. This caused the JMT version to desync with the block height. While this does not affect the operation of the chain (which is why the bug was not detected in initial testing), it does interfere with the upgrade mechanism.

This bug was fixed in v0.80.10, but validator operators who upgraded to v0.80.9 and then restarted their nodes may have corrupted JMT version indexing. See release notes for more details: Release v0.80.10 · penumbra-zone/penumbra · GitHub

Validators whose JMT version has desynced from the block height will not be able to upgrade cleanly, so unfortunately this is another thing for validator operators to coordinate.

1 Like
2

v0.80.9 issues

Unfortunately, the initial implementation of UIP-6 in pd v0.80.9 introduced a state corruption bug that did not affect the operation of the network, but does interfere with the upgrade mechanism. Essentially, writing the app version into the state (so the node could check that it was correct during an upgrade) was done in a way that incremented the Jellyfish Merkle Tree version, which is supposed to correspond 1:1 to the block height. This caused the JMT version to desync with the block height. While this does not affect the operation of the chain (which is why the bug was not detected in initial testing), it does interfere with the upgrade mechanism.

To add to what henry said about version desync, here is how you can determine if you are affected, and what to do:

  1. Did you run v0.80.9 at any point in the past?
  • No: You are unaffected, no action items.
  • Yes: You are most likely affected, continue to 2.
  1. Are you running v0.80.10?
  • No: continue to 3.
  • Yes: Then you can check your logs for an entry such as this one:

INFO **abci**:**EndBlock{**height=27063**}**: penumbra_app::server::consensus: ending block height=27063 latest_state_version=27062

This is an entry from a healthy node. The difference between block height (27063) and state version (27062) is exactly one. This is good. Any other value indicates a version desync. And you will have to acquire a snapshot, or resync your node.

  1. If you are running v0.80.9, you are affected. You can update the binary to v0.80.10 and check the logs. We are also going to release a small diagnostic utility that will enable you to detect this remotely using gRPC.
2 Likes
3

There is a utility available in v0.80.11: https://github.com/penumbra-zone/penumbra/releases/tag/v0.80.11:

pcli query chain detect-desync

Point it at the RPC you want to check and it will tell you whether you are affected or not.

1 Like