Update and Summary
The 0.80.0 upgrade successfully resolved the original transfer issue. Restoring packet commitments as part of an upgrade migration allowed relayers to clear stuck transfer packets and restore access to funds.
This issue was unexpected. As described in previous posts, there were two immediately contributing factors:
- An address format incompatibility created by Nobleās USDC-specific middleware, causing the Noble chain to reject transfers from Penumbra;
- Incorrect handling of error acknowledgements by the Penumbra chain, which caused these rejected transfers to be frozen instead of refunded.
In isolation, the first issue would have been inconvenient but not particularly severe, as transfers would have been immediately refunded. As described in previous posts, it was unfortunately not detected by manual testing, which exercised non-USDC withdrawals to Noble (which donāt trigger the custom middleware), or by automated testing via InterChainTest (which doesnāt have Nobleās middleware).
The second issue was with the Penumbra software. Regardless of the behavior of the Noble chain, the ICS20 transfer handler should have correctly handled errors reported by counterparty chains, as required by the ICS20 specification. Unfortunately, this was not detected by InterChainTest, which turns out not to test failure cases, or by previous security audits of the IBC code.
The discovery of this issue triggered a sudden loss of confidence in the previous assurance work done on Penumbraās IBC stack. In parallel with pursuing resolution paths in line with the community sentiment shared above, Penumbra Labs, a service provider to the Institute for Applied Numogrammatics, also began internally and externally re-auditing the IBC stack to discover any additional issues that might also have been missed up to that point. This uncovered two additional issues, one minor and one security-critical, both of which were fixed without incident as part of the 0.80.0 upgrade, approved by on-chain governance as Proposal #3 and enacted by the validators and full node operators who executed the upgrade.
Missing channel ID check in acknowledgement handler
The acknowledgement handler failed to check the counterparty channel ID when processing an acknowledgement, which could potentially allow a byzantine counterparty chain C to send acknowledgements on behalf of an unrelated counterparty chain B. However, this had no impact, as at the time, the acknowledgement handler was (incorrectly) a no-op. This issue was resolved by commit 0a0d955c3b8973b3c2c86b41cde963753d3aa094.
Incorrect balance handling for inbound transfers
As described in a previous post, Penumbra maintains value balance tracking of both native and non-native tokens. This allows the chain to ensure that a counterparty chain is not returning more of a native token than was originally sent out to that chain. For instance, an incoming transfer of 200 units of token transfer/channel-2/upenumbra along channel-2 should be recognized as a return of a native token called upenumbra and the balance should be decremented by 200. However, the logic for updating the value balance of native tokens was incorrect: while the decremented balance was computed correctly, it was written into the state as the balance for the prefixed token transfer/channel-2/upenumbra rather than the native token upenumbra.
This meant that the balance was not correctly decremented, and a byzantine counterparty chain could have improperly minted native tokens by repeatedly sending inbound transfer packets for which no outbound transfer existed. This would have required control over the counterparty chain itself, and is known not to have occurred, as at the time, the Penumbra chain only had five counterparty chains (CosmosHub, Dydx, Noble, Osmosis, and Neutron), none of which were malicious. To prevent the possibility of exploitation, a ā
+1 minority of stake weighted validators temporarily blocked new channel creation by patching their CheckTx handlers, until the upgrade was applied. This ultimately had no effect, as no new channel creation was attempted anyways.
This issue was resolved by commit 66ed6a57cef5bb51d62d885d6469214af973f7a5, which corrects the balance handling.
While balance tracking is correctly handled by the current code, so no issue exists for new channels, or for new transfers on existing channels, the chainās view of aggregate balances for existing counterparty chains may be slightly too high, as they were previously not decremented correctly when processing inbound transfers of native tokens. This could be corrected in a future chain upgrade. However, in the meantime, the remaining risk appears to be insignificant, as the gap is small and any attack would require, for example, controlling the entire Osmosis chain, at which point the attacker could steal a much greater amount of tokens directly.
DoS risk via zero-fee transactions
Finally, the upgrade also closed a DoS risk resulting from zero-fee transactions. Penumbra transactions pay fees based on their gas use. The gas use for the transaction is the sum of the gas use of each action. As fee payments require spends, in addition to the economic cost, any transaction with nonzero fees must also include a proof-of-work in the form of a SNARK proof, which is more expensive to generate than to verify. However, the chain rules did not explicitly prohibit transactions with no actions, which would pay no gas and do nothing. This would likely have been low-impact if it had been attempted, as these transactions would be cheap to process and easy to block by validatorsā CheckTx implementations.
This issue was resolved by commit ea50717bd3fd27a740b84d426d2c5cb4db3d0b84, which blocks transactions with no actions. Thanks to Github user @syvb for submitting the issue.
