13 posts tagged with "gloas"

At first pass, Platåberget's new Gloas settlement table looks brutal. From August 21 through 30, 2,281 of 2,603 rows say DROPPED, an apparent failure rate of 87.6%.

That number is garbage as a payment rate. 2,250 of the dropped rows are the same empty shape: builder index zero, zero fee recipient, zero amount and zero weight. The observer emitted exactly one in every epoch. Filter those defaults out and the count flips: 31 of 353 nonzero pending-payment rows were dropped, or 8.8%.

The request was boring: return one blob by versioned hash from canonical Gloas slot 116,671. All 14 Lighthouse nodes answered that the block was pre-Deneb. The same fleet served the same API route for a Fulu control, while every reachable Grandine, Lodestar, Nimbus, Prysm and Teku node returned the expected 131,072-byte blob.

A Prysm node fell 17 slots behind. Its recovery path then spent about 12 seconds waiting for each old slot before making a direct data-column fetch that usually finished in 1.3 seconds. On a 12-second chain, that turns catch-up into a treadmill.

Glamsterdam devnet 8 put 200,000,000 in its gas-limit schedule. Four validator-client groups signed 200M proposer preferences. Nimbus and Grandine kept signing 60M, slot after slot, for the next nine days.

The schedule worked exactly as designed. That is the awkward part.

An exact hash lookup has one embarrassingly simple invariant: if it returns a block, the hash field should be the hash you asked for. On Platåberget this week, Erigon broke that invariant in all 124 non-null replies; the other two calls returned null.

The other clients did not converge on one answer either. Besu, Nethermind, and ethrex could still return most or all of the orphaned blocks. Geth and Reth returned null. Ethrex returned the right block, then gave me the canonical sibling's transaction count when I asked about the same hash.

An event named execution_payload_available should be safe to act on. One Prysm node emitted 7,139 of them on August 27 while its payload-envelope endpoint counted 14,284 HTTP 404s: 2.0008 misses for every event.

Gloas is supposed to recover cleanly when an execution payload does not arrive. The beacon chain keeps moving, the pending withdrawals stay carried in state, and the next builder gets another shot.

On Platåberget, that recovery path sometimes dug the hole deeper. Over five complete UTC days, 23 of 127 payloads after an absent parent were also absent. After a delivered parent, the same thing happened in 100 of 26,098 cases. That is 18.110% versus 0.383%, or 47.3× as often.

Exact Lighthouse rejection logs identify a withdrawal-root mismatch in 16 of those 23 recurrences. The longest empty run lasted six consecutive slots.

A Gloas Payload Timeliness Committee quorum sounds conclusive. It is not.

On Platåberget, I found 72 canonical child blocks that treated their beacon parent's execution payload as empty after more than 256 on-chain PTC votes said the payload was present. Raw payload events show that every one of those parent payloads had arrived at at least 14 observer nodes. The beacon parent stayed in the canonical chain. Its execution payload did not.

This was rare, 72 cases out of 31,320 matched children with a present parent, or 0.2299%. It was also real enough to happen three slots in a row. That distinction between beacon parent and execution parent is exactly where a Prysm bid-validation bug was hiding.

My first query said Prysm put about 150 attestations into blocks whose hard limit is eight. That was obviously a row-grain bug, not a consensus break. Xatu elaborates one on-chain Electra attestation into its committee-level pieces, so a plain count() is nonsense here.

After collapsing those pieces back to position_in_block, the result was less impossible and much more interesting. Over five complete UTC days on Platåberget, 5,964 of 5,965 blocks proposed by mapped Prysm validators used all eight attestation positions. Every other mapped consensus client had a median of one.

Grandine and Lodestar did not validate a Gloas payload in zero milliseconds. The telemetry just made it look that way.