spike: investigate missing UTXO metadata causing storage root panic (PM-21781)

[m2ux]

Summary

Investigate the root cause of the storage root mismatch panic on QANet (Feb 10 2026). Initial hypothesis (non-deterministic dust fee calculation) was invalidated by domain expert review. The actual divergence source is unknown and requires log-level investigation.

🎫 Ticket 📐 Engineering

---

Motivation

On Feb 10 2026, QANet experienced a ~5-minute block time spike with all node types panicking: "Storage root must match that calculated." The warning "couldn't get metadata of input utxo, it's likely invalid" appeared in logs preceding the panic.

An initial fix targeting midnight-ledger dust fee calculation (PR #184) was closed after domain expert review confirmed the code path is deterministic across all nodes — if UTXO metadata is missing, all nodes (including the block author) return the same value, producing no state divergence. The WARN log was a red herring.

The actual root cause of the storage root mismatch remains unknown.

---

Investigation Plan

• Obtain DataDog access for QANet log analysis
• Enable debug logging on a QANet node to replicate the panic
• Gather complete logs for both entities with mismatching state (block author + importing node)
• Compare state roots at the divergence point to identify which storage key(s) differ
• Narrow down to the specific state transition producing non-deterministic results

Previous investigation (ruled out)

The generationless_fee_availability() code path in midnight-ledger/ledger/src/dust.rs was analyzed in depth and is not the cause. All nodes execute the same deterministic code. See closure report for details.

---

📌 Submission Checklist

• Changes are backward-compatible (or flagged if breaking)
• Pull request description explains why the change is needed
• Self-reviewed the diff
• I have included a change file, or skipped for this reason: spike/investigation only
• If the changes introduce a new feature, I have bumped the node minor version
• Update documentation (if relevant)
• No new todos introduced

---

🔱 Fork Strategy

• Node Runtime Update
• Node Client Update
• Other
• N/A

---

🗹 TODO before merging

• Root cause identified
• Fix implemented based on log evidence
• Ready for review

[github-actions]

KICS version: v2.1.16

Category Results CRITICAL 0 HIGH 0 MEDIUM 96 LOW 12 INFO 83 TRACE 0 TOTAL 191	Metric Values Files scanned 31 Files parsed 31 Files failed to scan 0 Total executed queries 73 Queries failed to execute 0 Execution time 8