Add support for stateful test suites and add chain tests#11
Add support for stateful test suites and add chain tests#11stringintech merged 6 commits intomainfrom
Conversation
|
Before implementing the object reference pattern with the Key differences: Old approach (implicit state):
{
"id": "assert_chain_height_zero",
"method": "btck_chain_get_height",
"params": {}, // Assumes global chainstate exists
"expected": {"success": true, "result": {"height": 0}}
}New approach (explicit references):
{
"id": "chain#5",
"method": "btck_chain_get_height",
"params": {"chain": "$chain_ref"}, // Explicit reference
"expected": {"result": 0}
}The reference pattern should provide more flexibility: handlers can manage multiple concurrent contexts/chainstates, dependencies between tests are explicit, and test scenarios are more composable. Note on option objects: I intentionally avoid exposing option objects (like |
|
Tested |
stickies-v
left a comment
There was a problem hiding this comment.
Concept ACK. This seems like a really elegant way to chain (hah) operations together, and wrt explicitness: I agree that the explicit approach is much better than the implicit one.
Working on other things atm but will test this out as soon as I get the chance (and having #6 addressed would make testing a lot easier).
Without this check, the code would proceed to call Normalize() on a null result, causing an unclear failure that doesn't explain the actual problem.
Sort test suites alphabetically based on their json file names before execution.
Introduces stateful test suites where tests depend on each other and execute sequentially. If any test fails in a stateful suite, all subsequent tests are automatically skipped and marked as failed. Handler processes are closed after stateful suites to force re-spawn and prevent state leaks to the remaining tests.
ad723e8 to
39bc313
Compare
|
Rebased to include changes in #12 - ID no longer required in response. Also added support for the requested feature in #6. Tested pre-release |
Introduces the object reference and registry pattern for persisting objects across requests. Test cases now include a ref field directly in the request when the expected result is a reference, allowing handlers to store and reuse objects like context, block, and chain throughout the tests. Extends response validation unit tests to test scenarios where reference is expected in response.
39bc313 to
9558afd
Compare
|
Addressed suggestion by @stickies-v in #11 (comment). Also addressed some nits and extended response validation unit tests to test scenarios where a ref object is required as result. Tested pre-release |
stickies-v
left a comment
There was a problem hiding this comment.
I think the chaining interface works really well, very nice work! I've implemented v0.0.3-alpha.3 in stickies-v/py-bitcoinkernel#42 and the interface feels very natural, I can't think of any improvements.
The verbosity flags and error message propagation have also been very helpful in implementing this, thanks.
Didn't look at the code changes or test vectors themselves, but I extensively used the handler-spec.md which made it easy to implement the interface (except for the one omission I found, as commented).
Probably won't leave any further comments on this PR, everything works nicely for me!
Adds chain.json test suite demonstrating stateful chain operations including block processing, height tracking, and verifying state after a reorg scenario. Documents the ref field and registry pattern in handler-spec.md. Restructures handler-spec.md with a method reference documenting supported operations (context management, chainstate manager, chain operations, block operations, and script verification) with their parameters, return values, and error conditions.
Add -v and -vv flags to print complete request chains that can be piped directly to a handler for manual debugging and reproduction. - Introduces DependencyTracker to build transitive dependency chains from ref usage - Switches to pflag for better CLI flag support - Adds unit tests for dependency tracking logic
9558afd to
a5976bb
Compare
|
Thanks a lot for looking into this @stickies-v and @janb84. Would merge in a couple of days if no further issues are found, to be able to add more coverage of the C API. |
This PR introduces stateful test suites and adds a new suite for testing chain operations.
Key additions:
chain.jsondemonstrates stateful testing with block processing, height tracking, and chain reorganization verification. The block data used in this suite can be deterministically regenerated by this functional test.handler-spec.mdwith detailed method reference for all supported operations.-vand-vvflags to print complete request chains that can be piped directly to a handler for manual debugging and reproduction - Add verbose modes for easier handler debugging #6.