diff --git a/contract_driven_development/service_virtualization.md b/contract_driven_development/service_virtualization.md index 16f4680e5..f37bb7355 100644 --- a/contract_driven_development/service_virtualization.md +++ b/contract_driven_development/service_virtualization.md @@ -46,6 +46,16 @@ Service Virtualization * [Transient Expectations (a.k.a. Transient Stubs)](#transient-expectations-aka-transient-stubs) * [Setting transient expectations](#setting-transient-expectations) * [Clearing Transient Expectations](#clearing-transient-expectations) + * [Request Matchers](#request-matchers) + * [Matcher: `$neq`](#matcher-neq) + * [Matcher: `$match`](#matcher-match) + * [Match an specific value a fixed number of times](#match-an-specific-value-a-fixed-number-of-times) + * [Match any unique value a fixed number of times](#match-any-unique-value-a-fixed-number-of-times) + * [Match any value a fixed number of times](#match-any-value-a-fixed-number-of-times) + * [Multiple `value: each` matchers working together](#multiple-value-each-matchers-working-together) + * [Simulating Downstream Dependency Failures using matchers](#simulating-downstream-dependency-failures-using-matchers) + * [First 2 Requests - Simulated Latency](#first-2-requests---simulated-latency) + * [Requests After Exhaustion - Recovery](#requests-after-exhaustion---recovery) * [Externalised Response Generation](#externalised-response-generation) * [Hooks](#hooks) * [Overview](#overview) @@ -505,7 +515,7 @@ For example, suppose that in the request we expect, the important values to be m Let's see how we can formulate an example that meets these requirements. -- Create a new file in the `employees_examples` directory named `any_name.json`: +- Create a new file in the `employees_examples` directory named `any_name.json` with the following contents: ```json { @@ -1054,6 +1064,7 @@ stub: {% endtabs %} **Note:** If the delay is specified in the example file, it will be used to simulate response times for that specific example. + Otherwise, the global delay will be applied. ## SSL / HTTPS Stubbing @@ -1250,7 +1261,7 @@ If your tests are written in Python, you can start and stop the stub server with ## Transient Expectations (a.k.a. Transient Stubs) -A transient mock disappears immediately after it has been exercised. +A transient mock becomes unavailable immediately after it has been exercised. ### Setting transient expectations @@ -1284,6 +1295,360 @@ To do that, make an API call to the path /_specmatic/http-stub/ To clear the transient mock in the above example, you would call http://localhost:9000/_specmatic/http-stub/123 with the DELETE verb. +## Request Matchers + +Specmatic provides powerful matchers that offer dynamic, fine-grained control over how your stub server responds to incoming requests, enabling you to simulate complex and evolving scenarios. + +{: .note} +Requests are validated against the specification first, before any matchers in examples are evaluated. + +### Matcher: `$neq` + +**Syntax:** `$neq()` + +Use this matcher to ensure that a field value in incoming requests is **not equal** to a specific value. + +For example, the following example matches only requests where `status` is not equal to `Pending`: + +```json +{ + "http-request": { + "method": "POST", + "path": "/verifyUser", + "body": { + "status": "$neq(Pending)" + } + }, + "http-response": { + "status": 200, + "status-text": "OK" + } +} +``` + +### Matcher: `$match` + +**Syntax:** `$match(exact: , dataType: , times: , value: each|any)` + +This matcher provides flexible options to control how request fields match example data and how many times a particular example can be used. + +#### Match an specific value a fixed number of times + +Consider the following **transient** example: + +```json +{ + "transient": true, + "http-request": { + "method": "POST", + "path": "/greet", + "body": { + "message": "$match(exact: hello, times: 2)" + } + }, + "http-response": { + "status": 200, + "body": { + "reply": "Hi there!" + } + } +} +``` + +This matcher will respond to two requests where `message` is exactly `hello`. After two successful matches, the matcher becomes **exhausted**, and subsequent requests with `{"message": "hello"}` will no longer match this example. + +**Example:** + +Request 1 payload: + +```json +{ + "message": "hello" +} +``` + +Request 2 payload: + +```json +{ + "message": "hello" +} +``` + +After these two requests, the matcher becomes **exhausted**, and this transient example will not match further requests such as: + +Request 3 payload: + +```json +{ + "message": "hello" +} +``` + +If there are multiple matchers in an example, the example remains active until **all** of its matchers are exhausted. + +#### Match any unique value a fixed number of times + +Consider the following **transient** example: + +```json +{ + "transient": true, + "http-request": { + "method": "POST", + "path": "/echo", + "body": { + "text": "$match(dataType: string, value: each, times: 2)" + } + }, + "http-response": { + "status": 200, + "body": { + "echoedText": "$(text)" + } + } +} +``` + +Each unique value for `text` can be matched twice. For example: + +Request 1 payload: `{ "text": "hello" }` + +Request 2 payload: `{ "text": "hello" }` + +After these two requests, the matcher for `text` with value `hello` becomes **exhausted**. + +Request 3 payload: `{ "text": "world" }` + +The value "world" has never been seen before, and hence this matcher will be active for "world", and will match it twice before getting exhausted. + +If there are multiple matchers in an example, the example remains active until **all** of its matchers are exhausted. + +#### Match any value a fixed number of times + +Consider this **transient** example: + +```json +{ + "transient": true, + "http-request": { + "method": "POST", + "path": "/echo", + "body": { + "text": "$match(dataType: string, value: any, times: 2)" + } + }, + "http-response": { + "status": 200, + "body": { + "echoedText": "$(text)" + } + } +} +``` + +Here, the matcher can match any value twice, regardless of what it is. After two matches, it becomes **exhausted**, and the example will no longer be used. + +#### Multiple `value: each` matchers working together + +Specmatic actually tracks the usage of each unique combination of matcher values in the presence of `value: each`. This can be better understood with an example which has multiple matching having `value: each`. + +Consider this **transient** example where both `id` and `details` use `value: each` with `times: 1`. + +```json +{ + "transient": true, + "http-request": { + "method": "PATCH", + "path": "/order/10", + "body": { + "id": "$match(dataType: integer, times: 1, value: each)", + "details": "$match(dataType: string, times: 1, value: each)" + } + }, + "http-response": { + "status": 200, + "body": { + "status": "Order updated" + } + } +} +``` + +Let's walk through a sequence of requests. + +Initial state: + +* `id` matcher: **Active**. +* `details` matcher: **Active**. + +Request 1 payload: + +```json +{ + "id": 10, + "details": "packed" +} +``` + +This is the first time the stub sees `id: 10` together with `details: "packed"`, so it matches. + +Matcher state after Request 1: + +* `id` matcher with value `10` given `details: "packed"`: used `/` (**Exhausted**) +* `details` matcher with value `packed` given `id: 0`: used `/` (**Exhausted**) + +Request 2 payload + +```json +{ + "id": 10, + "details": "packed" +} +``` + +This is the exact same pair as Request 1. + +Matcher state after Request 2: + +* Pair `id: 10`, `details: "packed"` is already 1/1 (**Exhausted**) → this request will **not** match this example. + +Request 3 payload +```json +{ + "id": 10, + "details": "shipped" +} +``` + +This is a new pair: `id: 10` with `details: "shipped"`. + +Matcher state after Request 3: + +* `id` matcher with value `10` given `details: "packed"`: used `/` (**Exhausted**) +* `id` matcher with value `10` given `details: "shipped"`: used `/` (**Exhausted**) +* `details` matcher with value `packed` given `id: 10`: used `/` (**Exhausted**) +* `details` matcher with value `shipped` given `id: 10`: used `/` (**Exhausted**) + +Even though `id = 10` was seen before, this request worked because this exact combination (`10` + `"shipped"`) had not been used previously. But with `times: 1`, this combination is now exhausted. + +Request 4 payload: + +```json +{ + "id": 10, + "details": "shipped" +} +``` + +This reuses the pair from Request 3. But given that these values have already been used together once, this request will not match. + +Matcher state after Request 4: + +* `id` matcher with value `10` given `details: "packed"`: used `/` (**Exhausted**) +* `id` matcher with value `10` given `details: "shipped"`: used `/` (**Exhausted**) +* `details` matcher with value `packed` given `id: 10`: used `/` (**Exhausted**) +* `details` matcher with value `shipped` given `id: 10`: used `/` (**Exhausted**) + +Request 5 payload +```json +{ + "id": 20, + "details": "packed" +} +``` + +Now we have a new pair: `id: 20` with `details: "packed"`. + +Matcher state after Request 5: + +* `id` matcher with value `10` given `details: "packed"`: used `/` (**Exhausted**) +* `id` matcher with value `10` given `details: "shipped"`: used `/` (**Exhausted**) +* `id` matcher with value `20` given `details: "packed"`: used `/` (**Exhausted**) +* `details` matcher with value `packed` given `id: 10`: used `/` (**Exhausted**) +* `details` matcher with value `shipped` given `id: 10`: used `/` (**Exhausted**) +* `details` matcher with value `packed` given `id: 20`: used `/` (**Exhausted**) + +This request matches because, although `"packed"` was seen earlier with `id = 10`, the combination `id: 20`, `details: "packed"` is new. + +In summary: + +* The stub accepts each unique `id` + `details` combination exactly once. +* Reusing the same pair again will not match the example. +* Reusing either `id` or `details` with a new partner is allowed until that new combination has also been used once. + +### Note On Matcher Exhaustion + +* An example will continue to match incoming requests as long as at least one of its matchers is not **exhausted** for the given request values. +* When all matchers in a transient example are exhausted for the incoming request, the example itself is considered **exhausted** and will not return match success, even though the actual values in the request align with the data in the example. + +### Simulating Downstream Dependency Failures using matchers + +Let's bring this together with a practical example. + +One of the most valuable applications of matchers is **resilience testing**, for example, by utilizing the `delay-in-seconds` feature within Specmatic stubs, you can verify that your API implementation enforces strict timeouts when downstream services become unresponsive or slow. + +**Scenario**: Your API depends on a payment service. You need to verify that your implementation does not wait indefinitely for a slow downstream response, which could lead to thread pool exhaustion or hung connections. Instead, your API should detect the latency, abort the connection early, and return a `429 (Too Many Requests)` or appropriate failure status to the client. + +#### First 2 Requests - Simulated Latency + +The stub introduces a 5-second delay before responding. The goal is to prove that your implementation does not wait the full 5 seconds. + +It should ideally timeout earlier (e.g., at 2 seconds) and release the connection. + +```json +{ + "transient": true, + "delay-in-seconds": 5, + "http-request": { + "method": "POST", + "path": "/payments", + "body": { + "amount": "$repeat(times: 2, value: any)" + } + }, + "http-response": { + "status": 200, + "body": { + "status": "delayed_success", + "transactionId": "txn-12345" + }, + } +} +``` + +Transient requests take priority over others. So this example will be matched first for the first 2 requests with any `amount` value. + +#### Requests After Exhaustion - Recovery + +Once the first example gets exhausted (after 2 requests), it will match no more requests. Then the next example kicks in. + +We'd also have an example available that simulates the downstream service recovering its performance, allowing your implementation to process requests normally. + +```json +{ + "http-request": { + "method": "POST", + "path": "/payments" + }, + "http-response": { + "status": 200, + "body": { + "status": "success", + "transactionId": "txn-12345" + } + } +} +``` + +**How this works**: +1. **Requests 1-2**: The stub receives the request and delays response by 5 seconds +2. **Verification**: Your API implementation, configured with a shorter timeout (e.g., 2000ms), should throw a TimeoutException or equivalent before the stub responds. +3. **Resource Management**: Instead of holding the connection open, your API catches the timeout, aborts the downstream call, and returns HTTP 429 to the client +4. **Requests 3+**: The first matcher is exhausted. So when the consumer retries, the second example responds immediately with a 200 OK, thus showing how the system returns to a healthy state once the downstream service recovers + +Specmatic can assist you in testing this scenario during contract testing. In the case of a `429 (Too Many Requests)` response, Specmatic will retry the request, adhering to the delay specified in the `Retry-After` header. For additional details, please refer to the [Smart Resiliency Orchestration](/contract_driven_development/contract_testing.html#smart-resiliency-orchestration) section. + ## Externalised Response Generation There may be circumstances where we need to compute the response or part of it based on the request in the expectation. Here is an example. diff --git a/features/dictionary.md b/features/dictionary.md index 719a8efba..fb585a61f 100644 --- a/features/dictionary.md +++ b/features/dictionary.md @@ -23,6 +23,7 @@ redirect_from: * [Examples](#examples) * [Generating the Dictionary](#generating-the-dictionary) * [Understanding the Dictionary](#understanding-the-dictionary) + * [Example Directory Parameter](#example-directory-parameter) * [Dictionary with Contract Testing](#dictionary-with-contract-testing) * [Run the tests](#run-the-tests) * [Generative Tests](#generative-tests) @@ -545,6 +546,14 @@ We have not included the invalid value of `department` in the bad-request exampl the command will include only valid values in the dictionary, allowing it to be executed even with invalid examples. {: .note } +### Example Directory Parameter + +If you have examples in some other existing directory, you could provide the path to that directory using the `--examples-dir` option as shown below: + +```shell +docker run --rm -v "$(pwd)/employees.yaml:/usr/src/app/employees.yaml" -v "$(pwd)/examples:/usr/src/app/examples" -v "$(pwd)/employees_dictionary.yaml:/usr/src/app/employees_dictionary.yaml" specmatic/specmatic-openapi examples dictionary --spec-file employees.yaml --examples-dir ./examples +``` + ## Dictionary with Contract Testing The Dictionary can be utilized in contract testing, allowing Specmatic to use the values defined in the dictionary when generating requests for tests. To illustrate this process, we will use the previous specification and dictionary as an example.