Product Testing

Product Testing lets you lock in expected calculation results as Test Cases and replay them automatically whenever a Product, ProductBrick, or DataPool changes — so regressions are caught the moment they're introduced, not after they reach a customer.

Overview

Orbit Products are made up of ProductBricks and DataPools. Even a small change — a price increase, a new diesel surcharge, a tweak to feasibility logic — can ripple through every booking that uses the product. Product Testing turns each saved booking into a deterministic regression test, so changes are validated before they're persisted.

A Test Case snapshots the inputs of a calculation (the full transport draft) together with the expected outputs (line items and feasibility) and the moment in time the calculation was performed. When you later edit the underlying Product, ProductBrick, or DataPool, Orbit replays every affected Test Case against the pending change and asks you to review the results before the save is committed.

Key highlights:

Save-blocking by design: Every change to a Product, ProductBrick, or DataPool runs the full regression suite. There is no path to save without reviewing results.

Deterministic by simulated time: Each Test Case runs against the system time at which it was recorded, so it pulls the versions of bricks and pools that were valid at that moment.

Cascading validation: When a DataPool changes, every Product is tested automatically.

Two-phase save: A soft-save runs the tests and produces a review summary; a hard-save commits the change and updates the expected results (the ground truth) for the test cases you keep.

API-first: Every flow available in Orbit MissionControl is also available through the Orbit API, so agents and integrations can author and run tests programmatically.

Concepts

Test cases

A Test Case is a frozen snapshot of one calculation:

Draft Snapshot: The full transport input (properties, timestamps, shipment data, transport shape) used as the deterministic input.

Assertions: The expected outputs of the calculation. Today this covers line items and feasibility.

Simulated time: The moment in time at which the calculation was originally performed. When the Test Case runs, Orbit treats this as the current system time, so the calculation pulls the brick and pool versions that were valid then.

A Test Case belongs to a Product. The Product is the natural aggregation point because it knows every ProductBrick and DataPool it depends on.

Cohorts

Test cases are organised into Cohorts. A Cohort groups the test cases that share the same simulated runtime, typically the test cases that were captured against the version of the Product that is currently live.

The active group is the current cohort. When you publish a new version of a Product, ProductBrick, or DataPool with a future validity date, the current cohort is archived and a new current cohort is created with timestamps shifted forward to match the new validity window. Archived cohorts remain available for inspection.

Soft-save and hard-save

Saving a Product, ProductBrick, or DataPool is split into two steps:

Soft-save: Orbit runs every affected Test Case against the pending change and returns a result summary. Nothing is persisted yet.
Hard-save: Once you confirm the soft-save results, Orbit applies the change, updates the ground truth for the test cases you kept, and archives any superseded cohort.

This split is what makes the feature both UI-friendly and agent-friendly: the same two steps drive the Review Test Results modal in Orbit MissionControl and the equivalent flow over the Orbit API.

Creating a Test Case

There are two entry points for capturing a Test Case.

From the Transport Composer (recommended)

This is the most natural way to build up a regression suite — capture realistic, present-day bookings as you encounter them.

1. In Orbit MissionControl, open the Transport Composer.

2. Enter a transport request and let the products calculate.

3. On the calculated product card, open the action menu and choose Save as Test.

4. Give the Test Case a descriptive name (for example, Same-day Berlin courier, 2 stops, 30 kg) and submit.

Orbit captures the full draft, the moment of calculation, and the calculated line items and feasibility as the expected output.

The button is intentionally tucked into the action menu so it does not disrupt the booking journey for everyday users.

From the Testing tab

Each Product has a Testing tab in Settings → Products. Use this surface to:

- Inspect the existing Cohorts for a Product.

- Open the draft snapshot of any Test Case to review its inputs.

- Run the current cohort manually with Run current test cases.

- Delete the current cohort if it has become irrelevant.

If a Product has no Test Cases yet, the tab points you back to the Transport Composer, where most authoring happens.

The save flow

Saving a Product, ProductBrick, or DataPool always opens the Review Test Results modal. The modal walks through three phases.

1. Tests run

Orbit resolves every Product affected by the change, including downstream products when a shared DataPool is edited, and runs the relevant test cases in parallel. While the run is in progress the modal shows the running state.

2. Results

For each affected Product, the modal shows a {passed} / {total} passed count and a list of Test Cases, each marked as Pass, Fail, or Error:

- Pass The pending change produces output that matches the recorded expected output.

- Fail The pending change produces a different result than what was originally expected.

- Error The calculation could not complete (for example, missing data or a runtime exception in a brick).

You can expand any Test Case to view its assertions and inspect the inputs. If a Test Case is no longer relevant, for example, because the change deliberately makes its scenario obsolete, hold to delete it before continuing.

3. Confirm save

If every Test Case passes, the save proceeds straight through. If any Test Case fails, you have to make an explicit decision: Update Ground Truth acknowledges that the new outputs are correct and re-records them as the new expected results; Keep Editing cancels the save and returns you to the editor.

In other words: a save with failing tests is allowed, but only after you've confirmed that every difference is intentional. The set of Test Cases you carry forward becomes the new baseline. There is no silent override.

Cohort shifts on new versions

When the change introduces a new validity window, for example, a DataPool valid from the first of next month, Orbit performs two operations on save:

1. The simulated time on every kept Test Case (and all timestamps inside its draft snapshot, such as pickup and delivery windows) is shifted forward by the delta between the old and new validity dates. The new Cohort continues to test the same scenarios, just under the new version.

2. The previous Cohort is archived. It remains visible in the Testing tab and is still used if you ever roll back to that earlier version.

If the validity shift is not a whole number of weeks, weekday-sensitive logic (for example, a Sunday surcharge) may now resolve to a different day of the week. The simulated date is shown next to each Test Case so you can spot this and adjust the validity window if needed.

Cascading changes

A DataPool can be referenced by many Products. When you edit one, the soft-save resolves every Product that uses it and runs all of their current cohort Test Cases. The Review Test Results modal groups the results by Product, so you can scan the impact of a single change at a glance.

ProductBrick and Product edits behave the same way, scoped to the Products that reference them.

Real World Example:

A logistics planner at Spaceport Shipping Co. in Paris is updating the diesel surcharge DataPool for the upcoming month. The current value is 7 %, the new value is 9 %, valid from the first of next month.

The planner opens Settings → Products → Pools, edits the DataPool, and clicks save. The Review Test Results modal opens and runs the current cohort against the pending change. The same-day courier Product shows 9 / 12 passed. Three test cases — all of them business-priority deliveries to ACME Ltd in Tokyo — are marked Fail because their total price has increased by 2 %.

The planner expands the failing test cases, confirms the new totals are exactly what the surcharge increase should produce, and clicks Update Ground Truth. Orbit commits the new pool version, archives the previous Cohort, shifts the simulated timestamps of the kept Test Cases forward, and re-records the new totals as the expected results. The next time anyone touches this product, those updated totals are the baseline.

API Access

The full Product Testing flow is exposed over the Orbit API under /v5/products/{productId}/test-cases, including:

- Listing, fetching, creating, and deleting Test Cases.

- Running the test suite for a Product.

- Updating the ground truth after a soft-save.

- Removing individual assertions from a Test Case.

This means an agent or integration can drive the same soft-save → review → hard-save loop programmatically: trigger a soft-save, inspect each Test Case result, decide which ones to keep, and commit the change. The Orbit MissionControl modal is one possible front-end for that flow — your own automation is another.

For request and response schemas, see the Orbit API Reference.

FAQ

Where do test cases live?

Test cases are attached to the Product. The Product knows every ProductBrick and DataPool it depends on, so it is the natural place to attach the regression suite.

Can I save a change without running the tests?

No. Every change to a Product, ProductBrick, or DataPool goes through the soft-save / hard-save flow. The tests always run, and you always confirm the result before the change is persisted.

What happens to my old test cases when I publish a new version?

The previous Cohort is archived and remains attached to the Product. The Test Cases you keep are copied into a new current Cohort with timestamps shifted forward by the validity-window delta. If you later restore an earlier version, its archived cohort is still there.

My test failed because the simulated date moved to a different weekday. What should I do?

The simulated date is shown next to each Test Case. If a weekday-sensitive rule (for example, a weekend surcharge) is the cause, adjust the validity window of your new version so the shift lands on the same weekday — typically by extending the delta to a multiple of seven days.

What kinds of assertions are supported today?

Line items and feasibility are captured automatically when you save a Test Case from the Transport Composer.

Can a test case call external APIs?

Tests are deterministic only as long as the ProductBrick code does not call out to live external services. Calls to external REST APIs cannot be replayed against a frozen point in time, so we recommend keeping that logic on the platform. But, of course, you may still call external API endpoints. Just make sure to notice that this might lead to flaky regression tests.

Is Product Testing available through the Orbit API?

Yes. The full create / list / run / update-ground-truth flow is available under /v5/products/{productId}/test-cases. See the Orbit API Reference for details.