How to test market data outages at an investment bank

Investment bank market data outage testing checks whether trading and risk systems detect unavailable, delayed, stale or invalid market data and respond according to approved controls. It also verifies recovery: services should resume from a known, consistent data state, with gaps and downstream effects identified before normal processing continues.

How to plan investment bank market data outage testing

Start by mapping the data path for each critical use case. A feed may pass through a vendor connection, feed handler, normalisation service, cache or distribution layer before reaching an order management system, pricing service, risk engine or trader display. Record which applications depend on each feed and what they are expected to do if it fails.

Define the expected behaviour with business, trading, market risk and technology owners. Controls may include marking data unavailable, displaying its last update time, blocking new orders that require a valid price, applying an approved fallback source, or escalating an alert. The response depends on the instrument, venue, data type and bank policy. A test should verify the configured rule rather than assume one response fits every desk.

Agree measurable acceptance criteria before injecting faults. For example, specify how quickly a feed-health alarm must appear, which actions must be blocked, how stale data is labelled, and what reconciliation is required after recovery. Include downstream checks: a feed can recover while a cache, valuation or risk position remains inconsistent.

Select outage and degradation scenarios

Use a controlled test environment with representative feed configuration, permissions and test data. Confirm that fault injection cannot affect a live market connection or production trading. Coordinate with system owners and set a stop condition, monitoring plan and rollback procedure. If a vendor simulator or service virtualisation is used, validate that it reproduces the relevant message and timing behaviour.

Vary one failure condition at a time so results identify the cause. Useful scenarios include:

• A complete disconnect or rejected connection.
• Delayed updates that exceed the configured freshness threshold.
• A feed that remains connected but repeats old values.
• Missing, duplicated or out-of-order messages.
• Invalid fields, unexpected precision or an unsupported instrument identifier.
• Loss of one venue or source while another remains available.
• Recovery after a sequence gap, including snapshot refresh or message replay.

Check both explicit errors and silent degradation. A connected status does not prove that prices are current, complete or usable. Verify source timestamps, receive timestamps, sequence numbers where available, instrument mapping and the freshness indicators presented to consuming systems.

Verify controls and recovery

For each scenario, compare observed behaviour with acceptance criteria across the full path. Confirm that alerts identify the affected source and consumers, that stale values cannot be mistaken for live values, and that any restricted action is actually prevented at the relevant service boundary. Check audit and operational logs for timestamps, fault details, response actions and recovery decisions.

Recovery testing should cover the transition back to normal processing. Verify how the service detects missing data, whether it requests a snapshot or replays messages, and how it handles duplicates during catch-up. Reconcile affected instruments and downstream positions against the authoritative test feed. Confirm that fallback data is removed or clearly identified when the primary source returns, and that the system does not silently retain a fallback value as current.

Record residual gaps, such as consumers outside the test scope or fallback sources that were unavailable. Repeat relevant scenarios after feed-handler, routing, timeout or consumer changes. This work complements [Trading platform testing] and [Banking third-party service outage testing], which cover broader trading workflows and external dependency failures.

Worked example: delayed price updates

Suppose a test equity feed continues sending heartbeats but stops updating prices for one instrument. Configure the test feed to hold that instrument’s value while other instruments continue normally. The expected result might be that the affected price is marked stale after the agreed threshold, an alert identifies the instrument and source, and an order flow that requires a current price is blocked or routed according to the approved rule.

After restoring updates, verify that the instrument becomes current only after the recovery condition is met. Check the displayed timestamp, downstream price and risk calculation, alert clearance and audit record. Also confirm that unaffected instruments continued to process normally. This isolates a partial-data failure that a simple connection-loss test would miss.

Evidence and risk control

Keep the scenario, environment version, feed configuration, injected fault, expected result and observed evidence together. Capture relevant logs, alerts, timestamps and downstream assertions so another tester can reproduce the outcome. Where the test uses synthetic data, record how it represents sequence, timing and instrument characteristics. Avoid conclusions based solely on a green health check: demonstrate that the consuming applications enforce the required behaviour.