Getting a strategy crash to a person within 60 seconds

Why it mattered

Dashboards work when someone is looking at them. During a session people are often busy with something else, and a crashed strategy or a misbehaving connector keeps costing money until somebody notices. The alert had to find people wherever they were, so it goes to Zoom and to email.

The watcher covers two kinds of failure: a strategy process that has crashed, and a connector that is behaving abnormally.

The constraint that shaped it

The watcher runs on the same latency-sensitive hosts as the processes it watches. Anything I put there competes with trading for CPU time and for cache. A monitor that slows down the thing it monitors is worse than having none, so the footprint mattered more than any feature.

How it works

The flow is short. A watcher on the host notices a crash or an anomaly and emits an event. A dispatch step turns the event into a message and sends it to Zoom and to email.

The 60-second budget is a choice about how often to look. Checking more often gets the alert out sooner and costs more on a host where every cycle counts. A minute is fast enough for a person to act on.

What 60 seconds measures

Under 60 seconds is time to notify: from the crash or anomaly to the Zoom or email message going out. It is not time to resolve. Someone still has to look at the process and decide what to do.

What I'd change

I would publish the watcher's own footprint next to the claim: CPU share and resident memory on a trading host, measured over a session. A small footprint is easy to say and easy to show.

I would add a scheduled self-test that starts a dummy process on one host, kills it, and checks the alert arrives inside the budget. That turns the 60-second figure from a design target into something checked every day.