A post-trade pipeline that shows the session while it runs

Why it mattered

The trading and operations teams wanted to see post-trade activity during the session. The useful view used to arrive in an end-of-day report, which is too late to act on anything that went wrong in the morning.

I built the collection and storage side: the agents on the source hosts, the batching, the two stores, and the Grafana dashboards on top.

Constraints

Two constraints shaped most of the decisions.

• Write volume follows the market. It rises and falls with activity, and ingest has to absorb the bursts without dropping rows.
• Collection runs on the same hosts as trading processes, so the agents have to stay light there.

The design

Telegraf agents on the source hosts collect post-trade records and metrics and write them in batches. High-volume post-trade rows go to ClickHouse. Time-series metrics go to InfluxDB. PostgreSQL is part of the stack as well. Grafana sits at the end and reads the stores directly.

The main decision was to split storage by the shape of the data. Post-trade rows arrive in volume, and most questions asked of them are aggregations over a session. A columnar store reads only the columns a query touches, which suits that work. Metrics are narrow series with a timestamp and a few tags, and InfluxDB is built for exactly that. Putting both in one store would have meant one of them living in the wrong engine.

Batching is the other trade. Writing each row on its own costs a round trip per row, and ClickHouse handles many small inserts badly. Sending rows in batches makes each insert cheap per row, but a row now waits until its batch is flushed. That wait comes out of the under-a-second budget, so batch size and flush interval are a latency setting as much as a throughput one.

Grafana is the query and visual layer only. It never sits in the write path, so a slow or broken dashboard does not stop rows from landing.

What the numbers measure

1.6M rows per second is the rate of rows written into ClickHouse across the ingest tier, counted at the database. It is a write rate, and it says nothing about how fast a dashboard renders.

Under one second is end to end: from a record leaving its source host to it being queryable from Grafana. It measures how fresh the data is. A dashboard panel on a 5-second refresh will show the row later than that, and the figure does not claim otherwise.

What I'd change

I would publish a small synthetic benchmark next to this write-up: trade-shaped rows into ClickHouse under Docker Compose, rows per second plotted against batch size. The production number cannot be shown, but the method can.

I would also write the measurement method into each dashboard's description, so anyone looking at a latency or throughput panel knows what it counts.