Mohan Kumar - Backend / Platform Engineer in New Delhi

Trading and operations teams needed to see post-trade activity while the session was still running, not in an end-of-day report.

• Write volume rises and falls with market activity. Ingest has to absorb the bursts without dropping rows.
• Collection runs next to trading processes and must stay light on those hosts.

• Telegraf agents on the source hosts collect post-trade records and metrics and write them in batches.
• High-volume post-trade rows go to ClickHouse (columnar), time-series metrics to InfluxDB. PostgreSQL is part of the stack as well.
• Grafana reads from the stores. It is the query and visual layer only, never in the write path.

If a strategy crashes and nobody notices, we lose money. People are not always watching Grafana, and the alert had to reach them wherever they were.

• Runs on latency-sensitive hosts. It must not compete with trading processes for CPU or cache.

• A small watcher that notices strategy-process crashes and connector anomalies and posts them to Zoom and email.
• It shares hosts with trading processes and is designed to minimise interference with them: a small CPU and cache footprint.

Reconciliation, context generation and reporting depend on post-trade data spread across many hosts. Every new pipeline used to need an engineer to write Airflow code.

• Workers run on latency-sensitive hosts close to the data and must stay out of the way of trading.
• Some pipeline authors are not engineers.

• Airflow pipelines with edge workers on latency-sensitive hosts, feeding reconciliation, context generation and reporting.
• Wrote a YAML-to-DAG parser so non-engineers can add an Airflow pipeline without writing Python.

Exchanges and regulators publish rule changes as PDFs, each on its own site. Someone had to read every one of them and spot the deadlines by hand.

• Several of the sites block plain HTTP scrapers.
• A summary must not invent content when a circular has no readable body.
• Everything runs on our own hardware, with no external LLM API.

• Scrapers for 16 sources, plus a headless-browser worker for the sites that block bots. Text and tables come out of PDF and HTML attachments in a memory-capped child process, so one bad file cannot take the parser down.
• Summaries and severity come from a local LLM. When a circular has no text, a rule-based summary is used instead and the model never sees it.
• Deadline extraction, theme tracking, relevance scoring per team, and email and chat alerts behind allowlists.
• A triage classifier. I wrote the labelling guidelines and checked agreement between labellers. Then I compared TF-IDF with logistic regression against a fine-tuned transformer, and the better model ran in shadow mode before anyone relied on it. Most of the gains came from better labels.

Operators need a shell on trading hosts, but full sudo is too much. Every action has to be auditable, and access has to be revocable at once.

• Disabling a user must end their sessions on every host within 5 seconds.
• An emergency kill must never wait on an approval, the CA or the logs.

• Login with short-lived SSH certificates from an internal CA, plus password and TOTP. A revocation list blocks a disabled user everywhere.
• A signed command catalogue and policy. Risky commands need a second approver. A kill never does.
• A hash-chained audit log with an offline verifier. A file guard puts back tampered policy and sshd files in under a second.
• All of it runs in a Docker sandbox with mock credentials, alongside security and compliance reviews, runbooks and a user handbook. It is a prototype and has not gone to production.