01 / systems / 03 of 05
Post-trade pipelines on Airflow, defined in YAML
Reconciliation, context generation and reporting all need post-trade data that is spread across many hosts. These Airflow pipelines move more than 500 GB of it a day, and a YAML-to-DAG parser lets people who are not engineers add a pipeline.
Why it mattered
Post-trade data is produced on many hosts, and several jobs downstream depend on it: reconciliation, context generation and reporting. Every new pipeline used to need an engineer to write an Airflow DAG in Python, get the imports right and ship it. Some of the people who knew what the pipeline should do could not write that code, so they waited for someone who could.
Constraints
- The workers run on latency-sensitive hosts, close to the data, and must stay out of the way of trading.
- Some pipeline authors are not engineers. Python is not an option for them.
- The volume is large: more than 500 GB a day across the hosts.
The design
Airflow schedules the work centrally. Edge workers run the tasks on the hosts where the data already sits. Moving the compute to the data avoids copying hundreds of gigabytes across the network before any useful work starts. The price is that those workers now share hosts with latency-sensitive processes, which is why the first constraint above matters so much.
The YAML layer is the part I would defend hardest in a design review. A pipeline author writes a spec. The parser reads it, fills in the Airflow imports and the Python boilerplate, and produces a DAG that Airflow picks up like any other. The author never touches Python.
A spec format limits what a pipeline can express. Anything the parser does not support needs an engineer again, either to extend the parser or to write that DAG by hand. I think that is a fair price, because a small format can be validated and reviewed before it runs.
What the numbers measure
500+ GB a day is the post-trade data read and written by the Airflow edge workers over a day. It is a daily total, not a rate, and it covers all pipelines together.
YAML is a mechanism, not a measurement. The point is who can author a pipeline, and the honest measure of that is how many pipelines non-engineers have written.
What I'd change
I would run the parser in CI against every spec and fail the build on unknown or missing fields, so a typo in YAML shows up in review instead of as a DAG that never appears in Airflow.
I would also record per-task resource use on the edge workers and keep it next to the pipeline, so the claim that they stay out of trading's way has a number behind it.