Reading every exchange circular so people don't have to

Why it mattered

A circular can move a deadline or change a rule the firm has to follow. Missing one is a compliance problem. Reading every one is slow, and the date that matters is often somewhere in the middle of a long PDF.

The tracker is an internal web app for the trading desk and the operations teams, deployed on an internal host. The history has 325 commits between 9 September and 9 October 2026: about 31,000 lines of Python on the backend and about 12,700 lines on the frontend.

Constraints

• Several of the sites block plain HTTP scrapers.
• A summary must not invent content when a circular has no readable body.
• Everything runs on our own hardware. No circular text goes to an external LLM API.

How it fits together

Scrapers run on a schedule (APScheduler). Most sites answer plain HTTP. The ones that block bots go through a separate browser path: headful Chrome driven over the DevTools protocol, plus a Playwright worker. Keeping that path separate means the easy sites stay cheap and only the hard ones pay for a browser.

Text and tables come out of PDF and HTML attachments with pdfplumber. Some PDFs are malformed or very large, and a parser that runs out of memory takes its whole process down with it. So the parser runs in a child process with a memory cap, and that process can be killed without touching anything else. One bad file fails on its own.

Summaries and a severity come from a local model, qwen2.5 served by Ollama. The rule I cared most about is simple: when a circular has no readable text, the model never sees it. A rule-based summary is written instead, so there is nothing for the model to make up.

On top of that sit deadline extraction, theme tracking and a relevance score for each team. Everything lands in PostgreSQL 16, with Meilisearch for full-text search. Alerts go out by email and chat, and only to allowlisted recipients, so a bad rule cannot message the whole firm.

Backend: FastAPI, async SQLAlchemy, Alembic. Frontend: Next.js 15 and React 19. It runs under Docker Compose behind nginx.

Triage, and why the labels mattered more than the model

A summary tells you what a circular says. It does not tell you whether it matters to you. I split triage into three classification tasks and built a labelled set for them.

I wrote versioned labelling guidelines. LLM labelling agents labelled each item twice, I measured agreement between them with kappa, and the items they disagreed on went through an adjudication pass.

Then I compared TF-IDF with logistic regression against a fine-tuned ModernBERT over several rounds. The better model runs in shadow mode behind a feature flag in the backend: it labels new circulars, and nobody relies on those labels yet.

The main finding was about the data. The labels were the bottleneck, and most of the gains came from making them better.

Tests before refactoring

The backend has 109 pytest files. Before refactoring older code I added characterization tests that pin down what it currently does, so a refactor that changes the output fails a test instead of reaching users.

What the numbers measure

16 is the number of exchange and regulator sites scraped on a schedule. 100+ files counts test files in the backend suite. It is a size, and it does not say how many tests pass or how much code they cover.

What I'd change

Since the labels turned out to be the bottleneck, next time I would write the guidelines and build the labelled set before any model comparison. Early model numbers on weak labels mostly measure the labels.

I would keep the empty-text rule in any LLM feature I build: decide in code when the model is allowed to answer, and give it nothing when there is nothing to summarise.