Until the end of June the platform could not send a single email, and a queue of features was waiting on it: password reset, invitations, sign-in codes, notifications, invoices. I built email once, as a subsystem, rather than inside the first feature that needed it. The capability and an admin view to watch it landed on the same day, with one reference template proving the pipeline end to end and no user-facing email yet.

An outbox rather than a queue#

The obvious design enqueues a send job at the moment the action happens. It loses mail in two directions. The job is enqueued and the transaction then rolls back, so an email goes out about something that never happened. Or the transaction commits and the queue is briefly unreachable, so an email that should have gone never does.

The platform writes an outbound row instead, in the caller's own database transaction, and only then enqueues the job that delivers it. A rolled-back action leaves no row; a committed one leaves a row that something will deliver. A cron sweeper picks up any row that has sat two minutes past its scheduled time, which covers the lost enqueue and the worker that died mid-send. The job locks its row, and the sweeper skips rows already locked, so the two never deliver the same one twice.

Each row walks queued → sending → sent. A transient SMTP error moves it to retrying, with a backoff that starts at a minute and doubles up to an hour. A permanent error, or the fifth failed attempt, ends it at failed, and the failed rows are a dead-letter queue I can query and replay. Since every attempt lives on a row, every email the platform sent or failed to send has an audit trail.

Templates that survive Outlook#

Each email is three files: a subject, an MJML body and a plaintext alternative. MJML because responsive HTML that Outlook renders correctly is a craft of its own, and MJML compiles to it. The reference MJML compiler is a Node program and the API is Python, so I used mrml, a Rust implementation with Python bindings: rendering an email needs no Node runtime beside the API.

Templates render in a sandboxed Jinja2 environment with autoescaping on the HTML parts, so a user whose name contains markup, or something that looks like a template expression, gets it printed as text. Strings translate through the same catalogues as the rest of the API, in English and French.

Security at the boundary#

Every check runs when a message is built, not when it is sent. A carriage return or line feed in a header is refused, which closes header injection. Addresses are validated, and attachments and the whole message have size caps. Outside production, recipients can be allow-listed or redirected, so a developer pointed at a real relay cannot mail real users. Logs mask every address. In production the configuration refuses to start with a plaintext SMTP connection or one whose certificate is not verified.

Per-recipient rate limits and idempotency keys are opt-in per call. A password reset uses both, so a double click sends one email and a scripted loop is throttled.

An admin view, the same day#

An outbox is only an audit trail if someone can read it. The rows became a resource of the platform's own CRUD framework, behind a new access right given to platform administrators: a list, a search and a read-only form, with two actions, resend and cancel, that refuse a row whose state they cannot apply to.

The security rules taught me something about my own access-control engine. I wanted operators to read every column and change only a handful, and wrote one rule for both. A field restriction in that engine applies to reads as well, so the single rule hid most of the row. It became two rules, one to read and one to update the scheduling fields. Rendered bodies and template context stay secret; the last error does not, because an operator needs to know why a send failed.

Local development gained Mailpit, a mail catcher, so an email sent from a laptop lands in an inbox I can open rather than in a log line.

What it took to deliver anything#

Tidy as it was, nothing was sending. Ten days later I found that nothing in local development, CI or the end-to-end suite ever started the worker process that drains the queue. The subsystem was fully tested against an in-memory backend and consumed by nobody.

Wiring the worker in turned up a subtler bug. Playwright starts each server a test run needs and waits for it to be ready, and the worker listens on no port, so I pointed its readiness check at the mail catcher's. Playwright reads an already-open port as a server that is already running and never runs the command, so in CI the worker was silently never launched. The check now waits for the line the worker prints when it starts, and an end-to-end test changes a password and reads the resulting email back out of Mailpit, subject and body.

Two days after that the design itself needed correcting. The send job held its row lock, and with it a pooled database connection, for the whole SMTP round trip, and the sweeper held one connection while delivering up to a hundred rows in one transaction. Under load the worker could drain the pool, and the API would start answering 503. Delivery became claim-then-send: mark the row sending and commit, send with no connection held, then record the outcome in a second short transaction. The sweeper stopped delivering anything and only re-enqueues. That makes delivery at-least-once, since a crash between the send and its commit can deliver an email twice, and a duplicate is the failure I would rather have than a lost message.

What it cost#

The first commit added about 2,400 lines, 61 tests and an operations guide among them. It was the right size. Password reset landed on it eleven days later, and every email since has been the same shape: three template files and one function call, with the retries, the audit trail and the admin view already there.

Transactional email as a subsystem