Selected work

Project / Distributed asynchronous processing

AI Incident Processing Pipeline

Personal project Production

Overview

A distributed AI incident-processing pipeline built with Celery, Redis, and FastAPI. It fans service analysis across parallel workers, coordinates results through a chord, and demonstrates retries, worker recovery, and idempotent state handling.

  • Celery
  • FastAPI
  • Redis
  • PostgreSQL
  • Python
  • TypeScript

Why

Long-running AI work needs a different execution model.

Long-running AI calls are a poor fit for synchronous request/response execution. An incident-analysis workflow can also fail partway through, lose a worker, or receive the same task more than once. The project explores how to move that work into an asynchronous pipeline without losing durable state or hiding partial failure from the user.

The deployed system

The workflow stays visible.

Real captures from the public demo. Select an image to view it larger.

The chaos workflow with two checkout-api retries visible while the Celery chord holds final synthesis.
The chaos workflow with two checkout-api retries visible while the Celery chord holds final synthesis.
The completed chain, parallel service group, chord convergence, and final synthesis.
The completed chain, parallel service group, chord convergence, and final synthesis.
Service-level results keep observed evidence separate from AI inference before the final incident intelligence report.
Service-level results keep observed evidence separate from AI inference before the final incident intelligence report.

Architecture

Fan out the analysis. Join it before synthesis.

FastAPI creates a durable job in PostgreSQL before Celery starts the work. Deterministic normalization and summarization run as a chain, then service analyses fan out dynamically through a Celery group. Redis transports tasks and temporarily coordinates chord results; the chord releases a final Gemini synthesis only after every service branch returns. PostgreSQL remains the durable source of truth throughout.

Retries are limited to transient provider and network failures, with bounded attempts, exponential backoff, and jitter. Late acknowledgements, reject-on-worker-lost behavior, a prefetch multiplier of one, terminal-state guards, and idempotent service-analysis persistence support safe recovery under at-least-once-style execution semantics.

  1. RequestBrowser → FastAPI
  2. Durable jobPostgreSQL
  3. Task transportRedis → Celery
  4. Deterministic workNormalize → Summarize
  5. Parallel workDynamic service-analysis group
  6. Fan-inRedis result backend → Celery chord
  7. Final workGemini synthesis → PostgreSQL

PostgreSQL is the durable source of truth. Redis carries tasks and holds temporary result state needed for chord coordination.

Reliability

Designed around partial failure and redelivery.

01

Selective retries

Only transient provider and network failures retry, using bounded attempts, exponential backoff, and jitter.

02

Worker-loss recovery

Late acknowledgements and reject-on-worker-lost behavior allow lost work to be redelivered. Worker prefetch is limited to one task.

03

Idempotent persistence

Service-analysis writes and terminal-state guards are designed for at-least-once-style execution, not exactly-once processing.

04

Durable history

Job state and execution events are persisted so progress and failure remain inspectable outside worker memory.

Observability / Demo

Follow the work, not just a spinner.

The deployed public demo exposes the workflow instead of hiding it behind a spinner. Visitors can follow normalization, summarization, parallel service branches, chord coordination, and final synthesis. A deterministic chaos mode injects two temporary checkout-api failures before Gemini is called, making the retries and third-attempt recovery visible.

Gemini responses use a standard JSON Schema and are validated with Pydantic. The interface deliberately separates observed evidence from AI inference.

  1. 01Normalize incident events
  2. 02Build a deterministic summary
  3. 03Fan service analyses out in parallel
  4. 04Wait for every branch in a chord
  5. 05Run final Gemini synthesis

Structured AI

A schema at the model boundary.

Gemini 3.5 Flash Lite receives a standard JSON Schema for each structured response. Returned data is then validated through Pydantic before it can be persisted or presented.

The interface keeps source observations visually distinct from model inference. That makes it clearer which claims came from the incident payload and which were proposed by the AI analysis.

Observed evidence

Facts preserved from the incident payload.

AI inference

Model conclusions returned through schema-constrained output and validated by Pydantic.

Deployment

Separate surfaces for the UI and asynchronous work.

Frontend
React, TypeScript, Vite, TanStack Query, Cloudflare Workers static assets
Railway
FastAPI service, separate Celery worker, managed Redis, managed PostgreSQL
AI execution
Gemini 3.5 Flash Lite calls from Celery workers

Technical stack

Backend
Python, FastAPI, Celery 5.6, SQLAlchemy, Alembic, Pydantic
State & coordination
PostgreSQL, Redis, Celery chain/group/chord
Frontend
React, TypeScript, Vite, TanStack Query
Deployment
Docker, Railway, Cloudflare Workers

Lessons

Durability and coordination are different jobs.

Reliable asynchronous work depends on explicit ownership boundaries. Redis is useful for task transport and temporary chord coordination, while durable job and event history belongs in PostgreSQL. At-least-once execution also makes idempotent state changes and terminal-state guards part of the core design rather than optional hardening.

Explore the pipeline

Run the chaos demo or inspect the implementation.

Back to selected work