Why
Long-running AI work needs a different execution model.
Long-running AI calls are a poor fit for synchronous request/response execution. An incident-analysis workflow can also fail partway through, lose a worker, or receive the same task more than once. The project explores how to move that work into an asynchronous pipeline without losing durable state or hiding partial failure from the user.
The deployed system
The workflow stays visible.
Real captures from the public demo. Select an image to view it larger.
Architecture
Fan out the analysis. Join it before synthesis.
FastAPI creates a durable job in PostgreSQL before Celery starts the work. Deterministic normalization and summarization run as a chain, then service analyses fan out dynamically through a Celery group. Redis transports tasks and temporarily coordinates chord results; the chord releases a final Gemini synthesis only after every service branch returns. PostgreSQL remains the durable source of truth throughout.
Retries are limited to transient provider and network failures, with bounded attempts, exponential backoff, and jitter. Late acknowledgements, reject-on-worker-lost behavior, a prefetch multiplier of one, terminal-state guards, and idempotent service-analysis persistence support safe recovery under at-least-once-style execution semantics.
- RequestBrowser → FastAPI
- Durable jobPostgreSQL
- Task transportRedis → Celery
- Deterministic workNormalize → Summarize
- Parallel workDynamic service-analysis group
- Fan-inRedis result backend → Celery chord
- Final workGemini synthesis → PostgreSQL
PostgreSQL is the durable source of truth. Redis carries tasks and holds temporary result state needed for chord coordination.
Reliability
Designed around partial failure and redelivery.
Selective retries
Only transient provider and network failures retry, using bounded attempts, exponential backoff, and jitter.
Worker-loss recovery
Late acknowledgements and reject-on-worker-lost behavior allow lost work to be redelivered. Worker prefetch is limited to one task.
Idempotent persistence
Service-analysis writes and terminal-state guards are designed for at-least-once-style execution, not exactly-once processing.
Durable history
Job state and execution events are persisted so progress and failure remain inspectable outside worker memory.
Observability / Demo
Follow the work, not just a spinner.
The deployed public demo exposes the workflow instead of hiding it behind a spinner. Visitors can follow normalization, summarization, parallel service branches, chord coordination, and final synthesis. A deterministic chaos mode injects two temporary checkout-api failures before Gemini is called, making the retries and third-attempt recovery visible.
Gemini responses use a standard JSON Schema and are validated with Pydantic. The interface deliberately separates observed evidence from AI inference.
- 01Normalize incident events
- 02Build a deterministic summary
- 03Fan service analyses out in parallel
- 04Wait for every branch in a chord
- 05Run final Gemini synthesis
Structured AI
A schema at the model boundary.
Gemini 3.5 Flash Lite receives a standard JSON Schema for each structured response. Returned data is then validated through Pydantic before it can be persisted or presented.
The interface keeps source observations visually distinct from model inference. That makes it clearer which claims came from the incident payload and which were proposed by the AI analysis.
Facts preserved from the incident payload.
Model conclusions returned through schema-constrained output and validated by Pydantic.
Deployment
Separate surfaces for the UI and asynchronous work.
- Frontend
- React, TypeScript, Vite, TanStack Query, Cloudflare Workers static assets
- Railway
- FastAPI service, separate Celery worker, managed Redis, managed PostgreSQL
- AI execution
- Gemini 3.5 Flash Lite calls from Celery workers
Technical stack
- Backend
- Python, FastAPI, Celery 5.6, SQLAlchemy, Alembic, Pydantic
- State & coordination
- PostgreSQL, Redis, Celery chain/group/chord
- Frontend
- React, TypeScript, Vite, TanStack Query
- Deployment
- Docker, Railway, Cloudflare Workers
Lessons
Durability and coordination are different jobs.
Reliable asynchronous work depends on explicit ownership boundaries. Redis is useful for task transport and temporary chord coordination, while durable job and event history belongs in PostgreSQL. At-least-once execution also makes idempotent state changes and terminal-state guards part of the core design rather than optional hardening.
Explore the pipeline