Workflow Orchestration · Netflix / Orkes Conductor

The server holds the state.
Workers just poll for work.

Conductor is an orchestration engine where workflows are JSON definitions interpreted by a stateless server. Every decision is derived from persisted state — which is why killing a worker, or even the server, never loses a workflow. Scroll down and run the simulation to see it happen.

01

The component map

Four planes: clients that start workflows, a stateless server that decides, external workers that execute, and storage that remembers. Hover each block for its job description.

Client / App

REST · gRPC · SDK
Starts workflows, queries status. Fire-and-forget: once started, the server owns the execution.

Conductor UI

:5000 / :8127
Visualizer + Executions tab. Reads the same APIs — nothing special happens here.
Conductor Server — stateless, scale horizontally :8080

API Layer

/api/workflow · /api/tasks
Entry point for clients and workers. Workers long-poll /tasks/poll/{type} here.

Decider

event-driven engine
On every event: load state + JSON definition → compute next tasks → push to queues. No replay — state is explicit.

Task Queues

dyno-queues / Redis
One logical queue per task type. Atomic hand-off guarantees a task goes to exactly one worker.

Workflow Sweeper

reconcile loop
Safety net: periodically re-runs decide() on every RUNNING workflow. Catches timeouts and lost events.

Workers

Python · Go · Java · any
Stateless processes anywhere with HTTP reach to the server. Poll → execute → POST result. Never receive inbound connections.

System Tasks

run inside server
HTTP, SWITCH, FORK_JOIN, SUB_WORKFLOW, WAIT, JQ — orchestration logic that needs no external worker.

Persistence — Redis / Postgres / Cassandra

THE source of truth: workflow state, task state, inputs/outputs, queues

Indexing — Elasticsearch / OpenSearch

search & execution history for the UI; not needed for correctness

every arrow into the server is initiated from outside · the server never dials out to workers

02

Run the Decider loop

This is the greet_flow you built. Start it and watch the full cycle. Then run it again and hit Kill worker while the task is IN_PROGRESS — the lease expires, the Sweeper catches it, and a retry is scheduled.

workflow: —
Client
POST /api/workflow/greet_flow
Persistence
workflow + task state
Conductor Server
API
:8080/api
Decider
evaluate → schedule
Queue · greet
Sweeper
reconcile every ~30s
Worker (Python)
long-poll /tasks/poll/greet
greet()
— idle —
task / data completion event worker long-poll timeout path

--:--:--conductor server ready · press “Start workflow”

03

Task lifecycle & the lease

A task is a lease with deadlines — like a DHCP lease. Silence past a deadline is treated as failure; nothing ever depends on a live connection to the worker.

SCHEDULED
worker polls
IN_PROGRESS
worker POSTs result
COMPLETED
pollTimeout — sits in queue, nobody polls → TIMED_OUT.
Your first experiment: worker off, task waits (default 0 = forever).
responseTimeoutSeconds — the lease. No result / no update in time → Sweeper marks TIMED_OUT → Decider applies retryCount / retryLogic / retryDelaySeconds → new attempt queued.
Design consequence — a worker may die at 90% done; the retry re-runs from zero. Task logic must be idempotent.
04

How Conductor talks to everything else

Four integration surfaces. Note the direction of each arrow — the only outbound calls the server ever makes are system tasks and event sinks it was explicitly told to call.

Polyglot workers inbound · poll

SDKs for Python, Java, Go, C#, JS/TS, Clojure. Each worker long-polls its task queue over HTTP — works from Kubernetes pods, VMs, behind NAT, from another site. One workflow can mix languages per task.

worker ⟶ GET /tasks/poll/greet ⟶ server

System tasks outbound

The server itself calls your services: HTTP task hits any REST endpoint (your FastAPI, NetBox, a device API gateway), SUB_WORKFLOW composes workflows, SWITCH/FORK_JOIN add control flow — zero worker code.

server ⟶ HTTP task ⟶ your REST API

Event handlers in / out

Eventing bridges message buses: an incoming Kafka / SQS / AMQP / NATS message can start a workflow or complete a WAIT task; an EVENT task publishes back to the bus. This is how Conductor joins event-driven pipelines.

Kafka topic ⟷ event handler ⟷ workflow

API & Webhooks inbound

Everything is a REST call: start/pause/terminate workflows, complete tasks from external systems (human approval, a ticketing hook), query state. Your CI, ChatOps bot, or an AI agent can drive workflows the same way the UI does.

external system ⟶ POST /api/... ⟶ server
05

The four ideas that explain everything

State lives in the DB

Server and workers are stateless. Restart all of them mid-flight: 1000 RUNNING workflows resume, because the Sweeper re-decides each one from persisted state.

Workflows are data, not code

A JSON definition interpreted by the Decider. No deterministic replay (Temporal's model) — the current state is stored explicitly, so “resume” is just “read and continue”.

Pull, never push

Workers dial in. No inbound ports on workers, trivial scale-out, and the queue guarantees exactly one worker picks up each task.

Events + reconcile

Event-driven Decider for speed, periodic Sweeper for completeness — the same watch-plus-reconcile pattern as a Kubernetes controller.