Why Conductor

Applies to both editions

This page describes the Conductor engine that powers both open-source Conductor and Orkes Conductor. Operational defaults mentioned here, such as Redis or Elasticsearch, are specifics of the open-source distribution; Orkes Conductor deployments run the Orkes platform stack.

The problem

Every distributed process has to survive failure. Without coordination, each service carries its own retry, timeout, and recovery logic. That logic gets duplicated everywhere and owned by no one.

One common proposed solution is choreography, where services react to each other's events with no central coordinator. This keeps services decoupled on paper, but the logic of the overall business process is not visible. The flow exists only as an implied chain of event contracts, so changing one service can break consumers it cannot see. Observing the process is also hard. For example debugging a failure means correlating logs across all of the services.

Orchestration is Conductor's approach. The overall business process is defined in one place, while the work itself stays distributed. Conductor is the orchestrator. It owns the flow, the state, and the recovery, so workers stay stateless and independent.

How it works

Conductor runs as a server that your workers connect to. The server schedules tasks, persists every state change, and applies retries and timeouts. Workers poll the server for tasks, run your business logic in any supported language, and report results back. State lives in the persistence store you choose.

%%{init: {'look': 'handDrawn', 'theme': 'base', 'themeVariables': {'primaryColor': '#eef2ff', 'primaryBorderColor': '#1e40af', 'primaryTextColor': '#1e293b', 'lineColor': '#1e3a8a', 'edgeLabelBackground': '#ffffff', 'clusterBkg': '#fbfcff', 'clusterBorder': '#2563eb', 'fontFamily': '-apple-system, system-ui, Segoe UI, Roboto, Helvetica, Arial, sans-serif', 'fontSize': '15px'}, 'flowchart': {'nodeSpacing': 50, 'rankSpacing': 58, 'padding': 14, 'htmlLabels': true, 'curve': 'basis'}}}%%
graph TD
    subgraph Workers
        A["Worker A<br/>(Python)"]
        B["Worker B<br/>(Java)"]
        C["Worker C<br/>(Go)"]
        D["Worker D<br/>(C#)"]
    end

    subgraph Server["Conductor Server"]
        S["Scheduling · State · Retries<br/>Persistence · Queuing"]
    end

    subgraph Storage["Persistence"]
        DB["Redis / PostgreSQL / MySQL / Cassandra"]
    end

    A -- "poll / complete" --> S
    B -- "poll / complete" --> S
    C -- "poll / complete" --> S
    D -- "poll / complete" --> S
    S --> DB

See Architecture for details.

What Conductor gives you

Durable execution

Every workflow execution is persisted, so progress survives failure. A failed task is retried under a configurable backoff policy, a crashed worker's task is rescheduled to another worker, and a server restart resumes executions from their last recorded state. Your code carries no retry logic, because Conductor applies it for you. The same guarantee extends to agents.

Language-agnostic workers

Workers can be written in Python, Java, Go, JavaScript, C#, or Clojure, and each task in a workflow can use a different language. Workers talk to Conductor over REST or gRPC, so they can run in containers, VMs, serverless functions, or on a laptop.

Built-in system tasks

Common steps ship with the server: HTTP calls, inline scripts, JSON transforms, event publishing, wait timers, and human approval gates. None of them require a worker. See System Tasks.

Flow control operators

Operators express control flow in the definition itself: fork and join for parallelism, switch for branching, do-while for loops, and sub-workflows for composition. Dynamic tasks let the graph be resolved at runtime. See Operators.

AI tasks and agents

LLM calls run as native system tasks. Configure a provider and model on the task, or bring a framework-authored agent into a durable Conductor graph. The LLM orchestration guide is the provider and capability reference.

MCP support is built in. LIST_MCP_TOOLS discovers a server's tools and CALL_MCP_TOOL invokes one, with the same retries and state tracking as any other task.

Vector search tasks support Pinecone, pgvector, and MongoDB Atlas, so a single workflow can index embeddings, run similarity search, and pass the results to an LLM. Content generation tasks produce images, audio, video, and PDFs. All AI tasks share the standard durability guarantees: automatic retries, timeouts, and a complete execution record.

Event-driven workflows

Workflows can be triggered by external events and can publish events of their own. Kafka, NATS, AMQP, and SQS are supported. See Event orchestration.

Full operational control

Any execution can be paused, resumed, restarted, retried, or terminated. Executions are searchable by status, time, correlation ID, or custom tags, and every task records its inputs, outputs, timestamps, retry history, and worker identity.

Horizontal scaling

Servers and workers scale independently. Task domains, rate limits, concurrency limits, and persistence configuration control throughput and isolation, and metrics expose how each queue is behaving.

When to use Conductor

Use case	Example
Microservice orchestration	Order processing: payment → inventory → shipping → notification
Workflow automation	Automate business processes with durable execution, retries, and full observability
Durable agents	Multi-step LLM chains with function calling, tool use, RAG, and human-in-the-loop — durable agents that survive crashes
Long-running workflows	Insurance claims, loan approvals, onboarding flows spanning days or weeks — async workflows that survive deploys
Event-driven automation	React to Kafka events, trigger workflows, publish results back
Batch processing	Fan-out work across thousands of parallel workers with dynamic fork
Saga pattern	Distributed transactions with compensation on failure
RAG applications	Build retrieval-augmented generation pipelines with vector search, embedding generation, and LLM completion as workflow tasks
Content generation pipelines	Generate images, audio, video, and PDFs using AI models orchestrated as durable workflows

Next steps

• Quickstart — run your first workflow in 2 minutes
• Workflows — how workflow definitions work
• Tasks — task types and configuration
• Workers — building workers in any language