Async/await for containers: how Trigger.dev suspends and resumes running tasks

I’ve been looking for a platform to run AI-powered workflows — LLM chains, data pipelines, agent loops — without building all the queue, retry, and scheduling infrastructure myself. That’s how I came across Trigger.dev, an open-source platform for running background tasks in TypeScript. It’s a nice project, but one feature in particular caught my attention.

If your task calls wait.for() or awaits a child task with triggerAndWait():

export const myTask = task({
  id: "my-task",
  run: async () => {
    // Container is suspended here — you pay nothing for the hour
    await wait.for({ hours: 1 });

    // Container is suspended while the child task runs in its own container
    const result = await childTask.triggerAndWait({ data: "some data" });
  },
});

the platform can checkpoint the task container, release its compute, and restore it when the wait resolves. You do not have to serialize local variables yourself. This applies to supported wait operations, not every JavaScript await; short timer waits may not checkpoint.

Trigger.dev calls this its Checkpoint-Resume System. A snapshot preserves process state such as memory and CPU registers. Storage and orchestration remain necessary while compute is suspended, and external services do not rewind with the process.

The analogy with async functions is useful: pause while waiting, then continue when ready. The difference is the level at which resources are released. A normal await leaves the process running; a supported Trigger.dev wait can let the platform suspend the task container and restore it on a compatible worker.

That sounded almost too good to be true, so I asked Claude to explore the source code to understand how it actually works. The answer involves CRIU (Checkpoint/Restore In Userspace), Docker’s experimental checkpoint API, Buildah for OCI image creation, and a carefully designed state machine to coordinate it all.

In this article I want to share what I found.

The problem: long waits in serverless tasks

Let’s first understand where checkpoints might be needed. Consider a task that processes a payment, waits for a confirmation, and then sends a receipt:

import { task, wait } from "@trigger.dev/sdk";

export const processPayment = task({
  id: "process-payment",
  run: async (payload) => {
    const charge = await chargeCustomer(payload);

    // Parent container is suspended while getConfirmation runs in its own container
    // Could take hours or days — you don't pay for the wait
    const confirmation = await getConfirmation.triggerAndWait({
      chargeId: charge.id,
    });

    await sendReceipt(charge, confirmation);

    return { success: true };
  },
});

A workflow that waits for hours can exceed a function invocation’s execution limit. Keeping a worker allocated throughout that wait also has a cost.

Common ways to handle the wait include:

1. Keep the container running while the confirmation arrives — you pay for compute the entire time, even though the parent task is doing nothing
2. Split into multiple tasks — break the workflow into chargeCustomer, a scheduled trigger, and sendReceipt, losing the simplicity of a single function. Workflow orchestrators like AWS Step Functions or Google Cloud Workflows can help, but you’re now debugging state machines instead of async functions
3. Serialize state to a database — save charge somewhere, schedule a follow-up job, deserialize on resume — now you’re building a workflow engine

Trigger.dev’s answer is option 4: freeze the container’s memory to disk, shut it down, and restore it later.

When you call triggerAndWait, it spawns the child task in a separate container, then the parent is checkpointed and suspended — releasing its compute and concurrency — until the child completes. The parent resumes with the child’s return value, as if it were a normal await. The same mechanism kicks in for timer waits like await wait.for({ hours: 24 }).

How this compares to traditional approaches

Workflow engines preserve progress in different ways. Persisting explicit application state is one option; event-history replay is another. Neither is the same as saving a process image.

Event-history replay, used by Temporal, reconstructs workflow state by replaying recorded events through deterministic workflow code. Completed activities are represented by recorded results rather than rerun as ordinary side effects. This does not require serializing arbitrary closures or open sockets.

Container checkpointing saves supported process state, including the memory holding local variables and closures. It can avoid replaying workflow code, but requires compatible operating-system and runtime support. Files, sockets, and other external resources still need explicit consideration.

The trade-off is between replay constraints and snapshot infrastructure: checkpointing reduces application-level state handling, while adding storage, transfer, and compatibility requirements.

CRIU: the technology underneath

CRIU is a Linux tool that saves and restores supported process state: memory pages, registers, file descriptors, and parts of the associated kernel state. It can operate on a process tree.

Because CRIU works below the language runtime, it can preserve programs written in many languages. That does not make every process checkpointable: devices, kernel features, namespaces, and runtime support constrain what can be restored.

Docker has experimental support for CRIU through docker checkpoint create, and Kubernetes supports it via the CRI (Container Runtime Interface) with crictl checkpoint. Trigger.dev uses both, depending on the deployment mode. When CRIU isn’t available at all — missing binary, unsupported kernel, or Docker experimental features not enabled — Trigger.dev falls back to docker pause, which suspends the container but doesn’t capture state. The workflow continues, but if the container dies, the run is lost. This fallback exists for development environments where setting up CRIU isn’t practical.

Trying it yourself

This demo runs CRIU inside a privileged Docker container. It requires a compatible Linux kernel and CRIU setup; --privileged alone does not guarantee it will work.

docker run -d --name criu-demo --privileged python:3.12-slim bash -c 'apt-get update -qq && apt-get install -y -qq criu > /dev/null 2>&1 && sleep infinity'

Copy the counter script — it just increments a number and writes it to a file every second:

docker exec criu-demo bash -c 'cat > /counter.py << "EOF"
import time
count = 0
while True:
    count += 1
    with open("/output.txt", "a") as f:
        f.write(f"count = {count}\n")
    time.sleep(1)
EOF'

Copy the demo script — it starts the counter, checkpoints it, then restores it:

docker exec criu-demo bash -c 'cat > /demo.sh << "EOF"
#!/bin/bash
python3 /counter.py &
PID=$!
disown
sleep 5

echo "--- before checkpoint ---"
cat /output.txt

mkdir -p /checkpoint
criu dump -t $PID -D /checkpoint --shell-job -v0

echo "--- checkpointed, process killed ---"
> /output.txt

criu restore -d -D /checkpoint --shell-job -v0
sleep 5

echo "--- after restore ---"
cat /output.txt
EOF
chmod +x /demo.sh'

Run the demo:

docker exec criu-demo /demo.sh

Output:

--- before checkpoint ---
count = 1
count = 2
count = 3
count = 4
count = 5
--- checkpointed, process killed ---
--- after restore ---
count = 7
count = 8
count = 9
count = 10
count = 11

The counter was checkpointed after writing count 5, the process was killed, then CRIU restored it from the checkpoint — and it resumed counting as if nothing happened. The variable count was sitting in Python’s heap memory, and CRIU captured and restored the entire memory state. This is exactly the mechanism Trigger.dev uses, just wrapped in more orchestration.

The demo checkpoints a process from inside its container. In the source implementation discussed next, the coordinator asks the container runtime to checkpoint the task from outside.

How the checkpoint flow works

Here’s the full checkpoint-resume flow for a parent task that triggers a child task (based on the diagram from Trigger.dev docs):

The diagram shows the flow for triggerAndWait, but the same mechanism applies to wait.for() — the only difference is what resolves the waitpoint (a timer vs a child task completing).

The source links below point to specific revisions consulted for this walkthrough. They describe that implementation, not a guarantee that current deployments use the same component layout.

The CR System in the diagram maps to the Coordinator container — here’s how these components are laid out on a worker VM:

Worker VM
├── Supervisor container (apps/supervisor)
│   ├── Dequeues runs from the platform
│   ├── Creates task containers on demand
│   └── Coordinates with Coordinator for checkpointing
│
├── Coordinator container (apps/coordinator)  ← "CR System" in the diagram
│   ├── Runs the Checkpointer (CRIU, Buildah)
│   ├── Has access to Docker daemon
│   └── Freezes task containers from the outside
│
└── Task container (ephemeral, one per run)
    ├── Controller (ManagedRunController)  [entry point]
    │   └── Signals when task is suspendable
    │
    └── Worker  [child process, forked via IPC]
        └── Your task code (task.run())

In this design, the controller signals readiness and the coordinator requests the checkpoint from outside the task container. The checkpoint includes the controller and worker state. Restoring on another worker also requires a compatible runtime and the necessary filesystem state.

Let’s walk through each step.

Step 1: Start execution

The supervisor running on the worker VM dequeues the run from the platform and creates a task container via workloadManager.create(), passing environment variables, e.g. TRIGGER_SUPERVISOR_API_DOMAIN, so the controller process inside the task container knows how to reach the supervisor’s Workload API over HTTP. The controller ManagedRunController is the main Node.js process inside the task container — it uses Node’s fork() to spawn a worker child process that runs your task code. The two processes communicate via Node.js IPC.

Step 2: Trigger child task

When your code calls await childTask.triggerAndWait(...), two things happen:

• the SDK running inside the worker process makes an API call directly to the Trigger.dev platform (the “Trigger.dev” actor in the diagram), bypassing the controller — the worker has its own HTTP client to the platform. This queues the child task for execution and creates a waitpoint — a record in the platform’s database that says “this run is waiting for this child task to complete.”
• the worker signals to the controller via IPC that it’s suspendable — ready to be frozen without data loss.

The parent doesn’t wait for the child to start; it just tells the platform “run this” and signals “I can be checkpointed now.” Note the split: the controller handles the execution lifecycle (suspendable signaling, snapshot management), but the SDK’s API calls to trigger tasks and create waitpoints bypass it entirely — they go straight from the worker to the platform over HTTP. For wait.for(), the waitpoint is a datetime instead of a child task, but the rest of the flow is identical.

Step 3: Request snapshot

The controller inside the task container calls suspendRun() on the supervisor’s Workload API (using the HTTP connection from Step 1). The supervisor delegates to the coordinator via the CheckpointClient. The coordinator invokes CRIU from the outside to freeze the task container. The checkpoint logic lives in checkpointAndPush() and has two modes:

Docker mode (local/development):

docker checkpoint create --leave-running <container-name> <checkpoint-name>

Kubernetes mode (production):

crictl checkpoint --export=/checkpoints/<identifier>.tar <container-id>

Both commands ask the runtime to create a process checkpoint. The Docker example uses --leave-running, so taking the snapshot does not itself stop the original container permanently; the surrounding orchestration handles suspension and cleanup.

Step 4: Store snapshot

In production (Kubernetes mode), the checkpoint is exported as a tar archive. The coordinator wraps it in an OCI container image using the Buildah class and pushes it to a registry:

buildah from scratch
buildah add <container> /checkpoints/<identifier>.tar /
buildah config --annotation=io.kubernetes.cri-o.annotations.checkpoint.name=<shortCode> <container>
buildah commit <container> <registry>/<namespace>/<project>:<version>.prod-<shortCode>
buildah push --tls-verify <imageRef>

The OCI registry transports the checkpoint artifact. Restoring it requires a compatible checkpoint-aware runtime; packaging a checkpoint as an OCI image does not make it an ordinary runnable image on any node.

Step 5: Release resources

Once the checkpoint image is stored, the platform:

1. Updates the run status to WAITING_TO_RESUME in the platform’s database (the same database where the waitpoint from Step 2 was created)
2. Stores a TaskRunCheckpoint record (type, location, image reference)
3. Releases all concurrency for this run

If you have a queue with concurrencyLimit: 5 and three tasks are suspended, those three slots are freed up. Suspended tasks consume zero compute and zero concurrency.

Step 6: Child task completes

The child task runs in its own container. When it finishes, its controller calls completeRunAttempt() on the supervisor, which reports the result to the platform. The platform resolves the parent’s waitpoint record created in Step 2, which triggers the restore flow. For wait.for(), this step is replaced by the timer expiring, which resolves the waitpoint the same way.

Step 7: Retrieve snapshot and restore state

The platform requests the checkpoint from the CR system, which retrieves the snapshot image from storage. A new container is started from the checkpoint image — CRIU restores all processes to their exact memory state.

The restored controller detects it’s been restored and calls continueRunExecution():

POST /api/runs/{runId}/continue
Body: { snapshotId: "...", workerId: "...", runnerId: "..." }

The restored container may run on a different compatible worker. A shared registry makes the artifact available; runtime and host compatibility determine whether restoration succeeds.

Step 8: Resume and complete execution

The backend validates the snapshot, updates the run to EXECUTING, and the task continues from the line after the await. From your code’s perspective, nothing happened — the await resolved with the child task’s return value, and execution continues normally.

What can go wrong

Checkpointing isn’t magic. There are edge cases:

Connections can become stale. Preserving a socket’s local state does not keep its remote peer alive. After a long wait, database, HTTP, or WebSocket connections may need reconnection. Application code should handle that possibility.

Process snapshots and filesystem snapshots are different. File descriptors refer to files whose required contents and metadata must also be available on restore. Whether writable container layers or temporary files are preserved depends on the runtime and deployment; do not infer it from the memory snapshot alone.

Larger memory footprints usually mean larger checkpoints. Snapshot size is not necessarily equal to allocated RAM: resident pages, compression, sparse files, and runtime options affect it. Large checkpoints can increase storage, transfer, and restore time.

CRIU requires kernel support. CRIU needs specific kernel features (namespaces, cgroups) and Docker’s experimental mode. In Kubernetes, the container runtime (CRI-O or containerd) must be configured for checkpoint support. This isn’t available everywhere, which is why Trigger.dev has the simulation fallback.

The same pattern, at the VM level

CRIU operates at the process level — it captures a single process tree inside a container. But the same checkpoint/restore pattern works at the VM level too. Firecracker, the microVM manager behind AWS Lambda and Fly.io, can pause an entire virtual machine, dump its full memory and device state to files, and restore from those files later — on a completely new Firecracker process.

I tested this on WSL2 with KVM enabled. The setup: Firecracker v1.12.0, an Alpine Linux rootfs with a shell counter incrementing every second, and a prebuilt Linux kernel. After booting the VM and letting the counter reach 20, I paused the VM and created a snapshot via Firecracker’s REST API:

# Pause
curl --unix-socket /tmp/firecracker.socket -X PATCH \
  http://localhost/vm -H 'Content-Type: application/json' \
  -d '{"state": "Paused"}'

# Snapshot
curl --unix-socket /tmp/firecracker.socket -X PUT \
  http://localhost/snapshot/create -H 'Content-Type: application/json' \
  -d '{"snapshot_type": "Full", "snapshot_path": "./snapshot_file", "mem_file_path": "./mem_file"}'

Then I killed the Firecracker process entirely, started a fresh one, and loaded the snapshot:

curl --unix-socket /tmp/firecracker.socket -X PUT \
  http://localhost/snapshot/load -H 'Content-Type: application/json' \
  -d '{"snapshot_path": "./snapshot_file", "mem_file_path": "./mem_file", "enable_diff_snapshots": false, "resume_vm": true}'

The counter resumed at 21. The restore took ~29ms.

Firecracker snapshots include guest memory and device state, but restoration still has CPU, version, and host compatibility requirements. AWS Lambda SnapStart uses snapshots of initialized environments to reduce startup latency. That is distinct from resuming a workflow halfway through execution, and does not promise sub-100ms starts for every Lambda function.

The elegance of the approach

What I find most compelling about this design is the abstraction boundary. From the developer’s perspective:

await wait.for({ hours: 24 });

The platform coordinates snapshot creation, storage, resource release, and restoration behind that wait call. The task retains ordinary async control flow, while still needing to handle external connections, side effects, and retries.