Background Tasks and Non-Blocking Workflows in FastAPI

What “background work” means in an API

In a REST API, you often want to acknowledge a request quickly, then perform extra work that does not need to block the client response. Examples include sending an email receipt, writing an audit log entry, or calling a third-party webhook. In FastAPI, BackgroundTasks lets you schedule small pieces of work to run after the response is sent.

The key idea: the client gets a response immediately, and the server continues doing the background action in the same application process.

Typical use cases

• Send an email or SMS notification after creating a resource
• Write analytics/audit logs
• Call a webhook callback after persisting data
• Warm a cache entry or precompute a small derived value

When BackgroundTasks is appropriate (and when it is not)

Good fit

• Short-lived tasks (usually seconds, not minutes)
• Low to moderate volume (a manageable number of tasks per process)
• Best-effort work where occasional failure can be retried later or is acceptable
• No strict delivery guarantees required

Not a good fit (use a queue + worker)

Use a dedicated job queue/worker system (e.g., Celery/RQ/Arq, or a managed queue service) when you need:

• High volume background jobs (thousands/minute)
• Long-running jobs (video processing, large exports, ML inference)
• Retries with backoff, dead-letter queues, or guaranteed delivery
• Horizontal scaling of workers independent from the API
• Isolation so background failures do not impact API process stability

Important operational note: BackgroundTasks runs in the same server process. If the process restarts, pending background work is lost. If you run multiple workers (e.g., multiple Uvicorn/Gunicorn workers), tasks run in whichever worker handled the request.

How BackgroundTasks works in practice

FastAPI injects a BackgroundTasks instance into your endpoint. You register callables with add_task(). After FastAPI finishes sending the HTTP response, it executes the registered tasks.

Continue in our app.

You can listen to the audiobook with the screen off, receive a free certificate for this course, and also have access to 5,000 other free online courses.

Or continue reading below...

Download the app

You can schedule:

• a regular function (def)
• an async function (async def)

Choose based on what the task does. If it performs blocking I/O (like using requests), prefer an async client (like httpx.AsyncClient) or run blocking work in a thread pool (but keep it small and controlled).

Practical example: trigger a background webhook callback

This example returns 202 Accepted immediately while a webhook callback runs in the background. It also demonstrates basic idempotency and error logging.

Step 1: Create a minimal in-memory “job store”

In production you would store this in a database or Redis. Here we use an in-memory dict to keep the example focused on background tasks behavior.

from fastapi import FastAPI, BackgroundTasks, Header, HTTPException, status, Response, Request
from pydantic import BaseModel, AnyUrl
import asyncio
import time
import uuid

app = FastAPI()

# In-memory stores for demo purposes only
JOBS: dict[str, dict] = {}
PROCESSED_IDEMPOTENCY_KEYS: set[str] = set()

class WebhookRequest(BaseModel):
    target_url: AnyUrl
    payload: dict

Step 2: Implement the background function

This task simulates a network call and updates job state. The try/except ensures failures are captured and do not crash the request handler after the response is sent.

async def deliver_webhook(job_id: str, target_url: str, payload: dict) -> None:
    JOBS[job_id]["status"] = "running"
    JOBS[job_id]["started_at"] = time.time()

    try:
        # Simulate I/O latency (replace with an actual async HTTP call)
        await asyncio.sleep(2)

        # Example: pretend the webhook succeeded
        JOBS[job_id]["status"] = "succeeded"
        JOBS[job_id]["finished_at"] = time.time()
        JOBS[job_id]["result"] = {"delivered_to": target_url}

    except Exception as exc:
        # Record failure for later inspection/retry
        JOBS[job_id]["status"] = "failed"
        JOBS[job_id]["finished_at"] = time.time()
        JOBS[job_id]["error"] = str(exc)

Step 3: Create an endpoint that schedules the background task

This endpoint:

• Accepts an optional Idempotency-Key header
• Creates a job record
• Schedules the background delivery
• Returns immediately with a job id

@app.post("/webhooks/deliver", status_code=status.HTTP_202_ACCEPTED)
async def schedule_webhook(
    body: WebhookRequest,
    background_tasks: BackgroundTasks,
    idempotency_key: str | None = Header(default=None, convert_underscores=False),
):
    # Basic idempotency: if the same key is reused, reject or return the existing job.
    # For a real system, store key->job_id in a persistent store with TTL.
    if idempotency_key:
        if idempotency_key in PROCESSED_IDEMPOTENCY_KEYS:
            raise HTTPException(
                status_code=status.HTTP_409_CONFLICT,
                detail="Duplicate Idempotency-Key",
            )
        PROCESSED_IDEMPOTENCY_KEYS.add(idempotency_key)

    job_id = str(uuid.uuid4())
    JOBS[job_id] = {
        "status": "queued",
        "created_at": time.time(),
        "target_url": str(body.target_url),
    }

    background_tasks.add_task(deliver_webhook, job_id, str(body.target_url), body.payload)

    return {"job_id": job_id, "status": "queued"}

Step 4: Add a status endpoint to verify non-blocking behavior

This lets you confirm the response returns immediately and the background task updates state afterward.

@app.get("/webhooks/jobs/{job_id}")
async def get_job(job_id: str):
    job = JOBS.get(job_id)
    if not job:
        raise HTTPException(status_code=404, detail="Job not found")
    return {"job_id": job_id, **job}

Step 5: Verify that the response returns immediately while work continues

Run your API server, then:

• Call POST /webhooks/deliver and measure that it returns fast (well under the simulated 2 seconds).
• Poll GET /webhooks/jobs/{job_id} and observe status transitions: queued → running → succeeded (or failed).

Example requests:

# Schedule
curl -X POST http://127.0.0.1:8000/webhooks/deliver \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: 7c0d2d2a-2f1b-4d3a-9b7f-1f2a4a1b9c01" \
  -d '{"target_url":"https://example.com/webhook","payload":{"order_id":123}}'

# Immediately check status
curl http://127.0.0.1:8000/webhooks/jobs/<job_id>

Design guidelines for safe background work

1) Keep tasks small and non-blocking

• Prefer async libraries for network calls (e.g., async HTTP clients).
• Avoid CPU-heavy work; it can slow down the API process.
• If you must do blocking I/O, consider moving it to a worker system or carefully using a thread pool with limits.

2) Treat background tasks as “best effort” unless you add durability

Because tasks run in-process, they can be lost on restart. If losing a task is unacceptable (e.g., billing, guaranteed notifications), use a durable queue and worker.

3) Error handling: capture, record, and expose

• Wrap task logic in try/except.
• Record failure state somewhere persistent (database/Redis) so you can inspect and retry.
• Log exceptions with enough context (job id, user id, target URL), but avoid logging secrets.

A practical pattern is to store a job record with fields like:

4) Idempotency: assume tasks may run more than once

Even without a queue, duplicates can happen (client retries, timeouts, load balancer retries). Make background work idempotent so repeating it does not cause double side effects.

Common idempotency techniques:

• Idempotency-Key: client sends a unique key; server stores key → result/job id.
• Natural idempotency: use a unique constraint (e.g., “email already sent for order_id”) and do nothing if it already exists.
• Upserts: write results with “insert or update” semantics.

For webhook delivery, idempotency might mean including an event id and ensuring the receiver can deduplicate, while you also deduplicate on your side.

5) Timeouts and retries (decide explicitly)

BackgroundTasks does not provide built-in retry policies. If you need retries:

• Implement limited retries inside the task (with small delays) for transient errors, or
• Persist a failed job and provide a manual/admin retry endpoint, or
• Move to a queue system that supports retries and backoff.

Be careful with retries inside the API process: too many retries can tie up resources and reduce API throughput.

6) Return the right HTTP semantics

• Use 202 Accepted when work is asynchronous and not completed yet.
• Return a job_id and provide a status endpoint if the client needs to track completion.
• If the background action is purely internal and the client does not need tracking, you can return 200/201 and omit job tracking, but still ensure errors are logged.