Fan out batch jobs across thousands of containers
Take any function and run it in parallel in the cloud. Beam containerizes your code, streams results back to your shell, and bills only for the seconds each task runs.
Data pipelines without the pipeline infrastructure
ETL, batch inference, scrape-and-process, cron — the primitives are built in. Beam runs the compute; you write the function.
“What would have taken us literally days with any other provider we've tried took only hours on Beam.”
Parallelize anything
Wrap the unit of work
Decorate the function that processes one item — a document, an audio file, a row batch — with its compute needs.
Fan it out
Call .map() over your inputs for one-off jobs, or deploy a managed task queue that accepts work over HTTP.
Collect results
Stream return values back to your shell, save files with Output, or get a webhook per completed task.
# For always-on pipelines, deploy a managed task queue
from beam import task_queue, Output
@task_queue(
cpu=1.0,
memory="2Gi",
callback_url="https://api.your-app.com/done",
)
def handler(document_url: str):
result = process(document_url)
Output(path=result).save()
# Deploy it
$ beam deploy queue.py:handler
# Enqueue work over HTTP — returns a task ID instantly
$ curl -X POST https://your-queue.app.beam.cloud \
-H 'Authorization: Bearer YOUR_TOKEN' \
-d '{"document_url": "https://example.com/report.pdf"}'Frequently asked questions
How many tasks can run in parallel?
.map() launches a container per item and the fleet scales with your workload — thousands of containers, if that's what the job needs. For deployed task queues, you control concurrency and autoscaling settings per deployment.
Does it work for jobs that aren't AI?
Yes. Web scraping, ETL, report generation, media transcoding — anything you can write in Python. Request a fraction of a CPU per container for light tasks, or attach a GPU per task for batch inference.
How do I monitor a task?
Every task gets an ID and a status you can check from the dashboard or query over the REST API, with logs streamed per task. Attach a callback_url to get a POST the moment each task finishes or fails.
Can I schedule recurring jobs?
Yes. The schedule decorator turns any function into a managed cron job — use standard cron expressions or shorthands like @daily and @hourly.
How do I get results out?
Three ways: return values stream back to the caller, files persist via Output, and callbacks POST to your webhook as each task completes.
Am I billed between tasks?
No. Containers scale to zero when the queue is empty and billing is per-second while tasks run.
Related use cases
Run GPU training jobs on-demand
Start training runs on cloud GPUs from Python — pay only while they run.
InferenceRun Whisper transcription behind your own API
On-demand Whisper transcription behind an autoscaling API.
Batch ProcessingRun OCR and document extraction at scale
Private OCR and extraction pipelines, one container per document.
Start shipping on infra
you won’t outgrow.
Run sandboxes and GPU workloads on your cloud, and scale out to ours when you need to. No infra to manage.