Fan out batch jobs across thousands of containers
Take any function and run it in parallel in the cloud. Beam containerizes your code, streams results back to your shell, and bills only for the seconds each task runs.
Data pipeline primitives
The primitives for ETL, batch inference, scrape-and-process, and cron are built in. You write the function and Beam runs the compute.
Fan-out with .map()
One line turns a for-loop into a parallel run across thousands of containers.
Explore docsManaged task queues
Deploy a queue that accepts tasks over HTTP and works through them on containers that autoscale.
Explore docsScheduled Jobs
Run tasks on a schedule.
Explore docsWebhooks on completion
Attach a callback URL and get a POST when each task finishes, so you don't have to poll.
Explore docsDurable outputs
Persist artifacts from any task with Output and retrieve them by URL when each batch is ready.
Explore docsRetries & timeouts
Configure per-task timeouts and automatic retries instead of writing your own task management script.
Explore docsWhat would have taken us literally days with any other provider we've tried took only hours on Beam.
Ryan Kerr, Senior Engineer, Magellan AI
podcast ads processed in one year
increase in throughput
production uptime
Parallelize anything
Wrap the unit of work
Decorate the function that processes one item (a document, an audio file, a row batch) with its compute needs.
Fan it out
Call .map() over your inputs for one-off jobs, or deploy a managed task queue that accepts work over HTTP.
Collect results
Stream return values back to your shell, save files with Output, or get a webhook per completed task.
# For always-on pipelines, deploy a managed task queue
from beam import task_queue, Output
@task_queue(
cpu=1.0,
memory="2Gi",
callback_url="https://api.your-app.com/done",
)
def handler(document_url: str):
result = process(document_url)
Output(path=result).save()
# Deploy it
$ beam deploy queue.py:handler
# Enqueue work over HTTP — returns a task ID instantly
$ curl -X POST https://your-queue.app.beam.cloud \
-H 'Authorization: Bearer YOUR_TOKEN' \
-d '{"document_url": "https://example.com/report.pdf"}'Frequently asked questions
How many tasks can run in parallel?
.map() launches a container per item, and the fleet scales with your workload to thousands of containers if the job needs them. For deployed task queues, you control concurrency and autoscaling settings per deployment.
Does it work for jobs that aren't AI?
Yes. Web scraping, ETL, report generation, media transcoding, or anything else you can write in Python. Request a fraction of a CPU per container for light tasks, or attach a GPU per task for batch inference.
How do I monitor a task?
Every task gets an ID and a status you can check from the dashboard or query over the REST API, with logs streamed per task. Attach a callback_url to get a POST the moment each task finishes or fails.
Can I schedule recurring jobs?
Yes. The schedule decorator turns any function into a managed cron job. Use standard cron expressions or shorthands like @daily and @hourly.
How do I get results out?
Three ways: return values stream back to the caller, files persist via Output, and callbacks POST to your webhook as each task completes.
Am I billed between tasks?
No. Containers scale to zero when the queue is empty and billing is per-second while tasks run.
Related use cases
Run GPU training jobs on-demand
Start training runs on cloud GPUs from Python and pay only while they run.
InferenceRun Whisper transcription behind your own API
On-demand Whisper transcription behind an autoscaling API.
Batch ProcessingRun OCR and document extraction at scale
Private OCR and extraction pipelines, one container per document.
Give your agents their own machines
Sign up, install the CLI, and launch a sandbox, VM, or GPU from your terminal or your coding agent.