Persistent VMs are now available.Learn more

Fan out batch jobs across thousands of containers

Take any function and run it in parallel in the cloud. Beam containerizes your code, streams results back to your shell, and bills only for the seconds each task runs.

Data pipeline primitives

The primitives for ETL, batch inference, scrape-and-process, and cron are built in. You write the function and Beam runs the compute.

What would have taken us literally days with any other provider we've tried took only hours on Beam.

Ryan Kerr, Senior Engineer, Magellan AI

Read the case study

600K

podcast ads processed in one year

50%

increase in throughput

99.9%

production uptime


Parallelize anything

  1. Wrap the unit of work

    Decorate the function that processes one item (a document, an audio file, a row batch) with its compute needs.

  2. Fan it out

    Call .map() over your inputs for one-off jobs, or deploy a managed task queue that accepts work over HTTP.

  3. Collect results

    Stream return values back to your shell, save files with Output, or get a webhook per completed task.

terminal
# For always-on pipelines, deploy a managed task queue
from beam import task_queue, Output

@task_queue(
    cpu=1.0,
    memory="2Gi",
    callback_url="https://api.your-app.com/done",
)
def handler(document_url: str):
    result = process(document_url)
    Output(path=result).save()

# Deploy it
$ beam deploy queue.py:handler

# Enqueue work over HTTP — returns a task ID instantly
$ curl -X POST https://your-queue.app.beam.cloud \
    -H 'Authorization: Bearer YOUR_TOKEN' \
    -d '{"document_url": "https://example.com/report.pdf"}'

Frequently asked questions

How many tasks can run in parallel?

.map() launches a container per item, and the fleet scales with your workload to thousands of containers if the job needs them. For deployed task queues, you control concurrency and autoscaling settings per deployment.

Does it work for jobs that aren't AI?

Yes. Web scraping, ETL, report generation, media transcoding, or anything else you can write in Python. Request a fraction of a CPU per container for light tasks, or attach a GPU per task for batch inference.

How do I monitor a task?

Every task gets an ID and a status you can check from the dashboard or query over the REST API, with logs streamed per task. Attach a callback_url to get a POST the moment each task finishes or fails.

Can I schedule recurring jobs?

Yes. The schedule decorator turns any function into a managed cron job. Use standard cron expressions or shorthands like @daily and @hourly.

How do I get results out?

Three ways: return values stream back to the caller, files persist via Output, and callbacks POST to your webhook as each task completes.

Am I billed between tasks?

No. Containers scale to zero when the queue is empty and billing is per-second while tasks run.

Pay as you go
billed by the second

Give your agents their own machines

Sign up, install the CLI, and launch a sandbox, VM, or GPU from your terminal or your coding agent.