Deploying LLMs with Streaming Responses

This is the second feature in our launch week, where we ship a brand new feature every day, five days in a row.

Background

Today we're officially releasing a realtime API to stream websocket responses from your Beam apps.

Technically, it's been possible to stream responses for a while. However previously, users had to write websocket logic into their Beam apps manually using FastAPI.

This wasn't terrible, but it introduced a lot of boilerplate into users' application code.

We wanted to hide this ugly websocket logic from users, so we decided to add a @realtime decorator which automatically creates and manages websocket connections from a Beam app.

Using Streaming Responses

Here's how to deploy a realtime endpoint on Beam. Under the hood, this is a wrapper on top of our ASGI decorator. All the features in ASGI apps -- such as concurrent requests, callback URLs, and custom base images - can be tacked onto your realtime apps.

Connecting to a realtime app from your client is straightforward:

Running Concurrent Requests

You can specify the number of concurrent requests your app can handle using the concurrent_requests parameter in the @realtime decorator.

This allows you to increase the number of requests your app can handle at once, which can help you achieve higher throughput. For instance, if your app is doing I/O-bound work, additional requests can be handled while your I/O operations complete in the background.

Get Started Today

To get started, create a Beam account and follow this guide in our docs. If you have any feedback on the workflow or feature requests, we’d love to hear from you in our Slack Community.

This is Launch 2/5 this week! You can follow along with our upcoming launches on Twitter.