Running Agents as Containers

[matoushavlena]

In Pre-alpha, agents are built similarly to Heroku buildpacks and run in non-isolated environment. The lack of strict isolation poses some security challenges. To address this, we're currently thinking about moving toward a containerized solution. While docker containers is a clear candidate, we're evaluating if using pure docker containers alone would suffice, or if adopting Kubernetes or potentially offering lighter alternatives like Lima would be more advantageous.

Please share your thoughts, recommendations, experiences.

[JanPokorny]

My opinions:

Security: blindly running external code is the de-facto standard nowadays (npx, uvx, curl http://example.com/install | sh etc.), it being an agent does not change much IMO. But isolation is very useful for user-generated code like in bee-code-interpreter. Docker supports some level of isolation, varying based on rootful/rootles, enabled capabilities, and runtime (podman/docker/gvisor).

Ease of use: Working in a container environment is a bit harder for agent creators (need to write Dockerfile, except maybe we can generate it for common cases?), but potentially simpler for agent users -- the only system dependency is a container runtime. As Matouš mentions, we may even set-up a VM through Lima (which is a static binary) and run a container runtime there, avoiding any setup steps for the user, but at the cost of running another VM if the user already has one through Rancher Desktop / Colima / etc. Or we could support both cases and detect if a viable container runtime is already present on the machine.

Kubernetes or plain containers?: The platform manages lifecycles of agents, potentially scaling them to zero when not needed. This can be done either over the container runtime directly, or we can adopt Kubernetes which already has a lot of related functionality. Adopting Kubernetes could also mean simpler daemon management (OpenTelemetry, Arize Phoenix, etc.), better potential for "scaling up", and better manifest management since Agent would just be a managed CRD, no need to reinvent the wheel with our custom manifest management. For local use, k3s is the most promising since it can run from any container runtime that supports nesting (DinD), or from Lima, and many runtimes already support it out of the box (like Colima and Rancher Desktop) or have an extension available (Podman Desktop). The question is whether Kubernetes isn't too heavy for a middle layer, given the nature of agent management -- maybe operating directly over the container runtime isn't much more complex, and having smaller stack means less potential for integration bugs.

[psschwei]

I think running in containers makes a lot of sense.

I also think that if we are ever planning to run the platform remotely (i.e. not just on the user's laptop), then running on Kubernetes is more or less a non-negotiable.

For running locally, I don't think it's any more difficult to use Docker[1] vs. Kubernetes. There's various options for running a local cluster (kind, minikube, k3s, ...), and a lot of the container runtimes have a Kubernetes option built in (Docker, Podman, Rancher, Colima). So basically, if you can run a container, you can run Kubernetes.

Also, in my experience working with folks who weren't primarily software developers, there isn't a substantial learning curve between either option [2], especially if the right tooling is put in place, which we'll want to do anyways for our own sanity.

Docker does provide some shortcuts that make getting running easier (especially around networking and exposing services), but that can be a double-edged sword: things that worked on your laptop may require substantial rewrites when you try to run them on your distributed environment. So IMO may as well get them "right" the first time.

---

[1] Here referring to any way to run containers locally: Docker, Podman, etc.

[2] Particularly if it's just a matter of running existing Kubernetes resources. Creating Kubernetes resources is a little trickier.

[maceip]

curious why webasembly component model wouldn't work here?

in my view agents should be lightweight and run anywhere without requiring a container runtime

[psschwei]

Wasm could absolutely work for agents, though I don't have any personal experience using it in this context. Any examples you could share?

You would still need some way to host agents, which would probably still entail Kubernetes for at least some use cases.

It's also worth noting that containers have a lot wider adoption than Wasm, for better or for worse.

That said, in an ideal world, as long as your agent implements the ACP spec, you should be good to go, whether you're using wasm or containers or even something else.

Thanks @maceip for the suggestion, definitely some good things to think about in this space!

[JanPokorny]

I love the concept of WASM as a technology, but the harsh truth is that most of the AI world uses Python, which as a scripting language isn't a great fit for WASM. Yeah, there's Pyodide, plus some WASI-based ports, even componentize-py which can directly compile (a subset of) Python. But all of the solutions have specific shortcomings that are hard to explain to a person not so deeply into the tech, like a researcher who just wants to use regular Python. Plus, the goal of the platform is to integrate existing agents, where porting to WASM would be a very high entry requirement.