Bottlenecks in Multi-Agent Systems

While developing the ZeroClaw multi-agent runtime, the biggest performance bottleneck has unequivocally been ‘inter-agent communication’. In our designed architecture, multiple dedicated agents (Workers) communicate with the main Hub to distribute tasks.

The initial design used simple JSON serialization and standard streams. However, as throughput increased, memory copying and serialization overhead began to pose problems. Specifically, latency was detected when relaying token streams of large models like LLMs in real-time, or when transferring large volumes of logs to file system agents.

This post introduces how we significantly improved inter-agent communication performance by leveraging Rust’s powerful features to implement Zero-Copy and Shared Memory.

Problem Diagnosis: Serialization and Copy Overhead

The existing communication method followed this flow:

1. Data Generation: Agent A creates a data struct.
2. Serialization: Converts to JSON using serde_json::to_string, etc. (consumes CPU resources).
3. Transmission: Sends byte streams via IPC channels (sockets, etc.).
4. Reception and Parsing: Agent B receives bytes and parses them using serde_json::from_str (consumes CPU resources).

In this process, data is copied at least three times between memory spaces. The cost of data on the heap being moved to the stack or buffer reallocations, due to Rust’s safety guarantees, was non-negligible.

Solution: Rust-based Zero-Copy IPC Design

We introduced serde’s zero_copy feature and the bytes::Bytes crate into ZeroClaw’s communication layer to eliminate unnecessary copies.

1. Buffer Management using Bytes and Arc

Rust’s bytes::Bytes is based on Arc (Atomic Reference Counting). When transferring data ownership, it copies only the pointer and metadata, not the data itself. This allows multiple agents to safely reference data in the same memory region.

use bytes::{Bytes, BytesMut, BufMut};
use serde::{Serialize, Deserialize};

// Message struct to be transmitted between agents
#[derive(Debug, Deserialize, Serialize)]
pub struct AgentMessage {
    pub id: u64,
    pub payload: Bytes, // Stores raw data
}

impl AgentMessage {
    // Directly wraps Bytes read from the network or file
    pub fn from_bytes(id: u64, data: Bytes) -> Self {
        Self { id, payload: data }
    }
}

2. Shared Memory IPC (IPC Channel) Implementation

Beyond simple byte transmission, for high performance, we can also consider using OS-level shared memory. Rust’s ecosystem provides the shared_memory crate for this. However, here we will apply a method to maintain Zero-Copy over the more general tokio::sync::mpsc channel.

use tokio::sync::mpsc;
use std::sync::Arc;

// Agent A (Sender)
pub async fn producer_task(tx: mpsc::Sender<AgentMessage>) {
    let large_data = vec![0u8; 8192]; // Example: 8KB data
    // Convert to BytesMut, then freeze to create immutable Bytes (wrapped in Arc)
    let shared_bytes = BytesMut::from(&large_data[..]).freeze();
    
    let msg = AgentMessage::from_bytes(1, shared_bytes);
    
    // When sending via tx, only the pointer within msg.payload (Bytes) is copied.
    // The actual 8KB data is not copied (Zero-Copy).
    tx.send(msg).await.unwrap();
}

// Agent B (Receiver)
pub async fn consumer_task(mut rx: mpsc::Receiver<AgentMessage>) {
    while let Some(msg) = rx.recv().await {
        // Here, msg.payload is a reference pointing to the original data.
        println!("Received message ID: {}, Payload Len: {}", msg.id, msg.payload.len());
        
        // Can be directly written to disk or sent over the network without further processing.
        // save_to_disk(msg.payload).await;
    }
}

The core of this code is that even when Bytes transfers ownership, it internally shares the heap data via Arc. This means that when tx.send() is called, the 8KB array is not copied; instead, the Arc’s count is incremented, and only the pointer is passed.

Performance Comparison and Measurement

To compare before and after improvements, we conducted benchmarks using Criterion.

• Environment: Apple M1 Pro, 16GB RAM
• Scenario: Transmitting 1MB of payload 10,000 times

As the data size increases (e.g., for LLM context transfer), the effect of Zero-Copy becomes maximized. The original method caused CPU spikes due to allocation/deallocation, while the improved version showed stable resource usage.

Conclusion: Completing ZeroClaw’s High-Performance Architecture

The Zero-Copy strategy, utilizing Rust’s ownership system with Bytes and Arc, is essential for multi-agent runtimes like ZeroClaw. Beyond simply being ‘fast’, it allows for more efficient use of server resources, enabling the execution of more agents concurrently.

In the future, the ZeroClaw project plans to further abstract this IPC layer and develop a macro that automatically generates optimized communication code simply by using #[derive(AgentMessage)], without the user needing to know the internal Rust implementation.

We hope this experience is helpful for those building high-performance Rust servers, and we recommend applying it to your projects, with actual code examples provided.

Reference Code Repository: ZeroClaw GitHub Repository Related Post: Introducing ZeroClaw - A High-Performance Rust Agent Runtime