Architecture on Yarang's Tech Lair

Building a Multi-LLM Distributed Orchestrator with NATS JetStream

Fri, 08 May 2026 21:57:11 +0900

Part 1 discussed the model-specific limitations discovered while running four AIs—Claude, ZAI, Codex, and Gemini—concurrently on the same tasks. This part is about “how we made it possible”—the system design and implementation story.

System Overview

AgentForge consists of three components.

[Task Publisher]
 │ NATS JetStream publish
 ▼
[NATS Broker] ─── af.worker.{id}.inbox
 │ JetStream consume (independent streams per worker)
 ▼
[Worker Pollers] × N (poller.py × 18 instances)
 │ LLM CLI Execution (claude / codex / gemini)
 ▼
[Result Return] af.task.{task_id}.completed

When a publisher posts a task to NATS, each worker, which is independently subscribed, receives the message on its inbox and executes the LLM CLI. The result is then published back to a completion topic.

Why NATS JetStream?

We considered several message broker options: Redis Streams, Kafka, RabbitMQ, and NATS JetStream.

Reasons for choosing NATS JetStream:

Single Binary — Operates with a single nats-server without requiring separate runtimes. It has no dependencies like Kafka’s ZooKeeper or RabbitMQ’s Erlang/OTP.
Built-in Persistence — JetStream is a streaming layer on top of NATS, storing messages to the filesystem. This ensures that unprocessed tasks are not lost even if a worker restarts.
NKey-based Authentication — We can issue independent Ed25519 key pairs for each worker. If one worker is compromised, the credentials of other workers remain valid.
Lightweight — Memory usage is around 30MB on a single server. Even with 18 workers connected, the broker load is minimal.

The Core: Backend Adapter in `poller.py`

The heart of the worker is poller.py. This single file handles NATS subscriptions, LLM CLI execution, and result returns.

Since LLMs have different execution methods, we separated them into a backend adapter dictionary.

_BACKENDS: dict[str, dict] = {
 "claude": {
 "bin": os.environ.get("CLAUDE_BIN", "/usr/local/bin/claude"),
 "tools": os.environ.get("ALLOWED_TOOLS", "Read,Edit,Write,Glob,Grep"),
 "model": os.environ.get("CLAUDE_MODEL", ""),
 },
 "codex": {
 "bin": os.environ.get("CODEX_BIN", "/usr/bin/codex"),
 "model": os.environ.get("CODEX_MODEL", ""),
 "sandbox": os.environ.get("CODEX_SANDBOX", "read-only"),
 },
 "gemini_cli": {
 "bin": os.environ.get("GEMINI_BIN", "/usr/bin/gemini"),
 "model": os.environ.get("GEMINI_MODEL", ""),
 },
}

The MODEL_BACKEND environment variable determines which LLM to use. This allows the same poller.py code to run different LLMs across 18 workers.

Claude Backend

async def run_claude(instructions: str, task_id: str) -> tuple[int, str]:
 cfg = _BACKENDS["claude"]
 cmd = [cfg["bin"], "--print", "--allowedTools", cfg["tools"]]
 if cfg.get("model"):
 cmd += ["--model", cfg["model"]]
 proc = await asyncio.create_subprocess_exec(*cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)

The --print flag is key. It runs Claude Code in non-interactive mode instead of conversational mode, ensuring the results are returned via stdout.

ZAI Backend

ZAI offers an Anthropic API-compatible endpoint, so it doesn’t require a separate backend. Routing is handled by two environment variables.

# /etc/agentforge/cc-zai-high-dev-01.env
ANTHROPIC_BASE_URL=<ZAI endpoint>
ANTHROPIC_AUTH_TOKEN=<ZAI API key>

By injecting this file using systemd’s EnvironmentFile= directive, the claude binary sends requests to the ZAI endpoint. This allows us to connect to a different LLM provider simply by changing environment variables, without altering the code.

Declarative Management: `fleet.yaml` × `servers.yaml`

Manually managing 18 workers is impractical. We declaratively defined the entire infrastructure using two YAML files.

`servers.yaml` — Server Inventory

servers:
 - name: worker-node-1
 role: worker-host
 services: [agentforge-worker, tunnel-arm1]

 - name: broker-host
 role: broker-host
 services: [nats-jetstream, postgres]

 - name: worker-node-2
 role: worker-host
 services: [agentforge-worker, tunnel-arm1]

`fleet.yaml` — Worker Placement

workers:
 - worker_id: cc-go-dev-01
 llm: claude-code
 model: claude-sonnet-4-6
 lang: go
 role: developer
 host: worker-node-1
 enabled: true
 create_pr: true

 - worker_id: codex-py-dev-01
 llm: codex
 model: gpt-5.5
 lang: python
 role: developer
 host: worker-node-1
 enabled: true
 create_pr: false

Changing just the host field moves a worker to a different server. Setting enabled: false stops the deployment script from starting that worker.

Worker Templating System: `provision_worker.py`

Manually writing systemd unit files for each new worker is prone to errors. We automated this using Jinja2 templates and a provisioning script.

Template Structure

templates/
 systemd/
 claude.service.j2 # For claude-code and ZAI alike
 codex.service.j2 # OpenAI Codex
 gemini.service.j2 # Google Gemini CLI

The core part of claude.service.j2:

Environment=MODEL_BACKEND=claude
Environment=CLAUDE_BIN={{ claude_bin }}
{% if claude_model %}
Environment=CLAUDE_MODEL={{ claude_model }}
{% endif %}
{% if env_file %}
EnvironmentFile={{ env_file }}
{% endif %}
Environment=WORK_BASE={{ work_base }}
Environment=WORK_DIR={{ work_base }}/repo
Environment="{{ 'ALLOWED_TOOLS=' + allowed_tools }}"
Environment=CREATE_PR={{ 'true' if create_pr else 'false' }}
{% if create_pr and github_remote %}
Environment=GITHUB_REMOTE={{ github_remote }}
{% endif %}

For ZAI workers, the env_file block is activated, adding the EnvironmentFile. For PR creation workers, github_remote is injected. Other settings use defaults.

`provision_worker.py` Usage

# Preview (no actual deployment)
python3 scripts/provision_worker.py --worker new-worker-id --dry-run

# Actual deployment (including NATS creds issuance)
python3 scripts/provision_worker.py --worker new-worker-id --issue-creds

# Bulk deployment for the entire fleet.yaml
python3 scripts/provision_worker.py --all

Internal operations:

Reads worker entries from fleet.yaml.
Reads target hosts from servers.yaml.
Renders Jinja2 templates.
Deploys /etc/systemd/system/{worker_id}-poller.service via SSH.
Creates the working directory.
Executes systemctl daemon-reload && enable --now.
(Optional) Issues NATS NKey with nsc add user → deploys creds → regenerates auth.conf.

Distributed Hosting: Adding Workers to a Second Server

Running all workers on a single server creates a single point of failure. We added Claude workers to a second host.

The method for workers on the second host to connect to the NATS broker is via an autossh tunnel.

[Unit]
Description=NATS Broker Tunnel
After=network-online.target

[Service]
ExecStart=/usr/bin/autossh -N \
 -L 4222:127.0.0.1:4222 \
 -i /home/ubuntu/.ssh/id_ed25519 \
 broker-host
Restart=always
RestartSec=10

With this configuration active, workers always connect to nats://127.0.0.1:4222. They don’t need to know the broker host’s address. As long as the tunnel is alive, it works the same way from any host.

NATS Credential Operations Experience

NATS NKey management was the most complex part of the implementation.

NATS JetStream’s authentication structure is hierarchical.

Operator (Root Signing Authority)
 └── Account: SYS (System Account)
 └── Account: Services (Worker Account)
 ├── User: cc-dev-01
 ├── User: cc-go-dev-01
 ├── User: codex-py-dev-01
 └── ...

Each worker has an independent User NKey and can publish/subscribe within the permissions scope (af.>, _INBOX.>, $JS.>) of the Services account.

Adding a new worker requires the Operator’s signing key. We initially made the mistake of not backing up this key, leading to its loss. Consequently, we had to regenerate the entire Operator and replace all worker credentials en masse. The service downtime was approximately 60 seconds.

# Regeneration procedure
nsc add operator AgentForge
nsc add account SYS
nsc add account Services
for worker in cc-dev-01 cc-go-dev-01 ...; do
 nsc add user --account Services --name $worker \
 --allow-pub "af.>,_INBOX.>,$JS.>" \
 --allow-sub "af.>,_INBOX.>,$JS.>"
done
nsc generate config --mem-resolver --sys-account SYS > auth.new.conf

Adding a New Worker: The Full Procedure

Since the completion of this system, adding a new worker is straightforward.

Step 1: Add an entry to fleet.yaml

- worker_id: my-new-worker
 llm: claude-code
 model: claude-haiku-4-5
 lang: multi
 role: developer
 host: worker-node-1
 enabled: true
 create_pr: false

Step 2: Preview

python3 scripts/provision_worker.py --worker my-new-worker --dry-run

Step 3: Actual Deployment

python3 scripts/provision_worker.py --worker my-new-worker --issue-creds

That’s it. Template rendering, SSH deployment, NATS credential issuance, and service registration are all handled by a single command.

Next Steps

The current system is structured such that workers process tasks independently. Future plans include:

Routing Policies: Automatically selecting the appropriate worker based on task characteristics (e.g., Go code → claude-go-dev, cost-first → ZAI lightweight tier).
Results Comparison Dashboard: A UI to display fan-out results side-by-side.
Cost Tracking: Aggregating API call costs per worker.

The code is publicly available on GitHub.

AgentForge Blog Automation Service: Full Architecture - From AI Comments to Translation and Post Generation

Tue, 05 May 2026 00:30:00 +0900

Running a blog involves three of the most tedious tasks: replying to comments, maintaining English translations, and consistently writing new posts. The AgentForge project automates all three with AI agents.

This post outlines the complete architecture of our blog automation service, which operates across two servers.

System Topology

┌─────────────────────┐ HTTPS ┌─────────────────────┐
│ arm1 server │ ──────────────▶ │ ec1 server │
│ (Agent Operator) │ │ (Blog Hosting) │
├─────────────────────┤ ├─────────────────────┤
│ blog-agent (:8081) │ │ Hugo (nginx) │
│ ├─ CommentHandler │ │ Blog API (:8000) │
│ ├─ TranslateHandler│ │ ├─ translator.py │
│ └─ PostGenerator │ │ ├─ blog_manager.py │
│ │ │ └─ git_handler.py │
│ NATS / PostgreSQL │ │ │
│ Prometheus / Grafana │ │ Git (yarang/blogs) │
└─────────────────────┘ └─────────────────────┘

Server	Role	Core Services
arm1	Agent Operator	`blog-agent.service` — Flask + Scheduler + LLM Client
ec1	Blog Hosting + API	Hugo (nginx) + `blog-api.service` (FastAPI)

Communication between the two servers is restricted to HTTPS API calls only. SSH access from arm1 to ec1 is blocked, so all integrations are done through the Blog API.

arm1: Unified Blog Agent

Why Unified?

Initially, comment response, translation, and post generation operated as separate processes (three systemd services). The issues were:

Using Claude Code CLI (--print) for calls resulted in a response time of 9.7 seconds and consumed 688MB of disk space.
Managing six systemd units was burdensome.
No state sharing between processes was possible.

By unifying these into one process and switching to direct LLM API calls, we achieved the following:

Metric	Before	After
Response Time	9.7s	1.7s
Disk Usage	688MB	~50MB
systemd Units	6	1
Processes	3	1

Architecture

class BlogAgent:
 """1 Process = Flask (webhook) + Scheduler (timer) + LLM Client"""
 
 def __init__(self):
 self.config = AgentConfig.from_credentials()
 self.llm = LLMClient(self.config) # ZAI glm-4.7
 self.api = BlogAPIClient(self.config) # ec1 Blog API
 
 # Handlers
 self.comment = CommentHandler(self.llm, self.config)
 self.translate = TranslateHandler(self.api)
 self.post_gen = PostGenerator(self.llm, self.api)
 
 # Scheduler
 self.scheduler = Scheduler()
 self.scheduler.every(hours=6, task=self.translate.check_and_sync)
 self.scheduler.daily_at(hour=9, task=self.post_gen.generate_and_publish)

Module Operations

1. CommentHandler — AI Comment Response

Receives Webhook events from GitHub Discussions to automatically generate AI comments.

[User Comment] → GitHub Webhook → arm1 Flask → CommentHandler
 → LLM Call (ZAI glm-4.7) → Generate Reply → Post Comment via GitHub API

Trigger: Webhook event-based (real-time)
Filtering: Skips blog owner comments and AI-generated comments.
Security: HMAC-SHA256 Webhook secret verification, Flask-Limiter applied.

2. TranslateHandler — Automatic Translation Trigger

Requests translation synchronization from ec1’s Blog API every 6 hours.

[Scheduler 6h] → TranslateHandler.check_and_sync()
 → POST /translate/sync → ec1 Blog API performs actual translation

arm1 does not perform the translation itself; it only sends a trigger to the ec1 API. The actual translation logic resides in translator.py on ec1.

3. PostGenerator — Automatic Post Generation

Automatically generates technical blog posts every day at 9 AM.

[Scheduler 09:00 KST] → PostGenerator.generate_and_publish()
 → Collect existing topics → Refer to RSS trends → Generate content with LLM
 → Deduplication Check → Publish via Blog API

Deduplication is key. It compares the similarity between new titles and the last 100 existing titles using difflib.SequenceMatcher:

def _is_duplicate_title(self, new_title, existing_titles):
 """Considers it a duplicate if the ratio is >= 0.6"""
 new_lower = new_title.lower().strip()
 for title in existing_titles[-100:]:
 ex_lower = title.lower().strip()
 ratio = difflib.SequenceMatcher(None, new_lower, ex_lower).ratio()
 if ratio >= 0.6:
 return True
 return False

ec1: Blog API Translation System

Transition to Gemini

Initially, translations were performed using ZAI (glm-4.7), but a critical issue arose:

glm-4.7 is a reasoning model, which first consumes its max_tokens budget for reasoning_content (internal thought process). If max_tokens=256, it uses all 256 tokens for reasoning, leaving the actual content as an empty string.

This led to an incident where nine English posts were translated with empty string titles.

Solution: Replaced with Gemini 2.5 Flash Lite.

Item	ZAI (Previous)	Gemini (Current)
Model	glm-4.7 (reasoning)	gemini-2.5-flash-lite
Translation Time	~30s/post	~8s/post
Cost	Paid API	Free (1,500 requests/day)
Empty Response Issue	Occurred	None

OpenAI-Compatible Endpoint

Gemini provides an OpenAI-compatible API. The existing code can be used without any changes by simply switching the base URL:

LLM_BASE_URLS = {
 "GEMINI": "https://generativelanguage.googleapis.com/v1beta/openai",
 "ZAI": "https://api.z.ai/api/coding/paas/v4",
}

Translation Matching Logic

Pairing Korean↔English posts uses date prefix matching:

ko: 2026-05-04-001-개발-생산성-17배-극대화-deepseek-v4와-...
en: 2026-05-04-001-개발-생산성-17배-극대화-deepseek-v4와-...
 ↑ Same prefix = Same post

Although the slugs might differ in language, if the YYYY-MM-DD-NNN part is the same, it’s recognized as the same post. The prerequisite for this method is that no two posts with the same date and number exist.

Title-in-Body Translation Technique

Translating the title via a separate API call caused issues with empty results from the reasoning model. The solution is to include the title as the first line of the body:

# When requesting translation
prompt = f"# {original_title}\n\n{original_body}"

# Extracting the title from the translation result
if translated.lstrip().startswith("# "):
 lines = translated.lstrip().split("\n", 1)
 extracted_title = lines[0].lstrip("# ").strip()
 translated_body = lines[1].lstrip("\n")

This translates the title and body simultaneously in a single API call, preserving context and saving tokens.

LLM Strategy: Role-Based Model Separation

Not all tasks are handled by a single LLM. Models are separated based on the nature of the task.

Task	Server	Model	Reason
AI Comment Response	arm1	ZAI glm-4.7	Conversational, excellent Korean quality
Post Generation	arm1	ZAI glm-4.7	Long-form content generation, creativity required
Translation (ko→en)	ec1	Gemini Flash Lite	Non-reasoning, fast and free

Core Principle: Do not use reasoning models for translation. Reasoning models consume tokens for internal thought processes, making non-reasoning models more suitable for simple conversion tasks.

Monitoring and Operations

Health Check Endpoints

# arm1 agent
curl http://arm1:8081/health
# → {"status":"healthy","agent":"blog-agent","scheduler_jobs":2,"uptime_sec":...}

curl http://arm1:8081/status
# → {"scheduler":[{"name":"auto-translate","last_run":...},{"name":"post-generator","last_run":"2026-05-04"}]}

# ec1 Blog API
curl https://blog.example.com/api/health
# → {"status":"healthy","version":"2.0.0"}

Observability Points

Metric	Normal Range	Alert Condition
arm1 uptime	>0	Service Down
scheduler_jobs	2	≠ 2
Translation Sync	ko post count = en post count	Discrepancy occurs
Post Generation	1 post daily	No posts for over 24 hours

Lessons Learned and Operational Tips

1. The Pitfall of Reasoning Models

It’s often not explicitly stated in documentation that max_tokens combines reasoning and content. If you get an empty response, check the finish_reason—if it’s "length", it indicates insufficient token budget.

2. Value of the OpenAI-Compatible Pattern

When switching translation providers from ZAI to Gemini, the code change was just one line for the base URL. Abstracting to an OpenAI-compatible interface from the start dramatically reduces LLM replacement costs.

3. Constraints of Date Prefix Matching

In the YYYY-MM-DD-NNN pattern, if two or more posts share the same date and number, translation matching will break. The PostGenerator must include logic to check the last number for that date and increment it when generating new posts.

4. Benefits of Process Consolidation

Consolidating three independent services into one resulted in:

State Sharing (LLM clients, configurations, API clients initialized only once)
Simplified Deployment (one systemd unit)
Easier Debugging (logs consolidated in one place)

Future Plans

Review the integration of arm1 agent’s LLM with Gemini.
Comment Quality Evaluation Pipeline (monitoring the appropriateness of auto-generated comments).
Automatic Translation Quality Verification (comparing with back-translation).
Expanding inter-agent collaboration through the AgentForge framework.

Blog automation aims not for “complete automation,” but for “minimal human intervention.” A structure where AI generates content, humans review it, and the system alerts operators to anomalies is the key to stable operation.

Multi-Model AI Agent Team Design: Composed Architecture and 5-Team Hierarchy

Mon, 30 Mar 2026 00:33:34 +0900

Overview

For building a blog system, I designed a multi-model agent team consisting of 14 AI specialists, 5 teams, and 4 LLM models. The core innovations are two:

Composed Agent: Separating role definitions from execution profiles for maximum reusability
Hierarchical Bridge Leadership: Dual membership of tech leads between upper and lower teams to resolve communication bottlenecks

This post covers the final structure, model distribution strategy, and the composed architecture design process.

Background: Why Multi-Model?

Using a single LLM for all tasks creates two problems:

Cost: Running 14 specialists on a Claude Opus-level model makes costs uncontrollable
Fit: Design needs fast reasoning, security analysis needs deep logic, implementation needs stable coding

So we distributed models based on task characteristics.

Final Team Structure

5 teams, 14 specialists, 4 models.

graph TD
 subgraph UPPER["Upper — Steering Team · consensus"]
 ORC["Orchestrator
relay:steering-orchestrator"]
 DES["Architect
gemini:gemini-2.5-flash"]
 SEC["Security Reviewer
codex:gpt-4o"]
 STL["Backend Tech Lead
relay:developer-zai"]
 FTL["Frontend Tech Lead
relay:developer-zai"]
 DTL["Desktop Tech Lead
relay:developer-zai"]
 INF["Infra Network
gemini:gemini-2.5-flash"]
 SAD["Server Admin
relay:developer-zai"]
 end

 subgraph LOWER_BE["Backend Team · leader_decides"]
 BTL["Backend Tech Lead"]
 BDEV["Backend Developer"]
 end

 subgraph LOWER_FE["Frontend Team · leader_decides"]
 FTL2["Frontend Tech Lead"]
 FDEV["Frontend Developer"]
 FUX["UX Designer"]
 end

 subgraph LOWER_DT["Desktop Team · leader_decides"]
 DTL2["Desktop Tech Lead"]
 DDEV["Desktop Developer"]
 DUX["UX Designer"]
 end

 subgraph LOWER_INFRA["Infra Team · leader_decides"]
 SAD2["Server Admin (Leader)"]
 INET["Cloud Network"]
 DBA["DB Architect"]
 end

 UPPER -.->|bridge| LOWER_BE
 UPPER -.->|bridge| LOWER_FE
 UPPER -.->|bridge| LOWER_DT
 UPPER -.->|bridge| LOWER_INFRA

 BTL --> BDEV
 FTL2 --> FDEV
 FTL2 --> FUX
 DTL2 --> DDEV
 DTL2 --> DUX
 SAD2 --> INET
 SAD2 --> DBA

Team Details

Team	Type	Decision-Making	Leader	Members
Steering Team	upper	consensus	Orchestrator	8 (including bridges)
Backend Team	lower	leader_decides	Backend Tech Lead	2
Frontend Team	lower	leader_decides	Frontend Tech Lead	3
Desktop Team	lower	leader_decides	Desktop Tech Lead	3
Infra Team	lower	leader_decides	Server Admin	3

The Infra Team Separation Decision

In the initial design, DB Architect and Server Admin were part of the backend team. But we separated them based on workspace.

graph LR
 subgraph BackendTeam["Backend Team"]
 B["API Code Writing
FastAPI, Python
workspace: VS Code / SSH"]
 end

 subgraph InfraTeam["Infra Team"]
 S["Server Management
Docker, Ubuntu, Nginx
workspace: SSH Terminal"]
 N["Cloud Network
Cloudflare Dashboard
workspace: Web Console"]
 D["DB Management
PostgreSQL, Migrations
workspace: psql / SSH"]
 end

 B -.->|API deployment| S
 B -.->|Query optimization| D

When workspaces differ, separation is more natural than keeping them together.

Model Distribution Strategy

pie title Specialists by Model
 "relay:developer-zai (GLM)" : 10
 "gemini:gemini-2.5-flash" : 2
 "codex:gpt-4o" : 1
 "zai:glm-4" : 1

Model	Specialists	Purpose	Why Chosen
relay:developer-zai	10	Implementation, ops, leads	Cost-effective, stable coding
gemini:gemini-2.5-flash	2	Design, infra network	Fast response, easy external API calls
codex:gpt-4o	1	Security review	High reasoning, OWASP knowledge
zai:glm-4	1	Context compression	Free tier, text summarization specialized

By assigning 10 implementation specialists to GLM, we achieved 60-70% total cost reduction.

Composed Agent Architecture

The core innovation is separating role definitions (Expert) from execution profiles (Definition).

With the traditional approach, role and execution logic are coupled — any change requires a full rewrite, and reuse is impossible.

Composed Approach

graph TD
 DEF["Definition
Backend Developer"]
 DEF --> BASE["Base: backend-core"]
 DEF --> CAP["Capabilities:
rest-api, crud, auth-jwt"]
 DEF --> PLAT["Platform: fastapi"]
 DEF --> POL["Policy: blog-default"]

 BASE --> |"compose"| RUN["Runtime Agent"]
 CAP --> |"compose"| RUN
 PLAT --> |"compose"| RUN
 POL --> |"compose"| RUN

Module Structure

agent-library/
├── definitions/ ← 14 agent definitions
├── modules/
│ ├── base/ ← 6 base modules
│ ├── capabilities/ ← 15 capability modules
│ ├── platforms/ ← 5 platform modules
│ └── policies/ ← 1 policy
└── runs/ ← execution history

Advantages

Reusability: rest-api capability module shared by backend developer and backend tech lead
Platform swap: Change platform: fastapi to platform: django for instant switching
Capability extension: Add a new capability module and connect it to the Definition
Policy unification: All agents follow the same blog-default policy

Expert-Definition Mapping

Expert	Definition	Base	Capabilities	Platform
Backend Developer	backend-developer	backend-core	rest-api, crud, auth-jwt	fastapi
Backend Tech Lead	backend-tech-lead	backend-core	rest-api, crud, code-review	fastapi
Frontend Developer	frontend-developer	frontend-core	markdown-renderer, list-filter-sort	nextjs
Server Admin	server-administrator	server-core	docker-management, nginx-config, postgres-admin	ubuntu
Infra Network	infra-network-admin	infra-core	dns-management, ssl-certificates, rate-limiting	cloudflare
Security Reviewer	security-auditor	specialist-core	security-audit	fastapi
Context Compressor	context-compressor	specialist-core	context-compression	markdown

TLS Certificate Strategy: Cloudflare Origin CA

For production TLS certificates, we chose Cloudflare Origin CA over Let’s Encrypt.

sequenceDiagram
 participant Client as Visitor
 participant CF as Cloudflare (Proxy)
 participant Nginx as Nginx (Origin)
 participant API as FastAPI

 Client->>CF: HTTPS request
 CF->>CF: Terminate with Cloudflare-managed cert
 CF->>Nginx: Encrypt with Origin CA cert
 Nginx->>API: HTTP (local)
 API-->>Nginx: Response
 Nginx-->>CF: Encrypted with Origin CA
 CF-->>Client: Response

Item	Let’s Encrypt	Cloudflare Origin CA
Validity	90 days (renewal required)	15 years (no renewal)
Issuance	ACME automation required	Manual from Dashboard
Complexity	certbot setup	Copy cert files only

Production architecture:

Oracle Cloud ARM (4 OCPU, 24GB)
├── PostgreSQL (installed directly on host)
├── Docker Compose
│ ├── blog-api (FastAPI)
│ ├── blog-frontend (Next.js standalone)
│ ├── MinIO (S3-compatible storage)
│ └── Nginx (Cloudflare Origin CA)
└── Cloudflare Proxy (Full Strict SSL)

Relay Plugin: Agent Invocation Mechanism

The team structure runs in Claude Code through the Relay plugin.

sequenceDiagram
 participant User as User
 participant Claude as Claude Code
 participant Plugin as Relay Plugin
 participant MCP as MCP Server
 participant LLM as External LLM

 User->>Claude: /relay:invoke-agent
 Claude->>Plugin: Load definition by expert slug
 Plugin->>Plugin: Compose Definition (base + capabilities + platform + policy)
 Plugin->>Plugin: Check backed_by

 alt relay:developer-zai
 Plugin->>Claude: Run internal agent
 else gemini:*
 Plugin->>MCP: Call gemini_mcp server
 MCP->>LLM: Gemini API
 LLM-->>MCP: Response
 MCP-->>Plugin: Result
 else codex:*
 Plugin->>MCP: Call codex_mcp server
 MCP->>LLM: OpenAI API
 LLM-->>MCP: Response
 MCP-->>Plugin: Result
 end

 Plugin-->>Claude: Final result
 Claude-->>User: Response

backed_by Namespaces

Namespace	MCP Server	Purpose
`relay:developer-zai`	internal agent	Implementation, ops (low cost)
`relay:steering-orchestrator`	internal agent	Coordination, final decisions
`gemini:gemini-2.5-flash`	gemini_mcp	Design, external APIs
`codex:gpt-4o`	codex_mcp	Security analysis
`zai:glm-4`	zai_mcp	Context compression

Design Decision Log

Decision	Alternative	Rationale
Separate infra team	Include in backend	Different workspaces (SSH vs IDE)
Cloudflare Origin CA	Let’s Encrypt	15-year validity, no renewal
PostgreSQL on host	Docker container	Memory efficiency on single server
Composed agent	Single-definition agent	Module reusability, easy platform swap
Assign many GLM	Assign many Claude	60-70% cost reduction

Retrospective: What I Learned

1. “Executable Structure” over “Perfect Structure”

Trying to design everything perfectly can prevent you from ever starting. It’s better to compromise and improve as you execute.

2. Workspace Defines Team Boundaries

People who write code and people who manage servers have different physical work environments — that’s a natural team boundary.

3. The Value of Composed Architecture

In an environment with 14 specialists, 5 teams, and 4 models interacting, module separation is essential. It minimizes the scope of changes and maximizes reusability.

4. Cost is Determined at Design Time

Asking “does this task really need a high-cost model?” every time naturally optimizes costs.

Next Steps

Start Phase 1 implementation: DB, Auth, Post/Category CRUD, Docker
Share team operation experience: Problems encountered during actual execution
Performance monitoring: Response time per model, cost vs quality analysis

This post summarizes my experience building an AI agent team using Claude Code + Relay plugin. It reflects learnings from applying this to a real project — different approaches may be more suitable depending on your situation.

[Discord MCP] Gateway Architecture Design

Sun, 01 Mar 2026 01:13:00 +0900

The Claude Code team designed a Discord Gateway Service for user communication via Discord. This article summarizes the key architecture decisions and user_comm Agent design.

1. Overall Architecture

Components

Layer	Component	Role
Discord	Bot, Channel, Thread	User interface
Gateway	WebSocket, REST API, SSE	Message routing
MCP	gcp-mcp, oci-mcp, db-mcp	Tool execution
user_comm	Discord Agent	User communication

Message Flow

flowchart LR
 subgraph D[Discord]
 U[User]
 end
 subgraph G[Gateway :8081]
 WS[WebSocket]
 API[REST API]
 SSE[SSE]
 end
 subgraph A[Claude Code Team]
 UC[user_comm]
 GCP[gcp-mcp]
 OCI[oci-mcp]
 DB[db-mcp]
 end

 U --> WS
 WS --> UC
 UC --> API
 API --> GCP
 API --> OCI
 API --> DB
 GCP --> UC
 OCI --> UC
 DB --> UC
 UC --> SSE
 SSE --> U

2. user_comm Agent (User Communication Handler)

Role and Responsibilities

The user_comm Agent is a member of the Claude Code team, communicating with users through Discord channels and collaborating with other agents.

Function	Description
Input Reception	Receive Discord messages and route to appropriate agent
Opinion Request	Query user opinions at other agents’ requests
Notification/Report	Send system status, warnings, reports
Team Communication	Exchange messages with other agents

Internal Structure

flowchart TB
 subgraph UC[user_comm Agent]
 DBot[Discord Bot]
 Agent[UserCommAgent]
 TComm[TeamCommunicator]
 DBot <--> Agent
 Agent <--> TComm
 end
 subgraph DAPI[Discord API]
 M[Messages]
 B[Buttons]
 T[Threads]
 end
 subgraph O[Other Agents]
 GCP[gcp-mcp]
 OCI[oci-mcp]
 AL[alert-mcp]
 end

 DBot <--> M
 DBot <--> B
 DBot <--> T
 TComm --> GCP
 TComm --> OCI
 TComm --> AL

Message Types

class MessageType(Enum):
 TASK_REQUEST = "task_request" # Task request
 NOTIFICATION = "notification" # Notification
 OPINION_REQUEST = "opinion_request" # Opinion request
 OPINION_RESPONSE = "opinion_response" # Opinion response
 STATUS_REPORT = "status_report" # Status report
 ERROR = "error" # Error

Key Function Flows

Input Reception (Discord -> Agent)

User: "@gcp-monitor check server status"
 -> DiscordBot.on_message
 -> UserCommAgent._parse_command (target: gcp-monitor)
 -> TeamCommunicator.send_to_agent
 -> Discord: "Request forwarded to gcp-monitor."

Opinion Request (Agent -> Discord)

gcp-monitor: "Clean up disk?" -> user_comm
 -> DiscordBot.ask_opinion (wait for button or reply)
 -> User response
 -> TeamCommunicator.send_to_agent (forward response)

Notification/Report (Agent -> Discord)

oci-monitor: "Disk 92% warning" -> user_comm
 -> DiscordBot.send_message (Embed format)

3. Lightweight Architecture Without Redis

Why Remove Redis?

Item	With Redis	With In-Memory
Thread Lock	Redis SET NX	Python dict
Event Distribution	Redis Streams	Direct SSE
State Storage	Redis Cache	Memory cache

Conclusion: In-memory is sufficient for single-instance environments

Gateway Structure

flowchart TB
 subgraph GW[Gateway Service]
 Bot[Discord Bot]
 subgraph C[Core Components]
 Lock[Thread Lock]
 Cache[Message Cache]
 SSEMgr[SSE Manager]
 end
 end
 DAPI[Discord API] <--> Bot
 Bot --> Lock
 Bot --> Cache
 Bot --> SSEMgr
 SSEMgr --> Clients[WebSocket Clients]

Component Details

Module	Role	Features
Discord Bot	WebSocket connection	Auto-reconnect
Thread Lock	Concurrency control	5min timeout
Message Cache	Message storage	Max 1000 items
SSE Manager	Real-time delivery	Broadcast to all MCPs

4. MCP Selection: 4-Stage Hybrid

Selection Priority

Rank	Method	Example	Description
1	Slash Command	`/gcp status`	Most explicit
2	@Mention	`@gcp-monitor status`	Natural conversation
3	Keyword Detection	`gcp server status`	Auto keyword recognition
4	Channel Assignment	#gcp-monitoring	Channel default MCP

Fallback Order

flowchart TD
 A[Message Received] --> B{Slash Command?}
 B -->|Yes| C[Route to MCP]
 B -->|No| D{Mention?}
 D -->|Yes| C
 D -->|No| E{Keyword?}
 E -->|Yes| C
 E -->|No| F{Channel Default?}
 F -->|Yes| C
 F -->|No| G[Broadcast]

Slash Command List

Command	MCP	Description
`/gcp status [server]`	gcp-mcp	GCP server status
`/gcp list`	gcp-mcp	GCP instance list
`/oci status [server]`	oci-mcp	OCI server status
`/oci list`	oci-mcp	OCI instance list
`/db query <sql>`	db-mcp	Execute DB query
`/db list`	db-mcp	DB list
`/alert check`	alert-mcp	Check alerts

5. Thread Lock Rules

Lock Behavior

sequenceDiagram
 participant M1 as gcp-mcp
 participant GW as Gateway
 participant M2 as oci-mcp
 participant DT as Discord Thread

 Note over M1,DT: Lock Acquisition
 M1->>+GW: Lock request
 GW-->>-M1: Lock acquired
 M1->>DT: Send response

 Note over M2,GW: Lock Conflict
 M2->>GW: Lock request
 GW-->>M2: Lock failed

 Note over M1,GW: 5min timeout
 GW->>GW: Auto-release lock

 Note over M2,GW: Lock Retry
 M2->>+GW: Lock request
 GW-->>-M2: Lock acquired

Lock API

Method	Endpoint	Description
`POST`	`/api/threads/{id}/acquire`	Acquire lock
`POST`	`/api/threads/{id}/release`	Release lock
`GET`	`/api/threads/{id}/lock`	Check lock status

Request/Response Example

Lock Acquire Request

POST /api/threads/123456/acquire
{
 "agent_name": "gcp-mcp",
 "timeout": 300
}

Lock Acquire Response

{
 "acquired": true,
 "thread_id": "123456",
 "agent": "gcp-mcp"
}

6. MCP Tools (8)

Tool List

Tool	Description	Key Parameters
`discord_send_message`	Send message	channel_id, content
`discord_get_messages`	Get messages	channel_id, limit
`discord_wait_for_message`	Wait for message	channel_id, timeout
`discord_create_thread`	Create thread	channel_id, message_id
`discord_list_threads`	List threads	channel_id
`discord_archive_thread`	Archive thread	thread_id
`discord_acquire_thread`	Acquire lock	thread_id, agent_name
`discord_release_thread`	Release lock	thread_id, agent_name

Tool Relationships

flowchart LR
 subgraph Lock[Lock Tools]
 Acquire[acquire_thread]
 Release[release_thread]
 end
 subgraph Thread[Thread Tools]
 Create[create_thread]
 List[list_threads]
 Archive[archive_thread]
 end
 subgraph Msg[Message Tools]
 Send[send_message]
 Get[get_messages]
 Wait[wait_for_message]
 end

 Acquire --> Create
 Create --> Send
 Send --> Release

Usage Examples

# Send message
discord_send_message(
 channel_id="123456789",
 content="Server status: OK"
)

# Create thread
discord_create_thread(
 channel_id="123456789",
 message_id="987654321",
 name="Status Check"
)

# Acquire lock
discord_acquire_thread(
 thread_id="111222333",
 agent_name="gcp-mcp",
 timeout=300
)

7. File Structure

flowchart TB
 subgraph P[server_monitor/]
 subgraph Gw[gateway/]
 M1[main.py]
 M2[discord_ws.py]
 M3[thread_lock.py]
 M4[sse.py]
 end
 subgraph Uc[user_comm/]
 A1[agent.py]
 A2[discord_bot.py]
 A3[team_comm.py]
 end
 subgraph Mcp[discord_mcp/]
 S1[server.py]
 end
 subgraph Sh[mcp_shared/]
 MO[monitor/]
 end
 subgraph Doc[docs/]
 D1[MCP_SELECTION_STRATEGY.md]
 D2[MCP_ROUTING_POLICY.md]
 end
 end

Directory Description

Path	Description
`gateway/`	Gateway Service (FastAPI)
`user_comm/`	User Communication Agent
`discord_mcp/`	MCP Server (8 tools)
`mcp_shared/`	Shared MCP tools
`docs/`	Documentation

8. How to Run

Local Execution

# Start Gateway Service
uvicorn gateway.main:app --host 0.0.0.0 --port 8081

# Start user_comm Agent (standalone mode)
python main.py --standalone

# Start user_comm Agent (team mode)
python main.py --team server-monitor

# Health check
curl http://localhost:8081/health

# Response
{"status": "healthy", "discord_connected": true}

Claude Code MCP Configuration

// ~/.claude/settings.json
{
 "mcpServers": {
 "discord-gateway": {
 "command": "python3",
 "args": ["/path/to/discord_mcp/server.py"],
 "env": {
 "GATEWAY_URL": "http://localhost:8081"
 }
 }
 }
}

9. Roadmap

Phase 1: Complete

FastAPI Gateway Service
Discord WebSocket connection
Thread Lock (In-Memory)
SSE broadcast
MCP Server (8 tools)

Phase 2: In Progress

user_comm Agent implementation
Slash command implementation
Channel default MCP settings
Keyword auto-detection
Routing configuration file

Phase 3: Optional

API authentication (API Key)
Rate Limiting
Message persistence (SQLite)

Conclusion

We chose a strategy of starting with a lightweight architecture and expanding when needed.

Item	Current	Future
State Storage	In-Memory	SQLite (if needed)
Distributed Lock	Not used	Redis (multi-instance)
Authentication	None	API Key (if needed)
User Communication	Gateway direct	user_comm Agent

The current structure is sufficient for single instances, and we plan to expand incrementally as traffic grows. The user_comm Agent systematizes user communication and improves collaboration with other agents on the team.

Korean Version: 한국어 버전

[Claude Code] Team Agent Communication Architecture

Sat, 28 Feb 2026 14:07:38 +0900

Claude Code Team Agent Communication Architecture Design

Overview

When multiple AI agents collaborate in Claude Code, an efficient communication method is needed. This article shares the communication architecture designed while building a Discord-integrated server monitoring team.

Problem Definition

Claude Code has two types of agents:

Python Processes (Daemon)
- Discord Gateway, user_comm_daemon, etc.
- Always running, filesystem access available
LLM Agents (Created as Tasks)
- gcp-monitor, oci-monitor, alert-manager, etc.
- Use SendMessage tool, Claude Code native communication

Communication between these two types was needed.

Considered Approaches

1. File-based Message Queue (Existing)

Pros: Simple implementation, no dependencies
Cons: Latency (~1s), file management required

2. WebSocket

Pros: Real-time bidirectional communication
Cons: Server/client implementation required

3. gRPC

Pros: Strong type checking, streaming
Cons: proto file management, high complexity

4. SQLite Message Hub (Adopted)

Pros: Fast (~10ms), transactions, no additional dependencies
Cons: Works only on a single machine

Final Architecture

┌─────────────────────────────────────────────────────────────────┐
│ SQLite Message Hub (messages.db) │
│ ~/.claude/teams/{team}/messages.db │
└───────────────────────────┬─────────────────────────────────────┘
 │
 ┌───────────────────┼───────────────────┐
 ↓ ↓ ↓
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Discord │ │ Python │ │ LLM │
│ Gateway │ │ Daemon │ │ Agent │
│ (Direct Access)│ │ (Direct Access)│ │ (MCP Tool) │
└──────────────┘ └──────────────┘ └──────────────┘

SQLite Message Hub Structure

Messages Table

CREATE TABLE messages (
 id INTEGER PRIMARY KEY AUTOINCREMENT,
 message_id TEXT UNIQUE NOT NULL,
 sender TEXT NOT NULL,
 recipient TEXT, -- NULL = broadcast
 message_type TEXT DEFAULT 'message',
 content TEXT NOT NULL,
 priority TEXT DEFAULT 'normal',
 metadata TEXT DEFAULT '{}',
 status TEXT DEFAULT 'pending',
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
 processed_at TIMESTAMP,
 expires_at TIMESTAMP
);

Agent Status Table

CREATE TABLE agent_status (
 agent_name TEXT PRIMARY KEY,
 status TEXT DEFAULT 'offline',
 last_heartbeat TIMESTAMP,
 metadata TEXT DEFAULT '{}'
);

Communication Methods

LLM Agent ↔ LLM Agent

Use SendMessage (Claude Code native)
Auto-managed, idle notification provided

Python ↔ LLM Agent

SQLite Hub + MCP Tool
Python accesses SQLite directly
LLM Agent accesses via MCP Tool

MCP Server Implementation

MCP tools for LLM Agents:

@server.list_tools()
async def list_tools():
 return [
 Tool(name="team_send_message", ...),
 Tool(name="team_read_messages", ...),
 Tool(name="team_broadcast", ...),
 Tool(name="team_get_status", ...),
 Tool(name="team_update_status", ...),
 ]

Message Flow Example

1. Discord user: "Hello"
 ↓
2. Discord Gateway → SQLite: INSERT message
 ↓
3. user_comm Daemon → SQLite: SELECT pending
 ↓
4. user_comm → SQLite: INSERT response
 ↓
5. Discord Gateway → SQLite: SELECT response
 ↓
6. Discord channel: "Hi there! 👋"

Performance Comparison

Method	Latency	Stability	Complexity
File-based	~1000ms	Medium	Low
WebSocket	~5ms	High	Medium
SQLite	~10ms	High	Low

Conclusion

The SQLite Message Hub + MCP Server combination was most suitable for Claude Code team agent communication:

Low Latency: 100x faster than file-based
Stability: Data integrity guaranteed through transactions
Simplicity: Uses Python’s built-in sqlite3 without additional dependencies
Flexibility: Both Python and LLM Agents use the same data source

Code

Core components:

message_hub.py: SQLite Message Hub
team_comm_mcp_server.py: MCP Server
discord_gateway.py: Discord integration gateway
user_comm_daemon.py: Message processing daemon

Korean Version: 한국어 버전

[ZeroClaw] Multi-Agent Architecture Design

Fri, 27 Feb 2026 22:58:08 +0900

ZeroClaw Multi-Agent Architecture Design

Overview

ZeroClaw’s multi-agent architecture implements an advanced collaboration system evolving from a single-agent model.

Current Status

Phase 1: In-Process Delegation (Complete)
Phase 2: File-Based Multi-Agent Architecture (In Development)

1. Design Philosophy

Principle	Application
KISS	Simple communication protocol
YAGNI	Implement only essential features
SRP	Single responsibility agents
Secure	Least privilege principle

2. Architecture Structure

Application Layer: Research │ Code │ Test Agent
Message Bus Layer: NATS/Redis Pub/Sub
Transport Layer: gRPC │ WebSocket │ Unix Socket

3. Agent Definition

agent:
 id: "researcher"
 name: "Research Agent"

execution:
 mode: subprocess

provider:
 name: "openrouter"
 model: "claude-sonnet-4-6"

4. CLI Commands

zeroclaw agent list
zeroclaw agent show <id>
zeroclaw agent run --agent-id researcher

5. Security

Mode	Isolation	Use Case
Subprocess	Process	Trusted agents
Docker	Container	File operations
Wasm	Memory	High security

Korean Version: 한국어 버전

[Multi-Agent] File-Based Architecture Design

Fri, 27 Feb 2026 09:30:57 +0900

File-Based Multi-Agent Architecture Design

Status: Design Proposal
Author: multi-agent-architect
Date: 2026-02-24
Type: Alternative Architecture (Process-Per-Agent)

Executive Summary

This document defines a file-based, process-per-agent architecture where each agent is defined as a separate configuration file and runs as an independent process. Agents are invoked on-demand and report results back to a main coordinator agent.

Core Principles

File-Based Agent Definition: Each agent is a .toml file in agents/
Process Isolation: Each agent runs in its own process
On-Demand Invocation: Main agent spawns subprocess agents as needed
Reporting Protocol: Structured result reporting via IPC
Zero-Config Discovery: Auto-discovery of agent definitions

1. Agent Definition File Structure

1.1 Directory Layout

~/.zeroclaw/
├── config.toml # Main configuration
├── agents/ # Agent definitions directory
│ ├── researcher.toml # Research agent
│ ├── coder.toml # Code generation agent
│ ├── tester.toml # Testing agent
│ ├── reviewer.toml # Code review agent
│ └── summarizer.toml # Summarization agent
├── agents.d/ # Optional: additional agent dirs
│ └── custom/
│ └── my_agent.toml
└── workspace/ # Shared workspace

1.2 Agent File Schema

# agents/researcher.toml

# Agent metadata
[agent]
id = "researcher"
name = "Research Agent"
version = "1.0.0"
description = "Conducts research on given topics using web search and knowledge bases"

# Execution configuration
[agent.execution]
# How to run this agent: "subprocess" | "wasm" | "docker"
mode = "subprocess"

# Command to spawn (template variables: {agent_id}, {workspace}, {config_dir})
command = "zeroclaw"
args = [
 "agent",
 "run",
 "--agent-id", "{agent_id}",
 "--config", "{config}/agents/researcher.toml",
 "--workspace", "{workspace}"
]

# Working directory for the subprocess
working_dir = "{workspace}"

# Environment variables for the subprocess
[agent.execution.env]
ZEROCLAW_AGENT_MODE = "worker"
ZEROCLAW_AGENT_ID = "researcher"

# Provider configuration (overrides main config)
[provider]
name = "openrouter"
model = "anthropic/claude-sonnet-4-6"
api_key = null # Inherit from main, or set agent-specific key
temperature = 0.3
max_tokens = 4096

# Tools available to this agent
[[tools]]
name = "web_search"
enabled = true

[[tools]]
name = "web_fetch"
enabled = true

# Tools explicitly denied to this agent
[[tools.deny]]
name = "shell"
reason = "Research agent should not execute shell commands"

# System prompt
[system]
prompt = """
You are a Research Agent. Your role is to:
1. Search for and gather information from credible sources
2. Synthesize findings into structured reports
3. Cite sources and provide references
4. Avoid speculation - stick to verified information
"""

# Memory configuration
[memory]
backend = "shared" # "shared" | "isolated"
category = "research"

# Reporting configuration
[reporting]
mode = "ipc"
format = "json" # "json" | "markdown" | "both"
timeout_seconds = 300

# Retry configuration
[retry]
max_attempts = 3
backoff_ms = 1000

2. Agent Registry

The AgentRegistry discovers and manages agent definitions:

pub struct AgentRegistry {
 agents_dir: PathBuf,
 agents: HashMap<String, AgentDefinition>,
 security: Arc<SecurityPolicy>,
}

impl AgentRegistry {
 pub fn new(agents_dir: PathBuf, security: Arc<SecurityPolicy>) -> Result<Self>;
 pub fn discover(&mut self) -> Result<()>;
 pub fn get(&self, id: &str) -> Option<&AgentDefinition>;
 pub fn list(&self) -> Vec<String>;
}

3. Inter-Process Communication (IPC)

Task Message (Main → Agent)

pub struct AgentTask {
 pub task_id: String,
 pub from_agent: String,
 pub to_agent: String,
 pub prompt: String,
 pub context: HashMap<String, serde_json::Value>,
 pub input: Option<serde_json::Value>,
 pub timestamp: chrono::DateTime<chrono::Utc>,
 pub deadline: Option<chrono::DateTime<chrono::Utc>>,
}

Result Message (Agent → Main)

pub struct AgentResult {
 pub task_id: String,
 pub agent_id: String,
 pub status: AgentStatus,
 pub data: Option<serde_json::Value>,
 pub output: String,
 pub error: Option<String>,
 pub metrics: AgentMetrics,
 pub artifacts: Vec<Artifact>,
 pub timestamp: chrono::DateTime<chrono::Utc>,
}

4. Security Considerations

Execution Mode	Isolation	Use Case
Subprocess	Process-level	Trusted agents
Docker	Container	Untrusted, file operations
Wasm	Memory-only	High security needs

5. CLI Commands

# List available agents
zeroclaw agent list

# Show agent details
zeroclaw agent show <agent_id>

# Run an agent directly (for testing)
zeroclaw agent run --agent-id researcher --prompt "Research X"

# Create a new agent from template
zeroclaw agent create --id my-agent --name "My Agent"

# Validate agent definition
zeroclaw agent validate <agent_id>

6. Success Criteria

Agent definition files validated on load
Subprocess agents spawn and complete tasks
IPC protocol supports tasks >1GB data
Main agent can invoke worker agents via tool
Failed agents don’t crash main agent
Security policy enforced per-agent
CLI commands for agent management

Korean Version: 한국어 버전

[ZeroClaw] Multi-Agent Protocol Design

Thu, 26 Feb 2026 23:37:59 +0900

ZeroClaw Multi-Agent Communication Protocol Design

Written on February 26, 2026

Overview

This is a communication protocol design for ZeroClaw’s multi-agent system.

1. Design Goals

Standardized Message Format - Consistent envelope-based messaging
Flexible Routing - Direct, broadcast, Pub/Sub patterns
Sync + Async - Support for blocking and non-blocking communication
Event-driven Coordination - React to system events
State Sharing - Safe shared memory and distributed state

2. Message Types (12)

Task, TaskResult, Query, QueryResponse, Stream, Event, Heartbeat, Ack, Error, Control, Sync, SyncResponse

3. Communication Patterns

Direct (1:1)
Broadcast (1:N)
Request-Response
Pub/Sub
Streaming

4. Implementation Roadmap

Phase	Content
1	Core messaging layer
2	State store
3	DelegateTool integration
4	Pub/Sub and streaming
5	Distributed environment support

This document was written through collaboration of the ZeroClaw development team.

Korean Version: 한국어 버전

[ZeroClaw] Codebase Architecture Analysis

Thu, 26 Feb 2026 10:07:29 +0900

ZeroClaw Codebase Architecture Analysis Report

Written on February 26, 2026

Overview

ZeroClaw is a high-performance autonomous agent runtime written in Rust. This report summarizes the analysis of the codebase structure and security architecture.

1. Core Architecture Patterns

ZeroClaw adopts trait-driven extensibility as its core design principle.

1.1 Major Traits (7)

Trait	Location	Role
`Provider`	src/providers/traits.rs	AI model provider interface
`Channel`	src/channels/traits.rs	Communication channel interface
`Tool`	src/tools/traits.rs	Tool execution interface
`Memory`	src/memory/traits.rs	Memory backend interface
`Observer`	src/observability/traits.rs	Observability interface
`RuntimeAdapter`	src/runtime/traits.rs	Runtime adapter interface
`Peripheral`	src/peripherals/traits.rs	Hardware peripheral interface

2. Security Architecture

ZeroClaw implements security through a Defense-in-Depth strategy.

2.1 Security Layers (6)

SecurityPolicy Core - Autonomy level management
Gateway Security - Bearer token authentication
Tool Validation - Injection prevention
Runtime Sandbox - Landlock/Firejail/Docker
Secret Management - ChaCha20-Poly1305
Audit Logging - Event tracking

2.2 Core Security Boundaries

Command allowlist: 15 items
Rate limiting: 20 requests/hour
Environment variables: only 8 allowed

3. Multi-Agent System

Implementation Roadmap

Phase	Duration	Content
1	1 week	Core traits, DelegateTool extension
2	2-3 weeks	Docker/Wasm execution modes
3	4+ weeks	Distributed message bus, consensus algorithm

Conclusion

ZeroClaw has an excellent architecture in terms of extensibility, security, performance, and maintainability.

This report was written based on collaborative analysis by the ZeroClaw development team.

Korean Version: 한국어 버전

[Multi-Agent] Communication Platform Design

Sat, 21 Feb 2026 21:00:00 +0900

Introduction

As AI agent systems evolve, the transition from single-agent to multi-agent systems is accelerating. This article explores how to design a platform architecture for efficient communication between multiple agents.

1. Architecture Overview

A multi-agent platform consists primarily of an Orchestrator, Message Bus, and various specialized agents.

Multi-Agent Architecture

Core Components

Component	Role
Orchestrator	Task coordination, agent scheduling
Message Bus	Inter-agent communication relay
State Store	Shared state management
Task Queue	Priority-based work queue
Agents	Specialized task execution

2. Communication Protocol Design

Inter-agent communication is designed based on asynchronous message passing.

Message Protocol

Communication Flow

Publish: Agent publishes task request to message bus
Route: Bus routes message to appropriate agent
Accept: Target agent accepts task
Acknowledge: Bus sends acceptance confirmation to requester
Execute: Agent performs actual work
Result: Result is published to bus
Deliver: Bus delivers result to requester

3. Message Structure

Design scalable and type-safe message structures.

Message Structure

Message Envelope

{
 "header": {
 "message_id": "uuid-v4",
 "timestamp": "2026-02-21T12:00:00Z",
 "source_agent": "agent-research-01",
 "target_agent": "agent-writer-01",
 "message_type": "TASK_REQUEST"
 },
 "metadata": {
 "priority": "normal",
 "ttl": 3600,
 "correlation_id": "parent-uuid"
 },
 "payload": {
 "content_type": "json",
 "encoding": "utf-8",
 "data": {
 "task": "summarize",
 "content": "..."
 }
 }
}

Message Types

Type	Purpose
`TASK_REQUEST`	Task request
`TASK_RESULT`	Task result
`STATUS_UPDATE`	Status change notification
`HEARTBEAT`	Liveness check
`QUERY`	Information lookup
`RESPONSE`	Query response

4. Agent State Management

Each agent has a clear state machine.

State Management

State Transitions

IDLE → RECEIVING → PROCESSING → EXECUTING → COMPLETED → IDLE
 ↘ ↘ ↘
 ERROR ──────→ retry ──→ IDLE

State Behaviors

State	Behavior
IDLE	Waiting for new task
RECEIVING	Message reception and validation
PROCESSING	Task analysis and preparation
EXECUTING	Actual work execution
COMPLETED	Result transmission and cleanup
ERROR	Error handling and retry

5. Communication Patterns

5.1 Request-Response

Synchronous 1:1 communication pattern.

# Request
response = await bus.request(
 target="agent-code-01",
 message=TaskRequest(task="generate_code", spec={...}),
 timeout=30
)

# Response
return TaskResult(status="success", data=generated_code)

Asynchronous 1:N communication pattern.

# Subscribe
@bus.subscribe("research.completed")
async def handle_research_result(message):
 # Handle research result
 pass

# Publish
await bus.publish(
 topic="research.completed",
 message=ResearchResult(data={...})
)

5.3 Broadcast

Used to send notifications to all agents.

# Broadcast
await bus.broadcast(
 message=SystemAlert(
 level="warning",
 message="High load detected"
 )
)

6. Core Design Principles

6.1 Loose Coupling

Agents should not know each other’s internal implementations. They communicate only through message interfaces.

6.2 Fault Tolerance

class Agent:
 async def execute_with_retry(self, task, max_retries=3):
 for attempt in range(max_retries):
 try:
 return await self.execute(task)
 except Exception as e:
 if attempt == max_retries - 1:
 await self.transition_to(State.ERROR)
 raise
 await asyncio.sleep(2 ** attempt) # Exponential backoff

6.3 Scalability

Horizontal scaling: Add agents with the same role
Load balancing: Automatic distribution by message bus
Sharding: Separate agent groups by task type

6.4 Observability

# Metrics collection
@metrics.track
async def process_message(self, message):
 with metrics.timer("process_duration"):
 result = await self.handle(message)
 metrics.increment("messages_processed")
 return result

7. Implementation Considerations

7.1 Message Serialization

Format	Pros	Cons
JSON	Readability, compatibility	Size, performance
Protocol Buffers	Performance, schema	Complexity
MessagePack	JSON compatible, size	Tool support

7.2 Backpressure

class Agent:
 def __init__(self, max_concurrent=10):
 self.semaphore = asyncio.Semaphore(max_concurrent)

 async def process(self, message):
 async with self.semaphore:
 return await self.handle(message)

7.3 Dead Letter Queue

Failed messages are moved to a separate queue for analysis and reprocessing.

8. Conclusion

Key elements of a multi-agent communication platform include:

Flexible message structure: Support for various communication scenarios
Clear state management: Predictable agent behavior
Diverse communication patterns: Request-response, publish-subscribe, broadcast
Fault-tolerant design: Retry, timeout, error handling

Based on these principles, scalable and reliable multi-agent systems can be built.

References

Korean Version: 한국어 버전

Architecture on Yarang's Tech Lair

Building a Multi-LLM Distributed Orchestrator with NATS JetStream

System Overview

Why NATS JetStream?

The Core: Backend Adapter in poller.py

Claude Backend

ZAI Backend

Declarative Management: fleet.yaml × servers.yaml

servers.yaml — Server Inventory

fleet.yaml — Worker Placement

Worker Templating System: provision_worker.py

Template Structure

provision_worker.py Usage

Distributed Hosting: Adding Workers to a Second Server

NATS Credential Operations Experience

Adding a New Worker: The Full Procedure

Next Steps

AgentForge Blog Automation Service: Full Architecture - From AI Comments to Translation and Post Generation

System Topology

arm1: Unified Blog Agent

Why Unified?

Architecture

Module Operations

1. CommentHandler — AI Comment Response

2. TranslateHandler — Automatic Translation Trigger

3. PostGenerator — Automatic Post Generation

ec1: Blog API Translation System

Transition to Gemini

OpenAI-Compatible Endpoint

Translation Matching Logic

Title-in-Body Translation Technique

LLM Strategy: Role-Based Model Separation

Monitoring and Operations

Health Check Endpoints

Observability Points

Lessons Learned and Operational Tips

1. The Pitfall of Reasoning Models

2. Value of the OpenAI-Compatible Pattern

3. Constraints of Date Prefix Matching

4. Benefits of Process Consolidation

Future Plans

Multi-Model AI Agent Team Design: Composed Architecture and 5-Team Hierarchy

Overview

Background: Why Multi-Model?

Final Team Structure

Team Details

The Infra Team Separation Decision

Model Distribution Strategy

Composed Agent Architecture

Composed Approach

Module Structure

Advantages

Expert-Definition Mapping

TLS Certificate Strategy: Cloudflare Origin CA

Relay Plugin: Agent Invocation Mechanism

backed_by Namespaces

Design Decision Log

Retrospective: What I Learned

1. “Executable Structure” over “Perfect Structure”

2. Workspace Defines Team Boundaries

3. The Value of Composed Architecture

4. Cost is Determined at Design Time

Next Steps

[Discord MCP] Gateway Architecture Design

1. Overall Architecture

Components

Message Flow

2. user_comm Agent (User Communication Handler)

Role and Responsibilities

Internal Structure

Message Types

Key Function Flows

Input Reception (Discord -> Agent)

Opinion Request (Agent -> Discord)

Notification/Report (Agent -> Discord)

3. Lightweight Architecture Without Redis

Why Remove Redis?

Gateway Structure

Component Details

4. MCP Selection: 4-Stage Hybrid

The Core: Backend Adapter in `poller.py`

Declarative Management: `fleet.yaml` × `servers.yaml`

`servers.yaml` — Server Inventory

`fleet.yaml` — Worker Placement

Worker Templating System: `provision_worker.py`

`provision_worker.py` Usage