PostgreSQL on Yarang's Tech Lair

pg_durable Analysis: Implementing Durable Function Execution with Rust

Sat, 06 Jun 2026 09:00:49 +0900

Recently open-sourced by Microsoft, pg_durable is an experimental project that enables ‘Durable’ function execution directly within the database. In traditional microservice architectures, business logic often resides in the application layer, with the database primarily serving as a state storage. However, pg_durable proposes an approach that integrates logic and data in close proximity, minimizing external system failures or network costs.

In this article, we will analyze the core mechanisms of pg_durable and explore how to implement a similar pattern using Rust and PostgreSQL.

Core Mechanisms of pg_durable

The most significant feature of pg_durable is that it stores the function’s state and execution flow itself as database records. In typical applications, if an HTTP request fails, the in-memory context disappears. However, pg_durable records the input and current step of the function to be executed in a table.

This approach offers the following benefits:

Auto-Recovery: Even if a server crashes, ’tasks to be executed’ remain in the database, allowing for continued execution after the server restarts.
Exactly-Once Semantics: By leveraging database transactions, state updates and logic execution can be handled atomically.

Implementing a Database Trigger-Based Worker with Rust

While the internal implementation of pg_durable is complex, we can simply replicate this pattern using Rust and PostgreSQL’s LISTEN/NOTIFY functionality. This approach utilizes the database as a ‘message broker’ while saving on separate infrastructure costs.

1. Database Schema Design

First, we create a table to store jobs.

CREATE TABLE durable_jobs (
 id SERIAL PRIMARY KEY,
 status TEXT NOT NULL, -- 'pending', 'running', 'completed'
 payload JSONB NOT NULL,
 created_at TIMESTAMP DEFAULT NOW()
);

2. Rust Worker Implementation (Using sqlx)

We will use Rust’s asynchronous runtime and sqlx to write a worker that detects and processes new jobs as they enter the database.

# Cargo.toml
[dependencies]
tokio = { version = "1", features = ["full"] }
sqlx = { version = "0.7", features = ["runtime-tokio", "postgres", "json"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

Now, let’s implement the worker in Rust code.

use sqlx::{PgPool, Listener};
use serde::{Deserialize, Serialize};
use tokio::time::{sleep, Duration};

#[derive(Debug, Serialize, Deserialize)]
struct JobPayload {
 user_id: i32,
 action: String,
}

#[tokio::main]
async fn main() -> Result<(), sqlx::Error> {
 // Create a database connection pool
 let pool = PgPool::connect("postgresql://user:password@localhost/mydb").await?;
 
 // Listener to detect database changes (using pg_notify)
 let mut listener = pool.acquire().await?.listen("job_channel");

 println!("Worker started, waiting for jobs...");

 loop {
 // Wait for notifications
 let notification = listener.recv().await?;
 println!("Received notification: {}", notification.payload);

 // Upon receiving a notification, attempt to process the actual job
 process_jobs(&pool).await?;
 }
}

async fn process_jobs(pool: &PgPool) -> Result<(), sqlx::Error> {
 // Within a transaction, fetch a 'pending' job and change its status to 'running' (Locking)
 let mut tx = pool.begin().await?;
 
 let job_opt = sqlx::query_as::<_, (i32, JobPayload)>(
 r#"
 UPDATE durable_jobs 
 SET status = 'running' 
 WHERE id = (
 SELECT id FROM durable_jobs 
 WHERE status = 'pending' 
 FOR UPDATE SKIP LOCKED 
 LIMIT 1
 )
 RETURNING id, payload
 "#
 )
 .fetch_optional(&mut *tx)
 .await?;

 if let Some((id, payload)) = job_opt {
 println!("Processing job {}: {:?}", id, payload);
 
 // Execute business logic (e.g., external API calls, etc.)
 // In a real environment, heavy tasks would be performed here.
 let result = simulate_heavy_task(&payload).await;

 match result {
 Ok(_) => {
 // On success, update status to 'completed'
 sqlx::query("UPDATE durable_jobs SET status = 'completed' WHERE id = $1")
 .bind(id)
 .execute(&mut *tx)
 .await?;
 }
 Err(e) => {
 // On failure, leave status as 'failed' or add retry logic
 eprintln!("Job {} failed: {}", id, e);
 sqlx::query("UPDATE durable_jobs SET status = 'failed' WHERE id = $1")
 .bind(id)
 .execute(&mut *tx)
 .await?;
 }
 }
 
 tx.commit().await?;
 }
 
 Ok(())
}

async fn simulate_heavy_task(payload: &JobPayload) -> Result<(), Box<dyn std::error::Error>> {
 // Example: Wait for 2 seconds
 sleep(Duration::from_secs(2)).await;
 println!("Task executed for user {}", payload.user_id);
 Ok(())
}

3. Job Registration Trigger

In practice, you can automate this by sending an INSERT query from the application level and then issuing a NOTIFY, or by using a trigger.

-- Example of creating a trigger function and trigger
CREATE OR REPLACE FUNCTION notify_job() RETURNS TRIGGER AS $$
BEGIN
 PERFORM pg_notify('job_channel', NEW.id::text);
 RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER job_notify_trigger
AFTER INSERT ON durable_jobs
FOR EACH ROW
EXECUTE FUNCTION notify_job();

Considerations for Applying the Architecture

When applying this pattern (the pg_durable style) to high-performance systems like ZeroClaw or MCP servers, consider the following:

Database Load: Since the database partially acts as an execution engine rather than just a storage, frequent polling by workers or executing heavy logic can degrade DB performance. It is essential to configure it to allow multiple worker instances to process tasks safely and in a distributed manner, using the SKIP LOCKED clause.
Scalability (Scale-out): Even when the Rust code shown above is run as multiple processes, it can safely distribute tasks thanks to FOR UPDATE SKIP LOCKED. In a Kubernetes environment, flexible responses can be achieved by increasing worker pods when the queue grows, using HorizontalPodAutoscaler (HPA).

Conclusion

Microsoft’s pg_durable showcases an evolved form of ‘Data-Centric Architecture’. Before introducing complex message queues (Kafka, RabbitMQ), why not achieve simple yet robust durability by actively leveraging the transactional capabilities of the database? The example code above is readily testable in practice, so we recommend applying it to your side projects or back-office systems.

Implementing Postgres-based Durable Workflows: Using a DB as a Workflow Engine

Fri, 29 May 2026 09:00:36 +0900

As we’ve been building a fast and lightweight microservice architecture recently, minimizing external dependencies has emerged as a critical challenge. Specifically, during the development of the ZeroClaw runtime, we considered introducing separate message queues like Kafka or Redis to ensure complex state machine management and reliable communication between agents, but we were concerned about operational complexity.

At this point, the “Postgres-based Durable Workflow” pattern, which has recently been a hot topic on Hacker News and similar platforms, caught our attention. This approach goes beyond simply storing data, leveraging the database itself as a workflow engine to handle state persistence and retry logic. In this post, we’ll share the core concepts of this pattern and a practical implementation method applied to the ZeroClaw project.

Why Postgres?

Traditionally, we might have used specialized workflow engines like Temporal.io or Cadence, or implemented event sourcing through Kafka. However, for small teams or personal projects, maintaining such infrastructure can be a significant burden.

PostgreSQL is already adopted as the default database for most services. It guarantees strong consistency through ACID transactions and allows for distributed lock implementation (claim-based lock) using the SKIP LOCKED feature. This means we can simplify our infrastructure stack by utilizing our “already trusted storage” as a queue, a scheduler, and a state store.

Core Architecture: Using Tables as a Queue

The core of this pattern lies in defining workflow states and execution tasks within tables. Let’s take the ZeroClaw agent task queue as an example.

1. Defining the State Table (SQL)

We designed the workflow_jobs table to store the current state of a job and its next execution time.

CREATE TABLE workflow_jobs (
 id SERIAL PRIMARY KEY,
 job_type TEXT NOT NULL, -- e.g., 'agent_task', 'discord_notify'
 payload JSONB NOT NULL, -- Data required for execution
 status TEXT NOT NULL DEFAULT 'pending', -- pending, running, succeeded, failed
 claimed_at TIMESTAMP,
 run_after TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- Timestamp for delayed execution or retries
 last_error TEXT,
 attempt_count INT DEFAULT 0,
 max_attempts INT DEFAULT 5,
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- State-based indexing (for efficient lookups)
CREATE INDEX idx_jobs_claim ON workflow_jobs (status, run_after) WHERE status = 'pending';

The status and run_after columns are key here. These two columns allow us to immediately find “jobs that need to be executed now.”

2. Scheduling and Executing Tasks (Rust Example)

Now, a worker must find a task, claim ownership, and execute it within a transaction. To prevent race conditions, we use the FOR UPDATE SKIP LOCKED clause.

Here’s an example implementation using Rust and sqlx:

use sqlx::{PgPool, Postgres};
use sqlx::types::chrono::{Utc, DateTime};
use serde_json::Value;

#[derive(Debug)]
pub struct Job {
 pub id: i32,
 pub job_type: String,
 pub payload: Value,
}

/// Function called by the worker in its execution loop.
/// Safely claims a job within a transaction.
pub async fn claim_job(pool: &PgPool, worker_id: &str) -> Result<Option<Job>, sqlx::Error> {
 let now = Utc::now();
 
 let mut tx = pool.begin().await?;
 
 // 1. Find an executable job and lock it.
 // SKIP LOCKED: Ignores rows that have already been locked by another worker.
 let query = r#"
 SELECT id, job_type, payload 
 FROM workflow_jobs 
 WHERE status = 'pending' 
 AND run_after <= $1
 ORDER BY created_at ASC
 FOR UPDATE SKIP LOCKED
 LIMIT 1
 FOR UPDATE OF workflow_jobs
 "#;

 let job: Option<Job> = sqlx::query_as::<_, Job>(query)
 .bind(now)
 .fetch_optional(&mut *tx)
 .await?;

 let job = match job {
 Some(j) => j,
 None => {
 tx.rollback().await?; // No job to claim (Rollback is not strictly necessary but explicit)
 return Ok(None);
 }
 };

 // 2. Change the job status to 'running'.
 sqlx::query(
 "UPDATE workflow_jobs SET status = 'running', claimed_at = $1, attempt_count = attempt_count + 1 WHERE id = $2"
 )
 .bind(Utc::now())
 .bind(job.id)
 .execute(&mut *tx)
 .await?;

 tx.commit().await?;

 Ok(Some(job))
}

/// Function to update status after job completion.
pub async fn complete_job(pool: &PgPool, job_id: i32, success: bool, error_msg: Option<String>) -> Result<(), sqlx::Error> {
 let status = if success { "succeeded" } else { "failed" };
 
 // For failures, exponential backoff can be considered to calculate run_after.
 let next_run = if success {
 None
 } else {
 // e.g., Retry after 10 seconds (actual implementation should include exponential logic based on attempt_count)
 Some(Utc::now() + chrono::Duration::seconds(10))
 };

 if success {
 sqlx::query(
 "UPDATE workflow_jobs SET status = $1, run_after = NULL WHERE id = $2"
 )
 .bind(status)
 .bind(job_id)
 .execute(pool)
 .await?;
 } else {
 sqlx::query(
 "UPDATE workflow_jobs SET status = $1, last_error = $2, run_after = $3 WHERE id = $4"
 )
 .bind(status)
 .bind(error_msg)
 .bind(next_run)
 .bind(job_id)
 .execute(pool)
 .await?;
 }

 Ok(())
}

3. Worker Loop Configuration

Now, using the functions above, we can run an infinite loop (or use Tokio’s tokio::time::interval) to process jobs.

async fn worker_loop(pool: PgPool, worker_id: String) {
 loop {
 match claim_job(&pool, &worker_id).await {
 Ok(Some(job)) => {
 println!("[{}] Job acquired: {:?}", worker_id, job.job_type);
 
 // Execute actual business logic (e.g., LLM inference, API calls)
 let result = execute_logic(job.payload.clone()).await;

 // Reflect the result
 let _ = complete_job(&pool, job.id, result.is_ok(), result.err().map(|e| e.to_string())).await;
 }
 Ok(None) => {
 // If there are no jobs to process, wait briefly (to prevent high CPU usage)
 tokio::time::sleep(tokio::time::Duration::from_millis(500)).await;
 }
 Err(e) => {
 eprintln!("[{}] Worker error: {}", worker_id, e);
 tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
 }
 }
 }
}

Pros, Cons, and Usage Tips

Pros:

Simplicity: Achieves reliability using only RDBMS transactions without complex distributed system theory.
Operational Efficiency: Backing up the DB also restores workflow states.
Strong Consistency: No worries about in-flight message loss.

Cons:

Throughput Limitations: Postgres has limitations on per-connection throughput, so a specialized MQ is better if you need tens of thousands of transactions per second.
DB Load: Workers continuously poll, so connection pools should be optimized using tools like pgbouncer.

Conclusion

In ZeroClaw’s multi-agent environment, the message processing throughput of individual agents wasn’t high enough to justify introducing Kafka, but state management reliability was crucial. Through the Postgres-based architecture, we were able to achieve both “stable state management” and a “scalable task queue” without introducing separate infrastructure.

If your project is grappling with the trade-off between “simplicity” and “reliability,” why not consider leveraging the readily available Postgres as a workflow engine before adopting heavy-duty solutions?

References

Building durable workflows on Postgres (Hacker News Discussion)
SKIP LOCKED and Distributed Lock Implementation