Database on Yarang's Tech Lair

pg_durable Analysis: Implementing Durable Function Execution with Rust

Sat, 06 Jun 2026 09:00:49 +0900

Recently open-sourced by Microsoft, pg_durable is an experimental project that enables ‘Durable’ function execution directly within the database. In traditional microservice architectures, business logic often resides in the application layer, with the database primarily serving as a state storage. However, pg_durable proposes an approach that integrates logic and data in close proximity, minimizing external system failures or network costs.

In this article, we will analyze the core mechanisms of pg_durable and explore how to implement a similar pattern using Rust and PostgreSQL.

Core Mechanisms of pg_durable

The most significant feature of pg_durable is that it stores the function’s state and execution flow itself as database records. In typical applications, if an HTTP request fails, the in-memory context disappears. However, pg_durable records the input and current step of the function to be executed in a table.

This approach offers the following benefits:

Auto-Recovery: Even if a server crashes, ’tasks to be executed’ remain in the database, allowing for continued execution after the server restarts.
Exactly-Once Semantics: By leveraging database transactions, state updates and logic execution can be handled atomically.

Implementing a Database Trigger-Based Worker with Rust

While the internal implementation of pg_durable is complex, we can simply replicate this pattern using Rust and PostgreSQL’s LISTEN/NOTIFY functionality. This approach utilizes the database as a ‘message broker’ while saving on separate infrastructure costs.

1. Database Schema Design

First, we create a table to store jobs.

CREATE TABLE durable_jobs (
 id SERIAL PRIMARY KEY,
 status TEXT NOT NULL, -- 'pending', 'running', 'completed'
 payload JSONB NOT NULL,
 created_at TIMESTAMP DEFAULT NOW()
);

2. Rust Worker Implementation (Using sqlx)

We will use Rust’s asynchronous runtime and sqlx to write a worker that detects and processes new jobs as they enter the database.

# Cargo.toml
[dependencies]
tokio = { version = "1", features = ["full"] }
sqlx = { version = "0.7", features = ["runtime-tokio", "postgres", "json"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

Now, let’s implement the worker in Rust code.

use sqlx::{PgPool, Listener};
use serde::{Deserialize, Serialize};
use tokio::time::{sleep, Duration};

#[derive(Debug, Serialize, Deserialize)]
struct JobPayload {
 user_id: i32,
 action: String,
}

#[tokio::main]
async fn main() -> Result<(), sqlx::Error> {
 // Create a database connection pool
 let pool = PgPool::connect("postgresql://user:password@localhost/mydb").await?;
 
 // Listener to detect database changes (using pg_notify)
 let mut listener = pool.acquire().await?.listen("job_channel");

 println!("Worker started, waiting for jobs...");

 loop {
 // Wait for notifications
 let notification = listener.recv().await?;
 println!("Received notification: {}", notification.payload);

 // Upon receiving a notification, attempt to process the actual job
 process_jobs(&pool).await?;
 }
}

async fn process_jobs(pool: &PgPool) -> Result<(), sqlx::Error> {
 // Within a transaction, fetch a 'pending' job and change its status to 'running' (Locking)
 let mut tx = pool.begin().await?;
 
 let job_opt = sqlx::query_as::<_, (i32, JobPayload)>(
 r#"
 UPDATE durable_jobs 
 SET status = 'running' 
 WHERE id = (
 SELECT id FROM durable_jobs 
 WHERE status = 'pending' 
 FOR UPDATE SKIP LOCKED 
 LIMIT 1
 )
 RETURNING id, payload
 "#
 )
 .fetch_optional(&mut *tx)
 .await?;

 if let Some((id, payload)) = job_opt {
 println!("Processing job {}: {:?}", id, payload);
 
 // Execute business logic (e.g., external API calls, etc.)
 // In a real environment, heavy tasks would be performed here.
 let result = simulate_heavy_task(&payload).await;

 match result {
 Ok(_) => {
 // On success, update status to 'completed'
 sqlx::query("UPDATE durable_jobs SET status = 'completed' WHERE id = $1")
 .bind(id)
 .execute(&mut *tx)
 .await?;
 }
 Err(e) => {
 // On failure, leave status as 'failed' or add retry logic
 eprintln!("Job {} failed: {}", id, e);
 sqlx::query("UPDATE durable_jobs SET status = 'failed' WHERE id = $1")
 .bind(id)
 .execute(&mut *tx)
 .await?;
 }
 }
 
 tx.commit().await?;
 }
 
 Ok(())
}

async fn simulate_heavy_task(payload: &JobPayload) -> Result<(), Box<dyn std::error::Error>> {
 // Example: Wait for 2 seconds
 sleep(Duration::from_secs(2)).await;
 println!("Task executed for user {}", payload.user_id);
 Ok(())
}

3. Job Registration Trigger

In practice, you can automate this by sending an INSERT query from the application level and then issuing a NOTIFY, or by using a trigger.

-- Example of creating a trigger function and trigger
CREATE OR REPLACE FUNCTION notify_job() RETURNS TRIGGER AS $$
BEGIN
 PERFORM pg_notify('job_channel', NEW.id::text);
 RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER job_notify_trigger
AFTER INSERT ON durable_jobs
FOR EACH ROW
EXECUTE FUNCTION notify_job();

Considerations for Applying the Architecture

When applying this pattern (the pg_durable style) to high-performance systems like ZeroClaw or MCP servers, consider the following:

Database Load: Since the database partially acts as an execution engine rather than just a storage, frequent polling by workers or executing heavy logic can degrade DB performance. It is essential to configure it to allow multiple worker instances to process tasks safely and in a distributed manner, using the SKIP LOCKED clause.
Scalability (Scale-out): Even when the Rust code shown above is run as multiple processes, it can safely distribute tasks thanks to FOR UPDATE SKIP LOCKED. In a Kubernetes environment, flexible responses can be achieved by increasing worker pods when the queue grows, using HorizontalPodAutoscaler (HPA).

Conclusion

Microsoft’s pg_durable showcases an evolved form of ‘Data-Centric Architecture’. Before introducing complex message queues (Kafka, RabbitMQ), why not achieve simple yet robust durability by actively leveraging the transactional capabilities of the database? The example code above is readily testable in practice, so we recommend applying it to your side projects or back-office systems.

The Evolution of Redis Arrays: An Architectural Analysis for Large-Scale Data Processing

Tue, 05 May 2026 09:00:52 +0900

The Evolution of Redis Arrays: An Architectural Analysis for Large-Scale Data Processing

Hello everyone! I recently came across an interesting article on Hacker News, written by Oran Agra, one of Redis’s core developers, titled “Redis array: short story of a long development process.” This wasn’t just a story about adding a new feature; it was a testament to the dedication of developers who tackled 25-year-old legacy code, ensuring performance, maintaining stability, and formatting a massive codebase overnight.

Today, based on this article, we’ll dive deep into how the Array data structure has evolved within Redis and what lessons we can learn for designing large-scale systems.

1. The Problem: The Shackle of 25-Year-Old Legacy Code

Redis’s LIST data structure internally uses QuickList. QuickList combines the advantages of ziplist and linkedlist, which are doubly linked lists. However, when dealing with massive lists containing tens of millions of elements, memory fragmentation and cache misses were causing significant performance degradation.

Specifically, when processing array-type data, the existing structure had the following bottlenecks:

Memory Overhead: Additional memory usage due to pointer connections.
Sequential Access Cost: Latency caused by inefficient use of cache lines.

To address this, the development team decided to overhaul the internal structure at the C language level. The biggest challenge here was the “legacy code that had to be changed.”

2. The Solution: Formatting a 25M-line Codebase

The most impressive part of the article was “Formatting a 25M-line codebase overnight.” The process of formatting and refactoring 25 million lines of code required more than just technical challenges; it demanded strategy akin to chess.

2.1. Preparations for Refactoring

The biggest fear in large-scale refactoring is “regression.” Modifying the array structure could affect hundreds of Redis commands (like LPUSH, RPUSH, LINDEX, etc.).

To mitigate this, the team adopted the following approach:

Expand Test Coverage: Ensure existing commands pass unit tests.
Strengthen CI/CD Pipeline: Implement benchmarking scripts to immediately detect performance degradation upon code changes.

2.2. The New Structure of Redis Arrays

The improved Array structure moved beyond simply allocating memory and was modified to maximize data locality. The core principle was “maximizing the use of contiguous memory blocks while allowing for segmentation and management when necessary.”

This yielded the following benefits:

Improved CPU Cache Hit Rate: Significantly increased L1/L2 cache hit rates due to contiguous memory access.
Memory Savings: Reduced actual data storage space by minimizing unnecessary pointer connections.

3. Practical Guide: Efficient Array Usage in Redis

Now that we’ve covered the theoretical background, let’s look at how to apply it in practice with code.

3.1. Problems with Existing List Usage

First, let’s consider the traditional way of adding tens of millions of items to a list. This operates based on QuickList, and as the number of items increases, the number of jumps also increases.

# Traditional Method (QuickList based)
# Add 10,000,000 items (potential for memory and speed degradation)
for i in {1..10000000}; do
 redis-cli LPUSH my_huge_list "item:$i"
done

3.2. Optimization using Streams and Hashes

While the internal improvements to Redis Arrays are transparent to users, when designing, we need to consider “data size” and “access patterns.” If simple sequential storage is all that’s needed, using the latest version of Redis alone will provide benefits.

However, if you need to search or modify data within the array, it’s advisable to use HASH instead of LIST.

import redis
import time

r = redis.Redis(host='localhost', port=6379, db=0)

# Scenario: Storing Log Data (Large Scale)
# 1. Using List (for sequential storage)
def push_to_list(count):
 start = time.time()
 for i in range(count):
 r.lpush("logs:timeline", f"log_entry_{i}")
 print(f"List pushed {count} items in {time.time() - start:.4f}s")

# 2. Using Hash (for search and modification)
def push_to_hash(count):
 start = time.time()
 pipe = r.pipeline()
 for i in range(count):
 pipe.hset("logs:details", f"entry_{i}", f"log_content_{i}")
 pipe.execute()
 print(f"Hash pushed {count} items in {time.time() - start:.4f}s")

if __name__ == "__main__":
 # Test inserting 100,000 data points
 push_to_list(100000)
 push_to_hash(100000)

Execution Result Analysis: In recent Redis versions (7.x and above), the internal Array structure is optimized, making LPUSH very fast. However, if you frequently need to retrieve data at a specific index, LINDEX has a complexity of O(N), making the O(1) approach using HGET much more advantageous.

4. Conclusion: The Harmony of Development Culture and Technology

The development process of Redis Arrays offers us important lessons:

Performance Isn’t Free: Improving 25-year-old code requires commensurate refactoring and testing costs.
Investment in Tools: This work was possible due to automated tools and a CI/CD environment capable of formatting 25 million lines of code.

When we design systems, we need to go beyond simply asking “Is it fast?” and consider “How can we achieve maintainable performance?” As the Redis team demonstrated, sometimes we must not shy away from large-scale improvements that shake the foundations of the architecture.

5. References

Thank you!