Very Fast Message Processor (VFMP)

A lightweight, high-throughput message broker built in Go. VFMP is designed to be simple to operate and scale horizontally, providing reliable at-least-once message delivery with minimal overhead.

Goals

• High throughput: In-memory linked-list queues with no serialisation overhead on the hot path
• At-least-once delivery: Lease-based acknowledgement ensures messages are redelivered if a consumer crashes or times out
• Simple operations: Single binary, YAML config, Prometheus metrics out of the box
• Scalable consumption: gRPC server-streaming lets consumers pull batches of messages efficiently

How it works

Publishers send messages over HTTP. Consumers pull messages via gRPC and acknowledge each one with a lease token. If a consumer fails to acknowledge within 10 seconds the message is automatically requeued. After 3 failed delivery attempts a message is moved to a dead-letter queue (DEADLETTER/<topic>) for inspection.

Publisher                  VFMP                     Consumer
   |                        |                           |
   |-- POST /messages/foo ->|                           |
   |                        |-- Consume(foo, n) ------->|
   |                        |<-- stream: msg+lease -----|
   |                        |                           | (process)
   |                        |<-- Ack(lease_token) ------|
   |                        |                           |

Quick start

# Start the server (HTTP :8080, gRPC :9090, metrics :5050)
just run

# Publish a message
curl -X POST http://localhost:8080/messages/my-topic \
  -H "X-Correlation-ID: $(uuidgen)" \
  -d "hello world"

# Check queue length
curl "http://localhost:8080/messages/my-topic?data=count"

# Consume messages (ACK each one)
just client -address localhost:9090 -topic my-topic

# Check queue length again
curl "http://localhost:8080/messages/my-topic?data=count"

Configuration

Configuration is loaded from a YAML file, environment variables, or flags — in that order of precedence.

Field	Env var	Default	Description
http_addr	HTTP_ADDR	:8080	HTTP API listener
tcp_addr	TCP_ADDR	:9090	gRPC listener
metrics_addr	PPROF_ADDR	:5050	Prometheus metrics listener
log_level	LOG_LEVEL	debug	Log level (debug/info/warn)
log_path	LOG_PATH	—	Optional log file path
shutdown_timeout	SHUTDOWN_TIMEOUT	10s	Graceful shutdown window

API

HTTP

Method	Path	Description
POST	/messages/{topic}	Publish a message (requires X-Correlation-ID header)
GET	/messages/{topic}?data=count	Queue length for a topic
GET	/messages/{topic}?data=peek	Peek at the next message
GET	/control/healthcheck	Health check
GET	/control/version	Server version

gRPC (vfmp.v1.MessageService)

RPC	Description
Consume	Server-streaming: receive up to N messages from a topic
Ack	Confirm successful processing — message permanently removed
Nck	Signal failure — message immediately requeued for redelivery
Dlq	Dead-letter a message — moved to DEADLETTER/<topic>

Proto definitions are in proto/vfmp/v1/messages.proto. Regenerate with just proto.