⚡  go-batcher

High-performance batch processing for Go applications

CI / CD		Quality
Security		Community

Project Navigation

📦 Installation	🎯 What's Inside	📚 Documentation
🧪 Examples & Tests	⚡ Benchmarks	🛠️ Code Standards
🤖 AI Usage	🤝 Contributing	👥 Maintainers
📝 License

🎯 What's Inside

Lightning-Fast Batch Processing in Action

package main

import (
    "fmt"
    "time"
    "github.com/bsv-blockchain/go-batcher"
)

func main() {
    // Create a batcher that processes items every 100ms or when batch size hits 1000
    b := batcher.New[string](
        1000,                                        // batch size
        100*time.Millisecond,                        // timeout interval
        func(batch []*string) {                      // processor function
            fmt.Printf("⚡ Processing %d items in one go!\n", len(batch))
            // Your batch processing logic here
            for _, item := range batch {
                fmt.Printf("Processing: %s\n", *item)
            }
        },
        true, // background processing
    )

    // Feed items - they'll be intelligently batched
    for i := 0; i < 5000; i++ {
        item := fmt.Sprintf("item-%d", i)
        b.Put(&item)
    }

    // Process any remaining items before shutdown
    b.Trigger()
    // Note: The batcher worker runs indefinitely - use context cancellation for cleanup
}

Constructor Variants

The go-batcher library provides several constructor options to fit different use cases:

// Basic batcher - simple batching with size and timeout triggers
b := batcher.New[string](100, time.Second, processFn, true)

// With slice pooling - reduces memory allocations for high-throughput scenarios
b := batcher.NewWithPool[string](100, time.Second, processFn, true)

// With automatic deduplication - filters duplicate items within a 1-minute window
b := batcher.NewWithDeduplication[string](100, time.Second, processFn, true)

// Combined pooling and deduplication - maximum performance with duplicate filtering
b := batcher.NewWithDeduplicationAndPool[string](100, time.Second, processFn, true)

Why You'll Love This Batcher

• Blazing Performance – Process millions of items with minimal overhead (benchmarks: 135 ns/op)
• Smart Batching – Auto-groups by size or time interval, whichever comes first
• Optional Deduplication – Built-in dedup variant ensures each item is processed only once within a time window
• Memory Pool Optimization – Optional slice pooling reduces GC pressure in high-throughput scenarios
• Thread-Safe by Design – Concurrent Put() from multiple goroutines without worry
• Time-Partitioned Storage – Efficient memory usage with automatic cleanup (dedup variant)
• Minimal Dependencies – Pure Go with only essential external dependencies
• Flexible Configuration – Multiple constructor variants for different use cases
• Production-Ready – Battle-tested with full test coverage and benchmarks

Perfect for high-throughput scenarios like log aggregation, metrics collection, event processing, or any situation where you need to efficiently batch operations for downstream systems.

📦 Installation

go-batcher requires a supported release of Go.

go get -u github.com/bsv-blockchain/go-batcher

📚 Documentation

• API Reference – Dive into the godocs at pkg.go.dev/github.com/bsv-blockchain/go-batcher
• Usage Examples – Browse practical patterns either the examples directory or view the example functions
• Benchmarks – Check the latest numbers in the benchmark results
• Test Suite – Review both the unit tests and fuzz tests (powered by testify)

Development Build Commands

Get the MAGE-X build tool for development:

go install github.com/mrz1836/mage-x/cmd/magex@latest

View all build commands

magex help

Repository Features

This repository includes 25+ built-in features covering CI/CD, security, code quality, developer experience, and community tooling.

View the full Repository Features list →

Library Deployment

This project uses goreleaser for streamlined binary and library deployment to GitHub. To get started, install it via:

brew install goreleaser

The release process is defined in the .goreleaser.yml configuration file.

Then create and push a new Git tag using:

magex version:bump push=true bump=patch branch=master

This process ensures consistent, repeatable releases with properly versioned artifacts and citation metadata.

Pre-commit Hooks

Set up the Go-Pre-commit System to run the same formatting, linting, and tests defined in AGENTS.md before every commit:

go install github.com/mrz1836/go-pre-commit/cmd/go-pre-commit@latest
go-pre-commit install

The system is configured via modular env files and provides 17x faster execution than traditional Python-based pre-commit hooks. See the complete documentation for details.

GitHub Workflows

All workflows are driven by modular configuration in .github/env/ — no YAML editing required.

View all workflows and the control center →

Updating Dependencies

To update all dependencies (Go modules, linters, and related tools), run:

magex deps:update

This command ensures all dependencies are brought up to date in a single step, including Go modules and any tools managed by MAGE-X. It is the recommended way to keep your development environment and CI in sync with the latest versions.

🧪 Examples & Tests

All unit tests and examples run via GitHub Actions and use Go version 1.24.x. View the configuration file.

Run all tests (fast):

magex test

Run all tests with race detector (slower):

magex test:race

⚡ Benchmarks

Run the Go benchmarks:

magex bench

Benchmark Results

Benchmark	Description	ns/op	B/op	allocs/op
BenchmarkBatcherPut	Basic Put operation	135.1	8	0
BenchmarkBatcherPutParallel	Concurrent Put operations	310.0	9	0
BenchmarkPutComparison/Put	Put operation (non-blocking)	300.7	9	0
BenchmarkPutComparison/PutWithPool	Put with slice pooling	309.9	1	0
BenchmarkWithPoolComparison/Batcher	Standard batcher	171.2	18	1
BenchmarkWithPoolComparison/WithPool	Pooled batcher	184.0	9	1
BenchmarkTimePartitionedMapSet	Map Set operation (bloom filter)	366.7	147	6
BenchmarkTimePartitionedMapGet	Map Get operation (bloom filter)	80.5	39	2
BenchmarkBatcherWithDedupPut	Put with deduplication	740.1	166	7
BenchmarkBatcher	Full batch processing (1M items)	1,081ms	710MB	1.9M
BenchmarkBatcherWithDeduplication	Deduplication processing	90.7	13	0

Performance benchmarks for the core functions in this library, executed on an Apple M1 Max (ARM64). The benchmarks demonstrate excellent performance with minimal allocations for basic operations.

🛠️ Code Standards

Read more about this Go project's code standards.

🤖 AI Usage & Assistant Guidelines

Read the AI Usage & Assistant Guidelines for details on how AI is used in this project and how to interact with AI assistants.

👥 Maintainers

Siggi	Dylan	MrZ

🤝 Contributing

View the contributing guidelines and please follow the code of conduct.

How can I help?

All kinds of contributions are welcome 🙌! The most basic way to show your support is to star 🌟 the project, or to raise issues 💬.

📝 License