parse-rust

[ En | 中 ]
The inverse of str.format() — extract structured values from text with Rust-speed parsing.

Introduction

parse (~1.8k stars, MIT-licensed) is the inverse of Python's str.format(). Given a format template and a string, it extracts structured values by matching the template against the input and converting captured fields to their declared types. Its core capabilities include:

• Core API: parse() extracts values from a full-string match, search() finds the first match anywhere in the text, findall() returns all non-overlapping matches, and compile() pre-compiles a format string into a reusable Parser object.
• Format mini-language: Full support for [[fill]align][sign][0][width][grouping][.precision][type], the same specification used by Python's str.format().
• 25 format specifier types: 7 string-like types (default, s, w, W, l, S, D), 5 integer types (d, b, o, x, n), 5 floating-point types (f, F returning Decimal, e, g, % auto-dividing by 100), and 8 datetime types (ti, te, ta, tg, th, tc, ts, tt plus custom %... strptime formats).
• Result encapsulation: Result objects support indexed access (result[0]), named access (result["name"]), .fixed tuple, .named dict, and .spans position information.
• Advanced features: evaluate_result=False deferred evaluation, extra_types custom type converters, with_pattern decorator for custom match patterns, case_sensitive control, repeated named-field semantics with RepeatedNameError.

Typical application scenarios include log analysis, data extraction, ETL pipelines, and template matching — anywhere structured data needs to be extracted from semi-structured text.

parse-rust reimplements the full parsing engine in Rust while preserving the Python API. All parsing, pattern compilation, and type conversion logic runs in safe Rust. The only non-Rust component is FixedTzOffset — a ~20 line Python bridge class required because CPython mandates that datetime.tzinfo subclasses be created via pure Python single-inheritance.

Innovation & Optimization

Compared to the original Python implementation, parse-rust delivers improvements across four dimensions:

1. Language switch: Python → Rust + PyO3

The core parsing engine is written entirely in Rust and exposed to Python via PyO3. This architecture gains Rust's performance and memory safety at compile time, while retaining seamless access to the Python ecosystem. The engine is organized into four modules — compiler, types, parser, and result — with zero Python fallback paths. Custom type converters (extra_types, with_pattern) are dispatched in Rust; user-provided Python converter functions are invoked as callbacks from Rust, not via a fallback path.

2. Performance

All benchmarks use release builds with 5 independent runs reporting mean ± std, measured against Python parse 1.20.2 as baseline:

• Throughput: 4.2x–10.8x speedup across parse/search/findall paths. Log-style parsing reaches 10.8x, simple string extraction 6.6x.
• P99 tail latency: 13–29x improvement (e.g., log-style: 1.01 μs vs 29.02 μs). Rust's P50/P95/P99 are tightly clustered within 0.2–0.5 μs — no GC pauses, no long-tail jitter.
• Memory efficiency: 4.2x–7.7x higher throughput per MB of RSS. RSS shows zero growth from 100K to 300K inputs, confirming bounded heap allocation.

3. Security

• 0 unsafe blocks — memory safety guaranteed at compile time.
• 32/32 adversarial input tests pass — covering large inputs (1 MiB), null bytes, Unicode edges (emoji, combining chars, RTL, surrogates), regex injection resistance, 8-level deep nesting, malformed formats, type conversion boundaries, and search/findall edge cases. Zero crashes, zero panics, zero hangs.
• Static analysis: 0 Clippy correctness/safety warnings; dependency audit is clean; the only prior finding (pyo3 < 0.24.1) has been resolved by upgrading to pyo3 0.24.

4. Architecture upgrade

The modular Rust design separates concerns cleanly: compiler (format string → regex, 700 LOC), types (25 specifiers, 463 LOC), parser (core engine, 555 LOC), and result (encapsulation + datetime normalization, 598 LOC). A global parser cache (OnceLock<Mutex<ParserCache>>) eliminates redundant format compilation, making cold/warm/compiled throughput differences < 10%. The compilation overhead of Rust's regex DFA is nanosecond-level, compared to Python re.compile's millisecond-level cost.

Installation

From source (recommended)

git clone https://github.com/coooodeer/parse-rust && cd parse-rust
pip install maturin
maturin develop --release

Build wheel

cd parse-rust
maturin build --release
# wheel is at target/wheels/

Verify

python -c "from parse_rust import parse, compile; print('OK')"
# Expected: OK

Requirements: Python ≥ 3.8, Rust ≥ 1.75 (stable), maturin ≥ 1.0

Quick Start

from parse_rust import parse, search, findall, compile

# Basic parsing — {} matches greedily
result = parse("Hello, {}!", "Hello, World!")
assert result[0] == "World"

# Named fields with type conversion
result = parse("User {name:w} is {:d} years old", "User Alice is 30 years old")
assert result["name"] == "Alice"   # :w — word characters
assert result[0] == 30             # :d — integer

# search finds the first match anywhere in the text
found = search("{:d}", "I have 3 cats and 5 dogs")
assert found[0] == 3

# findall returns all non-overlapping matches
values = [r[0] for r in findall("{:d}", "I have 3 cats and 5 dogs")]
assert values == [3, 5]

# compile for repeated use
parser = compile("User {name:w} is {:d} years old")
assert parser.parse("User Alice is 30 years old")["name"] == "Alice"
assert parser.parse("User Bob is 42 years old")[0] == 42

Supported API

Top-level functions

Function	Signature
parse	parse(format, string, extra_types=None, evaluate_result=True, case_sensitive=False)
search	search(format, string, pos=0, endpos=None, extra_types=None, evaluate_result=True, case_sensitive=False)
findall	findall(format, string, pos=0, endpos=None, extra_types=None, evaluate_result=True, case_sensitive=False)
compile	compile(format, extra_types=None, case_sensitive=False)
with_pattern	with_pattern(regex, regex_group_count=1) — decorator
extract_format	extract_format(format_string) — structured format spec parse

Parser object

Attribute / Method	Description
.parse(string, evaluate_result=True)	Parse a single string
.search(string, pos=0, endpos=None, evaluate_result=True)	Search for first match
.findall(string, pos=0, endpos=None, evaluate_result=True)	Return iterator of all matches
.format	The format template string
.pattern	The compiled regex pattern
.named_fields	List of named fields
.fixed_fields	Number of fixed-position fields

Result object

Attribute / Method	Description
result[i]	Integer index access (supports negative indices and slices)
result["name"]	Named field access
.fixed	Tuple of fixed-position field values
.named	Dict of named field values
.spans	Dict mapping field keys to (start, end) character offsets

Match object (when evaluate_result=False)

Method	Description
.evaluate_result()	Execute type conversion and return a Result

Supported Format Types

Type	Characters Matched	Python Output
(default) / s	Any characters (greedy)	str
w	Letters, numbers, underscore	str
W	Not letters, numbers, underscore	str
l	Letters (ASCII)	str
S	Non-whitespace	str
D	Non-digit	str
d	Integer (optional sign, digits)	int
b	Binary number	int
o	Octal number	int
x	Hexadecimal number	int
n	Number with thousands separators	int
f	Fixed-point number	float
e	Floating-point with exponent	float
g	General number format	float
%	Percentage (value / 100.0)	float
F	Decimal number	Decimal
ti	ISO 8601 date/time	datetime
te	RFC 2822 email date/time	datetime
ta	US (month/day) date/time	datetime
tg	Global (day/month) date/time	datetime
th	HTTP log date/time	datetime
tc	ctime() date/time	datetime
ts	Linux syslog date/time	datetime
tt	Time only	time
%...	Custom strptime format	date, time, or datetime

The format specification mini-language ([[fill]align][sign][0][width][grouping][.precision][type]) is fully supported, including alignment, fill character, width, thousands grouping, and precision.

Usage Examples

Datetime parsing

from parse_rust import parse

# ISO 8601
r = parse("At {:ti}", "At 1972-01-20T10:21:36Z")
assert r[0].year == 1972

# RFC 2822 email format (with timezone)
r = parse("At {:te}", "At Mon, 20 Jan 1972 10:21:36 +1000")
assert r[0].hour == 10

# Custom strptime format
r = parse("On {:%Y-%m-%d}", "On 2023-11-25")
assert r[0].day == 25

# Time only
from datetime import time, timezone, timedelta
r = parse("At {:tt}", "At 10:21:36 PM -0530")
assert r[0] == time(22, 21, 36, tzinfo=timezone(timedelta(hours=-5, minutes=-30)))

Custom types with extra_types and with_pattern

from parse_rust import compile, with_pattern

@with_pattern(r"[ab]")
def ab(text):
    return {"a": 1, "b": 2}[text]

parser = compile("test {result:ab}", extra_types={"ab": ab})
assert parser.parse("test a").named == {"result": 1}
assert parser.parse("test b").named == {"result": 2}
assert parser.parse("test c") is None  # regex mismatch

# Custom types can override built-in types
@with_pattern(r"\d+")
def doubler(text):
    return int(text) * 2

result = parse("{:d}", "42", extra_types={"d": doubler})
assert result[0] == 84

Deferred evaluation (evaluate_result=False)

from parse_rust import parse

match = parse("hello {}", "hello world", evaluate_result=False)
# match is a Match object — type conversion not yet performed
result = match.evaluate_result()  # triggers conversion
assert result.fixed == ("world",)

Span positions

result = parse("User {name:w} is {:d} years old", "User Alice is 30 years old")
print(result.spans)
# {0: (12, 14), 'name': (5, 10)}

Repeated named fields

from parse_rust import parse, RepeatedNameError

# Same value → match succeeds
result = parse("{name:w} {name:w}", "Alice Alice")
assert result.named == {"name": "Alice"}

# Type mismatch in repeated field → RepeatedNameError
try:
    parse("{name:w} {name:d}", "Alice 30")
except RepeatedNameError:
    pass  # expected

Case sensitivity

# Default: case-insensitive
assert parse("hello {}", "HELLO World") is not None

# Strict matching
assert parse("hello {}", "HELLO World", case_sensitive=True) is None

Search with position constraints

from parse_rust import search

# Start searching from character position 10
result = search("{:d}", "I have 3 and 5", pos=10)
assert result[0] == 5

Architecture

The core parsing engine is organized into four Rust modules, exposed to Python via PyO3:

Module	Source	Effective LOC	Responsibility
compiler	src/compiler.rs	700	Format string tokenization, format spec parsing, regex escaping, custom type registration
types	src/types.rs	463	Regex patterns for all 25 format specifiers, type conversion (int/float/Decimal/datetime), custom %... strptime translation
parser	src/parser.rs	555	Parse/search/findall orchestration, regex matching, repeated-name equality checks, deferred evaluation
result	src/result.rs	598	Result/Match encapsulation, ISO datetime normalization, microsecond precision, nested dict expansion

Python interface layer (src/lib.rs, 694 effective LOC): PyO3 bindings, RepeatedNameError exception, global parser cache (OnceLock<Mutex<ParserCache>>), and FixedTzOffset.

Performance

All benchmarks use a release build (maturin develop --release), 5 independent runs reporting mean ± std, with Python parse 1.20.2 as baseline.

Throughput (100K lines)

Path	Workload	Python baseline	Rust compiled	Speedup
parse	simple string extraction	177K l/s	1,164K l/s	6.6x
parse	mixed type parsing	77K l/s	812K l/s	10.5x
parse	log-style parsing	95K l/s	1,024K l/s	10.8x
search	simple string extraction	181K l/s	1,127K l/s	6.2x
search	log-style parsing	99K l/s	938K l/s	9.5x
findall	scan	34K l/s	145K l/s	4.2x
fallback	custom types	678K l/s	2,080K l/s	3.1x

P99 tail latency

Workload	Python baseline	Rust compiled	Improvement
simple string extraction	14.66 μs	1.10 μs	13.3x
mixed type parsing	29.51 μs	1.33 μs	22.2x
log-style parsing	29.02 μs	1.01 μs	28.7x

Rust's P50/P95/P99 latencies are tightly clustered (0.2–0.5 μs spread) — no GC pauses, no long-tail jitter.

Memory efficiency

Rust achieves 4.2x–7.7x higher memory efficiency (throughput per MB of RSS) vs. Python. RSS shows zero growth from 100K to 300K inputs, confirming bounded heap allocation and no per-record leaks.

Security

• 0 unsafe blocks — all Rust code runs within the safe subset, with memory safety guaranteed at compile time
• 32/32 adversarial input tests pass — covering large inputs (1 MiB), null bytes, Unicode edge cases (4-byte emoji, combining chars, RTL, surrogate pairs), regex injection resistance (*+?|^$\ all literalized), 8-level deep nesting, malformed format strings, type conversion boundaries, and search/findall edge cases. Zero crashes, zero panics, zero hangs.
• Clippy: 0 correctness/safety warnings (15 style/idiom-level findings only)
• Dependency audit: dependency audit is clean; pyo3 upgraded to 0.24 to resolve the only prior finding

Testing

Three-layer test evidence chain:

Layer	Command	Result
Rust unit tests	cargo test --quiet	58 passed (57 #[test] + 1 doctest)
Python integration tests	python -m pytest -q tests	151 passed, 1 skipped
Security tests	python -m pytest -q tests/test_security_inputs.py	32 passed

The 1 skip (test_parse.py::test_too_many_fields) is due to a Python version difference in upstream test conditions, not a compatibility failure.

Python integration tests comprise:

• Project API integration tests (22 tests): end-to-end coverage of all public APIs
• Upstream migration tests (7 files, 129 tests): cross-implementation behavioral consistency verification, migrated from parse-original/tests/ with minimal import changes

Project Structure

parse-rust/
├── Cargo.toml              # Rust project config
├── pyproject.toml          # Python package config (maturin)
├── build.rs                # PyO3 build script
├── LICENSE                 # MIT
├── src/                    # Rust core implementation
│   ├── compiler.rs         # Format string compiler
│   ├── lib.rs              # Python module entry point & top-level API
│   ├── parser.rs           # Core parsing engine
│   ├── result.rs           # Result / Match objects
│   └── types.rs            # Type system (25 format specifiers)
├── tests/                  # Python integration tests
│   ├── test_python_api_integration.py
│   ├── test_parse.py
│   ├── test_search.py
│   ├── test_findall.py
│   ├── test_bugs.py
│   ├── test_result.py
│   ├── test_parsetype.py
│   ├── test_pattern.py
│   └── test_security_inputs.py
└── benches/                # Benchmark scripts & results
    ├── benchmark_common.py   # shared framework
    ├── benchmark_parse_throughput.py
    ├── benchmark_search_throughput.py
    ├── benchmark_findall_throughput.py
    ├── benchmark_parse_latency.py
    ├── benchmark_fallback_throughput.py
    └── results/

License

MIT. See LICENSE.