grobid-papers

A production-grade Rust library for extracting structured data from scientific research papers in PDF format. Built on GROBID with GraalVM native compilation for high performance and seamless Rust integration.

Features

• Comprehensive Extraction: Extract titles, authors, abstracts, sections, references, and metadata from scientific PDFs
• Structured Output: Parse TEI XML into strongly-typed Rust domain models with full serialization support
• High Performance: Native code compilation via GraalVM with embedded JVM runtime (no external Java dependency)
• Production Ready: Type-safe error handling, extensive logging, and thread-safe concurrent processing
• Zero Configuration: Automatic GraalVM download and setup on first build
• Rich Metadata: Extract DOIs, ArXiv IDs, publication dates, keywords, and bibliographic references
• Hierarchical Sections: Preserve document structure with nested sections and bounding box coordinates
• Reference Parsing: Detailed extraction of citations including titles, authors, venues, and dates

Quick Start

Add to your Cargo.toml:

[dependencies]
grobid-papers = "0.1.0"

Extract data from a PDF:

use grobid_papers::{PaperProcessor, Result};

fn main() -> Result<()> {
    // Create processor with default configuration
    let processor = PaperProcessor::new();

    // Extract structured data from a PDF
    let paper = processor.extract_from_file("paper.pdf")?;

    // Access extracted data
    println!("Title: {:?}", paper.title);
    println!("Authors: {}", paper.authors.len());

    for author in &paper.authors {
        println!("  - {}", author.full_name());
    }

    println!("Sections: {}", paper.sections.len());
    println!("References: {}", paper.references.len());

    if let Some(doi) = &paper.doi {
        println!("DOI: {}", doi);
    }

    // Serialize to JSON
    let json = serde_json::to_string_pretty(&paper)?;
    println!("{}", json);

    Ok(())
}

Installation

Prerequisites

• Rust 1.70+ (2021 edition)
• 2GB disk space for GraalVM (auto-downloaded on first build)
• Internet connection for initial setup

Supported Platforms:

• Linux x86-64 (amd64)
• Linux ARM64 (aarch64) - includes custom-built native dependencies
• macOS x86-64 and ARM64 (M1/M2/M3)
• Windows x86-64

First Build

The first build will automatically:

1. Download GraalVM JDK if not present
2. Compile GROBID to native code via Gradle
3. Build the Rust crate

This takes 5-10 minutes. Subsequent builds are fast (~30 seconds).

cargo build --release

Manual GraalVM Setup (Optional)

If you prefer to manage GraalVM yourself:

export GRAALVM_HOME=/path/to/graalvm
cargo build --release

Usage Examples

See the examples/ directory for complete working examples:

• extract_paper.rs - Basic PDF extraction with detailed output
• benchmark.rs - Performance benchmarking and concurrent processing

Quick example:

use grobid_papers::PaperProcessor;

let processor = PaperProcessor::new();
let paper = processor.extract_from_file("paper.pdf")?;
println!("Title: {:?}", paper.title);
println!("Authors: {}, Sections: {}, References: {}",
    paper.authors.len(), paper.sections.len(), paper.references.len());

For advanced configuration, async processing, and accessing document structure, see the examples directory.

Architecture

The library uses a layered architecture for maintainability and performance:

┌─────────────────────────────────────┐
│   Public API (PaperProcessor)       │
├─────────────────────────────────────┤
│   Domain Models (Paper, Author)     │
├─────────────────────────────────────┤
│   GROBID Integration (JNI Layer)    │
├─────────────────────────────────────┤
│   Native Library (libtika_native)   │
│   Embedded GraalVM + GROBID         │
└─────────────────────────────────────┘

Key Components

• Public API: PaperProcessor - Simple, ergonomic interface for users
• Domain Models: Strongly-typed Rust structs with Serde support
  • Paper - Complete paper with all extracted data
  • Author, Section, Reference - Domain entities
  • PaperMetadata, RawMetadata - Metadata types
• GROBID Integration (internal): JNI bindings to GROBID Java code
  • extractor.rs - Calls GROBID via JNI to extract TEI XML
  • parser.rs - Parses TEI XML into Rust domain models
  • Uses embedded GraalVM runtime from native library
• Configuration: GrobidConfig for extraction options

Technology Stack

• GROBID: Machine learning-based PDF extraction (Java)
• GraalVM Native Image: Compiles Java to native code with embedded runtime
• JNI: Java Native Interface for Rust-Java interop
• quick-xml: Fast XML parsing for TEI documents
• Serde: Serialization/deserialization support

Native Dependencies

GROBID requires platform-specific native libraries:

• Wapiti - CRF sequence labeling for ML models (custom-built for ARM64)
• pdfalto - PDF to ALTO XML converter (custom-built for ARM64)

These are included in grobid-home/ and automatically loaded at runtime. ARM64 builds were custom-compiled from source for Linux ARM64 support.

Configuration

GrobidConfig Options:

• consolidate_header - Enrich metadata with CrossRef (slower but more accurate)
• consolidate_citations - Enrich references with external bibliographic databases
• include_raw_citations - Keep raw citation strings from PDF
• segment_sentences - Enable sentence-level segmentation

See API documentation for complete configuration details.

Environment Variables:

• GROBID_HOME - Path to GROBID models directory (auto-detected if not set)
• GRAALVM_HOME - Path to GraalVM installation (auto-downloads if not set)
• JAVA_HOME - Path to Java installation (optional)
• RUST_LOG - Logging level (e.g., RUST_LOG=info)

Building from Source

Standard Build

git clone https://github.com/9prodhi/grobid-papers.git
cd grobid-papers
cargo build --release

Running Examples

# Extract a paper and print results
cargo run --example extract_paper -- /path/to/paper.pdf

# With debug logging
RUST_LOG=debug cargo run --example extract_paper -- paper.pdf

# Run performance benchmarks
cargo run --release --example benchmark -- tests/papers --threads 8

Running Tests

cargo test

Clean Build

# Remove all build artifacts
cargo clean
rm -rf target/

# Remove downloaded GraalVM (workspace-level)
rm -rf ../.build-deps/

Performance

• Extraction Time: 0.9-5 seconds per paper (varies with PDF complexity)
• Memory Usage: ~500MB-1GB (includes JVM and ML models)
• First Extraction: +21 seconds overhead for JavaVM initialization and model loading (one-time cost)
• Concurrent Processing: Fully supported (each thread gets its own JNI environment)
• Throughput: 17.4 papers/minute (single-threaded), 86.2 papers/minute (8 threads)

Benchmarks

Real-world benchmarks on Linux ARM64 system with 20 cores, tested with 17 scientific papers (0.4-46 MB):

Operation	Time	Notes
Simple paper (10-20 pages)	0.9-2.5s	Basic structure, moderate references
Medium paper (20-40 pages)	3.0-4.8s	Complex structure, many sections
Large paper (40+ pages)	3.7-4.8s	Dense references, figures, tables
First extraction (cold start)	+21s	JavaVM init + ML model loading (one-time)
Single-threaded throughput	17.4 papers/min	Average: 3.4s/paper, Median: 2.3s/paper
Concurrent (8 threads)	86.2 papers/min	Wall-clock: 11.8s for 17 papers (5x speedup)

Run Your Own Benchmarks:

# Download test papers (18 recent ML papers from arXiv)
python3 scripts/download_papers.py

# Run comprehensive benchmarks
cargo run --release --example benchmark -- tests/papers --threads 8 --detailed

See scripts/download_papers.py for the paper download utility and examples/benchmark.rs for benchmark implementation.

Performance Notes:

• Cold start (21s) only occurs on first extraction after process start
• Subsequent extractions are fast (0.9-4.8s) depending on paper complexity
• File size doesn't strongly correlate with processing time
• Concurrent processing shows excellent scaling (5x speedup with 8 threads)
• Processing time depends more on document structure complexity than file size

Troubleshooting

"GROBID not available" error

Cause: GROBID models not found

Solution:

# Check that grobid-home exists
ls grobid-home/

# Verify it contains models (~500MB)
du -sh grobid-home/

# Set GROBID_HOME explicitly if needed
export GROBID_HOME=/path/to/grobid-home

Build fails with "GraalVM not found"

Cause: Cannot download GraalVM or GRAALVM_HOME is invalid

Solution:

# Ensure internet connection for auto-download
# Or set GRAALVM_HOME manually
export GRAALVM_HOME=/path/to/graalvm-jdk-21
cargo build

Native library link errors

Cause: Native library compilation failed

Solution:

# Clean and rebuild
cargo clean
cargo build --release

# Check build output for Gradle errors
# Look for "Successfully built tika_native libs" message

Extraction fails with JNI errors

Cause: JVM initialization or JNI call failed

Solution:

# Enable debug logging
RUST_LOG=debug cargo run --example extract_paper -- paper.pdf

# Check GROBID_HOME is valid
echo $GROBID_HOME

# Verify native library exists
find target/ -name "libtika_native.*"

Out of memory errors

Cause: Processing very large PDFs or many concurrent extractions

Solution:

• Process PDFs sequentially or limit concurrency
• Increase system memory
• Split large PDFs into smaller chunks

Thread Safety

• PaperProcessor is Clone and thread-safe - share across threads freely
• The global JavaVM is initialized once and shared safely
• Each thread gets its own JNI environment automatically
• Concurrent processing fully supported (see examples/benchmark.rs)

Project Status

This library is in active development and suitable for production use.

Completed:

• Core extraction pipeline
• Domain models with full Serde support
• Configuration management
• Error handling
• GROBID JNI integration
• GraalVM native compilation
• Thread-safe concurrent processing
• Comprehensive examples

Planned:

• Docker image with pre-built native libraries

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Guidelines

• Follow Rust idioms and style (use cargo fmt)
• Add tests for new features
• Update documentation
• Ensure cargo clippy passes
• Run cargo test before submitting

Origin and Attribution

This library is based on the extractous project and builds upon:

• GROBID - Machine learning for scholarly documents
• Apache Tika - Content detection and extraction
• GraalVM - High-performance JVM and native compilation

Acknowledgments

• The GROBID team for their excellent machine learning models
• The Extractous project for inspiration and initial implementation
• The Rust community for amazing tools and libraries
• GraalVM team for native compilation technology

---