PhotoTools - Raw Image Processing Utilities

A set of Python scripts for finding and converting raw image files to JPEG format, designed to help photographers consolidate and manage their local archives while reducing dependence on cloud storage providers.

Purpose and Vision

This project was created to help the author (Brian Nitz) declutter and consolidate his photo archive, making it easier to manage large collections of raw files locally. By providing tools to identify missing JPEG counterparts and convert raw files efficiently, photographers can:

• Reduce cloud dependency: Maintain complete local archives with both raw and JPEG versions
• Streamline workflow: Automate the process of ensuring every raw file has a corresponding JPEG
• Save storage costs: Avoid paying for cloud storage of redundant files
• Improve accessibility: Ensure JPEG versions are available for quick viewing and sharing

I hope this tool is useful to others facing similar challenges with photo archive management.

⚠️ EXPERIMENTAL SOFTWARE WARNING

This is experimental software provided as-is, without warranty.

• Data Safety: Always backup your raw image files before running these scripts
• Testing: Test on a small sample of images first before processing large collections
• Dependencies: Requires external tools (dcraw, cjpeg) that may not be available on all systems
• Performance: Processing large numbers of raw files can be time-consuming and resource-intensive

Description

This project contains two main scripts:

1. find_raw_without_jpeg.py - Scans directories for raw image files that don't have corresponding JPEG files
2. create_jpeg_from_raw.py - Converts raw image files to JPEG format using dcraw and cjpeg

Supported Raw Formats

The scripts support 21+ raw image formats including:

• .pef (Pentax)
• .raf (Fuji)
• .cr2, .cr3 (Canon)
• .nef, .nrw (Nikon)
• .arw, .srf, .sr2 (Sony)
• .dng (Adobe/Universal)
• .orf (Olympus)
• .rw2 (Panasonic)
• .raw (Generic)
• And more...

Dependencies and Sources

External Dependencies

This project depends on the following external tools:

dcraw (Raw Image Converter)

• Purpose: Converts raw camera files to PPM format
• Source: Original tool by Dave Coffin
• Website: https://www.cybercom.net/~dcoffin/dcraw/
• License: Public domain / free software
• Installation:

  # Ubuntu/Debian
  sudo apt-get install dcraw
  
  # macOS
  brew install dcraw
  
  # Windows (WSL)
  sudo apt-get install dcraw

cjpeg (JPEG Creation Tool)

• Purpose: Converts PPM files to JPEG format
• Source: Part of libjpeg-turbo / Independent JPEG Group
• Website: https://github.com/libjpeg-turbo/libjpeg-turbo
• License: BSD-style license / custom IJG license
• Installation:

  # Ubuntu/Debian
  sudo apt-get install libjpeg-progs
  
  # macOS
  brew install jpeg
  
  # Windows (WSL)
  sudo apt-get install libjpeg-progs

Python Dependencies

• Python 3.6+: Core runtime environment
• Standard Library Only: No external Python packages required
  • pathlib - File path handling
  • subprocess - External process execution
  • sys - System interface
  • os - Operating system interface
  • argparse - Command line argument parsing
  • typing - Type hints

Source Code References

The Python scripts in this project are original work but incorporate concepts from:

Raw Format Support

• Source: Common raw format extensions from industry standards
• Reference: Camera manufacturer specifications and dcraw documentation
• Formats Supported: 21+ raw formats including Canon, Nikon, Sony, Fuji, Pentax, etc.

Pipeline Architecture

• Concept: Unix philosophy of piping data between processes
• Implementation: Python subprocess with stdin/stdout communication
• Inspiration: Traditional command-line tool design patterns

File Processing Logic

• Base name grouping: Standard practice for matching raw/JPEG pairs
• Recursive directory scanning: Common filesystem traversal pattern
• Error handling: Best practices for external process execution

Installation

System Requirements

• Operating System: Linux, macOS, or Windows (with WSL)
• Python: Version 3.6 or higher
• Disk Space: 50MB for tools + space for converted JPEG files
• Memory: 512MB minimum (more for large directories)
• CPU: Multi-core recommended (dcraw is CPU-intensive)

Step-by-Step Installation

1. Install Python (if not already installed)

Ubuntu/Debian:

sudo apt-get update
sudo apt-get install python3 python3-pip

macOS:

brew install python3

Windows:

• Install Python from https://python.org
• Or use Windows Subsystem for Linux (WSL)

2. Install External Dependencies

Ubuntu/Debian/WSL:

sudo apt-get update
sudo apt-get install dcraw libjpeg-progs

macOS:

brew install dcraw jpeg

3. Download and Setup PhotoTools

# Clone the repository
git clone <repository-url>
cd phototools

# Or download and extract ZIP file
# Then navigate to the phototools directory

# Make scripts executable (Linux/macOS)
chmod +x find_raw_without_jpeg.py
chmod +x create_jpeg_from_raw.py

4. Verify Installation

# Test dcraw
dcraw --version

# Test cjpeg
cjpeg -version 2>/dev/null || echo "cjpeg available"

# Test Python
python3 --version

# Test scripts
python3 find_raw_without_jpeg.py --help
python3 create_jpeg_from_raw.py --help

Windows-Specific Setup

Option 1: Windows Subsystem for Linux (Recommended)

1. Install WSL:

   wsl --install

2. Install Ubuntu (or other distribution) from Microsoft Store

3. Install dependencies in WSL:

   sudo apt-get update
   sudo apt-get install python3 dcraw libjpeg-progs

4. Use PhotoTools from WSL terminal

Option 2: Native Windows

More complex - requires compiling dcraw and cjpeg from source:

1. Install Visual Studio Build Tools
2. Download dcraw source from https://www.cybercom.net/~dcoffin/dcraw/
3. Download libjpeg source from https://ijg.org/
4. Compile both tools (advanced users only)

Docker Installation (Alternative)

FROM ubuntu:20.04

RUN apt-get update && apt-get install -y \
    python3 \
    dcraw \
    libjpeg-progs \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY . .

CMD ["python3", "find_raw_without_jpeg.py", "--help"]

Build and run:

docker build -t phototools .
docker run -v /path/to/photos:/photos phototools python3 find_raw_without_jpeg.py /photos

Usage

Step 1: Find Raw Files Without JPEGs

# Basic usage - show missing JPEGs
python find_raw_without_jpeg.py /path/to/your/photos

# Output raw file paths for piping (recommended)
python find_raw_without_jpeg.py /path/to/your/photos --pipe

Step 2: Convert Raw Files to JPEG

# Pipe from find script (recommended workflow)
python find_raw_without_jpeg.py /path/to/photos --pipe | python create_jpeg_from_raw.py

# Specify JPEG quality (1-100, default 85)
python find_raw_without_jpeg.py /path/to/photos --pipe | python create_jpeg_from_raw.py --quality 90

# Manual usage with specific files
echo "/path/to/photo.CR2" | python create_jpeg_from_raw.py

Examples

# Find and convert all raw files without JPEGs
python find_raw_without_jpeg.py ~/Pictures --pipe | python create_jpeg_from_raw.py

# High quality conversion
python find_raw_without_jpeg.py ~/Pictures --pipe | python create_jpeg_from_raw.py --quality 95

# Check what's missing first, then convert
python find_raw_without_jpeg.py ~/Pictures
python find_raw_without_jpeg.py ~/Pictures --pipe | python create_jpeg_from_raw.py

How It Works

find_raw_without_jpeg.py

1. Recursively scans the specified directory
2. Groups files by base name (filename without extension)
3. Identifies base names that have raw files but no JPEG files
4. Can output either human-readable reports or raw file paths for piping

create_jpeg_from_raw.py

1. Reads raw file paths from stdin (one per line)
2. For each raw file:
   • Checks if JPEG already exists (skips if present)
   • Uses dcraw to convert raw to PPM format
   • Uses cjpeg to convert PPM to JPEG
   • Saves JPEG with same base name as raw file

Performance Notes

• Large directories: Scanning directories with 100,000+ files may take several minutes
• Memory usage: Minimal - processes files one at a time
• CPU usage: High during conversion (dcraw is CPU-intensive)
• Disk space: JPEG files are typically 1-10MB each

Troubleshooting

Common Issues

1. "dcraw: command not found"

   • Install dcraw: sudo apt-get install dcraw

2. "cjpeg: command not found"

   • Install libjpeg tools: sudo apt-get install libjpeg-progs

3. Permission errors

   • Ensure write permissions in the target directory
   • Use appropriate file permissions for raw files

4. Memory errors with large directories

   • Process directories in smaller chunks
   • Close other applications while processing

Debug Mode

Both scripts output progress information to stderr. Use shell redirection to separate progress from output:

python find_raw_without_jpeg.py /path/to/photos --pipe 2>find_errors.log | python create_jpeg_from_raw.py 2>convert_errors.log

License

This project is licensed under the MIT License - see the LICENSE file for details.

Development Attribution

This software was developed using Windsurf IDE with assistance from multiple AI models:

• Windsurf IDE (Codeium) - Primary development environment and AI assistant
• Cascade - AI agent for code generation and debugging
• SWE-1.5 - Software engineering model for code analysis and optimization
• swe-grep - Code search and pattern recognition model
• Claude 3.5 Sonnet - Advanced reasoning and documentation assistance

The AI tools helped with code structure, optimization, debugging, and comprehensive documentation, while the human developer (Brian Nitz) retained full creative control and approval of all generated code.

See the LICENSE file for complete attribution details.

Contributing

This is experimental software. If you encounter issues:

1. Check the troubleshooting section above
2. Test with a small sample of files first
3. Ensure all dependencies are properly installed
4. Report issues with detailed error messages and system information

Disclaimer

This software may damage or corrupt your image files. Always backup your photos before use. The author is not responsible for any data loss or damage that may occur from using this software.