🚀 MiniETL

A lightweight, modern ETL (Extract, Transform, Load) pipeline built with .NET 8 for processing Excel data into SQLite databases.

✨ Features

• 📊 Extract data from Excel files using ExcelDataReader
• 🔄 Transform data with custom business logic
• 💾 Load data into SQLite database
• 🏗️ Modular architecture with separate projects (Core, DB, Console, Reports)
• ⚙️ Configuration-driven with appsettings.json
• 🚀 Modern C# with .NET 8 and async/await patterns

📋 Table of Contents

• Architecture
• Quick Start
• Project Structure
• Usage
• Technology Stack
• License

🏗️ Architecture

MiniETL/
├── ETLCore/        # Core ETL logic and interfaces
├── ETLDB/          # Database operations and initialization
├── ETLConsole/     # Console application entry point
└── ETLReports/     # Report generation using ClosedXML

🚀 Quick Start

Prerequisites

• .NET 8 SDK
• Visual Studio 2022 or VS Code (recommended)

Installation

# Clone the repository
git clone https://github.com/yourusername/MiniETL.git
cd MiniETL

# Restore dependencies
dotnet restore

# Build the solution
dotnet build

# Run the ETL console
cd ETLConsole
dotnet run

💻 Usage

The application will:

1. Extract data from ./data/sales.xlsx
2. Transform sales data into structured records
3. Load data into SQLite database at ./data/mini_etl.db

# Run the ETL process
dotnet run --project ETLConsole

Sample Output

Database initialized and 'Sales' table ready.
Transformed 150 valid records.
Successfully read 150 rows from Sheet1
Sample Data:
Product A | 10 | 29.99
Product B | 5 | 49.99
...

📁 Project Structure

MiniETL/
├── ETLCore/
│   ├── IJob.cs              # Job interface
│   ├── JobRegistry.cs       # Job registration
│   ├── JobRunner.cs         # Job execution engine
│   └── TransformServices.cs # Data transformation logic
│
├── ETLDB/
│   ├── DbInitializer.cs     # Database setup
│   ├── LoadService.cs       # Data loading
│   └── ExcelReaderService.cs # Excel reading
│
├── ETLConsole/
│   └── Program.cs           # Application entry point
│
└── ETLReports/
    └── ReportGenerator.cs   # Report generation

🔧 Technology Stack

• .NET 8 - Modern, cross-platform runtime
• ExcelDataReader - Excel file parsing
• Microsoft.Data.Sqlite - SQLite database operations
• ClosedXML - Excel report generation
• Microsoft.Extensions.Configuration - Configuration management

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE.txt file for details.

👨‍💻 Author

Made with ❤️ by [Mohd]

---