electricity_generation

Overview

This project is an end-to-end data engineering pipeline that ingests global electricity generation data from the Ember Energy API, processes it using PySpark, and stores it in a Delta Lake table for analytics and dashboarding.

The goal is to analyze and compare electricity generation across Africa and Europe, broken down by energy source (Solar, Wind, Coal,   Bioenergy, etc.) and measured in TWh (terawatt-hours).

Objectives

Ingest electricity generation data from an external API
Handle multi-country and paginated API responses
Transform and clean raw data using PySpark
Enrich dataset with region mapping (Africa vs Europe)
Store processed data in Delta Lake (Unity Catalog)
Enable analytics-ready datasets for dashboards

Architecture

Ember Energy API
        
Python (requests)
        
Raw JSON response
        
PySpark Transformation Layer
        
Data Cleaning + Region Mapping
        
Delta Lake (Bronze Table)
        
Analytics / Dashboard (Power BI / Databricks SQL)

Tech Stack

Python (requests, pandas)
PySpark (Databricks)
Delta Lake
Databricks Unity Catalog
Ember Energy API
SQL (for exploration & dashboards)

Dataset Description

Each record represents electricity generation for a country by energy source.

 Key Columns:
    Column	            Description
    entity	            Country name
    entity_code	ISO     country code
    date	              Year of observation
    source	            Energy type (Solar, Wind, Coal, etc.)
    generation_twh	    Electricity generated (TWh)
    share_of_generation_pct	  Percentage share of total generation
    region	            Africa or Europe
    ingestion_time	    Timestamp of data load