Focused on building scalable data pipelines, robust ETL/ELT workflows, and cloud-based data solutions in large-scale environments.
The main tools and technologies I utilize to ensure data integrity and workflow efficiency.
- 🎓 Focused on Data Engineering and Big Data Systems architecture.
- 💡 Interested in scalable pipelines, real-time streaming (Kafka), and Cloud Data Warehousing (AWS/GCP).
- 🔍 Prioritizing data integrity, automated workflow orchestration, and efficient infrastructure management.
- 📈 Driven by passion for building systems that enable data-driven decision making.
- Email: [email protected]
- GitHub: github.com/lyraa88
- Established a Docker-based deployment (Serving) environment for ML models and implemented CI/CD pipeline concepts.
- Emphasis on reproducibility and operational management. - Github Repo
- Designed and implemented a real-time ingestion and streaming pipeline for sensor/audio data using Kafka. - Built automated batch ETL jobs (using Spark/Airflow) and established an AWS S3 data lake. - Github Repo
- Developed Airflow DAGs to automate the collection, transformation, and loading of user data into a PostgreSQL data mart. - Focus on designing data validation and quality check processes. - Github Repo
- Developed scripts for efficient collection, integration, and normalization of fragmented marketplace data. - Emphasis on the data preparation stage and schema design for analysis. - Paper Link