This data pipeline will programmatically fetch the latest list of Combined Print & E-Book Fiction and Combined Print & E-Book Nonfiction through NYTimes Books API and send the results to some email recipients.
Make sure you download Docker Desktop.
First, clone this repo
git clone https://github.com/zheyanliu99/NewYorkTimes-bestseller-book-pipeline.git
Make a new folder 'logs' in current directory
mkdir logs
Then, initialize docker with
docker-compose up airflow-init
Do this only if you are Linux/Mac user
echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
Docker-compose airflow (this can take 10 minutes for the first time)
docker-compose up -d
When all containers are healthy, Open http://localhost:8080
Import variables in the UI by Admin -> variables, variables file is config/variables_config.json. If you don't do this step, it will use the default email list ['[email protected]', '[email protected]', '[email protected]']. You succeed if you see something like this.
And the final directory should look like this
That's all for the setup, contact me if you met any issues or refer to this video https://www.youtube.com/watch?v=aTaytcxy2Ck&t=403s.
Graph View of the DAG
Email sent
Raise error if all the user in the email_list are sent this week
