Data Science Workflow Template

Security and Collaboration

This template implements a comprehensive security strategy for managing sensitive credentials while enabling safe collaboration with external contributors and clients. The approach uses GitHub repository permissions to create clear boundaries between code access and credential access, enabling different collaboration models for core teams, external contributors, and client stakeholders.

📋 Complete Security and Collaboration Plan - Detailed guide covering access control strategies, GitHub Actions integration, and local development workflows.

Set-up and installation

• Use this template to create your own project:
  • Option 1: Using GitHub website
    • Visit the template repository at https://github.com/n8layman/project-template
    • Click the green "Use this template" button and select "Create a new repository"
    • Fill in your repository name and description
    • Choose public or private visibility as needed
    • Click "Create repository from template"
  • Option 2: Using GitHub CLI
    • Install GitHub CLI if you haven't already (https://cli.github.com/)
    • Run the following command in your terminal:

      gh repo create my-new-project --template n8layman/project-template --private --clone
      

    • This will create a private repository called "my-new-project" and clone it to your local machine
    • Modify the command as needed, changing the repository name and visibility option
• If you didn't use the --clone option, clone your new repository:
  • In the terminal enter git clone https://github.com/yourusername/your-repo-name.git in a suitable directory

Common Features

Shiny Dashboard Development

This template is configured for developing interactive Shiny dashboards that can be deployed using Shinylive for zero-infrastructure, browser-based execution.

Dashboard Features

• Zero Infrastructure: Runs entirely in the browser using Shinylive
• Easy Sharing: Simple URL distribution to clients
• Automatic Updates: Deploys on code push via GitHub Actions
• No Setup Required: Works in any modern browser
• Private Repository with Public Dashboard: Source code remains confidential while dashboard is publicly accessible (requires GitHub Pro - ~$4/month)
• Secure API Management: API keys and sensitive credentials stored in GitHub Secrets

Development Workflow

1. Build dashboard with reactive components using Shiny
2. Test locally during development
3. Push to repository triggers automated deployment
4. Share GitHub Pages URL with clients

The dashboard runs client-side with no server connection required after initial load, making it perfect for client deliverables.

GitHub Actions Integration

This project comes with pre-configured GitHub Actions workflows to automate routine tasks and ensure project quality. GitHub Actions help maintain consistency and reduce manual effort through continuous integration and automation.

Automatic Bibliography Generation

As an example of how this works, one of the workflows in this template automatically processes academic papers and generates a BibTeX bibliography. When you add PDF papers to the resources/papers/ directory and push to GitHub, the workflow:

1. Extracts metadata from the PDFs
2. Searches for DOIs and retrieves complete citation information
3. Generates/updates a consolidated BibTeX file
4. Commits the changes back to the repository

This feature is particularly useful for:

• Maintaining an up-to-date references for your project
• Ensuring consistent citation formatting
• Making citations immediately available to all team members
• Enabling easy citation in R Markdown and Quarto documents

*The workflow uses the pdftools, rcrossref, and bibtex R packages to process papers. You can view the full implementation in .github/workflows/process-papers.yml and R/process_papers.R.

Here's the resulting bibtex file generated from one of my papers!

Secure Credential Management

This project uses GitHub Secrets to securely store sensitive information like API keys and access tokens. Secrets are automatically available to GitHub Actions workflows and can be accessed locally using the act tool.

Setting up GitHub Secrets

1. Navigate to your repository's Settings → Secrets and variables → Actions
2. Add your API keys and sensitive credentials as repository secrets
3. Reference them in workflows using ${{ secrets.SECRET_NAME }}

Local Development with Act

For local testing of GitHub Actions workflows that use secrets:

1. Install Act: Follow instructions at nektos/act
2. Create secrets file: Create .secrets file in your project root (add to .gitignore)
3. Run workflows locally: act -s GITHUB_TOKEN=your_token --secret-file .secrets

This allows you to test your complete pipeline locally before pushing to GitHub, including access to the same secrets used in production.

VSCode Integration

This project is configured to work seamlessly with Visual Studio Code for both R and Python development. Notes for getting that going are available here

R Workflow

Environment Setup

This pipeline was created using R version 4.4.2 and the {renv} framework to record R package dependencies and versions. Packages and versions used are recorded in the renv.lock.

• Duplicate the R environment used for the analysis:
  • This project was created using R version 4.4.2. This and other versions of R are available on the R Archive Network
  • This project uses the {renv} framework to record R package dependencies and versions. Packages and versions used are recorded in the renv.lock file.
  • To install the {renv} package run install.packages("renv")
  • Run renv::hydrate() to copy whatever packages are already available in your user / site libraries into the project library
  • Run renv::restore() to install any remaining required packages into your project library.

Machine Learning with tidymodels

This project uses the {tidymodels} framework for modeling and machine learning workflows. {tidymodels} is a collection of packages for modeling and machine learning using tidyverse principles. More information on {tidymodels} can be found here.

Targets workflow

The project pipeline has been automated using the the {targets} pipeline management tool. {Targets} breaks down the analysis into a series of discrete, skippable computational steps. Individual steps can be loaded into memory using tar_load(target_name). An overview of the pipeline can be found in the _targets.R file in the main project folder. Each component, such as data ingest, is further broken out into its own collection of targets as indicated in the _targets.R file.

Targets are organized into distinct groups, including:

1. Data ingest targets
2. Data processing targets
3. Analysis targets
4. Output targets

All targets are defined within the _targets.R file.

Re-running computationally expensive targets

Some targets are computationally intensive and long-running. The output of these targets has been saved in the \data folder as compressed RDS files. These files end in '.gz' and can be manually accessed using read_rds() or automatically through the targets pipeline. By default these steps will not be re-computed and are disconnected in the pipeline DAG unless a flag is set in the .env file. A description of these flags can be found at the top of the _targets.R and in the .env file.

Dockerized RStudio server

A dockerized container is available that provides a pre-configured environment with RStudio Server based on rocker/rstudio:latest, optimized for machine learning workflows. It comes with {targets}, {tarchetypes}, {tidymodels}, {dbarts}, and {mgcv} pre-installed, enabling users to quickly build, tune, and deploy predictive models without the hassle of configuring dependencies or installing packages manually. The Dockerfile can be built on both arm64 and x86 architectures.

A pre-built image for x86 systems is available at n8layman/docker-rstudio. This image is automatically built and deployed whenever changes are made to the Dockerfile. More information on how to start up rocker based containers including using environment variables to set the container user and the password is available at rocker-project.org.

An example docker-compose file is also available in the docker/rstudio folder.

Python Workflow

Environment Setup with Pixi

This project uses Pixi for environment management. Pixi is a package management and environment tool that ensures reproducible Python environments across different systems.

Setting up the Python environment

1. Install Pixi
   • Follow the installation instructions at Pixi's official documentation
   • In most systems, you can install with: curl -fsSL https://pixi.sh/install.sh | bash
2. Activate the Python environment
   • Navigate to the project directory
   • Run pixi shell to activate the environment
   • This loads all Python dependencies specified in the pixi.toml file
3. Available Python modules
   • The environment includes commonly used data science packages:
     • numpy, pandas, scikit-learn, matplotlib
     • jupyter, ipykernel
     • dvc for pipeline management
     • Other specialized packages for geospatial analysis and machine learning

Python Pipeline Management with DVC

For Python projects, this template uses DVC (Data Version Control) for pipeline management, providing similar functionality to R's {targets} framework:

• Dependency tracking: Automatically runs only changed pipeline stages
• Reproducible workflows: Ensures consistent results across environments
• Experiment tracking: Compare different models and parameters
• Version control integration: Works seamlessly with Git workflows
• Flexible for epidemiological forecasting: Supports data ingestion → cleaning → modeling → forecasting → visualization pipelines

Working with Python notebooks

• With the Pixi environment active, use jupyter notebook or jupyter lab to access interactive notebooks
• Notebooks are stored in the notebooks/ directory
• All Python dependencies are locked in the pixi.lock file, ensuring reproducibility

Mixed R and Python Workflows

• Integrating R and Python:
  • Use the {reticulate} package in R to call Python functions
• Environment compatibility:
  • The Pixi environment is configured to be compatible with the project's Docker container

Misc

A hook to prevent users from commiting files greater than GitHub's 100Mb file size limit is available in the .githooks folder. To enable this copy the .githooks/pre-commit file to the .git/hooks directory by running the following command in the terminal within the project base directory

cp .githooks/pre-commit .git/hooks/pre-commit

Going Further

This template provides a solid foundation for reproducible research and analysis projects. Here are some suggestions for extending the template based on your project's specific needs:

Deployment & Hosting

• Shiny Applications: Add deployment configurations for shinyapps.io or Shiny Server
• Plumber APIs: Create RESTful APIs to expose your models
• Quarto Publishing: Set up workflows for publishing interactive documents and dashboards
• Static Site Generation: Configure GitHub Pages for documentation sites

Advanced CI/CD

• Add GitHub Actions workflows for:
  • Automated testing of R and Python code
  • Scheduled data updates
  • Container building and registry publishing
  • Automatic documentation updates

Database Integration

• Set up connections to:
  • SQL databases with {duckdb} and {dolt}
  • {arrow}

This project uses

• targets to ensure that the analysis is reproducible.
• gitflow to manage project development.
• renv to manage the analysis environment and package versions
• rocker-project.org based container images
• pixi to manage Python environments and dependencies
• DVC for Python pipeline management and reproducible workflows
• This project supports Visual Studio Code for integrated development.
• tidymodels for machine learning workflows.
• act for local GitHub Actions testing.