Skip to content

project-environment-variables.md

Latest commit

 

History

History
114 lines (71 loc) · 3.99 KB

project-environment-variables.md

File metadata and controls

114 lines (71 loc) · 3.99 KB
Raw

Project environment variables

To run either locally or on Azure DevOps, this set of scripts need a few environment variables to be set.

This document lists and describes each one of them.

Cluster Management Environment Variables

DATABRICKS_DOMAIN

The Databricks instance name.

DATABRICKS_ACCESS_TOKEN

The Databricks Personal Access Token you have generated on your workspace.

DATABRICKS_CLUSTER_NAME_SUFFIX

Optional. A suffix you can use to identify all of your clusters. If you choose not to set one, all of your clusters will be called aml-cluster.

DATABRICKS_CLUSTER_ID

Use this variable if you prefer to use an existing cluster to run your train pipeline. Here, you can set the chosen cluster id. Check this doc for more information about cluster ids.

Train Environment Variables

AML_WORKSPACE_NAME

The name of your Azure Machine Learning Service Workspace. Amongst all the generated infrastructure resources, this will have the -AML-WS suffix.

RESOURCE_GROUP

The name of the Azure Resource Group that contains all of your workloads (Databricks Workspace, Azure ML Service, etc.). Amongst all the generated infrastructure resources, this will have the -AML-RG suffix.

SUBSCRIPTION_ID

The ID of the Azure Subscription that is being used to run this infrastructure.

TENANT_ID

The ID of the Azure Active Directory Tenant associated with the Azure Subscription

SP_APP_ID

The Application ID of the Service Principal you have created to run this code.

SP_APP_SECRET

The Application Secret (or password) of the Service Principal you have created to run this code.

SOURCES_DIR

The root folder of this code.

On a local development environment

For example: if you have cloned this repository on /home/username/projects and you have not customized the name of the repository local folder, the SOURCES_DIR would be /home/username/projects/MLOpsDatabricks.

On Azure DevOps

If you're associating this repo with an Azure DevOps pipeline without much customization to clone the repo, this variable can set as $(Build.SourcesDirectory).

TRAIN_SCRIPT_PATH

If you use this code structure as is, the current train script path is src/train/train.py.

DATABRICKS_WORKSPACE_NAME

The name that was given to your Azure Databricks Workspace. Amongst all the generated infrastructure resources, this will have the -AML-ADB suffix.

DATABRICKS_COMPUTE_NAME_AML

The name you want to give to the compute target that will be associated with the Azure ML Service Workspace. This will attach the Databricks Cluster to the Azure ML Service Workspace.

MODEL_DIR

The directory to save the trained model to, on the Databricks Cluster. It's recommended to use /dbfs/model.

MODEL_NAME

The name to give to the trained model. For example: my-trained-model.

Read more