Databricks as a shared service in TRE core

[m1p1h]

I'm interesting in exploring the use of databricks as a shared workspace service that sits with in the TRE core vnet to act as a data lakehouse, cataloguing and orchestration service to provision data into workspaces.

The idea being that external data ingress to the databricks service would still be performed via the existing gateway/firewall with any critical modifications needed.

From workspaces, data discovery and access to datasets in the DB lakehouse would be possible by leveraging unity catalog and workspace binding capabilities in databricks to pull in datasets.

From an implementation perspective, given there is already a databricks service bundled into AzureTRE as a workspace service (and a databricks-auth shared service) what would it take to repurpose this as a core shared service?

Would be interested to get thoughts, particularly @marrobi as I know you've looked at integrating FlowEHR, on both the feasibility/implications of doing this and more short-term what would need to be developed/adapted to repurpose the databricks bundle.

Thanks,
Mike

[marrobi]

Hi @m1p1h,

So the first thing to ask is, why would you want to deploy it via Azure TRE, rather than independently alongside the TRE using Terraform?

TRE service are primarily when we need to enable self service, as you know it adds additional layers of complexity into the deployment process. The approach usually taken when Databricks is needed as part of a Data platform/data orchestration layer is to deploy separately, as with FlowEHR. FlowEHR isn't something that we would ever deploy using AzureTRE.

The workspace service could be repurposed, albeit I think there might be some complexities around additional network subnets being required. and we don't have the functionality to add additional address spaces/subnets in core via the APIs.

Hope that helps, @tanya-borisova, @damoodamoo, @martinpeck might be able to add insight as worked on FlowEHR on a couple of recent projects.

Thanks,

Marcus

[tanya-borisova]

Just some context from a past project we did:

We did not integrate Databricks Unity catalog permissions with TRE permissions. Instead, there was a small number of users with elevated permissions (staff, or honorary contract holders), and only they could run queries on data without restrictions. The result of the final query was then copied into the TRE Workspace created for researchers, and so it could not be accessed anywhere else.
It perhaps is not ideal from the user experience POV and may not work for you, but this did provide us the guarantees we were looking for.

[m1p1h]

hi @tanya-borisova, thanks for sharing. How much of the workflow was automated or was it mainly a series of manual steps given the small number of users? Something like:

1. Researcher -> Submits data request
2. authorised user -> Runs query in databricks
   -> Moves query result to a storage container
   -> Creates workspace for researcher (or researcher self-serves themselves a workspace?)
   -> Sends researcher SAS URI to storage container (or is the workspace populated with the data at creation?)
3. Researcher -> accesses data container in permissioned TRE workspace

An approach where a TRE workspace is pre-canned with a descriptive catalog that researchers can access for discovery purposes and then a data request mechanism where a authorised user/data engineer is able to run the query on the databricks lakehouse to get the data and make it available in the requesters TRE workspace would help simplify the security model.

Cheers,
Mike

[damoodamoo]

Our process went:

• TRE workspace created first
• researcher worked 'offline' with a data manager to define the exact cohort query.
• researcher uses a separate hand-made portal to create a data access request, pasting in the query
• data manager / approver approves the request
• ADF pipeline invoked, passing in the TRE workspace ID and query.
• databricks step in the pipeline gets parameters, runs query, pushes the data to the storage account in the workspace.
• researchers in the TRE are given data reader permissions to the workspace storage account, so they can use storage explorer from their VM to download the data to their VM.

There's lots in there that is not ideal - the access request being a separate portal, having to have the workspace created already, the offline work needed to get the query. Lots of rough edges but it did the job for an MVP.

I probably have a vanilla demo of the work we did I could share privately - email me at [email protected] if interested to see?

[tanya-borisova]

@m1p1h from the process @damoodamoo outlined above, the last 3 steps are automated. The workspace is presumed to already be created, but the data provisioning happens in an automated way. We have, and can share, a workspace template we used for this, as well as the separate portal for access requests, and the Databricks pipeline used for provisioning. Happy to talk through this on a call if this helps - my email is [email protected]

[vvcb]

Thanks to everyone for helping us navigate this. The NWSDE is (or possibly not) in a slightly unusual situation with regards to where the datasets that are eligible for discovery through a metadata catalogue are located. This data will predominantly be outside the NWSDE.

What is the NWSDE ? NWSDE = AzureTRE and Gold/Gold-plus Lakehouse and Data Pipelines.

The NWSDE Lakehouse is meant to be thin/light - both in how much data will be in there and what operations will be performed on this data. The data will already have been curated and cohorted at source(s) based on the data access request. 'Project-specific' datasets will need to be moved into the NWSDE data lake for a small number of checks and processes before being moved into their corresponding workspace.

Loading

flowchart LR

GM[Greater Manchester - SnowFlake]:::srcClass 
LSC[Lancs & South Cumbria - Azure DataBricks]:::srcClass 
CM[Cheshire & Merseyside and AGEM National Data Assets - Azure ]:::srcClass 
HDRUK[HDRUK Gateway]
NWSDEGold[NWSDE DataLake Gold Layer]
NWSDEGoldPlus[NWSDE DataLake Gold+ Layer]
TRE[TRE Workspace]

GM -.->|metadata|HDRUK 
LSC -.->|metadata|HDRUK 
CM -.->|metadata|HDRUK 

GM -->|Project-specific data|NWSDEGold
LSC -->|Project-specific data|NWSDEGold
CM -->|Project-specific data|NWSDEGold

subgraph Azure-DataBricks
  NWSDEGold --> NWSDEGoldPlus
end

NWSDEGoldPlus --> TRE

classDef treClass fill:#f96
classDef srcClass fill:#cf0

What we are trying to decide is where the bit in the yellow box should sit in relation to AzureTRE - especially if it is not acting as a massive data store.

The flow of project-specific data happens after the research team's data access request has been approved.

1. Should we make it part of the AzureTRE infrastructure - reusing the firewall, networking, etc.?
2. Should we build a separate data platform infrastructure for maximum flexibility but associated costs - engineering, maintenance, consumption, etc?