Harnessing the power of AIRR supercomputers for trusted research

FRIDGE

Harnessing the power of AIRR supercomputers for trusted research

Jim Madge

Alan Turing Institute

Why sensitive data?

Data Ecosystem by Scriberia. Used under CC-BY 4.0 10.5281/zenodo.13882307

Why trusted research environments?

Sensitive Data by Scriberia. Used under CC-BY 4.0 10.5281/zenodo.13882307

Why HPC?

Machine Learning Reusable Pipelineby Scriberia, modified. Used under CC-BY 4.0 10.5281/zenodo.13882307

TREs are not HPC

Security is top priority
Lack HPC hardware
Scaling is expensive
Often designed for a single domain

- **TREs are not HPC** - Put security above all else - Generally lack HPC hardware - Often build on cloud systems - Great ability for "X" as code (policy, infra, networking) - Performance hit through virtualisation - Poor scaling - Cloud costs for HPC/GPU nodes are high, and availability may be low - Scaling on prem can be expensive, capital cost for hardware, operating costs power, cooling - Can be inflexible, designed for a small group of users or domain with only the tools they need

HPC is not a TRE

Designed for performance or throughput
Shared systems
Public
Few controls for data ingress and egress

FRIDGE

A ready-to-use SATRE compliant TRE that can be deployed to AIRR

Shared responsibility

Need agreement between TRE operator and infrastructure provider

Extend TRE governance to a secure enclave on HPC (the TRE tenancy)

Delegate information governance to the TRE operator

TRE operator 🧑‍⚖️

User management
Data processing
Data ingress and merging
Release of results

Infrastructure provider 🧑‍🔧

Securing TRE tenancy
Providing FRIDGE-compatible K8s
- GPU passthrough
- Compatible CNI

FRIDGE architecture

- TRE tenant, secured/configured by HPC provider, which FRIDGE is deployed in to - Connected to external TRE by VPN - Users in TRE submit jobs or request outputs - Advantages - Considered part of existing TRE, under it's governance - Outputs do not need to be classified, they go to a TRE - Small footprint on HPC - Promote reproducibility with workflows - Disadvantages - "I want a notebook" - "I want a desktop" - Probably slow for iterative, exploratory work - **LINK**: In more detail, our implementation

FRIDGE implementation

Security

Inside K8s 🚢

RBAC
Pod security standards
Network policy
Automatic encryption of volumes

On the host 🏢

Site-to-site VPN
TRE tenancy isolation

- Inside K8s - Users jobs have a service account which prevents them from interacting with the kubeAPI (RBAC) - PSS ensures pods have no privileges on the host, _e.g._ root, networking, except to use GPUs - Traffic within the cluster is heavily restricted by network policy, to only what is necessary - K8s volumes are automatically encrypted when provisioned - Securing the boundary - Site-to-site VPN means no endpoints are exposed publicly - Hypervisor isolation of K8s host - The physical host is not not compromised if the VM is - All of this means, - Users have very little privilege - They cannot access the host or modify the K8s spec - If they do somehow manage to compromise the VM, physical host is still protected, tenants are still isolated - If the host system is compromised, the data is encrypted with keys not available - **LINK**: Where have we gotten to?

Progress

Project

Shared responsibility model ✅
TRE tenancy ✅
MVP implementation on Azure ✅
Job submission API 🚧

on Dawn

On-demand K8s cluster ✅
GPU passthrough ✅
Fridge development deployment ✅
S2S VPN 🚧
PVC encryption 🚧

Lessons

Multi-party governance is difficult
Self-service HPC has arrived
K8s implementations differ
Don't reinvent the wheel

Future

Cardiac digital twin

🫀

Scaling to many nodes

🚀

Confidential computing

🔐

- **What's next*** - Workflow using cadiac data as part of a digital twin project on Dawn - Necessary to prove we can resolve governance! - To truly make the most of HPC, we need to show we can scale to tens or hundreds of nodes/GPUs - There are challenges in keeping a tenancy secure across many nodes we need to explore - As it stands, a privileged user on the host _could_ compromise FRIDGE - We need to trust the HPC provider - Confidential computing technology could remove the need for that trust, even a motivated and privileged attacker couldn't read FRIDGE data in memory - Also adds support for bring your own compute, when you can't or don't want to configure host for high security

Acknowledgements

Banner Thanks by Scriberia. Used under CC-BY 4.0 10.5281/zenodo.13882307

Development Team

DARE UK

Funding this work through the DARE UK Early Adopters programme

Scriberia and The Turing Way

Who we work with to make the excellent, openly licensed, illustrations

Try it out and contribute

alan-turing-institute/fridge

Explore other DARE UK activities

Early Adopters
TREvolution

Get involved

uktre.org
satre-specification.readthedocs.io