eospilot/eulake configuration documentation

.github/workflows/merge-check-paths.yml

@@ -5,10 +5,12 @@ on:
   pull_request:
     paths:
       - 'infrastructure/cluster/flux/**'
+      - '**.tf'
   push:
     branches:
       - main
     paths:
+      - '**.tf'
       - 'infrastructure/cluster/flux/**'
 
 jobs:

README.md

@@ -10,9 +10,11 @@ VRE links:
  - Code: https://github.com/vre-hub/vre/ 
  - User documentation: https://vre-hub.github.io/ 
  - Technical documentation: https://github.com/vre-hub/vre/wiki
+   - :construction: Ongoing migration: https://vre-hub.github.io/docs/tech-docs/home 
  - VRE file transfer monitoring: https://monit-grafana-open.cern.ch/d/PJ65OqBVz/vre-rucio-events?orgId=16
  - Live status of the VRE services: https://vre-hub.github.io/status/ 
- - VRE Slack channel: [invitation link](https://join.slack.com/t/eosc-escape/shared_invite/zt-1zd76ivit-Z2A2nszN0qfn4VF6Uk6UrQ).
+ - ESCAPE Mattermost Team: [invitation link](https://mattermost.web.cern.ch/signup_user_complete/?id=zqaa9p5fqfd9bnnc64at4b5aye&md=link&sbr=su).
+   - :exclamation: Afterwards please join the `VRE Support` channel
 
 
 [![flux check pipeline](https://github.com/vre-hub/vre/actions/workflows/merge-check-paths.yml/badge.svg)](https://github.com/vre-hub/vre/actions/workflows/merge-check-paths.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) 
@@ -31,4 +33,4 @@ To cite us, please use the latest publication:
 
 ## Contact 
 
-Email the CERN VRE team: `escape-cern-ops'at'cern.ch`
+Email the CERN VRE team: `escape-cern-ops'at'cern.ch`

containers/rucio-noise/produce_noise.sh

@@ -21,32 +21,38 @@ echo '*   RUCIO_SCOPE = '"$RUCIO_SCOPE"''
 echo '*   FILE_LIFETIME = '"$FILE_LIFETIME"''
 
 upload_and_transfer_and_delete () {
+
     for (( i=0; i<$len; i++ )); do
 
-        if [ $1 != $i ]; then
+        echo '*** ======================================================================== ***'
+        echo '*** '"${rses[$i]}"' ***'
+
+        RANDOM_STRING=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
+        echo '*** generated random file identifier: '"$RANDOM_STRING"' ***'
+        filename=/home/auto_uploaded_${RANDOM_STRING}_source${rses[$i]}
+        did=auto_uploaded_${RANDOM_STRING}_source${rses[$i]}
+
+        echo '*** generating '"$FILE_SIZE"' file on local storage ***'
+        head -c $FILE_SIZE < /dev/urandom  > $filename
+        echo '*** filename: '"$filename"' ***'
+
+        echo '*** uploading filename: '"$filename"' to '"${rses[$i]}"' ***'
+        rucio -v upload --rse ${rses[$i]} --lifetime $FILE_LIFETIME --scope $RUCIO_SCOPE $filename
+
+        for (( j=0; j<$len; j++ )); do
 
-            echo '*** ======================================================================== ***'
+            if [ $i != $j ]; then
 
-            RANDOM_STRING=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
-            echo '*** generated random file identifier: '"$RANDOM_STRING"' ***'
-	        filename=/home/auto_uploaded_${RANDOM_STRING}_source${rses[$1]}
-            did=auto_uploaded_${RANDOM_STRING}_source${rses[$1]}
-
-            echo '*** generating '"$FILE_SIZE"' file on local storage ***'
-            head -c $FILE_SIZE < /dev/urandom  > $filename
-            echo '*** filename: '"$filename"''
+            echo '*** adding rule from '"${rses[$i]}"' to '"${rses[$j]}"' ***'
+            rucio -v add-rule --lifetime $FILE_LIFETIME --activity "Functional Test" $RUCIO_SCOPE:$did 1 ${rses[$j]}
 
-            echo '*** uploading to rse '"${rses[$1]}"' and adding rule to rse '"${rses[$i]}"'' 
-            rucio -v upload --rse ${rses[$1]} --lifetime $FILE_LIFETIME --scope $RUCIO_SCOPE $filename && rucio add-rule --lifetime $FILE_LIFETIME --activity "Functional Test" $RUCIO_SCOPE:$did 1 ${rses[$i]}
+            fi
 
-            #echo 'sleeping' sleep 3600 
+        done
 
-            echo '*** removing all replicas and dids associated to from rse '"${rses[$1]}"' and adding rule to rse '"${rses[$i]}"'' 
-            echo '*** testing if `rucio erase` is able to remove all the replicas too ***'
-            rucio -v erase $RUCIO_SCOPE:$did
+        echo '*** Uploaded files and replicas should disappear after '${FILE_LIFETIME}' seconds ***'
+        # echo '*** Otherwise do a `rucio -v erase $RUCIO_SCOPE:$did` ***'
 
-            rm -f $filename
-	    fi
     done
 }
 

containers/rucio-noise/rses.txt

@@ -1,5 +1,4 @@
-CERN-EOS
-CESNET-S3
+CERN-EOSPILOT
 CNAF-STORM
 CC-DCACHE
 PIC-DCACHE

infrastructure/cluster/flux/eos/README.md

@@ -11,35 +11,46 @@ Any user connected to the VRE would be able to make use of the extension to acce
 
 ## EOS EULAKE instance
 
-### Configuration
+### ~~`eos/eulake`~~ `eos/pilot/eulake` configuration
 
-## `eos/pilot/eulake` configuration
+During the summer of 2024, the eulake instance was transferred into the EOS pilot instance. The snippets below have been edited acording to these changes.
 
-During the summer of 2024, the eulake instance was moved into the EOS pilot instance. The snippets below have been edited acording to this changes.
+> [!IMPORTANT]
+> The `eospilot` instance is not configured by default on the CERN OpenStack clusters - in general, they are deployed with `cern-magnum` charts that brings EOS deployments to the cluster. Because `eulake` is a subdirectory within `eospilot`, note that the following snippets are set up to point to the `eospilot/eulake` subdirectory instead to `eospilot`.
 
-The `eulake` instance is not configured by default on the CERN OpenStack clusters - The cluster are deployed with EOS deployments, though. To do so, path the `eos-csi-dir-etc-eos` configmap to add the eulake instance into your cluster. Modified the various mount directories as you wish.
-
-Then, add the `eulake` keytab secret as described below. The keytab sercret string can be find on the CERN-VRE `tbag`.
+ To add the `eospilot` instance to the EOS deployment, patch the `eos-csi-dir-etc-eos` configmap as shown below. No `ssskeytab` is further needed - as `eulake` used to require - `eospilot` uses the commn eos keytab.
 
 ```bash
-# charts `eosxd-csi-1.3.1` are deployed with k8s clusters v1.29.
-> kubectl -n kube-system patch configmap eos-csi-dir-etc-eos -p '{"data": {"fuse.eulake.conf": "{\"name\": \"eulake\", \"hostport\": \"eospilot.cern.ch\", \"remotemountdir\": \"/eos/pilot/eulake/escape/data/\", \"localmountdir\": \"/eos/eulake/\", \"auth\": {\"ssskeytab\": \"/etc/eos.keytab\"}}"}}'
-
-> kubectl -n kube-system patch secret eos-csi-file-etc-eos-keytab -p '{"stringData": {"fuse.sss.keytab": "<KEYSTAB_SECRET_STRING>"}}'
+# charts `eosxd-csi-1.3.1` are deployed with k8s clusters v1.29.2 and cern-magnum-0.15.2.
+> kubectl -n kube-system patch configmap eos-csi-dir-etc-eos -p '{"data": {"fuse.pilot.conf": "{\"name\": \"pilot\", \"hostport\": \"eospilot.cern.ch\", \"remotemountdir\": \"/eos/pilot/eulake/escape/data/\", \"auth\": {\"ssskeytab\": \"/etc/eos.keytab\"}}"}}'
+```
+```yaml
+# Patch also the following line into the big chunk of the `auto.eos` section below the rest of eos instances
+data:
+  auto.eos: |
+    (...)
+    pilot -fstype=eosx,fsname=pilot  :eosxd
+    (...)
 ```
 
-Now you can add this volumes on the jupyter hub deployment to access the instance from any pod or jupyter session. On the jupyter hub helm release
+Now you can add this volume on the jupyterHub deployment to access the instance from any jupyter/pod session. On the jupyterHub helm Helm charts add:
 
 ```yaml
         extraVolumes:
           - name: eulake-cern-eos-rse 
             hostPath:
               # This is pointing to /eos/pilot/eulake/escape/data, as defined on the eos-csi-dir-etc-eos/configmap  
-              path: /var/eos/eulake 
+              path: /var/eos/pilot 
         extraVolumeMounts:
           - name: eulake-cern-eos-rse # mounts the EOS RSE needed for the Rucio JupiterLab extension
-            mountPath: /eos/cern-eos-rse
+            mountPath: /eos/eulake
             mountPropagation: HostToContainer
             readOnly: true 
 ```
 
+> [!IMPORTANT]
+> Please note that within this configuration there are two things happening.
+> 1. The propagation of a volume into the cluster (mounting a specific subdirectory of `eospilot`). 
+> 2. The user authentication & authorisation to that subdirectory - which is not detailed here, and needs to be done from the eos server side. 
+>
+> If A&A is not correctly given/propagated, users won't be able to access `/eos/eulake` from their session.

infrastructure/cluster/flux/eos/eos-client.yaml

@@ -1,20 +1,20 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: eos-client
-  namespace: default
-spec:
-  containers:
-  - name: my-container
-    image: gitlab-registry.cern.ch/linuxsupport/alma9-base:latest
-    imagePullPolicy: IfNotPresent
-    command: ["sleep", "inf"]
-    volumeMounts:
-    - name: eos
-      mountPath: /eos
-      mountPropagation: HostToContainer
-  volumes:
-  - name: eos
-    hostPath:
-      path: /var/eos
-      type: Directory
+# apiVersion: v1
+# kind: Pod
+# metadata:
+#   name: eos-client
+#   namespace: kube-system
+# spec:
+#   containers:
+#   - name: my-container
+#     image: gitlab-registry.cern.ch/linuxsupport/alma9-base:latest
+#     imagePullPolicy: IfNotPresent
+#     command: ["sleep", "inf"]
+#     volumeMounts:
+#     - name: eos
+#       mountPath: /eos
+#       mountPropagation: HostToContainer
+#   volumes:
+#   - name: eos
+#     hostPath:
+#       path: /var/eos
+#       type: Directory

infrastructure/cluster/flux/eos/eosxd-patches.yaml

@@ -5,22 +5,5 @@
 #   namespace: kube-system
 # spec:
 #   data:
-#     fuse.eulake.conf: "{
-#       \"name\": \"eulake\", 
-#       \"hostport\": \"eospilot.cern.ch\", 
-#       \"remotemountdir\": \"/eos/pilot/eulake/escape/data\", 
-#       \"localmountdir\": \"/eos/eulake\", 
-#       \"auth\": {
-#         \"ssskeytab\": \"/etc/eos.keytab\"
-#         }
-#       }"
-# ---
-# apiVersion: apps/v1
-# kind: Secret
-# metadata:
-#   name: eos-csi-dir-etc-eos
-#   namespace: kube-system
-# spec:
-#   stringData:
-#     fuse.sss.keytab: |
-#       <KEYSTAB_SECRET_STRING>
+#     fuse.eulake.conf: | 
+#       '{"name":"eulake_test","hostport":"eospilot.cern.ch","remotemountdir":"/eos/pilot/eulake/escape/data/","localmountdir":"/eos/eulake/","auth":{"ssskeytab":"/etc/fuse.sss.keytab"}}'

infrastructure/cluster/flux/jhub/jhub-configmap-profiles.yaml

@@ -22,6 +22,10 @@ data:
           description: "ROOT v6.26.10 as well as a ROOT C++ and a python-3.8 kernel."
           kubespawner_override:
             image: ghcr.io/vre-hub/vre-singleuser-root:sha-c94d95a
+        - display_name: "ROOT Higgs 2024 environment"
+          description: "ROOT v6.32.04, and a python-3.11 kernel."
+          kubespawner_override:
+            image: ghcr.io/vre-hub/vre-singleuser-root-base:latest
         - display_name: "VIRGO - WDF environment"
           description: "Contains the full WDF v2.2.1 environment - Python 3.9 kernel."
           kubespawner_override:
@@ -53,4 +57,4 @@ data:
         - display_name: "Python 3.11 environment"
           description: "quay.io/jupyter/scipy-notebook:python-3.11 image"
           kubespawner_override:
-            image: quay.io/jupyter/scipy-notebook:python-3.11.8
+            image: quay.io/jupyter/scipy-notebook:python-3.11.8

infrastructure/cluster/flux/jhub/jhub-release.yaml

@@ -122,30 +122,35 @@ spec:
       defaultUrl: "/lab"
       # The liefcycle hooks are used to create the Rucio configuration file,
       # and the token file by copying the REFRESH_TOKEN from the environment variable to the token file.
-      startTimeout: 600
+      startTimeout: 1200
       lifecycleHooks:
         postStart:
           exec:
             command:
               - "sh"
               - "-c"
-              - >
-                mkdir -p /certs /tmp;
-                echo -n $RUCIO_ACCESS_TOKEN > /tmp/rucio_oauth.token;
-                echo -n "oauth2:${EOS_ACCESS_TOKEN}:iam-escape.cloud.cnaf.infn.it/userinfo" > /tmp/eos_oauth.token;
-                chmod 0600 /tmp/eos_oauth.token;
-                mkdir -p /opt/rucio/etc;
-                echo "[client]" >> /opt/rucio/etc/rucio.cfg;
-                echo "rucio_host = https://vre-rucio.cern.ch" >> /opt/rucio/etc/rucio.cfg;
-                echo "auth_host = https://vre-rucio-auth.cern.ch" >> /opt/rucio/etc/rucio.cfg;
-                echo "ca_cert = /certs/rucio_ca.pem" >> /opt/rucio/etc/rucio.cfg;
-                echo "account = $JUPYTERHUB_USER" >> /opt/rucio/etc/rucio.cfg;
-                echo "auth_type = oidc" >> /opt/rucio/etc/rucio.cfg;
-                echo "oidc_audience = rucio" >> /opt/rucio/etc/rucio.cfg;
-                echo "oidc_polling = true" >> /opt/rucio/etc/rucio.cfg;
-                echo "oidc_issuer = escape" >> /opt/rucio/etc/rucio.cfg;
-                echo "oidc_scope = openid profile offline_access" >> /opt/rucio/etc/rucio.cfg;
-                echo "auth_token_file_path = /tmp/rucio_oauth.token" >> /opt/rucio/etc/rucio.cfg;
+              - |
+                if [ "${SKIP_POSTSTART_HOOK}" = "true" ]; then
+                  echo "hello world";
+                else
+                  mkdir -p /certs /tmp;
+                  echo -n $RUCIO_ACCESS_TOKEN > /tmp/rucio_oauth.token;
+                  echo -n "oauth2:${EOS_ACCESS_TOKEN}:iam-escape.cloud.cnaf.infn.it/userinfo" > /tmp/eos_oauth.token;
+                  chmod 0600 /tmp/eos_oauth.token;
+                  mkdir -p /opt/rucio/etc;
+                  echo "[client]" >> /opt/rucio/etc/rucio.cfg;
+                  echo "rucio_host = https://vre-rucio.cern.ch" >> /opt/rucio/etc/rucio.cfg;
+                  echo "auth_host = https://vre-rucio-auth.cern.ch" >> /opt/rucio/etc/rucio.cfg;
+                  echo "ca_cert = /certs/rucio_ca.pem" >> /opt/rucio/etc/rucio.cfg;
+                  echo "account = $JUPYTERHUB_USER" >> /opt/rucio/etc/rucio.cfg;
+                  echo "auth_type = oidc" >> /opt/rucio/etc/rucio.cfg;
+                  echo "oidc_audience = rucio" >> /opt/rucio/etc/rucio.cfg;
+                  echo "oidc_polling = true" >> /opt/rucio/etc/rucio.cfg;
+                  echo "oidc_issuer = escape" >> /opt/rucio/etc/rucio.cfg;
+                  echo "oidc_scope = openid profile offline_access" >> /opt/rucio/etc/rucio.cfg;
+                  echo "auth_token_file_path = /tmp/rucio_oauth.token" >> /opt/rucio/etc/rucio.cfg;
+                fi;
+
       networkPolicy:
         enabled: false
       storage: