TIL How to mount a google cloud storage bucket as root on a VM
Editing /etc/fstab and the gcsfuse option -o allow_other solves the issue of users not being
able to read a bucket mounted by gcsfuse during a startup script (which runs a root)
Ordinarily one would mount a bucket on startup using a script like
1#!/bin/sh
2# (Assuming Debian base)
3
4echo "Adding APT source for gcsfuse"
5wget -O - https://packages.cloud.google.com/apt/doc/apt-key.gpg | tee /usr/share/keyrings/cloud.google.asc
6GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
7echo "deb [signed-by=/usr/share/keyrings/cloud.google.asc] https://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
8
9echo "Installing gcsfuse"
10apt-get update
11apt-get install -y gcsfuse
12
13echo "Mounting ds-workspace-storage"
14mkdir /mnt/<bucket-name>
15# Lets try to make sure that the permissions are set properly!
16chown root:google-sudoers /mnt/<bucker-name>
17chmod 775 /mnt/<bucker-name>
18gcsfuse \
19 --file-mode=775 \
20 --dir-mode=775 \
21 --gid=$(cat /etc/group | grep google-sudoers | cut -d: -f3) \
22 <bucket-name> /mnt/<bucket-name>But logging into the server as a user and running ls /mnt/<bucket-name> results in question marks:
user@server$ ls /mnt/<bucket-name>
ls: cannot access <bucket-name>: Permission denied
total 0
drwxr-xr-x root root yyy xxxx-xx-xx xx:xx .
drwxr-xr-x root root yy xxxx-xx-xx xx:xx ..
?????????? ? ? ? ? <bucket-name>So I made the following edits to the script:
1...
2
3echo "Mounting ds-workspace-storage"
4# >>>>
5echo "
6user_allow_other" >> /etc/fuse.conf
7# <<<<
8mkdir /mnt/<bucket-name>
9chown root:google-sudoers /mnt/<bucker-name>
10chmod 775 /mnt/<bucker-name>
11gcsfuse \
12 --file-mode=775 \
13 --dir-mode=775 \
14 --gid=$(cat /etc/group | grep google-sudoers | cut -d: -f3) \
15 -o allow_other \
16 <bucket-name> /mnt/<bucket-name>And this fixes the problem: Users in the google-sudoers group can view/read/write files in the
bucket!
Although unfortunately after all this effort, I realise that Google File Store might be better for the ML/DS workload I’m building infrastructure for…