TIL How to mount a google cloud storage bucket as root on a VM
Editing /etc/fstab and the gcsfuse option -o allow_other solves the issue of users not being
able to read a bucket mounted by gcsfuse during a startup script (which runs a root)
Ordinarily one would mount a bucket on startup using a script like
#!/bin/sh
# (Assuming Debian base)
echo "Adding APT source for gcsfuse"
wget -O - https://packages.cloud.google.com/apt/doc/apt-key.gpg | tee /usr/share/keyrings/cloud.google.asc
GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb [signed-by=/usr/share/keyrings/cloud.google.asc] https://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
echo "Installing gcsfuse"
apt-get update
apt-get install -y gcsfuse
echo "Mounting ds-workspace-storage"
mkdir /mnt/<bucket-name>
# Lets try to make sure that the permissions are set properly!
chown root:google-sudoers /mnt/<bucker-name>
chmod 775 /mnt/<bucker-name>
gcsfuse \
--file-mode=775 \
--dir-mode=775 \
--gid=$(cat /etc/group | grep google-sudoers | cut -d: -f3) \
<bucket-name> /mnt/<bucket-name>But logging into the server as a user and running ls /mnt/<bucket-name> results in question marks:
user@server$ ls /mnt/<bucket-name>
ls: cannot access <bucket-name>: Permission denied
total 0
drwxr-xr-x root root yyy xxxx-xx-xx xx:xx .
drwxr-xr-x root root yy xxxx-xx-xx xx:xx ..
?????????? ? ? ? ? <bucket-name>So I made the following edits to the script:
...
echo "Mounting ds-workspace-storage"
# >>>>
echo "
user_allow_other" >> /etc/fuse.conf
# <<<<
mkdir /mnt/<bucket-name>
chown root:google-sudoers /mnt/<bucker-name>
chmod 775 /mnt/<bucker-name>
gcsfuse \
--file-mode=775 \
--dir-mode=775 \
--gid=$(cat /etc/group | grep google-sudoers | cut -d: -f3) \
-o allow_other \
<bucket-name> /mnt/<bucket-name>And this fixes the problem: Users in the google-sudoers group can view/read/write files in the
bucket!
Although unfortunately after all this effort, I realise that Google File Store might be better for the ML/DS workload I’m building infrastructure for…