Backing up a Digital Ocean VM instance

(haskal actually write a blog post challenge 2020)

I figured out one way to image Digital Ocean VMs (if you for example, intend to destroy them but want a disk image). Because DO wants you to pay as much money to them as possible, you can't just download disk images from the web UI because of course that would be too easy. So instead, you can use a tool called takeover.sh to pivot from the main root filesystem to a backup root, then umount the main root, and then you'll be able to image it somewhere convenient.

First, prepare an alternate root. Here I use pacstrap but you could use the recommended SystemRescueCD.

# Create a new tmpfs
mkdir /takeover
mount -t tmpfs tmpfs /takeover
# Initialize
pacstrap -c /takeover base openssh lsof vim sshfs partclone
cp /etc/resolv.conf /takeover/etc/resolv.conf
# Add takeover.sh
git clone https://github.com/marcan/takeover.sh
gcc -static -o /takeover/fakeinit takeover.sh/fakeinit.c
cp takeover.sh/takeover.sh /takeover
# Replace this with the latest version, if  you want
wget https://busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-x86_64 /takeover/busybox
chmod +x /takeover/busybox

Edit /takeover/ssh/sshd_config and make sure PermitRootLogin is yes and PasswordAuthentication is yes.

If you intend to do weird funky stuff like FUSE (which i used for sshfs) load it now

modprobe fuse

Stop as much of the running system as possible. For example, on systemd you should stop all server services other than the systemd builtin ones like dbus and journald. Specifically, make sure port 80 is free because this is what takeover.sh uses for alternate SSH.

Run takeover.sh. What it will do is mount dev, proc, sys, and friends inside the alternate root, start an alternate sshd on port 80, then request that init (systemd) replace itself with the fakeinit inside the new root. When you reach the prompt to log into the new ssh, do it but leave the existing session open too. ssh -p 80 root@.... If this fails, do not enter OK at the prompt. You can enter anything else and takeover.sh will abort.

If alternate ssh works, you should have a session inside the new root. Complete the takeover.sh prompts, wait for it to finish, and then close the old session. Now you need to clean up the old root, which will be located on /old_root. Use lsof | grep old_root or your favorite commands to find and kill remaining processes located on old_root. Then, unmount any mounts in mount -l | grep old_root. Now you should be able to unmount /old_root itself. If that works, the system is free to run whatever you want on the old root filesystem, like partclone. I mounted my server over sshfs and then executed partclone to image the root filesystem to the server.
For example:

e2fsck -f /dev/whateverN # make sure it's ok. this will also error if it's still mounted
partclone -c -s /dev/whateverN -x zstd -N -o /somewhere/yourimage.img.zst

That's basically it. Congrats on getting around DO being spooky about backups ✨