Today I had to recover a Debian server after the /var partition was accidentally deleted. All the user generated data (primarily website content and databases) was backed up, and I also had access to a backup copy of the installed package list as generated by dpkg --get-selections. The instructions below don’t go into full detail, but hopefully they will set you on the right track. They should also work for Ubuntu installs.
If this happens to you, don’t panic, and whatever you do, don’t shut down or reboot the server!
To recover the server, I took the following steps:
- Create a bootstrapped Debian environment on a separate server
- Use the installed package list backup to install the same packages into the bootstrap environment
- Transfer the contents of
/varacross from the bootstrap onto the broken server - Restore user data from backup
- Fix various file permissions
- Restart the services
- Check the logs and systemd status for any remaining problems
- Reboot the server to check that everything comes back up as normal
All commands below run as root. Be careful not to make any mistakes!
To create a bootstrapped Debian environment, run the following:
apt-get install debootstrapmkdir /bootstrapdebootstrap stretch /bootstrap/cp dpkg-selections-backup.txt /bootstrap/chroot /bootstrap/mount -t proc none /proc
Now you can install the packages using the steps below. This took a bit of fiddling around but there are a number of helpful posts about the likely problems if you get stuck. IMPORTANT NOTE: if you are prompted, DO NOT install grub to the MBR! This might break your existing install!
dpkg --set-selections < dpkg-selections-backup.txt- Edit
/etc/apt/sources.list(bootstrap only includesmainby default, so you may need to add insecurity,non-free, etc) apt-get updateapt-get install dselectdselect updateapt-get dselect-upgrade
You can now rsync the data across from /bootstrap/var/ to /var on the broken server. Your life will be much easier if you use the --archive option. You may need to temporarily enable root SSH access in order to do this, e.g. with PermitRootLogin without-password.
Once I’d done that, I had to reinstall the mariadb-server packages on the broken server using apt-get install --reinstall mariadb-server mariadb-server-10.1 mariadb-server-core-10.1.
The next step is to restore your user data. This will be specific to the structure that you use, so I can’t help with instructions for that. Once you’ve done that, check the permissions of the files in /var/lib and /var/log against a working server (or perhaps your bootstrap) to check that they are correct. It’s quite likely that the user IDs of the system accounts will be different between the server and the bootstrap environment.
Now you should check the various services that are installed, using service --status-all. Not all of the services should necessarily be enabled, again I recommend comparing against a working server. Try to restart any services that aren't currently running but should be. If all the permissions are correct and the relevant user data (if any) has been restored then the service should start successfully. If it doesn’t, check the systemd status with systemctl status <service-name>, the systemd logs with journalctl -xe and the service’s log files in /var/log. Those should give you enough information to track down any remaining problems.
At this point, all of your services should be back up and running normally. The final step is to reboot your repaired server and verify that everything comes back up as expected after a restart.
Last but not least, if you’re trying this but getting stuck, we’re happy to help! Drop us a line at our contact page.