One can never have too many backups. There’s always room for one more backup. Backups are what keep computers running (most of the time)… Except when backups are not set up. At all. Not even a single antiqued copy exists of what broke down…
Let’s imagine that worst case scenario. Let’s imagine that:
- rpmdb (/var/lib/rpm/rpm__*) is broken
- /var/lib/rpm/Packages is broken, missing or empty
- no backup available for the above files
- no /var/log/rpmdb.log(.*) files available
Recovering package lists
All you got to go on now are the yum logs, found in several /var/log/yum.log(.*) files. These logs will list ALL installed or updated packages (including previously removed versions) and some verbose logging text. Save all these files to a safe location as soon as possible. Later combine them into one and clean up all extraneous information leaving them a pure package list, similar to:
This will contain multiple versions of the same package, but that does not matter much.
Prepare the workplace
You hopefully have yum fully functional on the system, but you’ll need a yum plugin that is most likely not installed. Unfortunately, you cannot use yum to install it since in lack of a rpm database you’ll be prompted to install about half of the packages on the server… again.
The manual approach of downloading the rpm file for yum-plugin-downloadonly (from the CentOS repositories) and manually installing it with rpm (skipping dependencies) is needed:
rpm -i yum-plugin-downloadonly-VERSION.rpm --nodeps
Prepare a work folder on the system:
Place the previously compiled packages list file in this folder in rpm-repair/list.txt
Grab packages and re-enter them into the db
Create a rpm-repair/grab.sh file with the following contents:
for word in $(cat list.txt); do yum install $word -y --downloadonly --downloaddir=/rpm-repair/packages/ ; done;
and make it executable (chmod +x grab.sh).
Create a rpm-repair/repair.sh file with the following contents:
for f in `ls packages`; do rpm -ivh --justdb --force --nodeps packages/$f; done
Also make this file executable (chmod +x repair.sh).
One you’ve got all these set up, run ./grab.sh and sit back and wait (you may want to run this is in a screen if you’re working remotely to avoid having to start again if connection drops).
Once all rpm files are downloaded, run ./repair.sh and wait some more.
When both operations are complete, your rpm db should be at least partially functional. However there will most likely be multiple versions of the same package registered in the db. This can be cleaned up with:
for f in `package-cleanup --dupes`; do echo $f; rpm -e --justdb --nodeps $f; done;
package-cleanup is part of the yum-utils package (which at this stage can be installed directly with yum, yay!)
There may be a couple of packages not registered (maybe they weren’t present in the log, maybe their rpm was no longer available at the particular logged version), but these should not be critical and might get updated/installed the next time you run yum update. The end result may not be perfect but it’s a lot better than being left with a non-update-able system (or having to reinstall it from scratch at this very moment).