One can never have too many backups. There’s always room for one more backup. Backups are what keep computers running (most of the time)… Except when backups are not set up. At all. Not even a single antiqued copy exists of what broke down…

Let’s imagine that worst case scenario. Let’s imagine that:

  • rpmdb (/var/lib/rpm/rpm__*) is broken
  • /var/lib/rpm/Packages is broken, missing or empty
  • no backup available for the above files
  • no /var/log/rpmdb.log(.*) files available

Recovering package lists

All you got to go on now are the yum logs, found in several /var/log/yum.log(.*) files. These logs will list ALL installed or updated packages (including previously removed versions) and some verbose logging text. Save all these files to a safe location as soon as possible. Later combine them into one and clean up all extraneous information leaving them a pure package list, similar to:

[...]
ethtool-3.5-1.4.el6_5.x86_64
ethtool-3.5-1.el6.x86_64
ethtool-3.5-5.el6.x86_64
expat-2.0.1-11.el6_2.x86_64
file-libs-5.04-15.el6.x86_64
file-libs-5.04-21.el6.x86_64
filesystem-2.4.30-3.el6.x86_64
finger-0.17-40.el6.x86_64
finger-server-0.17-40.el6.x86_64
fipscheck-1.2.0-7.el6.x86_64
fipscheck-lib-1.2.0-7.el6.x86_64
fontconfig-2.8.0-3.el6.x86_64
fontconfig-2.8.0-5.el6.x86_64
[...]

This will contain multiple versions of the same package, but that does not matter much.

Prepare the workplace

You hopefully have yum fully functional on the system, but you’ll need a yum plugin that is most likely not installed. Unfortunately, you cannot use yum to install it since in lack of a rpm database you’ll be prompted to install about half of the packages on the server… again.

The manual approach of downloading the rpm file for yum-plugin-downloadonly (from the CentOS repositories) and manually installing it with rpm (skipping dependencies) is needed:

rpm -i yum-plugin-downloadonly-VERSION.rpm --nodeps

Prepare a work folder on the system:

mkdir /rpm-repair
mkdir /rpm-repair/packages

Place the previously compiled packages list file in this folder in rpm-repair/list.txt

Grab packages and re-enter them into the db

Create a rpm-repair/grab.sh file with the following contents:

for word in $(cat list.txt); do yum install $word -y --downloadonly --downloaddir=/rpm-repair/packages/ ; done;

and make it executable (chmod +x grab.sh).

Create a rpm-repair/repair.sh file with the following contents:

for f in ls packages; do rpm -ivh --justdb --force --nodeps packages/$f; done

Also make this file executable (chmod +x repair.sh).

One you’ve got all these set up, run ./grab.sh and sit back and wait (you may want to run this is in a screen if you’re working remotely to avoid having to start again if connection drops).

Once all rpm files are downloaded, run ./repair.sh and wait some more.

Some cleanup

When both operations are complete, your rpm db should be at least partially functional. However there will most likely be multiple versions of the same package registered in the db. This can be cleaned up with:

for f in package-cleanup --dupes; do echo $f; rpm -e --justdb --nodeps $f; done;

package-cleanup is part of the yum-utils package (which at this stage can be installed directly with yum, yay!)

Conclusion

There may be a couple of packages not registered (maybe they weren’t present in the log, maybe their rpm was no longer available at the particular logged version), but these should not be critical and might get updated/installed the next time you run yum update. The end result may not be perfect but it’s a lot better than being left with a non-update-able system (or having to reinstall it from scratch at this very moment).

References: (1), (2), (3), (4), (5)

4 Comments

  1. What happens if you have multiple Kernel packages, or for example you switched to CloudLinux or another kernel. Will these create any issues? I have this problem right now and I have the yum logs. What happens if the packages that are deleted I take those out?

    Alex
  2. This is exactly the problem we have been having, so I built the scripts and ran them. However, I’m getting a constant segmentation fault on random rpms. This causes me to have to move that rpm out of the folder, delete the Packages db file, and kill the tasks locking it. then re-run the script until it seg faults again.

    Any ideas as to what might be causing that or another possible option? I’m really not wanting to have to rebuild the servers affected by this problem.

    Thanks so much for this write-up though! It is so close to fixing our problem, but we still get a yum stopiteration error after importing everything we can to the db.

    Thanks!

    Clark
      1. We didn’t notice any memory or cpu issues while the repair script was running. One package would get a seg fault, then next would seem to be running, and then it locks up. Locks up the process and it’s parent, as well as corrupting the Packages db file

        Clark

Leave a Reply