How to optimize used space in flash storage images

How to optimize used space in flash storage images

Due to their write-sensitive nature, flash storage (both USB sticks and SD cards) normally don’t get zeroed out when data is deleted, making any images created for backup purposes contain random data in the empty sectors. This makes images take up the full size of the disk/card even if very little is actually used from it and also makes image poorly compressible.

Even worse, if the image file was already created at some earlier point and the original source medium is no longer available, zeroing the free space and recreating the image is not an option. It’s also not the best idea to zero out free space on flash storage because it adds wear through (unnecessary) erase-write cycles.

The solution for both situations is to zero out the unused space directly in the image file, preferably while being stored on a mechanical hard drive (as to not simply transfer the write wear to different flash storage). Linux has all the tools necessary for this task readily built-in (and usually available without the need of installing additional packages). If a Linux install is not available, a bootable self-contained system such as SystemRescue can be used instead.

Once you’re in a shell/prompt, start by getting the image file on-hand and informing the kernel of its content:

partx -a -v myveryimportant.img

Notice the device name the imagine is assigned to – it can be loop0, loop1, so on.

In my example the original disk (an SD card used with Raspberry PI) contained two partitions – a boot partition and a main root partition so I’ll have to run everything in duplicate.

Prepare some mount points (these can be anywhere on your storage):

mkdir /mnt/part1
mkdir /mnt/part2

Then mount the two partitions (notice the partitions are identified in a similar numbered fashion to regular storage devices):

mount /dev/loop1p1 /mnt/part1
mount /dev/loop1p2 /mnt/part2

We can explore /mnt/partX to check the content if we want and then use the most basic tool to fill a file on each partition with zeros:

dd if=/dev/zero of=/mnt/part1/zero.file bs=10M status=progress; sync; rm /mnt/part1/zero.file
dd if=/dev/zero of=/mnt/part2/zero.file bs=10M status=progress; sync; rm /mnt/part2/zero.file

Be very careful about the parameters you run dd with as in case of insufficient attention it will overwrite important data without remorse.

Once its done the partitions can be unmounted:

umount /dev/loop1p1
umount /dev/loop1p2

Then any record of them erased (from the kernel’s memory):

partx -d -v /dev/loop1

Sources: (1)

Leave a Reply