1. Introduction
APT (Advanced Packaging Tool) is one of the main package managers among the major Linux distributions. In particular, it’s the default for Debian-based systems. As such, scripts and commands that extract package information in such environments usually employ the APT system.
In this tutorial, we talk about ways to check and potentially migrate the current repository information within a Debian-based Linux installation. First, we briefly refresh our knowledge about the structure of APT repositories on the machine. Next, we discuss a fairly simplistic way of listing repositories and their keys. After that, we explore a tool for the same but without handling security. Then, we go over a custom script that solves this problem. Finally, we show a robust and fairly minimal way to preserve a complete copy of the current repository setup.
We tested the code in this tutorial on Debian 12 (Bookworm) with GNU Bash 5.2.15. It should work in most POSIX-compliant environments unless otherwise specified.
2. APT Source Structure
Before attempting to extract repository information, we need to know where it resides.
2.1. Source Configuration
APT stores all sources at two locations:
- /etc/apt/sources.list file
- /etc/apt/sources.list.d/ directory
The files have a defined format:
$ cat /etc/apt/sources.list
#deb cdrom:[Official Debian GNU/Linux Live 2023-06-26T06:56:00Z]/ bookworm main non-free-firmware
deb http://deb.debian.org/debian stable main
deb-src http://deb.debian.org/debian stable main
deb http://deb.debian.org/debian-security/ stable-security main
deb-src http://deb.debian.org/debian-security/ stable-security main
deb http://deb.debian.org/debian stable-updates main
deb-src http://deb.debian.org/debian stable-updates main
The format includes the repository type, source URL, branch, and other specific information. The syntax is similar for a Personal Package Archive (PPA) as well but involves a ppa.launchpad.net or ppa.launchpadcontent.net domain name.
2.2. Repository Keys
Many public and private repositories have GPG keys to validate their identity to clients:
Index of /debian/dists/stable
[ICO] Name Last modified Size
[PDR] Parent Directory -
[ ] ChangeLog 2023-12-10 17:19 2.2M
[ ] InRelease 2023-12-10 17:49 148K
[ ] Release 2023-12-10 17:44 146K
[ ] Release.gpg 2023-12-10 17:49 1.8K
[DIR] contrib/ 2023-12-10 17:44 -
[DIR] main/ 2023-12-10 17:44 -
[DIR] non-free-firmware/ 2023-12-10 17:44 -
[DIR] non-free/ 2023-12-10 17:44 -
Here, we see the Release.gpg key of the Debian stable repository at https://ftp.debian.org/debian/dists/stable/.
Naturally, key configuration is part of the initial repository setup procedure. In fact, the system should store all known keys in two locations:
- /etc/apt/trusted.gpg file (not used in recent Linux versions)
- /etc/apt/trusted.gpg.d/ directory
Further, these keys are checked for each operation. So, to avoid security issues, we need the keys along with the repositories.
3. Extract Simple Lists
After getting to know the repository and security information, we can attempt a basic extraction.
3.1. Source Configuration
Knowing the structure and location of Debian repositories, we can simply list the data from all repository files via cat:
$ cat /etc/apt/sources.list /etc/apt/sources.list.d/*
This simple list has a couple of potential flaws:
- includes empty and commented lines
- may show errors or warnings
Let’s remedy both by filtering through grep:
$ cat /etc/apt/sources.list /etc/apt/sources.list.d/* 2>/dev/null | grep --invert-match '^ *#|^ *$'
Here, we use the –invert-match switch to avoid lines that ^ start with an # octothorp, potentially after some spaces, or which comprise only whitespaces from ^ beginning to $ end. Further, we redirect the stderr output to /dev/null.
Now, we should no longer see anything but repository configuration lines.
3.2. Repository Keys
To get the repository keys, we can employ a similar approach to the one above:
$ cat /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d/* 2>/dev/null
In this case, we expect no comments or empty lines. Yet, some keys might be in binary, so a Base64 encoding and decoding pipeline could be needed.
The major inconvenience with this approach is that the output format doesn’t distinguish where each line comes from. Thus, we end up with rows of repository configuration and key lines that we either need to distribute manually or place within the same file on another machine, if migrating.
Because of this, such a simple approach is often best suited for an overview and not migration.
4. add-apt-repository
Notably, the software-properties-common package contains a script for APT repository management:
$ apt-get install software-properties-common
At this point, we should be able to use the add-apt-repository, often aliased as apt-add-repository.
Once installed, we can use the fairly recent –list flag to show all configured repositories:
$ add-apt-repository --list
By redirecting this output to a file, we can make a snapshot of all repositories:
$ add-apt-repository --list > apt-repos.list
At this point, we can just rename the file and add it to another /etc/apt/ directory. However, we would have to handle the keys separately.
5. Custom Script
As with many other activities, we can attempt a custom script solution to our problem:
$ cat aptTransfer.sh
#!/usr/bin/env bash
for aptfile in $(find /etc/apt/sources.list /etc/apt/sources.list.d/ -type f); do
cat $aptfile | grep '^deb ' | while read aptline; do
aptline="${aptline:4}"
IFS='/'
read fields <<< "$aptline"
fields=($fields)
host=${fields[2]}
user=${fields[3]}
ppa=${fields[4]}
if [ "ppa.launchpad.net" = "$host" ] || [ "ppa.launchpadcontent.net" = "$host" ]; then
echo "sudo add-apt-repository ppa:$user/$ppa"
else
echo "sudo add-apt-repository '${aptline}'"
fi
done
done
In particular, we use a command substitution to find and list all [-f]ile[-type] filesystem objects in /etc/apt/sources.list*, storing them in $aptfile one by one. Then, we show only lines that start with deb followed by a space. For each such line, the first four characters (the deb-space prefix) are removed. After that, we read the rest as a / forward-slash-separated string by modifying $IFS. Next, we split the resulting $fields, reading each as expected for a Personal Package Archive (PPA).
Finally, we check whether the domain we extract as $host matches one of the known PPA hosting domains. If so, we assume the other extracted fields match the user and specific PPA.
Notably, we then generate a line that either adds the respective repository type via the add-apt-repository command. That should take care of the keys for each source.
6. Clone Directory Structure
To get all source and repository security information in a single package, we can use a basic tar command:
$ tar --verbose --create --file=apt-repos.tar /etc/apt/sources.list* /etc/apt/trusted.gpg*
Thus, we –create the apt-repos.tar TAR –file with all necessary files:
- /etc/apt/sources.list
- files within /etc/apt/sources.list.d/
- /etc/apt/trusted.gpg
- files within /etc/apt/trusted.gpg.d/
To deploy, we –extract on the destination machine while in the / root directory, since the archive contains full paths:
$ tar --verbose --extract --file=apt-repos.tar --directory=/
While this approach exhibits no flaws in terms of transferring the information properly and without errors, we still might need to avoid it due to potentially unwanted side effects:
- overwrites repositories already present on the system
- in rare cases, custom permissions can prevent the migration or processing of files
This means that the method is best suited for migrating an existing repository setup to vanilla installations instead of their defaults. Still, this is perhaps the most robust and simple way to ensure a smooth and complete bug-free setup.
7. Summary
In this article, we talked about ways to list and migrate APT repositories.
In conclusion, although we can extract all sources, keys might have to be handled separately, depending on the method we choose.