2017-08-23 Securely Syncing the Gentoo Repository via a Local Server

Figure: Using a local server for the Gentoo repository takes pressure off of the (often relatively slow) Internet connection and places it onto the (often relatively fast) LAN, especially important for those whose Internet is metered.

Running multiple Gentoo machines normally means keeping a copy of the Gentoo repository on each machine; by default this also means updating each copy of the repository by having each machine call out to the Gentoo mirrors on a regular basis for updates. This is a waste of both local and upstream bandwidth and resources, because, within a given time period, the repository is the exact same regardless of the type of machine it is on; instead, it is more efficient to set up one machine as a server that synchronizes with upstream, then have the other local machines synchronize to said local server. This guide will explain how to do that securely.

Though there are multiple protocols that Portage can use for synchronization, this guide will cover only SSH, and rsync+stunnel. Both of these are secure, allowing synchronization over an untrusted network, albeit the local server must be trusted. Securely syncing the server to the upstream repository with FEATURES=webrsync-gpg is not covered here, as it is instead covered here.

SSH Synchronization

One of the pros of SSH-based synchronization is that it requires no extra server configuration for machines that already have SSH access to the server. The downside is that SSH access means shell access, which might not be desirable if any of the local machines is not fully trusted, for example, because it is running proprietary applications for, say, video games; one could try and circumvent this potential security weakness by giving the SSH user a restricted shell such as rssh, but I was unable to get rssh working with the rsync arguments that Portage wanted to use (note: Portage calls rsync over the SSH session). Thus I prefer rsync+stunnel, but am including the SSH method for completeness and possible future reference.

Server

Assuming that sshd is already set up (find one of many guides if it is not), the only thing that may need to be done is to add a special user for synchronization. In order to add the user, run

	# useradd -m -s /bin/bash rsyncuser
	# passwd rsyncuser

...then edit the SSH daemon configuration file, usually /etc/ssh/sshd_config, to contain the following line:

	AllowUsers rsyncuser

Depending on the selected authentication method for SSH, this may be enough to give the user access; however, in the case that only key-based authentication is allowed (usually via something like PasswordAuthentication no & ChallengeResponseAuthentication no in sshd_config) an SSH keypair will need to be generated for the user and the private key copied to each client that wishes to synchronize with the server. This can be done with the following commands:

	# cd /home/rsyncuser
	# su rsyncuser
	$ umask 0066
	$ mkdir -p .ssh
	$ ssh-keygen -a 16 -t ed25519
	$ cp .ssh/id_ed25519.pub .ssh/authorized_keys

This will create a keypair for the rsyncyser and authorize that keypair for SSH login. The .ssh/id_ed25519 file (the private key) will then need to be copied to the client machines, probably via a USB storage device:

	# cp .ssh/id_ed25519 /path/to/probably/usb/storage/device

This ends the server-side configuration.

Clients

By default, Portage knows how to use SSH for synchronization, so basic setup is quite simple, although some of the details are rather esoteric.

In order to tell Portage to synchronize over SSH, if it hasn't been done already, set up the main repository configuration via:

	# mkdir -p /etc/portage/repos.conf
	# cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf/gentoo.conf

This creates, at the time of this writing, at least, a sane configuration file for the default repository that can then be reconfigured. With this file created, Portage should function as it always has; the defaults must be overridden in order to actually change its behavior. In order to use SSH, edit the /etc/portage/repos.conf/gentoo.conf file by changing the sync-uri option under the [gentoo] section to read:

	sync-uri = ssh://rsyncuser@192.168.1.1/usr/portage

...where 192.168.1.1 is replaced with the network address of the local server. Portage will now begin trying to sync via SSH.

"Trying" may be the modus operandi depending on whether or not the local server strictly requires keys or will accept user passwords; it is thus useful to understand where Portage stores its SSH information, as its home directory is not /home/portage, but, as /etc/passwd reveals, /var/tmp/portage, thus SSH configuration goes into /var/tmp/portage/.ssh. In order to get Portage to use rsyncuser's key, run the following:

	# mkdir -p /var/tmp/portage/.ssh
	# umask 0066
	# cp /path/from/probably/usb/device/with/rsyncusers/key/id_ed25519 /var/tmp/portage/.ssh/id_ed25519
	# chown -R portage:portage /var/tmp/portage/.ssh

Portage should now pickup rsyncuser's key when it tries to synchronize.

Admittedly, storing valuable authentication information in a /var/tmp subdirectory seems risky at best. Another option for storing the key is to place the key at a more stable location and then tell Portage where to find it. Start by placing the key somewhere safer:

	# mkdir -p /usr/local/portage/.ssh
	# chmod 700 /usr/local/portage/.ssh
	# chown -R portage:portage /usr/local/portage/.ssh
	# mv /var/tmp/portage/.ssh/id_ed25519 /usr/local/portage/.ssh/

...then tell Portage where to find the key by adding the following line to /etc/portage/make.conf:

	PORTAGE_SSH_OPTS="-i /usr/local/portage/.ssh/id_ed25519"

Portage will now look for the rsyncuser's key at the previously-specified location.

This solves the rsyncuser's key problem, but what about the important known_hosts file? Well, I didn't get that far. I don't fully trust of all my machines, and do not want to give all of them shell access to my server, and, since I couldn't get rssh to work with Portage and was looking for an excuse to play with stunnel, I decided to work on it instead, thus rsync+stunnel is the topic of the next section.

Rsync+stunnel Synchronization

This method works by using an rsync daemon to synchronize between the local server and client; however, because rsync is unencrypted, the connection is wrapped via the stunnel daemon, which uses TLS for encryption and authentication. stunnel must be configured on both the local server and each client.

Server

Start by setting up the rsync daemon without stunnel. By default, Gentoo comes with a disabled but sane configuration file at /etc/rsyncd.conf. Adding a few extra lines gives, excluding comments, the following:

	pid file = /run/rsyncd.pid
	use chroot = yes
	read only = yes
	hosts allow = 192.168.1.1/16
	reverse lookup = no
	timeout = 60

	[gentoo-portage]
	path = /usr/portage
	comment = Gentoo Portage tree
	exclude = /distfiles /packages

By default this configuration file creates an rsync module that clients can synchronize with; this module is read-only and excludes directories that are not a part of the repository. The extra options are: hosts allow = 192.168.1.1/16, which tells rsync to only accept connections from hosts in the specified block and must be appropriate to the local network; reverse lookup = no, which disables reverse-DNS lookups of client IP addresses; and timeout = 60, which sets a small timeout for unresponsive clients. See man 5 rsyncd.conf for details; the extra options are not strictly necessary for the purposes of this guide.

The next step is to install and configure stunnel. This involves four steps: installing the program, configuring the service, generating certificates, and creating the service: To install stunnel, run:

	# emerge -av stunnel

For convenience, the Gentoo developers created a useful example configuration file for rsync+stunnel which may be installed by running:

	USE="stunnel" emerge -av rsync

This creates the file /etc/stunnel/rsyncd.conf. Move it to /etc/stunnel/rsyncd-stunnel.conf for init scripts (later), and reconfigure it such that, excluding comments, it has the following options:

	foreground = no
	pid = /var/run/stunnel/rsyncd-stunnel.pid
	socket = l:TCP_NODELAY=1
	socket = r:TCP_NODELAY=1
	setuid = root
	setgid = root

	[rsync]
	accept = 874
	cert = /etc/stunnel/rsyncd-stunnel.crt
	key  = /etc/stunnel/rsyncd-stunnel.key
	client = no

	exec = /usr/bin/rsync
	execargs = rsync --server --daemon --config=/etc/rsyncd.conf .

Most of these options come with the default configuration file, but make sure to change the pid option. The cert and key options specify where the X.509 certificate and its key are to be loaded from (they will be generated later). Port 874 is used in place of 873, as it is not a plaintext rsync service. Client verification is not wanted here, so make sure to remove any verify and CAfile lines.

In order to generate the server's certificate, run the following series of commands:

	# cd /etc/stunnel
	# umask 0077
	# openssl genrsa -out rsyncd-stunnel.key 4096
	# openssl req -key rsyncd-stunnel.key -new -x509 -days 7200 -sha512 -out rsyncd-stunnel.crt -subj '/CN=rsyncd-stunnel/'

The short story here is that this will create a certificate and key pair for TLS that will last for almost 20 years, see man genrsa and man req for details.

Finally, create the service with:

	# cd /etc/init.d
	# ln -s stunnel rsyncd-stunnel

This uses the default stunnel init script, except it will look for a configuration file at /etc/stunnel/rsyncd-stunnel.conf, hence the earlier renaming that was done. To start the service, run rc-service rsyncd-stunnel start, and don't forget to add it to the default runlevel via:

	# rc-update add rsyncd-stunnel default

The server should now be configured and ready for synchronization. The clients must now be configured to trust and use the server.

Clients

The client must now be configured so that when Portage attempts to sync it will use the secure connection and it will validate that it is syncing with the server, otherwise a malicious actor could imitate the server and send a compromised repository. This involves installing and configuring stunnel, configuring Portage, then creating the service.

Begin by installing stunnel:

	# emerge -av stunnel

...then configure stunnel so that, without comments, the configuration file /etc/stunnel/stunnel.conf reads:

	setuid = stunnel
	setgid = stunnel
	pid = /run/stunnel/stunnel.pid
	socket = l:TCP_NODELAY=1
	socket = r:TCP_NODELAY=1

	[rsync-stunnel]
	client     = yes
	CAfile     = /etc/stunnel/rsyncd-stunnel.crt
	verifyPeer = yes
	accept     = 127.0.0.1:873
	connect    = 192.168.1.200:874

This creates an stunnel service. The client argument lets stunnel know to run rsync as a client rather than as a server. The CAfile argument specifies the location of the server certificate to use (note that it has not yet been copied over to the client). The verifyPeer is especially important as enabling it will actually validate the server certificate; without this enabled a malicious actor could intercept and modify the traffic. The accept argument specifies where the stunnel program will listen for client connections, Portage will later be configured to make connections here. The connect argument specifies where the stunnel program will connect to; it must be set to the address and port of the local server. The other options are fairly standard. Note that, unlike the server, this setup is using the default stunnel settings, configuring multiple stunnel services at the OpenRC level would require modifying a few parameters, such as the PID (Process ID) and the filename of the configuration.

Now that stunnel has been configured, make sure to copy the certificate from the local server to the client. This should be done via a trusted storage device (USB stick, SD Card, &c.) or the certificate retrieved via a command such as OpenSSL's s_client (usage not covered here) and its fingerprint manually verified.

	# cp /path/from/secure/storage/device /etc/stunnel/rsyncd-stunnel.crt

stunnel now has all that it needs to securely tunnel a connection to the server.

Portage must now be configured to use the secure tunnel. As with SSH configuration, begin by overriding the default repository configuration:

	# mkdir -p /etc/portage/repos.conf
	# cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf/gentoo.conf

...then modify the sync-uri argument to correspond to the accept argument configured in stunnel:

	sync-uri = rsync://127.0.0.1:873/gentoo-portage

This will synchronize with the gentoo-portage module defined on the local server's /etc/rsyncd.conf. Note how the use of stunnel is completely transparent to Portage. This has the minor disadvantage that error diagnostics must be found in log message from stunnel and not the Portage frontend, but is usually not a problem once things have been configured properly.

Next, start the service and add it to the default runlevel:

	# rc-service stunnel start
	# rc-update add stunnel default

...then try it out with:

	# emerge --sync

Portage should now be securely synchronizing to the local server! Repeat these steps for each client that needs to synchronize with the local server.