2019-07-20 Minetest 5.0.1 Upgrade and Server Hosting

Having finally fixed the tree-growth issue, it was time for me to upgrade my Minetest world to the newest version, including updating all of its mods. In addition, I had also long desired to host the world on a permanent server rather than just my gaming rig, because, as said rig has bright blue LEDs on it which keep me up at night, it makes a poor server. Hosting took a rather large amount of work, because, in addition to provisioning the server, I also: abused Portage into acting like a configuration manager, restricted access to authorized accounts only, created a bridge to my IRC server, configured the server to auto-restart on crash, and created a custom backup solution. Despite this being a large number of tasks, they all had to be done in one go, making for a rather tiring, though engaging, stretch of work and a lengthy blog.

Updating

Part of what made upgrading my Minetest world tricky was that the world actually consisted of a number of separate components: the game engine, the "game" (a default set of mods for the game, formally called minetest_game or "Minetest game"), and, finally, all of the individual mods that I'd added to my world. I decided to begin by reading the commit logs of the respective repositories, and then doing the actual upgrade.

The game engine changes were mostly incomprehensible to me, although I did recognize that a new type of map-generation, Carpathian, was added (see below). The default game's changes were much more recognizable and included things such as: dungeon loot, more flower types, torches being flooded by water, blueberry bushes, fireflies and butterflies, binoculars, and minimap kits. The changes seemed reasonable enough, so I went ahead and did the upgrade. Next up were the extra mods I'd installed.

Figure: Landscape generated by the new Carpathian mapgen.

Unlike the Minetest engine and game, the extra mods that I'd installed did not all come from a single source and were thus far more varied in their changes. The cme mod (Creatures Mod Engine) had been abandoned by its creator and thus had no changes. craftguide, however, had seen numerous improvements, such as: "progressive mode" (which only shows recipes for items that the player has ever had in their inventory, rather than all recipes by default), cooking results and time, a wall-mounted sign rather than a book, a /craft command, and more. Since I maintain a fork of farming_plus, there were no additional changes. The mapfix mod, which fixes weird lighting and fluid propagation issues, had mostly fixes. pyramids, a fun mod which generates its namesake in deserts, was created by the same author as the cme mod mentioned earlier and was also abandoned, so no changes were made. Slightly more confusing was the signs_lib mod, which draws written text directly on the sign, as the upstream repo at https://github.com/minetest-mods/signs_lib no longer existed! After some hunting around I eventually settled on https://github.com/zeuner/signs_lib, which, in addition to various bug fixes, added support for steel signs in addition to wooden ones.

There were, of course, a few problems with some of the mods after I'd upgraded. First, all existing instances of the Crafting Guide from the craftguide mod were now appearing as "Unknown Node"; it turned out that both the item had been renamed from xdecor:crafting_guide to craftguide:book and the alias had been removed in-between when I first installed the mod and when I'd done the upgrade. I submitted a pull request, but the author wasn't interested in the compatibility, so I created fork with the change. Next, and I feel lucky to have discovered this in my limited testing, the pyramids mod would crash the server when the falling nodes should have been activated; thankfully, this was a simple fix: changing a function call from nodeupdate() to minetest.check_for_falling(). Last, a visually displeasing bug in cme would cause the creatures to simply vanish on death, dropping loot, rather than play through their death animation. This had actually cropped up in my 0.4.14 to 0.4.16 upgrade, but I hadn't gotten around to it. After a few hours of searching, I eventually took a lucky guess: I changed the set_hp(0) method call to set_hp(1) and the creatures began animating properly. It appeared that newer versions of the engine automatically remove things at 0 health. I was also lucky in that the mod was written such that the creature entered a "stunned" state and could no longer cause damage while going through their death animation, otherwise the fix would have been much more intrusive.

With the engine, game, and mods upgraded, there was one last little bit of clean-up that needed to be done; whenever I'd bring up the server I'd get the following unhappy, yellow messages:

	WARNING[Main]: /!\ You are using old player file backend. This backend is deprecated and will be removed in a future release /!\
	WARNING[Main]: Switching to SQLite3 or PostgreSQL is advised, please read http://wiki.minetest.net/Database_backends.

It turned out that the engine wanted me to change both the authentication database and player database, so I ran the following commands in order to do so:

	minetest --server --migrate-auth sqlite3 --worldname myworld
	minetest --server --migrate-players sqlite3 --worldname myworld

That was painless enough! At last, upgrade completed, it was time to start looking at hosting my own server.

Hosting

For me, it wasn't enough to simply spin up a server, manually copy files over, launch the Minetest daemon, and consider it "hosted"; at work I've been using Chef for configuration management in order to codify my work, and I decided to see if I could use Portage for a similar purpose here. In addition, I wanted to make the server more manageable by adding all of the features that I mentioned earlier.

Configuring the Games

The fun part of using Chef at work is being able to point it at a node and have it automatically do things like install packages, configure files, and add users, but using it for my hobby environment seemed to be overkill, like swinging a sledgehammer at a nail. In addition, despite its power, Chef must also consider cross-OS compatibility, which means that it must add a layer of abstraction, needlessly complicating things in my homogeneous environment. Instead, at some point I had realized that tasks such as installing packages, installing configuration files, and adding users are also done by Gentoo's package-manager, Portage! This meant that I could get the benefits of configuration management while becoming more familair with my system and without having to install extra software. I began by trying this method with my customized version of the Minetest game.

When I began creating the game, I had previously placed my desired modules into the mods directory rather than creating a formal game for them. Now, I forked the minetest_game repo and added my desired modules as git submodules, with the expected struggles of using git submodules, of course. In addition, I disabled the carts mod as I believe that automatic, cost-free travel takes something out of the game (but that point could be a whole 'nother blog post). An additional problem remained: I had created my first world, "Cereal", on an older version of the game and, as a result, it had some strange deviations from the base game; specifically, stone bricks did not have a default orientation and thus, through consistent placement on my end, faced west rather than the default of north; furthermore, many area were generated before tin ore was added, thus crafting copper ingots became needlessly difficult in newer versions. Patching the game was simple enough, but the issue was fundamentally specific to that world and not any future worlds that I might create, thus I ended up creating a different branch for each game: one for a generic world, called, uncreatively, frostsnow and another for the aforementioned world, called, even less creatively, cereal. For clarity, I also slightly modified the game icon to make the games distinguishable in the client GUI.

When I said that I was using Portage for configuration management-like uses, it's more that I had this in mind than I was able to realize it right away, as nothing about the game ebuilds is specific to configuration management rather than package installation. Since I was pulling the game from a git repo, Portage had a useful helper called an eclass which I could use with inherit git-r3. Then all I needed to do was to define EGIT_REPO_URI to be the location of the game's git repo and then define EGIT_BRANCH and EGIT_COMMIT to be the branch and its corresponding commit, respectively, and Portage would then correctly pull the repo (not too unlike the way a Chef resource might). After that I copied the src_install function from the default games-action/minetest_game ebuild in order to install the cloned game version. The end result for the "frostsnow" game can be found here.

A surprisingly nuanced issue then came up with versioning. Since my custom games are based off of the default game, it meant that I had to add a sub-minor version, making for a rather lengthy version string. Also, because the game version depends on the engine version, I needed to add a dependency, which I did with RDEPEND=">=games-action/minetest-${PV/%.?/}" in order to truncate the sub-minor version (at least, until after sub-minor version 9, which probably won't happen). All of this was straightforward enough, but, as I wanted old ebuilds to continue working even after the game had been upgraded, I realized that rebasing my games off of the latest upstream game would break that compatibility. It got trickier, too, because, while I'd like to have the cereal and frostsnow branches point to the latest version of their respective games, the eclass that I was using required the commit to be on the same branch as specified in the ebuild. Consider the following scenario:

Figure: Simplified layout of the Minetest game git repo using hypothetical 6.0.0 game version. Note that, because of the way ebuilds specify commits, each version would require a separate branch with this method.

In this case, even if I rebased my patches on the new game version without removing the old patches, I'd have to create a separate branch for each game type and version, whose name is the type and version of the game without the sub-minor version; this is in addition to a tag for each type and sub-minor version, resulting in cumbersome branch & tag specifications that pollute the branch namespace. This is what I initially did, but, now that I've had time to reflect and write this blog, I think it'll be far more sane to just do an actual merge into the respective game type's branch; that will allow me to keep the history intact and the branch specification sane. Thus, this was a great example of over-thinking as a result of using a process which doesn't fit the problem.

With the games now in an easily-installable state thanks to the repos and ebuilds, it was time to tackle my authorization problem.

Preventing Unauthorized Access

Some admins leave their servers open to the public as a service to the community; kudos to those people. While I enjoy having such services around, I was not willing to put the time and effort into moderating and maintaining such a service, and so what I needed was some kind of authorization system. By default, Minetest allows users to arbitrarily create new accounts, so I needed some way to inhibit that.

What form the authorization solution would take wasn't known to me at the beginning, hence I prepared to buckle in for the long haul. I began by checking out the Minetest Wiki server hosting page, but didn't find anything relevant to my needs. Checking the minetest.conf example in /usr/share/doc/minetest-${VER} showed a default_password field which meant users would have to know the password in order to create an account; while that would help somewhat, people might give the password to a friend, who would then give it to a friend, who would give it to a friend, &c., and before I'd know it some really strange people would be joining. No, that wouldn't work for me. Next, I checked the lua_api.txt file in the aforementioned documentation folder, and found in there a section on authentication! Perhaps, in true Minetest style, what I needed would be in a mod, so I began to search the Mod Releases subforum, where I found the auth_rx mod. After reading the forums post and documentation, I determined that this mod would likely suit my needs and began to try it out.

Turned out that auth_rx mod used a custom authentication database format. In order to migrate, a tool called convert.awk was provided in auth_rx/tools; the tool worked by reading the plaintext auth.txt file and converting it to the mod's alternate auth.db file. You may recall that I had previously migrated from plaintext to SQLite3; well, as far as I could tell, the tool only worked on the plaintext format, so I had to re-migrate from SQLite3 back to normal plaintext, then migrate to the mod's format. I also ran the advanced import in order to collect some metadata, though the data was somewhat tainted due to the fact that the import did not appear to take into account that multiple worlds (mostly test ones) had all been logged to the same file. With the authentication data in place, I had to write some login rules.

The auth_rx mod used a Domain-Specific Language (DSL) known as the Minetest Authentication Ruleset Schema (MARS). In addition to specifying the conditions that would allow or prevent users from logging in, the rulesets also allowed specifying custom error messages when access was denied. Luckily for me, the very first example showed how to prevent users from creating new accounts! Combining that with the custom error message, I was able to both block new, unauthorized users and to provide a message telling them how they could contact me in order to get an account created. There was one caveat, though: how was I supposed to create an account now? In-game, there are commands to remove an account, but must have been assumed that users would just create accounts by logging in, so there was no corresponding command to create an account. For now, I used the power of MARS in order to implement a quick hack: logins from my IP address would be allowed to create an account. Sure, it's not secure against someone who can spoof my IP address, but, dear reader, please promise not to tell those people until I implement a proper solution. One last enhancement I made was, again following the examples in the documentation, to rate-limit logins after consecutive failures, much like fail2ban; I set a fail all rule to check for $ip_attempts gt 0, $ip_failures gte 7, and age($ip_newcheck) lt 5m to throttle players for 5 minutes after 7 failed login attempts. The full configuration file, with my IP and e-mail redacted, was thus:

# Disable new player joins.
try "You must have an account created in order to join this server.  If you would like an account created, please e-mail the server admin at 'redacted@redactions.org' with your desired account name and, optionally, your desired password."
fail all
# FIXME: Implement non-hacky way to allow new player creation.
unless $addr eq 1.2.3.fake
if $is_new eq $true
continue

# Ban after frequent failures.
try "Too many consecutive login failures, please wait 5 minutes before trying to log in again"
fail all
if $ip_attempts gt 0
if $ip_failures gte 7
if age($ip_newcheck) lt 5m
continue

# Allow player login attempt.
pass now

Figure: New users must have an account created for them.

Though I now had what I wanted for logins, all was not yet well, as, whenever I logged into the server, I always had the creative privilege, giving me unlimited resources and mining power, though other users did not. This was bad because I wanted to be able to actually play the game as a regular user, but still have admin privileges in case they were needed. Revoking the offending privilege via the /revoke ${NAME} creative did not work; I also tried manually editing the authorization files, but nothing seemed to work. Perhaps I'd gotten the server into a broken state with all of my trials? I tried wiping it and re-migrating, but still had the same issue. What did work, though, was to remove the auth_rx mod, but that was hardly an acceptable solution, so I decided to dig into the mod's code in order to see if I could fix the issue. Searching for the authentication APIs mentioned in Minetest's lua_api.txt documentation file quickly led me to some code with the telling comments "grant server operator all privileges", "(TODO: implement as function that honors give_to_admin flag)", and "server operator's privileges are immutable"; it turned out that the code determined who the admin was from the name variable in minetest.conf and then gave them all privileges immutably, not even storing them in the authorization file. While the comments suggested that there's a proper solution using a give_to_admin flag, I wasn't keen on trying to figure it out at this point time, so I simply made a fork with the offending code removedhttps://github.com/clinew/auth_rx/commit/583daec42043141eacd0860e3115b6c10d7cab80. This meant that I had to grant myself all privileges before adding the modified version of the mod, otherwise I wouldn't be able to get them back, but that was a simple one-time setup. It wasn't the most elegant solution, but it was good enough for now.

With the authentication mod figured out, I at last added it to my games. At this point, even though I didn't have all of my required features, I had the ones that would allow me to feel comfortable hosting a live server while I worked on the other needed features, so it was time to prepare the server.

Preparing the Server

This is the sort of boring but nuanced work that makes life a bit more tedious than one might like. I'm not going to cover generic server set-up (SSH, fail2ban, logrotate, &c.) besides noting that DNS did not seem to like having two separate A records (minetest.frostsnow.net and mt.frostsnow.net) pointing to the same IP address, so I used a CNAME record for one of them instead. After doing generic server set-up, within the Minetest world itself I had to make a couple of changes, such as setting the gameid variable in world.mt to point to my mt_cereal game, giving myself an actual password by logging in and running /setpassword, and setting my privileges and migrating the authentication database as described in the previous section (I was actually using a temporary, test instance earlier). Lastly, I decided to set a static spawn point away from all the stuff that I'd already built; this was done by setting static_spawnpoint in minetest.conf, but, as the setting applies to all worlds created using that configuration file, and not just my current world, that meant that, in the future, I'd have to make sure any additional worlds can either use that spawnpoint or install said worlds into a different location. I then built a small house with some basic server rules and a craftguide at the spawnpoint.

Figure: New users spawn on the stone block in the center of the hut, where they can read the server rules on the left, rest in the bed in the corner, or view the craftguide sign on the right.

Actually, things didn't go quite this straightforwardly, as I encountered the following error when trying to install Minetest: /var/tmp/portage/games-action/minetest-5.0.1-r1/work/minetest-5.0.1/src/irrlichttypes.h:33:10: fatal error: irrTypes.h: No such file or directory. Looking into the matter, it appeared that Irrlicht was the game engine upon which Minetest was built, and the unstable ebuild I was using did not require it when the dedicated USE flag was specified. Now, I could have tried to fix this properly, but, again, please forgive me, I simply installed dev-games/irrlicht manually (in fact, it appears that newer revisions of the ebuild have done away with the flag entirely). Once Minetest was installed I configured it to use my world by editing the ARGS parameter in /etc/conf.d/minetest-server to include --world /var/lib/minetest/.minetest/worlds/cereal, and also added --logfile /var/log/minetest/minetest-server.log, since the default location under /var/lib for log files is blasphemous in my eyes. A couple weeks later, when the ebuild was updated (as previously hinted), I also needed to change the MINETESTBIN variable from /usr/bin/minetestserver to /usr/bin/minetest and add --server to ARGS. It appeared that the ebuild was marked as unstable for a reason!

Now the server was installed, configured, and running. I'd gotten far, but there were still a number of features that I wanted to add before I would consider things to be good enough for now.

Building the IRC Bridge

One feature that I considered an absolute must was to link my Minetest server with my IRC server. This would make it easier both to socialize with my friends and to monitor the server for any suspicious activity. Thankfully, a mod for this already existed. Since the mod didn't come with my distribution, I had to manually install its luasocket dependency, found at dev-lua/luasocket. Then I added the mod to my custom games as yet another submodule, and configured my world to trust it via setting secure.trusted_mods to include irc in minetest.conf.

Though I had the mod installed, connecting to my server is not straightforward, as I both require the use of TLS and a properly-signed client certificate. From the mod's documentation it did not appear that the mod supported specifying a client key and certificate, but that was okay, because I could use stunnel as a workaround. I've written about using stunnel in a previous blog post, and so won't cover the basics again here, but, of course, it seems impossible to touch TLS without getting burned somehow, and this experience was no exception. Instead of connecting like it should have, I kept getting the following error in my Minetest log: ERROR[Server]: IRC: Connection error: 127.0.0.1: /usr/share/minetest/games/test/mods/irc/irc/init.lua:138: closed -- Reconnecting in 600 seconds.... I couldn't figure out what was wrong with the connection, so I set debug = 7 in stunnel.conf, tried reconnecting, and then opened up stunnel.log, which showed: LOG3[1]: SSL_connect: 14077410: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure. Well, that still didn't help, so I checked ircd.log on my IRC server and saw: OpenSSL handshake error 'error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher' for '1.2.3.fake' port '6697'. "no shared cipher"? I had nothing set explicitly in stunnel.conf, but checking stunnel.log showed Ciphers: HIGH:!aNULL:!SSLv2:!DH:!kDHEPSK and InspIRCd was configured to use DHE-RSA-AES256-GCM-SHA384. I tried setting ciphers = ALL in stunnel.conf, which then connected successfully using..."TLSv1.2 ciphersuite: DHE-RSA-AES256-GCM-SHA384 (256-bit encryption)". Seriously? I was hardly in the mood to truly deal with this weirdness, so I configured both stunnel and InspIRCd to use HIGH:@STRENGTH as their cipher list, which actually worked as expected. Maybe one day I'll take another look at this, and what exactly TLS considers "HIGH"-strength ciphers.

As if dealing with perplexing cipher-negotiation errors wasn't enough, I also ran into a permissions issue which required a patch. Rather than have the client certificate and key be readable by all users, I wanted to harden my setup a little by making it only readable by the user which the server is run as; in addition, rather than run stunnel as root, I wanted it to also run as the aforementioned user. Running it as a different user was easily done by setting setuid and setgid in stunnel.conf, but after I did this, stunnel no longer started! Reading stunnel.log this time showed Cannot create pid file /run/stunnel/stunnel.pid and create: Permission denied (13). Checking permissions on the /run/stunnel directory showed that it was owned by stunnel:stunnel and, logically enough, did not have write access for other. Easy enough fix, right? I ran chown on it in order to fix the permissions, re-launched stunnel, and it died again! Why? Same reason! Looking at the directory again showed that its permissions had been reverted! After some digging I found in /etc/init.d/stunnel the line checkpath -d -m 0755 -o stunnel:stunnel -q $(dirname ${PIDFILE}); turns out that the daemon initialization script was helpfully correcting permissions on that directory but did not take into account that the owner and group might not be the default. The solution was easy enough: in the initialization script, grab the values of setuid and setgid from stunnel.conf in the same manner that the init script's PIDFILE value was grabbed, then use those values for the permissions of the directory; I filed a bug & patch for the issue, but, at the time of this writing, have yet to hear back. Well, I got it working on my system, so good enough for now.

After I'd established my TLS tunnel it was time to focus on configuring the bridge just the way I wanted it. Like the static spawnpoint mentioned in the previous section, these settings went into minetest.conf and so would apply to any world created under that configuration. Obvious enough was pointing irc.server to the address of the TLS tunnel (otherwise I wouldn't have been able to test it, obviously), but less conspicuous was irc.port, which was strangely absent from the README but useable nonetheless. Next, I set the bridge's IRC nickname with irc.nick, joined channel with irc.channel, and made sure to register both the nick and the channel on the server. By default, Minetest players are able to disconnect from the bridge with /part, but, as part of the reason for the bridge is to monitor the Minetest server, I disabled this by setting irc.enable_player_part to false; I also set irc.send_kicks to true so that any kick messages would be sent to the IRC channel. An interesting feature of the bridge is that it allows users to opt out of bridging their communication by prefixing their message with [off]; I wanted this feature for the IRC channel, but not for the Minetest server. I couldn't find an existing option for this in the mod, so I created a fork with the change, and incorporated that fork into my games instead.

Figure: The IRC bridge in action on both the Minetest server (above) and IRC server (below).

There was one, in my opinion, sub-optimal thing about this solution overall: the tunnel was currently bound to the loopback interface, which meant that, in the case of multiple servers, I'd have to take care to avoid having multiple tunnels bind to the same port, and, theoretically, malicious programs running on the node could connect to the server. What might work instead for the Minetest side of the tunnel would be to bind to a UNIX domain socket rather than the loopback interface; stunnel appears to support this, but the Minetest IRC mod does not appear to. Perhaps it would make a good addition someday, but, for now, I wanted to move onto the next task.

Recovering from Crashes

One of the mods I use has this bug which, once in a great while, causes the server to crash. I'm not sure what exactly causes the bug, but it's rare, and the server is just fine after a restart. This isn't a big deal when I'm playing on my own as I can simply restart the server, but I'd rather it not happen to another player when I'm gone and unable to restart it. Gentoo had a solution to this, but it would have been rather tacky, in my opinion, to test how well a production server handled crashes by, well, crashing it. Instead, what I needed was a test server.

Rather than configuring a test server all by hand, I decided to follow-through on my package-manager as configuration-management plans. I began by creating a games-action/mt_test ebuild, and had it create a custom user for the test server by invoking enewgroup and enewuser in pkg_setup(). Some weirdness followed, as Portage expected a directory by the package name to be created, so I ran mkdir ${WORKDIR}/${P} in src_unpack() in order to hackishly create it. Then I copied the daemon's configuration file, minetestserver.confd, from the regular Minetest ebuild, but it needed a few modifications; I changed USER to be modifiable, gave PIDFILE a world-specific suffix, and changed the ARGS logfile argument to both point to /var/log rather than /var/lib and to contain a world-specific suffix for its log directory. I then had the ebuild set these values via sed as part of src_prepare(). One of the default Gentoo services, net, allowed one to create specific network interfaces by creating symbolic links to its initialization script in /etc/init.d (for example, creating the symbolic link from net.eth0 to net would create the eth0 interface); mimicking this idea, I had the ebuild create the service minetest-server.test by creating a symbolic link to minetest-server in the src_install() function and then installed my custom .confd with the same service name in /etc/conf.d. The logging directory was created via running keepdir /var/log/minetest-${WORLD} in the same src_install() function. This created a skeleton configuration for the game.

Now, Portage doesn't have anything that I know of that'd be like Chef's data bags for hosting the actual world data, but one could probably emulate that using some kind of hosting service. I didn't want to do that, however, so the actual game data I copied manually from the production world. Surprisingly enough, after changing the port variable in minetest.conf things seemed to work pretty well, though I quickly changed irc.nick and irc.channel as well in order to prevent the name collision and main channel spam. The ebuild created for the job can be found here. Satisfied with my test server setup, it was time to actually configure auto-restart.

By default, Portage would use the start-stop-daemon in order to control services, but there was another option, the supervise-daemon, which would monitor and restart crashed services; this could be configured by the service's initialization file in /etc/init.d as documented in openrc-run(8). The default Minetest server had defined custom start() and stop() functions for the actions of the same name. I removed the symlink which I had created earlier, replacing it with a copy of the file it was previously pointing to, and then tried to modify these functions in order to suit my needs, but kept getting weird results, such as the service respawning forever, even after I had told it to actually stop. Eventually, I tried removing these custom functions (leaving the service to run its default ones) and instead only defining the relevant variables based on their values in the /etc/conf.d file, and it worked! The resulting file can be found here; the main trick was to set the supervisor variable to supervise-daemon so it'd monitor and restart the service. After that, I could crash the service (via kill, or it could crash on its own) and it'd restart automatically. Once I'd ported the changes over to the production server, I had one final task left.

Creating a Backup Solution

There was no way I was going to go through all of this hosting effort and not have some kind of backup solution, but settling on a particular method was tricky. I wasn't keen to store all of my data in the cloud where I may be cut off from it, but that meant I'd somehow need to retrieve it from the Minetest server onto my home server, thus making the solution more complex. I also didn't want to use large amounts of data storing backups, nor set up a particularly complex data delta and deduplication scheme right then, so what I ended coming up with were some handwritten scripts to create, retrieve, and cycle the backups. In addition, I tried to make my life easier in the long run by using Portage to install and configure the scripts by creating ebuilds for them. I began by working on the Minetest server side.

Generating the backup itself was simple enough. There was some discussion on the forums regarding whether one could safely run backups without shutting the game down first, but without much consensus, so I decided to play it safe and shutdown the game server when creating any backups. The steps were simple: clean any existing backup copy, shutdown the server, create the backup with tar, restart the server, then compress the backup with bzip2. Since I would eventually like to be able to host multiple instances, I parameterized the script to accept the service name as the first argument and the Minetest user's home directory as the second, but the script would use a sane default if these were not specified.

With the backup-generating script created, it was time to install it, but, rather than manually install it, I decided to use an infrastructure-as-code approach and have an ebuild named app-backup/minetest-backup do so for me! As with the test server, I used enewgroup and enewuser within src_unpack in order to create a special user for the task, in this case named mtbackup. Since I (currently) wanted the backup script to run daily with the default arguments, I could install it directly by placing the script into the files subdirectory of the ebuild, then running exeinto /etc/cron.daily followed by doexe ${FILESDIR}/mtbackup in src_install(); this would run the backup (at 3:10 a.m.) as the root user rather than the mtbackup user, but I needed that level of permission in order to control the service anyways. In addition to generating the backup, I would also need to pull it, preferably securely and with proper authorization, so I chose to use OpenSSH's scp; this meant I'd have to authorize the mtbackup user for login. Now, Portage has a useful sandbox feature to prevent mis-behaving ebuilds from installing packages where they shouldn't, but I wasn't using Portage like I should have, so I had to disable the sandbox feature and face Portage's unhappy messages. In the ebuild, I disabled the sandbox with addwrite /home/mtbackup, set the insertion directory with insinto /home/mtbackup.ssh, added the authorized keys file with doins -r ${FILESDIR}/authorized_keys, and finally made mtbackup the owner of their configuration with fowners -R mtbackup:mtbackup /home/mtbackup/.ssh; the user's key was now authorized! Lastly, as a courtesy, I added a motd entry to the minetest.conf file warning users of the daily downtime for backups. Though I was now able to generate a nightly backup, I would still need to pull and store them.

Retrieving backups ended up being a bit trickier due to my space constraints and workaround; what I desired was to keep 7 daily backups, 5 weekly backups, and infinite monthly backups. This would make space-usage manageable while still allowing a decent timeframe for catching any kind of errors or allowing world-restoration in the case of out-of-control vandalism. In the hopes that I wouldn't have to roll my own script, I decided to see if I could trick logrotate into doing what I wanted for me. At first things seemed to be going well, I could set rotation periods such as daily, weekly, and monthly, rotation count with rotation #, and timestamp the files with dateext and dateformat. Unfortunately, each of these rules expected a target, and subsequent definitions overrode previous ones, so pointing multiple rules at the same file wouldn't work; I could have made multiple hard links to the same file contents and pointed a rule to each of them, but it would have gotten ugly and cluttered quickly. In addition, I had to specify an alternate state file for keeping track of rotations, couldn't figure out a way to keep an infinite number of rotations (for the monthly backups), and found testing to be rather difficult. I abandoned the idea and wrote a script instead.

The script itself is pretty average: there's a rotate function for its namesake, I use hard links for the backups so as not to duplicate data, and allow SOURCE and DESTINATION parameters as the first and second arguments to the script, respectively; I also had to add -o BatchMode=yes to scp in order to prevent the script from potentially stalling. Now, creating the ebuild, app-backup/minetest-backup-client, was way more fun. As before, I added a new user and group, this time called mtbackupc, then I added SSH configuration again, except this time I added the known_hosts file containing the Minetest server's public key rather than the authorized_keys file. The backup script needed to be handled a bit differently this time, though, as I wasn't going to have it race with backup creation; I needed to run it at a later time. I began by installing the script as a regular executable file in /usr/bin called mtbackupc, then, after some digging, found a way to manually insert a cron job for the mtbackupc user at /var/spool/cron/crontabs/mtbackupc as file contents 0 4 * * * mtbackupc in order to have the script run daily at 4:00 a.m. (if the backups take over 50 minutes to complete on the Minetest server I'm totally hosed here); this was followed by a call to fowners mtbackupc:crontab in order to fix permissions on the file. Last, I would have to manually install the key, as posting a private key publicly as part of the repo would be foolish, but, in case I ever forgot to install the key, I added a check in pkg_postinst() that would warn me during ebuild installation if no file was located at /home/mtbackupc/.ssh/id_ed25519. With both pieces now in place, I now had a working backup solution.

Developing the backup solution raised an interesting question about how the backup ebuilds ought to be designed that I've yet to fully resolve in my head. The question is one about how configuration-dependent the ebuilds ought to be; for example, installing a known_hosts file for a specific key, while convenient, is very much dependent on a particular key, in addition, the script itself currently uses a hardcoded mt.frostsnow.net domain. It might be better practice to include all generic bits in one ebuild, then create another ebuild, perhaps under something like sys-conf, with all configuration-dependent stuff that would then require the previous ebuild (except made generic) as a dependency. Similarly, should Minetest ebuilds like games-action/mt_test include a skeleton configuration for the server or pull in the actual data as well? I'm currently leaning towards a skeleton configuration, so that others can reproduce the world if they wish to but my instance of it will be privately managed by myself. The world is also too large to put directly in a repo.

One last note of amusement for me is that I actually developed the ebuilds on a test VM before deploying them to production, and, when I uninstalled their packages, Portage removed the files it had installed as well! This was unlike Chef, where I would have needed to write a separate recipe that would undo the work of the previous one. Though, Portage did leave the users, and, in fact, failed to clean the directory /home/mtbackupc, as I had actually logged into that user in order to do some debugging, thereby creating a .bash_history file and causing Portage to complain that the directory was not empty and would thus not be removed (but the files which had been installed there by the ebuild were removed). Pretty neat, though not entirely relevant to my needs and with arguable robustness.

Future Work

As answers often beget more questions, so, too, did this work beget more work. First, I needed a sane way to create a new user account as admin, probably from an in-game command, though it would also be useful if I had a bash command to do so. Second, I'd like a proper fix for the privilege issue in auth_rx that would allow me to arbitrarily add and revoke privileges on my admin account. Thirdly, as a nice-to-have, have the auth_rx mod properly detect new servers, as right now it fails on new servers because there's no player database to initialize from, but there also isn't one to import! Fourth, getting the irc mod to connect to UNIX domain sockets would allow me to easily set up new server IRC bridges via stunnel without worrying about port collisions and providing a layer of access control (I think). Fifth, I'd like to create a proper mt_cereal ebuild in the same way I have mt_test. Finally, it might be useful to pipe the Minetest log file to a locked IRC channel so I can monitor the game from IRC, but that may actually be a bit too much.

Figure: Screenshot of the now properly-hosted game world.

That will all have to wait for now, though. Things appear to be running smoothly, and hopefully all of the work I put into configuration management will pay off in the long run (it certainly made installing the test server a bit easier). My current goal is to keep things running smoothly while I take a break to work on other things, and then get back to this in due course.

Addendum 2019-07-21

Shortly after posting this blog, my friend ushered me back to my main base, saying that he had a gift for me! After a bit of nagging, I walked back to my base and saw:

Figure: Crystal Maiden attentively watches over my base.

He'd created a pixel art version of DotA2's Crystal Maiden as decoration for my base! Super cool, and a nice gift to receive after having finished the blog!