Every so often I find myself with an idle hour and decide to use that time to package some new software for Linux. A common activity among nerds, I’m sure ;). This blog post is a write up of what I did, and why, which may be useful to others with time on their hands.
I keep meaning to live stream when I do, but on this occasion I
had a bad hair day was also listening to and engaging with a podcast, so it wasn’t practical. In leiu of that, while the information is fresh in my head - the morning after - I thought I’d write down some notes here.
Why do this
My Linux distro of choice is Ubuntu. A significant number of Ubuntu users are on “old” Long Term Support (LTS) releases. Some estimates put it at around 10x or 20x more users are on Ubuntu LTS releases than the interim non-LTS releases. It’s hard to get new software into the repositories of old Ubuntu releases. While it’s possible for developers to create PPAs or repositories for their software, that’s hard for new users to discover. If I make and help publish a snap for something new, that can get in users hands within hours (or even minutes), with updates coming as they’re published. I wrote a (work) blog post titled “Snaps: How we got here” if you want to know more about “why?”.
Selecting a target
There’s always something to package. Out there a new piece of software has been published, or an old program needs preservation. Perhaps someone explicitly asked for help, or there just something I think is cool, that should be easier to install for Linux users.
I use f5bot which notifies me via email when the keyword ‘snapcraft’ is mentioned on Reddit, HN or Lobsters. I’ll also get pinged by a number of friends online when they discover something and think it should be snapped. You know who you are, thanks for keep my “fun software pipeline” full. :) Sometimes I’ll browse the GitHub Trending page to see what’s new in the world. That’s a great way to find new stuff, especially
On this occasion it was a mention in a Telegram channel I saw scroll past.
Some things make sense to snap, others less so. For example, a ‘leaf’ application which is relatively new, fast moving, has complex installation instructions (for novices) might be a good candidate. A library which depends on specific kernel modules, and has esoteric hardware requirements might be less interesting, difficult, or flat out impossible.
Indeed some classes of applications are easier to package as a snap than others. Command-line utilities written using ‘modern’ languages like Go and Rust are way eaiser than large graphical applications built using bleeding-edge versions of Qt or GTK. If I only have an hour to spare, I’m likely to go for easy pickings.
From a political point of view, I’m also unlikely to touch an application where the developer has previously expressed that they don’t want their software packaged in such a way. No point wasting time packaging something if a pull request is going to be rejected. Most developers aren’t that way inclined though, and welcome having their software in more hands.
In this case the application - t-rec - is new (to Linux), is actively developed, and the developer specifically asked for help in packaging. The fact that it’s made using Rust is a bonus here. All this was gleaned in a few moments by browsing the repo. I’ve not technically committed to this project yet. This is important to note, because I will often look at something, and reject it because it will take to long, or maybe it’s a commercial application, or has some weirdness that means it’ll be a timesink. This wasn’t one of those cases, so I moved on to snapping it.
“But Alan, it’s Rust, anyone can just use curl to install rustup, then get rust, the build dependencies, and build the software from source themselves”.
You might do that, millions of others won’t / can’t / don’t.
Fork - clone - branch
Step one! Fork!
The repository is on GitHub, so my first step is to fork the repo, clone it to my laptop and create a branch. All standard GitHub workflow stuff.
Aside: I don’t always do this. Sometimes I’ll start by creating a folder locally, and build the snap by pointing it at the upstream source. If I’m confident I can prep a PR and have it pushed in short-order, I’ll go the fork-clone-branch route. If I think this might take longer, and be something I will have to publish myself (under my own name) then I’ll build locally without forking. Maybe I’ll do a separate tutorial for that sometime.
Create a snapcraft.yaml
snap/snapcraft.yaml is the file that drives the
snapcraft tool to build and package the snap.
Inside the project folder, I’ll just
snapcraft init which looks like this:
$ snapcraft init Created snap/snapcraft.yaml. Go to https://docs.snapcraft.io/the-snapcraft-format/8337 for more information about the snapcraft.yaml format.
The initial yaml looks like this:
name: my-snap-name # you probably want to 'snapcraft register <name>' base: core18 # the base snap is the execution environment for this snap version: '0.1' # just for humans, typically '1.2+git' or '1.3.2' summary: Single-line elevator pitch for your amazing snap # 79 char long summary description: | This is my-snap's description. You have a paragraph or two to tell the most important story about your snap. Keep it under 100 words though, we live in tweetspace and your description wants to look good in the snap store. grade: devel # must be 'stable' to release into candidate/stable channels confinement: devmode # use 'strict' once you have the right plugs and slots parts: my-part: # See 'snapcraft plugins' plugin: nil
We’re going to almost completely replace this, but it’s a nice starter, as it reminds of the minimum bits needed to build a snap.
This is the easiest bit! I just yoink the details from the upstream repository. Typically this is copy/paste territory. The one you need to be careful of however, is the
name:. This is the registered name of the application in the store, and shouldn’t already be taken by someone else. I didn’t actually check, but
t-rec seemed pretty unlikely to be used already. (I have subsequently checked by running
snapcraft register t-rec and it wasn’t taken (but is now :))).
name: t-rec summary: Terminal Recorder - t-rec description: | Blazingly fast terminal recorder that generates animated gif images for the web written in rust.
We also have
adopt-info which I’ll come back to below.
Choose a base
core18. That means when this snap is built it will do so (by default) inside an Ubuntu 18.04 LTS VM (or container). When the user installs the snap they’ll need the
core18 snap too (which will be installed automatically) which contains a super minimal (~ 58MiB) Ubuntu 18.04 base system.
Ubuntu 18.04 LTS is supported, but it’s coming up on 3 years old now. So I decided to go for
core20 for my
base. There’s a few side-effects of this. Firstly, logically perhaps, it will build the snap inside an Ubuntu 20.04 LTS VM or container, not Ubuntu 18.04 LTS. On the end user system, they’ll need the minimal (~ 64MiB)
core20 to match - which again, will be automatically installed for them.
A second effect of choosing
core20 is that
snapcraft (the tool) has had some behavioural changes, and updates to features between
core20 based snaps. There are a few features missing in snapcraft when building a
core20 snap as opposed to a
core18 based snap. One example is the currently missing “GNOME Extension” - but I don’t need that because this is a command-line utility, not a GTK / GNOME application.
core18 is recommended. I’m only doing this because I know I don’t need anything that’s currently ‘missing’ from snapcraft for
Snaps have (simply) two confinement models.
strict confinement in which you (as the publisher) specify a set of
plugs which interface the application to resources on the system such as
audio-playback and so on.
classic confinement is (essentially) unconfined, so the application being snapped behaves much like any other binary on the host, unrestricted by
strict snaps are.
We usually urge developers to aim for
strict confinement. That’s not going to work for
t-rec because the nature of the application requires it to be able to run any arbitrary binary on the system.
strict confinement would block this, making the application pretty much useless.
If this snap works, and the developer accepts the PR, and decides to publish it in the store, then they’re going to have to request classic confinement. You can’t just make a classic snap and publish it in the stable channel of the Snap Store, the review tools on the backend won’t allow it. There’s a process for vetting applications and publishers, which the
Future Alan with a spoiler here: The upstream publisher accepted the PR, pushed the snap to the Snap Store, and has requested classic confinement. Once Canonical sleep off the new year, they’ll/we’ll action that.
Just set this to
stable and we’ll move on. I personally wish this would go away. Just set it and forget it, that’s what I do.
Ok, here’s where we get to the meat and vegetables of the snap. This is where we define how to build the software we’re planning to put inside the snap.
t-rec is a Rust application, which
snapcraft knows how to build. So we just specify the
rust plugin, location of the source files -
. - because we’re sat in the repo.
Recall earlier where I mentioned sometimes I create a folder and build the snap locally without forking. In that case I’d set
source: https://github.com/sassman/t-rec-rs.git, and at build time
snapcraft would get the source directly from the main branch (or whatever
source-branch I specify).
parts: t-rec: plugin: rust source: .
You can run
snapcraft help rust to learn more about that
The upstream developer has a nice README.md which details the additional requirements needed when building the snap, and things required at runtime. It’s a pretty minimal list, which is neat.
sudo apt-get install libx11-dev imagemagick
Isn’t it handy that developers use Ubuntu to build their software ;). I don’t need to translate package names from some other distro here!
Note that one is a
-dev package - which will be required at build-time, the other is a toolkit of image manipulation utilities, which I guess we need at runtime to create the .gif this application is designed to make. So we specify those like this:
build-packages: - libx11-dev stage-packages: - imagemagick
At build time
snapcraft will pull in
libx11-dev in the build VM (or container) to ensure
t-rec builds okay. It will also stuff
imagemagick into the final snap, so
t-rec can find it.
When the snap builds, it needs a version number to stamp on the file. A typical snap filename looks like
snapname_version_architecture.snap. So mine will end up as
t-rec_v0.4.0_amd64.snap, hopefully. There’s a variety of ways to enumerate the version number. I’m going to just use the git tag. We do this in the
override-pull: section, which is just a shell script that gets run at build time by
snapcraft just prior to the actual build itself. We use
snapcraftctl pull to tell
snapcraft to just do whatever it does to pull the source for this part specified in
source: earlier, then we jump in and set the version for the project. This is consumed by
adopt-info up the top of the yaml metadata.
override-pull: | snapcraftctl pull snapcraftctl set-version "$(git describe --tags)"
Sometimes developers have weird build systems, or sets of commands that need to be run, which diverge from a typical
autotools plugin expectations. This is fine, because we can override the build process, just as we can override the pull stage. In this case I’m going to create a symlink because
t-rec expects a binary called
usr/bin/convert to exist, which (for whatever reason) doesn’t happen when we staged
imagemagick earlier. I use an environment variable set inside
snapcraft at build-time called
SNAPCRAFT_PART_INSTALL which points to the internal directory where the final assets for this
part are assembled. So we run the build as per the plugin expectations with
snapcraftctl build then we jump in after and ninja our symlink.
override-build: | snapcraftctl build cd $SNAPCRAFT_PART_INSTALL/usr/bin ln -s ./convert-im6.q16 ./convert
I discovered this after doing a successful build, but the application would error, saying it couldn’t fine
usr/bin/convert. I did a quick
find /snap/t-rec/current on the installed snap (although
less t-rec_v0.4.0_amd64.snap would also have worked) to see what files were delivered. Perhaps
convert was in another location. Then I remembered that the
convert binary on Debian based systems points to
/etc/alternatives/convert which in turn points to
alan@robot:~$ which convert /usr/bin/convert alan@robot:~$ ls -l /usr/bin/convert lrwxrwxrwx 1 root root 25 Aug 17 14:54 /usr/bin/convert -> /etc/alternatives/convert alan@robot:~$ ls -l /etc/alternatives/convert lrwxrwxrwx 1 root root 24 Aug 17 14:54 /etc/alternatives/convert -> /usr/bin/convert-im6.q16
imagemagick deb is pulled into the snap, this dance doesn’t happen. So I worked around that with the symlink.
At this point I typically try building the snap - even though the yaml isn’t actually finished yet. More on that below. But for now, let’s try and build the snap.
Note: By default
snapcraft will use Multipass. I prefer to use LXD. I already have
lxd configured on my desktop, but if I didn’t then
snap install lxd and
sudo lxd init --auto followed by logging out/in (to add me to the
lxd group) usually works for me.
Ok, from the repo directory I run this:
snapcraft --use-lxd --debug --shell-after
--use-lxd- will spin up a container using
lxd. The OS release will match the
basespecified earlier, in this case Ubuntu 20.04 LTS.
--debug- if the build fails for whatever reason,
snapcraftwill leave me ‘inside’ a shell in the container (or indeed the VM if using multipass) in which I can poke around to debug why it failed. I can also re-run
snapcraftinside the container which saves a little time as it won’t need to start it up again then shell in. The project repo folder is mapped into the container (or VM) so it’s possible to continue editing the yaml from ‘outside’ the environment, and iterate by re-running
snapcrafton the ‘inside’. I find this a super quick way to work on a snap.
--shell-after- is exactly the same as
--debugbut it always drops you to a shell inside the container, even if the build was successful. I do this because sometimes I want to poke around and see what built even if it worked fine.
Here’s some of the end of the output of it running the build:
Compiling crossbeam-deque v0.8.0 Compiling rand_chacha v0.2.2 Compiling rand v0.7.3 Compiling tempfile v3.1.0 Compiling x11rb v0.7.0 Compiling jpeg-decoder v0.1.20 Compiling tiff v0.6.1 Compiling image v0.23.12 Finished release [optimized] target(s) in 1m 04s Installing /root/parts/t-rec/install/bin/t-rec Installed package `t-rec v0.4.0 (/root/parts/t-rec/build)` (executable `t-rec`) warning: be sure to add `/root/parts/t-rec/install/bin` to your PATH to be able to run the installed binaries Staging t-rec Priming t-rec Snapping | Snapped t-rec_v0.4.0_amd64.snap snapcraft-t-rec #
Note here I’m left in the container, which is named
snapcraft-(snapname). Just CTRL+D to drop back to my main machine.
snapcraft-t-rec # exit alan@robot:~/Source/popey/t-rec-rs$
Ok, so we have a snap which builds, but it won’t actually work! That’s because despite building a
t-rec binary, and bundling some binaries which came with
imagemagick, we haven’t exposed them to the world outside the snap. We do that with an
apps stanza. We only really need to expose one binary
t-rec from the snap to the host.
apps: t-rec: command: bin/t-rec
I didn’t specify any
plugs: for this command because it’s a
classic confined snap, so is effectively unconfined. If it were strictly confined I’d consult the documentation to identify the right interfaces to make the application function while confined. Not needed here.
The app name matches the snap name. As such, once installed a binary called
t-rec will appear in
/snap/bin which will setup the environment and run this rust binary we built. If we wanted to (for some reason) expose the
usr/bin/convert binary to the outside world we could do this:
convert: command: usr/bin/convert
However, as that command doesn’t match the name of the snap, and may collide with another binary on the host of the same name, it gets namespaced on install. So that would be
t-rec.convert. I don’t need to expose that binary, so I didn’t do this.
apps: stanza added, we re-run
snapcraft as above, and we get a successful build.
Install & Test
Installing locally is done with:
snap install t-rec_v0.4.0_amd64.snap --dangerous --classic
--dangerous- is you acknowledging that this snap didn’t go through the Snap Store review process. It could just as easily have been handed to you on a USB key, or maybe I downloaded it from a random URL. I’m saying I trust it, given I built it.
--classic- is you acknowledging that this is an unconfined application, and as such has free reign over whatever your user is able to do. Just like binaries you install from a PPA or the Ubuntu repository ;).
I tested the application by opening a new terminal, ran the application for the first time and got this:
Narrator: It worked.
Here’s the final
yaml I submitted as a pull request to the upstream project.
name: t-rec base: core20 adopt-info: t-rec summary: Terminal Recorder - t-rec description: | Blazingly fast terminal recorder that generates animated gif images for the web written in rust. grade: stable confinement: classic parts: t-rec: plugin: rust source: . override-pull: | snapcraftctl pull snapcraftctl set-version "$(git describe --tags)" override-build: | snapcraftctl build cd $SNAPCRAFT_PART_INSTALL/usr/bin ln -s ./convert-im6.q16 ./convert build-packages: - libx11-dev stage-packages: - imagemagick apps: t-rec: command: bin/t-rec
Once the upstream developer has done the necessary dances to get
classic confinement approval and publish to the Snap Store, the t-rec snap will be available for anyone running Ubuntu 14.04 through 20.10 and a ton of other distros too, with a simple
snap install t-rec. I did see an amusing suggestion on reddit that they should have called it
t-recs, which does indeed sound better. If they want to do that, there’s only a few simple changes required in the yaml above.
These rust-based snaps are super fun to make. I cloned the repo at 19:32 last night and pushed the PR at 20:21. Not bad given I was also chatting on a podcast at the same time.
Hope that’s interesting to some of you. Happy snapping. 🥰