I was recently interviewed by Joe Ressington for Late Night Linux Extra episode 14. Here’s a transcription I typed up, which may be useful. I used an automated tool to create the transcription, then tidied it up myself. If you spot anything which doesn’t match the audio, and is materially important, do feel free to propose an edit on GitHub (link at the top of the page).
Joe: This episode is about snaps, quite the controversial topic in some circles, and who better to talk about it than Alan Pope - “popey” as he’s affectionately known by almost everyone. According to his twitter bio he is a Developer Advocate at Canonical working on Snapcraft and Ubuntu. Now, in case you don’t know popey is a good friend of mine and I’ve done many podcasts with him over the years, but this, he’s Alan, he’s Alan Pope of Canonical, he’s not my friend. It’s a work thing for him, because you know, I wanted to ask him some fairly difficult questions and try and clear up some of the misapprehensions and just get to the bottom of what is going on with snaps.
Joe: Now, because it was work for him, I made sure to do it during his work day which means I had to get up “early” - it was the afternoon really, but it was early for me. I’d had no sleep, I was feeling very rough that morning. I say morning for me, You know what I mean? Some of this sounds a little bit low and I have got a cold or something. Don’t worry about that. It wasn’t that at all. Thank you everyone for supporting us on PayPal and patreon. It’s really really appreciated. Especially as we’ve now gone weekly you can go to latenightlinux.com/support and you’ll find details there. You can get an advert free RSS feed for five dollars or more on patreon. So do check it out.
Joe: Alright, then without further ado. Let’s hear me and popey talking first thing in my morning. Thanks for joining me Alan.
Alan: You’re welcome.
J: Never called you that before. I don’t think.
A: No nobody does.
J: So I’ve got you on to talk about snaps very brief overview for anyone who doesn’t know what snaps are, a universal packaging system.
A: Yeah software packaging for Linux.
J: Cross-platform cross distro. The goal of it, right?
A: Well, there are lots of goals. That’s one of them. Yeah its so that developers can create a package and we take care of the cross distro part. They just take care of packaging their software.
J: Yeah. Okay. Now one of the reasons that I’ve got you on is because there has been a growing PR problem as far as I can see with snaps. I personally use them and they work pretty well for me and I don’t have many problems with them, but there seems to be a vocal if small group of people on the internet who are just sort of inherently against snaps and it seems to be a PR problem in general. Is that something that you acknowledge?
A: I can certainly acknowledge that there are a group of opinionated people in the Linux community who like to repeatedly stress their opinions no matter what someone might say that counters those opinions sure.
J: And some developers have had backlash for releasing their software as a snap and “why isn’t it a flatpak”, “why isn’t it available as x,y & z?”.
A: Yeah, that’s been interesting to see companies who’ve made the effort to package their software in such a way that more people can get it more easily and discover it and get software updates, you know, nice and promptly and the first response on their blog is from some rando on the internet saying, you know, “why didn’t you do it differently?” But I don’t think that’s a thing that’s unique to software packaging. There’s a lot of opinionated nerds out there who feel it’s okay to express themselves, you know in comments or whatever medium they like and everyone’s entitled to their opinion for sure, but you can’t badger people into working on a particular thing. The fact is that we work with some companies and understand what their problems are. That’s what try to do is understand what their software distribution problems are and we try and solve them and having someone come along after you’ve spent months and months working with someone in order to solve their packaging woes and just leave her one line comment on a blog post saying or “what about flatpak?” is, you know, it’s a little sad really.
J: Yeah, but do these companies take note of that feedback or do they just brush it off?
A: So I think it depends on the developer really if it’s a one-person band who’s developing a little bit of software and they don’t have the time, and this this works both ways as well. So if a developer doesn’t have the time to support lots of different packaging methods, they might just put one out there plus a source tarball and say have at it do it yourself and that’s been the case for software packaging on Linux for years. This is not a new thing. I’m not a new phenomenon that you package your software in one way and there are people out there who want it done a different way. That’s not new. It’s just different package name is inserted into the argument, that’s all. And so for a smaller organisations where they don’t have the time and resources to do it then often. They’ll just say yeah go for it and someone in the community can maintain it. If it’s a larger organisation then they may well want to support all the different packaging formats because they don’t want to miss out on those potential users who support one system or another and that’s the same, you know, deb or rpm snap and flatpak and appimage and nix and whatever other packages are.
J: Do you think that the negative feedback has put any companies off though from releasing their software as a snap?
A: That’s a tricky one to answer. I can certainly say there’s there’s people I’ve talked to and said “Hey, would you like to make a snap of this thing?” And they say “No, thank you very much.”, you know, “Thanks for getting in touch. But no, thank you. We’ve chosen this other option”. Whatever other option is, it could be appimage, they could be just a deb or it could be just a tarball. Some companies say “no we don’t make binary packages.”, “Here’s the software make it yourself.” And so every company is different. I wouldn’t necessarily say there’s a glut of companies or developers out there who’ve seen the feedback and all the writing’s on the wall. We’re not going to do it. There’s a mixture of feelings out there because everyone has a different perspective different amount of resources, different aims and goals with their software. So yeah, probably there is someone somewhere who’s looked at this and said no, I’m not going to touch that because some people are angry about it on the internet, but equally there are others who say screw those guys we’ll do what we want and you can choose to use it or you can choose not to just don’t use it if you don’t like it.
J: Yeah, but there are an increasing number of distros who have not necessarily been hostile to snaps but of not necessarily facilitated the installation of them. And have gone more the flatpak route or the you know, strictly deb repo brutes. You know, I’m thinking Linux Mint who have been actively hostile and elementary who have sort of been like well, you know do what you want but we’re going to go with flatpak by default.
A: Yeah and everyone who has a Linux distro has their own goals and has their own mission. And yeah, I completely understand why elementary have made the decisions they have they’ve done that absolute best to make sure that everything on the desktop is open source, and that everything on the desktop complies with the elementary human interface guidelines. And I think they want to own every single part of the stack if they possibly can aside from the fact that a lot of is hosted on GitHub that aside they want to own as much of the stack as they possibly can and so they chose to be able to have a lot more control over what they do whereas others don’t mind
J: Well, you mentioned the open-source aspect. I mean that is somewhat of the elephant in the room here that the snap store is not open source, and you’ve discussed that in other places at length, so we probably don’t need to go too far down that road but it’s not possible is my understanding to use snaps with completely free software. Is that still the case?
A: I take a kind of opposition to “snaps are closed source” and “the snaps store is closed source”, and the blanket statements. There is a small part of it, that is non-free. Yes, that’s true. And some people have a philosophical objection to that, fair enough. Everyone’s entitled to their opinion. Some people run Trisquel, some people run Ubuntu with an Nvidia driver. Okay. Everyone’s got their own mission in life. And now I don’t want to go through that that whole argument again. Yeah, there are places I’ve written it but I think the main point is is that we’re doing what a lot of people do on other platforms. We created a central store because it makes it a lot easier for people to discover software and it makes it a lot easier for developers to know they got one place to put their software where they don’t have to publish in eight different places. They just put their snap in one place and they’re done. It makes it easier for users to find stuff and it makes it easier for developers to publish stuff. I can completely understand the argument. For having lots of separate repo’s, but then we tried that with ppas and we found that that wasn’t discoverable and was frustrating for users having to manage ppas and then discover that a ppa has disappeared or all the packages are deleted from a PPA and that’s frustrating for the user and we’re trying to make it less frustrating for the user not more frustrating for the user. So yeah, we wanted a central store and the people who design such things have decided it should be non free that small part of it. The vast majority of it is open source. There’s a tiny bit which is effectively a web server that is non-free. But the vast majority of it is open and you can install snaps and build snaps on a completely open source system. That’s not the case that you have to use non-free software to build snaps. That’s not true at all.
J: But don’t need to pull some snaps down to build snaps.
A: Not always, no. So a snap is really just a compressed file. It’s like it’s an archive is such a zip file, right, and at the lowest level you can put anything you like inside a snapm now we’ve made convenience tools like snapcraft and online build service that make it easier to do that. And yes, they leverage stuff in the store which is snaps, you know to make another snap the build on the technology we’ve already made you don’t have to use that you could literally compress a bunch of files and make a snap and there’s no requirement for any external service or external software to do that you basically just use mksquashfs and boom you’ve got snap
J: Right and then you could use snapd to install that.
A: Yeah, you can do snap install ./filename.snap. You could totally do that and you could put them all on a web server and call it your store. It is technically possible to do this. We’ve you know, we’ve had a lot of complaints from people who said oh well, yeah, it’s non-free. Well wright one that’s free. Like you could it’s not a huge amount of work. It’s basically just a web server with a bunch of files a bit of authentication to make sure that each developer owns their own thing. You don’t have to use it.
J: Well speaking of developers. It’s my understanding that anyone can upload to the snap storage that true
A: Anyone who’s got an account and who accepts the terms of service, yeah.
J: Yeah, right. So how do you stop people putting malware on there then?
A: So it depends what your definition of malware is really. Like most software stores we have have tools at the back end that scan things and will reject things that are doing things that we don’t want to happen in the store. So there are some back-end scanning tools that look for stuff. But also if a user sees something that they don’t like in the store they can report it and that comes to us and if it goes against our terms of service, then we’ll remove it and we’ve done that with a bunch of snaps not a tremendous amount because there haven’t been a tremendous number of what you might call the blanket term “dodgy” things that have been uploaded but those that have we yoink them, get rid of them.
J: Because something that a new user to Linux ask me about was whether he could trust a snap. I think it was the signal snap. Maybe that was built by snapcrafters. I think that’s what it says when you snap search Signal Desktop, and he asked me well. “Who is that? Who is snapcrafters?”. I looked it up and it’s it’s quite a loose group of individuals. It’s not Canonical employees necessarily. So how do you know that you can trust that snap?
A: How do they know they can trust any other software package on this system? I don’t see how snap is any more special than any other software package, they’ve got on their system.
J: Well, I would say it’s different in that if I get something from the Ubuntu repo’s specifically not universe, let’s say but the you know, The main repo then I know that Canonical has built that and they are taking responsibility for that and if there’s any issues with it, if there’s you know, someone’s snuck a crypto minor in there, for example, then it’s Canonical whose head will be on the chopping block for that. They will take responsibility. Whereas with the snap store that’s not as clear to me. It seems to be sort of up to the developer to take responsibility because it’s directly published by the developer.
A: Yes. That’s one of the goals of snap store to not be a blocker in between developers and users. So a developer who wants to publish their software or publish updates to the software can just do that. They’re self-serving they can publish on their schedule if they want to publish at 3 a.m. They can do it. They don’t have to wait for a Debian developer to approve it and allow it into a repository. That’s why the name of the owner of the package the publisher of the package use presented to you in the snap store is so that you can go and investigate that person and yeah all you got to do is go to GitHub. Look at the snapcrafters repo and you can see what’s in it. You can see the contributors to it. You say it’s a loose collection of people, you know, there are GitHub accounts. You can just like any other piece of software you can go back through the blame logs and the commit history and see who contributed to it. So it’s not like some super secret black hole that stuff comes out of it’s like any other software project is hosted on a source repos somewhere and you can go and find it and go and investigate for yourself rather than speculate and say what if and this shadowy snapcrafters organization, it’s not it’s just a bunch of people on GitHub like any other.
J: Okay. This episode is sponsored by DataDog, the performance monitoring and analytics solution for real-time visibility into a Linux environment combining metrics traces and logs in one unified platform allows you to get a bird’s-eye view of your entire infrastructure. You can also see any underutilized cloud or on-premise servers via the real-time auto-generated host map DataDog machine learning based alerts eliminate false positives and make sure that you only receive alerts on issues that matter. You can automatically detect unanticipated outliers anomalies and errors with Watchdog the auto detection engine that surfaces performance problems in your applications without any manual setup or configuration start your free DataDog trial today by visiting DataDog start your free trial create One dashboard and you’ll get a free day to dog t-shirt. That’s datadog.com/latenightLinux
J: You use the word blocker a blocker between the developer and the user but is it is drawn up a buffer between them
A: Can be.
J: Isn’t the point of distro to be that filter to take applications and package them in a way where they’re going to work together and integrate with desktops and all operating systems and make a cohesive experience. Isn’t that the value that a distro brings to software and open source software and if you just totally bypassed that then don’t you end up with just a bunch of disparate software that doesn’t necessarily work together.
A: That’s the glass half-empty version the glass half-full version is if I’m a developer and I want to publish my software and I’m unable to do that because someone has to write a file and then I have to wait for it to be uploaded then wait, that I have to wait for somebody else. Then I’m just not going to bother and we’ve had experience of that in the past with when we tried to the Ubuntu software store early on we asked developers to upload packages as debs and it went through a whole review process and it was so painful because there were so many people trying to upload, to get the things into Ubuntu and we just didn’t have the capability to scale that high to all those developers who are trying to iterate on their things and those developers had to learn how to make debs which is frustrating for the developer because it’s pretty opaque be making debs and they had to do it again every time there was a new release. So we were trying to remove all of that frustration. So I think the glass half-full side is a developer can go from idea to source code to a software package and get it in a store within like 24 hours. And I know that because me and Stuart did that this week. I gave him an idea for a piece of software. He developed it. I packaged it. We put it in the storm within about 24 hours. It was up and available and that’s really empowering for a developer.
J: Yeah, I get that but that buffer is also a security buffer as far as I’m concerned.
A: Is it, are you really sure that every single one of the packages that you’ve installed from your distro and it doesn’t matter which distro it is, are you certain that they’ve all been security reviewed because I don’t think they have.
J: It’s not necessarily that they’ve been reviewed.
A: Why would you care then? I mean I get I get kind of cross because people have this view of distros as the maintainers of the packages in the distros are meticulously going through every single change. I can every single line of the software that they’re packaging. Often they’re not, they’re just taking the upstream software bumping the number pushing it and it built yet. That works. Okay, and push it into the repository. That’s largely what software packagers do now. Sometimes it’s a little bit more complicated than that and especially more complicated if they’re creating a new package from scratch, but the vast majority of time for traditional Linux distributions the people are doing the packaging who are some kind of buffer are actually not reviewing the code at all.
J: But it’s about a chain of trust, isn’t it? They’re building software from someone that they person they trust and if they don’t trust them then they will meticulously look through it until that person gains their trust. Whereas with a snap any Tom, Dick and Harry can sign up and stick a snap in the store and you you know, we talked about how you people can report them or whatever, but that’s this seems to be a lack of accountability and responsibility there to me that you do have with a distro and the distros main repos.
A: No, I think you’re thinking of a hypothetical dream world that doesn’t exist, and this is the problem. I keep seeing this cited on the Internet as all these packages in all these repositories on all these distros have been through some kind of vetting and I can totally trust them. I guarantee you there are packages in the universe or universes like repositories that have CVE’s in them all over them, but you allow that because it was, you know, open source software and they’re all community and we’re all trying to work together on this same set of packages and that works okay up to a point, like a piece of software that goes without the security audit and goes without someone maintaining it or that individual who was packaging the thing is too busy to package that thing anymore. It can get orphaned and that does happen. Packages get orphaned all the time or people move on and that’s that’s a sad state of your software development people move on. And so I think you’ll not accurate in your assessment that these people are doing any kind of serious security audit of the software. Some packages are but I think the vast majority are not and so I feel like the accountability is with the developer, the developer who has a piece of software that they publish themselves. They are accountable their name is on it and if a CVE comes out they can rebuild it and publish it within minutes - depends how long it takes to build the software - but you know within minutes of the software being built with the security fixes, they can push it to the store and know that however many users they’ve got whether it’s tenm a hundred or a hundred thousand those users the next time they turn the computer on within some hours will get that security update and they can get the warm fuzzy feeling that there aren’t this long tail of insecure old versions out there.
J: I need to talk to you about two other things: performance and theming. There was a blog post put up by your colleague Igor on New Year’s Eve and that was about themes and what you’re doing about that but it’s funny that in the very first sentence the second word was performance. It was the very first sentence was an acknowledgement that performance is an issue for users. I know that there are things being done about that with different compression and stuff. But it that is one of the big drawbacks of snaps for me.
A: Yeah it can be for certain classes of snaps, I wouldn’t say every single snap is slow because that’s just not true. The problem is the highly visible snaps of high-profile applications that are well-known have had problems. And there’s a number of issues. It’s not just the compression scheme used was certainly one thing that we investigated and there’s threads all over the forum detailing all the investigations that were done and there’s a lot of time spent to try and improve that but there are other things like more recently we change the way font caching is done. So when you install a piece of software it often will regenerate a font cache so that you know when it’s writing text on the screen, it takes the text rendering from a cache file on disk and font cache building takes time and we were doing it on launch. So at the point when you open the application, it detects that font cache needs updating because either the font cache tool has changed or the number of fonts you have has changed and that was one of the things that was causing the snaps take a long time to start now that only happens with graphical applications. And so if you’re a desktop user you would see that and it was super frustrating and yeah, I get it and I’ve been working with people trying to solve this problem for a while and sometimes it recurs and we have to work out other ways to fix it. But one of the ways we fixed it is moving that font cache generation to the point at which the snap is installed rather at the point when you launch the application, so that font cache does still get built but it gets built at the point where you install it. And so you may notice that some snaps now take a little bit longer to install a little bit longer time to install but actually start faster, but there are other snaps which are like command line utilities which start pretty much instantly like just the same as a native app because snaps and not just graphical applications. There’s a giant pile of snaps out there that are command line utilities or demons and servers and stuff that don’t suffer from this problem at all. But yeah, we’re aware of it totally and we’re trying to fix it. But yeah, it is only so many engineers and there’s a lot of stuff to do
J: As far as my last question is the rest of the office suite and Adobe Creative Cloud, when?
A: Well, we’re not responsible for any of those things. We’re empowering developers. If those developers want to publish that software in the snap store. We’d love to help them do it.
J: Right very diplomatic answer presumably while working on that though trying to get them to do it.
A: So we have conversations with lots of developers all the time. Like it’s our job and some of those are lengthy conversations that require legal agreements. There’s one notable snap, whose name will remain nameless, but it took a year for us to get it in the store. Because frankly they didn’t have a huge number of Linux users and they didn’t want to spend a huge amount of time working on something which was only going to benefit maybe 1% of their user base. And so, you know, you have to be rather diplomatic when you’re talking to a developer and say, you know, you could make it easier to install your software if you did it like this and they say, yeah, okay we make it easier for maybe half a percent to one percent of our user base, is that worth doing you know, and so when we talk to developers, yeah, we want to try and get them to get their software in the store because we want to put that software in front of users. We want users to have a wide choice of software available to them. But sometimes that’s difficult or almost impossible and sometimes it takes a long time. So, you know, every time someone asks me about some super popular piece of software. Maybe we’re talking to that developer. Maybe we’re not I don’t know. Often times. We are it just you know, big enterprises often move quite slowly and have if they’ve got you know popular brands, they’ll have legal agreements that need signing. So yeah, it can take a long time.
J: Right, next week then, excellent. All right. Well, thanks a lot for joining me in an official capacity for once and we’ll have to speak again sometime.
A: Yeah new show when?