Testing Ubuntu Apps As A Service

tl;dr. Stuart Langridge and I made an simple, easy to use, experimental app tester called Marvin, for Ubuntu Click Packages, which emails you screenshots and logs of your app while running on a real device you may not own.

Screenshot of email from Marvin

I frequently get asked by new developers in the community to help test their apps on Ubuntu Phone. Typically they don't want extensive testing of all features, just a simple "does it start and what does it look like on the device you have?".

Often they don't have a physical device when developing on the desktop with our SDK, but want an on-device sanity check before they upload to the store. Sometimes they have one device such as a phone, but want to see what their app looks like on a different one, perhaps a tablet.

I've been happy to help developers test their apps on various devices, but this doesn't scale well, is time consuming and relies on me being online and having a phone which I'm happy to install random click packages on.

Meanwhile, at OggCamp I gave a short talk about our recent security incident on Ubuntu phone. During the Q&A and in the bar afterwards a couple of people suggested that we should have some system which enables automated testing of devices. They were coming at it from the security point of view, suggesting heavy instrumentation to find these kinds of issues before they hit the store.

While we (Canonical) already have tools which review apps before they go in the store, we currently don't actually install and execute the apps on devices, and have no plan to implement such a service (that I know of).

I'm aware that other platforms have implemented automated systems for testing and instrumenting apps and wondered how hard it would be to setup something really basic to cover at least one of the two use cases above. So I took to Telegram to brainstorm with my good friend Stuart Langridge.

Snippet of conversation with Stuart

We thrashed out what was needed for a 'minimum viable product' and some nice-to-have future enhancements. Pretty soon after, with a bit of python and some hacked-together shell scripts, 'Marvin' was born. I then approached Daniel McGuire who kindly provided some CSS to make it look prettier.

Screenshot of upload page

A developer can upload a click package to the site, and specify their email address & one or more of the available devices. Some time later they'll get an email showing a few screenshots of the app or scope running on a device and pertinent logs extracted after it ran. While the developer waits, the website shows the current status as 'pending' (you're in a queue), 'claimed' (by a device) and 'finished' (check your inbox).

This fulfills the simplest of use cases, making sure the app starts, and extracting the log if it didn't. Clearly there's plenty more it could potentially do, but this was our first target met.

Under the covers, there's a device attached to a computer which checks periodically for uploaded clicks and processes them in sequence. In between each run the phone is cleaned up, so each test is done on a blank device. Currently it tests traditional apps/games and scopes, webapps are rejected, but may be supported later.

The reason we reject webapps is because currently the devices have no network access at all - no wifi or cellular data. So running webapps would just result in this:-

Screenshot of webapp with no network access

It's experimental so not completely robust, being a prototype hacked together over a couple of weekends/evenings, but it works (for the most part). There's no guarantees of availability of the service or indeed the devices. It could go offline at any time. Did I mention it's experimental?

Significantly, I've disabled network access completely on the device, with no SIM inside, so any app which requires external network access is going to have a bad day. Locally installed apps however, will work fine.

We currently don't do any interaction with the uploaded applications, but simply run them and wait a few seconds (to give it time to quiesce) then take a screenshot. The image at the top of this post shows what a typical email from Marvin looks like.

It contains:-

  • click-review.txt - The output from running the click review tools
    • Note: Apps which fail the click review process (the same one run by the click store) will not be installed or tested.
  • install.txt - Output from the commands used to install the click on the device - good for debugging install failures
  • Screenshot-0 - What the "home" app scope looks like with the click installed - useful for showing the icon and description
  • Screenshot-1 - What happens immediately after starting the app, showing the splash screen
  • Screenshot-2 - The app after 5 seconds
  • Screenshot-3 - The app after 10 seconds
    • Note: We attempt to de-duplicate the screenshots so you may not get all four if any are identical
  • application-log.txt - The actual output (stdout) from the application, pulled from ~/.cache/upstart
  • dmesg.txt - Any kernel logging generated from the app during the app run
  • device-version.txt - The output of 'system-image-cli --info' run on the device, so the developer knows what OTA level, channel and device it ran on

There's clearly a ton of other things that could be added to the mail, or extra items which could be instrumented or monitored, and features we could add. Off the top of my head we could potentially add:-

  • Scripted touch/gestures
  • Networking
  • VPN endpoints (so the phone looks like it's in a particular region)
  • Orientation changes
  • Faked GPS location
  • Video/screencast recording during runtime
  • Input from microphone / camera(s)
  • Specify which release / channel to flash on the device prior to testing

Clearly all of these need some careful thought and planning, especially those enabling network access from the device.

We're interested in feedback from developers who might use Marvin, and suggestions for improvements we might make. There are a limited number of devices in the pool, and not all supported devices are currently available. In the future we may have more devices connected to Marvin as they become available.

So go and test your apps at marvin.popey.com!


Comments powered by Disqus