Making a Portable Persistent Ubuntu USB Stick

I recently wanted to make a slightly modified persistent bootable USB stick running a recent version Ubuntu. I made some notes and have put them here in case they’re useful to anyone else. It’s a bit of a manual process which could probably be streamlined / automated. This was just what I did as a one-off, take from it what you will.

USB3 sticks in a USB3 port work best as USB2 can be a bit on the slow side, especially for IO intensive operations like package installation or compiling.

Note: A few people have pointed out the fragility and short lifespan of USB sticks. This same procedure can be used to install on a hard disk or SSD in a USB enclosure. Once the image is copied to the external storage, simply use gparted to resize it up to take all available space.

The goal I had was to make an image which can be copied to USB stick to provide a persistent bootable Ubuntu SDK development environment. This could be useful for people who don’t run Ubuntu as their primary OS (Yes, these people exist, I know right!?) but want to dabble in Ubuntu application development. It’s also handy if you’re running an App Dev School where the computers aren’t yours, or run some other OS. The students could potentially take the sticks away with the full OS and all their work on. Just make the image and then copy it to multiple sticks before the class starts.

I also wanted to make it ask for locale and user details on first boot, so it could be easily configured and used in any language. This is pretty easy given the Ubuntu installer has all of that built in.

I used Ubuntu 14.10 i386 (but also tried with Ubuntu 14.04.1 LTS) and an 8GB USB stick which leaves a couple of GB over for work. Obviously a larger stick gives more space to the user. It turned out though that using an 8GB USB stick was a bit tight for SDK work. I ended up with 76MB left after creating one 15.04 armhf kit. Maybe 8GB is good for desktop and qml/html5 only development (although still a bit tight), but not for cross architecture or other binary builds. 16GB would have enough room for multiple kits and could build binaries for devices.

Some of these steps can be done in the background while you do other things. It’s not a massively time consuming task if you have a decent connection and fast USB stick / hard disk, but as I mentioned, is a bit manual.

The result is a USB stick which you can boot from and work off with data saved to the stick. You can optionally enable home directory encryption during the final end-user setup if that’s important to you.

Step 1 – Prep

Have an 8GB (or larger) USB 3 stick handy. I am using Kingston 8 GB USB 3.0 DataTraveler G4 Flash Drive and later Kingston Technology 16GB Data Traveler G4 USB 3.0 Flash Drive. Faster sticks are available of course, but I wanted something cheap to prototype on.
Have a laptop with a USB 3 port (or ports) and supports kvm. I did all this on my Ubuntu Vivid Vervet (15.04) Thinkpad X220 laptop which has a single USB3 port.
Make a directory on a local disk to store scratch image – will need 16GB or more space
Install qemu-kvm and gddrescue on host
Download ubuntu-14.04.1-desktop-i386.iso from http://releases.ubuntu.com/14.10/. (torrent link).

Step 2 – Installation of base system

Make a blank image on local disk
dd if=/dev/zero of=./disk_image bs=1M count=7500
This should result in a file a bit under 8GB.
e.g.

alan@deep-thought:/data/usb⟫ dd if=/dev/zero of=./disk_image bs=1048576 count=7500
7500+0 records in
7500+0 records out
7864320000 bytes (7.9 GB) copied, 34.468 s, 228 MB/s

Install Ubuntu into the image using kvm
sudo kvm -m 2048 -cdrom ~/Downloads/ubuntu-14.10-desktop-i386.iso -hda ./disk_image -boot d
This should boot off the ISO

QEMU_425

At the A11Y (person = keyboard) icon, hit space

QEMU_426

At the boot menu, choose language (this is just language for the installer, user will later choose which language to use)

QEMU_427

Press F3 and choose keyboard layout

QEMU_428

Press F4 and choose OEM install

QEMU_429

Pick “Install” from the menu.

QEMU_430

QEMU_431

Follow the installer prompts as normal. I configured with no swap, but use the entire disk for an ext4 volume for the root filesystem.
Set a password for the oem user, which will be thrown away later, and the user will get to set their own password.
Shut-down at the end

Step 3 – Install the SDK

This is the part where you make the modifications to the image (if any). I wanted to install the Ubuntu SDK.

Optionally at this point, make a backup of your cleanly installed Ubuntu 14.10.1 system
cp ./disk_image ./ubuntu_14.10_install_backup

Boot the previously created install (note the additional options – these are handy)
sudo kvm -m 2048 -hda ./disk_image -chardev stdio,id=mon -mon mon
Once booted to the desktop, in the terminal on the host at the (qemu) prompt type this to switch the VM to the console (which is faster to do stuff than the GUI :) ):-
(qemu) sendkey ctrl-alt-f1
Login to the tty with the oem user/password set in Step 2.
Follow the usual guide to install the SDK and update the system:-
sudo add-apt-repository ppa:ubuntu-sdk-team/ppa
sudo apt-get update
sudo apt-get install ubuntu-sdk
sudo apt-get dist-upgrade
sudo apt-get clean
sudo apt-get autoremove

Shut down the vm
sudo shutdown -h now

Optionally at this point, make a backup of your “SDK-installed” (or modified in whatever way you choose) OEM mode Ubuntu 14.10 system
cp ./disk_image ./ubuntu_14.10_install_sdk_oem_backup

Note: At this point you can boot the disk image and do further customisation – maybe adding other packages which may be of use, but I stopped here.

Step 4 – Prepare the OEM image for ‘shipping’

This is the point where we flip the switch in the installed image before handing it off to another user. On first boot they will get prompted to set locale and configure a new user.

Boot the previously created install which has the SDK installed
sudo kvm -m 2048 -hda ./disk_image
Click the “Prepare for shipping to end user” icon on the desktop – this sets the system to be ready for the first-boot experience for a new user
Shut down the system

Step 5 – Test this all worked

Make a copy of the master image for testing
cp ./disk_image ./testing_oem_install
Boot the test image to try it out
sudo kvm -m 2048 -hda ./testing_oem_install
At this point you should be prompted for the usual post-install setup tasks including language / locale / username & password. Setup as you would a normal machine
Open the SDK (or whatever you installed), test it all works
I tried creating a kit and do other SDK related things
Shutdown when done
Delete the test image
rm ./testing_oem_install

Step 6 – Copy the OEM image to a USB stick for shipping / use

Now we have a ‘final’ image (and optionally some backups) we can copy this to a stick for use by us / someone else. We can of course make more than one by doing this step multiple times with different sticks. On my system as you can see it took ~30 mins to copy the image to the stick. Faster, more expensive sticks may be better, these were pretty cheap.

Copy the disk image to an appropriately sized USB stick
sudo ddrescue -d -D --force ./disk_image /dev/sdX

e.g.

alan@deep-thought:/data/usb⟫ time sudo ddrescue -D -d --force disk_image /dev/sdc
GNU ddrescue 1.19
Press Ctrl-C to interrupt
rescued: 7864 MB, errsize: 0 B, current rate: 1966 kB/s
ipos: 7864 MB, errors: 0, average rate: 4884 kB/s
opos: 7864 MB, run time: 26.83 m, successful read: 0 s ago
Finished

real 26m51.682s
user 0m1.212s
sys 0m30.084s

Step 7 – Test & use the stick

Put the USB stick in a computer set to boot from external media.
Test that you get a desktop and the usual OEM prompts you got in Step 5.
If that works then you can do step 5 again for the same stick or as many sticks as you have.

Success!

Comments and suggestions welcome!

FOSDEM 2015 Field Report

A little late, but here’s my report from attending FOSDEM 2015 back at the start of February.

I’ve been to FOSDEM a few times in the past, but not for the last few years. It happened to co-incide with other events, or other family things took priority and then got out of the habit, so I’d not ended up going for 6 years or so!

This year at the last minute I applied for some funds to go, was accepted and made the most of it. I’ve listed below all the talks & sessions I went to, with some brain dump notes I took about some of them. At the bottom I listed some sessions which I didn’t enjoy or get a lot out of, but are listed for completion. I’ve also linked directly to the videos of the talks if they’re available at the time of writing. If not I’ve linked to the directory they should appear in at some point in the future.

A GPS watch made of free software and hardware – Federico Vaga, Matthieu Cattin
Video: http://video.fosdem.org/2015/main_track-hardware/gps_watch__CAM_ONLY.mp4

Recycle your Android devices for anything: run real Linux on them – David Greaves
Talk video:- http://video.fosdem.org/2015/devroom-embedded/sailfishos.mp4

  • Room was pretty packed with quite a few Jolla / SailfishOS friendly people at the front – managed to score myself a Jolla beanie hat from them :)
  • I was interested in this talk because it was focussed mostly on porting Mer & SailfishOS to Android devices.
  • We have a similar porting guide in Ubuntu but ours was outdated and inaccurate. This was fixed recently.
  • The Mer project had a nice overview of ports status online (https://wiki.merproject.org/wiki/Adaptations/libhybris), which I’ve shamelessly stolen for Ubuntu (https://docs.google.com/spreadsheets/d/1uUHF463g4f4L5ljWZf0l7b4VAevM-twHag4ZoEd_TNc/edit?usp=sharing)
  • Interested in the community they’re building specifically around porting – dedicated irc channel, irc meetings and real world meet-ups are something we should probably consider for our (Ubuntu) porters because right now I think our porters feel a bit helpless at times

Are distributions really boring and a solved problem? – Lucas Nussbaum
Video: http://video.fosdem.org/2015/devroom-distributions/distributions_boring_solved_problem.mp4

  • Illuminating talk about some of the problems the Debian project faces currently
  • As I’m not a Debian developer it was interesting to me to learn a bit about the Debian Sausage Factory and some of the issues they/we face

GCompris goes Qt Quick with the help of KDE – Bruno Coudoin
Video: http://video.fosdem.org/2015/devroom-desktops/gcompris_goes_qt_quick_with_the_help_of_kde__CAM_ONLY.mp4

  • A quick talk from Bruno about the work he’s done to port GCompris to the Qt framework
  • Learned that the development model for GCompris includes using In-App-Purchases on non-free platforms (Google Play store) to fund development of Open Source projects.

Copyleft in Europe: How does copyleft interact with Exhaustion Of Rights – Amanda Brock, Andrew Katz
Video: http://video.fosdem.org/2015/devroom-legal_and_policy_issues/copyleft_in_europe__CAM_ONLY.mp4

  • This was a super fascinating and slightly complex legalese talk which I admit I didn’t completely follow, but Amanda and Andrew made it digestable for a layman such as myself :)

Fork and Ignore: Fighting a GPL Violation By Coding Instead – Bradley M. Kuhn
Video: http://video.fosdem.org/2015/devroom-legal_and_policy_issues/fork_and_ignore__CAM_ONLY.mp4
Slides: http://ebb.org/bkuhn/talks/LinuxCon-North-America-2014/kallithea.html

  • Brilliant talk from Bradley, one of the highlights from FOSDEM. Slides above are from a different event, but same talk as far as I can tell.
  • Was enlightening to see an alternative way to deal with a GPL violation than the default which seems to be to take legal action and sue
  • Not all forks are bad :)

Ubuntu on phones and beyond – Michał Sawicz
Video: http://video.fosdem.org/2015/devroom-desktops/ubuntu_on_phones_and_beyond__CAM_ONLY.mp4

  • Michał gave a brief talk to a pretty packed room about the current state of Ubuntu for devices, and the future direction
  • Turned into more of a Q&A, which was beneficial as much of the audience had questions to ask
  • Was interesting to see some of the same questions come up again and again
  • Depressing to still get “Haha! LOL Amazon shills” comments and questions

Mobile == Web – Stormy Peters
Video: http://video.fosdem.org/2015/devroom-desktops/mobile_web__CAM_ONLY.mp4

  • Call to arms to developers to make their sites work well on the web for the next generation of smart phone users
  • Disappointed to see yet another Mozilla person preach about openness from the confines of a MacBook running OSX

Maintaining & Growing a technical community – Ali Spivak
Video:- http://video.fosdem.org/2015/devroom-mozilla/ud2218a_maintaining_growing_technical_community__CAM_ONLY.mp4

  • Ali gave some illuminating stats about contributions to the Mozilla project and how that fluctuates over time
  • Lots of info about what motivates people to contribute to open source projects and Mozilla in particular
  • “Being open is no good if nobody can find your resources” – we (Ubuntu) have some fixes to do there

Internet all the things – using curl in your device – Daniel Stenberg
Video:- http://video.fosdem.org/2015/devroom-embedded/curl_device.mp4

  • This was one of my favourite talks from FOSDEM. Learning how many projects use a simple (hah) tool like curl
  • Even learned about some command line options in curl that I’d not seen or used before.
  • Worth a watch

Living on Mars: A Beginner’s Guide – Ryan MacDonald
Video:- http://video.fosdem.org/2015/keynotes/closing_fosdem__CAM_ONLY.mp4

  • Fantastic talk to round off FOSDEM 2015. Ryan gave a fast-paced & entertaining talk about the Mars One mission plan and his part in it.

Porting Tizen:Common to open source hardware devices – Phil Coval
Video:- http://video.fosdem.org/2015/devroom-embedded/open_hw_tizen.mp4

  • Another porting talk (to go along with the Jolla one above) specifically talking about porting Tizen to Open Hardware
  • Interesting to hear about the Sunxi community build up around porting

Reached milestones and ongoing development on Replicant – Paul Kocialkowski
Video:- http://video.fosdem.org/2015/devroom-embedded/replicant_embedded_freedom.mp4

  • Fascinating talk detailing what lengths Paul goes to in order to develop a fully Free Software implementation of Android
  • In short, it’s not ‘finished’, and with the limited number of devices it’s possible to ‘open up’ probably won’t be any time soon

GNOME – creating ripples in the Linux eco-system – Sri Ramkrishna
Video:- http://video.fosdem.org/2015/devroom-desktops/gnome_creating_ripples_in_the_linux_ecosystem__CAM_ONLY.mp4

  • I was somewhat disappointed in this talk as it seemed to be billed as showcasing the best of what GNOME have done over the years, but came across as poorly thought through and the points a bit laboured.

Servo (the parallel web browser) and YOU! – Josh Matthews
Video:- http://video.fosdem.org/2015/devroom-mozilla/ud2218a_servo_and_you.mp4

  • Josh did a great job of showing the state of Servo right now and where gaps exist in the functionality, certainly worth watching for the Servo demo alone

Awesome Community is Awesome

While flash sales, conferences and trade shows happen, the Community Core Apps project rolls forward landing updates and fixes to Ubuntu Phone applications. Some are default apps on the devices bq customers will soon receive, others are easily installable from the click store. All are maintained by a community of Free Software developers. I’m incredibly proud of the work these people are doing and wanted to highlight some recent updates.

As with all the applications below, if you’d like to get involved in design, documentation, testing, translation or just plain writing code either get in contact with me, or with any of the developers listed below. They’re all very nice people.

Weather Reboot

Andrew Hayzen, Victor Thompson and Nekhelesh Ramananthan got together to help Martin Borho reboot the Weather App with new designs created in collaboration with the Canonical Design Team. With a cleaner look, easier to use user interface, and fewer poke-your-eyes-out colour gradients, the new Weather app is coming along nicely!

device-2015-03-08-230329

device-2015-03-08-230338

device-2015-03-08-230352

device-2015-03-08-230404

device-2015-03-08-230420

device-2015-03-08-230426

Doc Viewer

Stefano Verzegnassi has been working hard on the usability and performance in the Doc Viewer. It now directly pulls documents from ~/Documents on the device and can receive files via Content-Hub such as those downloaded in the browser. Rendering performance has been improved, using multi-core page rendering to make use of the grunt in powerful phones. It’s also now possible to manage documents directly in the app, so you can delete files from ~/Documents with a long-press and some taps, no file manager required. There’s also a Table of Contents revealed from a bottom edge swipe, and a new document grid view.

device-2015-03-04-130648

device-2015-03-04-191710

device-2015-03-04-130657

device-2015-03-04-130658

device-2015-03-04-130702

device-2015-03-04-130704

device-2015-03-04-130705

device-2015-03-04-130703

device-2015-03-04-130701

While not currently a default app in devices, it’s in the store and works really well on all Ubuntu devices. Upcoming is a refinement of the full screen reading experience to remove the header and some improvements to document zooming. Longer term we’re also investigating and prototyping support for common office file formats.

Terminal

Filippo Scognamiglio recently updated the Terminal app to support configurable keyboard overlays. The overlay is a strip above the default OSK (on screen keyboard) which can be populated with keyboard shortcuts or full commands. These are designed to supplement the OSK and allow fast access to complex keyboard combinations or frequently typed commands. The Terminal app ships with a small number of sample keyboard overlays, but it’s easily user-expandable. Filippo blogged about it, read more there. If you create a cool overlay, do share it, and we may include it in a future update as a default. This update is in the store, available to install on Ubuntu devices.

device-2015-03-04-130603

device-2015-03-04-130610

device-2015-03-04-130622

Upcoming is a user interface change to easily switch on and off the keyboard overlays. So you may have 20 installed, but rarely use some, so can easily switch them off so they don’t clutter the main UI. In addition Filippo is working on breaking up the terminal app into a re-usable component to make it easy to ship click packages containing a command line tool and terminal together. So for example a developer could bundle mutt+terminal in a click package in the store. Clearly not a typical use case for many phone users, but it’ll certainly be useful for our early adopter hacker types :)

Calendar

Kunal Parmar has updated the Calendar app with some nice UI and performance improvements. If you sync events with the Calendar then they should show up quicker in the app than previously. He also implemented the ability to move events around the calendar with a long press and drag. More recently we’ve noticed a couple of crashers which we’re trying to get to the bottom of. Renato has a patch for EDS which seems promising! The latest Calendar update is in the store and available for testing as always.

device-2015-03-08-222643

device-2015-03-08-222616

device-2015-03-08-222344

device-2015-03-08-222754

Calculator Reboot

Riccardo Padovani and Bartosz Kosiorek have been completing work on the Calculator ‘reboot’. We received a new design a while back which has been almost completed. We have some more design changes to make, but we plan to release this version to the store shortly as the default Calculator. You can currently test it by installing “Calculator Reboot” from the click store on your Ubuntu device. Feedback (and bugs) welcome as always.

device-2015-03-08-223548

device-2015-03-08-223524

device-2015-03-08-223515

Reminders

Michael Zanetti and Riccardo Padovani have been really active with the Reminders app the last few weeks. Many bugs fixed, and a new offline mode for those that don’t want to connect to Evernote. There’s so many improvements which will require some additional QA before we upload to the store which should happen next week. This is a major update which has been a long time coming, and I’ll talk more about that in detail when it’s closer to landing.

Update: This update has landed in the store now!

device-2015-03-04-130852

device-2015-03-04-130953

device-2015-03-04-131028

device-2015-03-04-131102

device-2015-03-04-131109

device-2015-03-04-131132

Dekko

Dan Chapman and Boren Zhang have been implementing designs provided by the Canonical design team in their Dekko Email app. The changes are coming thick and fast, and if you’d like to test some of the new features, then install the “Dekko (beta)” app from the click store.

device-2015-03-04-134507

device-2015-03-04-134514

device-2015-03-04-134519

device-2015-03-04-134550

device-2015-03-04-134606

device-2015-03-04-134622

Clock

Nekhelesh Ramananthan pushed a small but important update to the Clock app recently which makes translatable the cities which are listed in the app. Once that landed we asked the Ubuntu translators if they could kindly translate the 200+ strings. Over night ~30 languages were done with more coming in over the weekend. It never ceases to amaze me how fast and attentive the Ubuntu translation communities are!

device-2015-03-08-224804

A massive thank you to all the developers, designers, testers and translators who have helped improve all of these apps. As always, get in touch if you’d like to get involved!

Scopes Contest Mid-way Roundup

I recently blogged about my Ubuntu Scopes Contest Wishlist after we kicked off the Scopes Development Competition where Ubuntu Phone Scope developers can be in with a chance of winning cool devices and swag. See the above links for more details.

As a judge on that contest I’ve been keeping an eye out for interesting scopes that are under development for the competition. As we’re at the half way point in the contest I thought I’d mention a few. Of course me mentioning them here doesn’t mean they’re favourites or winners, I’m just raising awareness of the competition and hopefully helping to inspire more people to get involved.

Developers have until 3rd December to complete their entry to be in with a chance of winning a laptop, tablet and other cool stuff. We’ll accept new scopes in the Ubuntu Click Store at any time though :)

Robert Schroll is working on a GMail scope giving fast access to email.

gmail

Bogdan Cuza is developing a Mixcloud scope making it easy to search for cool songs and remixes.

Screenshot from 2014-11-10 18:57:32

Sam Segers has a Google Places scope making it easy to find local businesses.

Places-scope002

Michael Weimann has been working on a Nearby Scope and has been blogging about his progress.

Main

Dan has also been blogging about the Cinema Scope.

img2

Finally Riccardo Padovani has been posting screenshots of his Duck Duck Go Scope which is already in the click store.

divergentduck

I’m sure there there are other scopes I’ve missed. Feel free to link to them in the comments. It’s incredibly exciting for me to see early adopter developers embracing our fast-moving platform to realise their ideas.

Good luck to everyone entering the contest.

Ubuntu Scopes Contest Wishlist

We’re running a Scope Development Competition with prizes including a laptop, tablets, and a bunch of cool Ubuntu swag. Check the above link for details.

I’m one of the judges, so I’m not allowed to enter which is both good and bad news. Good because then you won’t see my terrible coding quality, but bad because I would really love one of these sweet Dell XPS laptops! :)

I do have things I’d like to see made as scopes, and some ideas for making ones that I might do in the future when time permits, and I thought I’d share them. As a judge I’m not saying “make this scope and I’ll vote for your entry” of course, I simply figured I can give people some ideas, if they’re stuck. We do have a set of criteria (see link above) for rating the scopes that are submitted, and those will be used for judging. None of those criteria are “whether it was on the list on popey’s blog”. These are just ideas to get people thinking about what might be possible / useful with a scope.

Surfacing my data

One of the goals of scopes is to enable users to easily and quickly get access to their data. That could be local data on the device or remote data in a silo somewhere online. Typically on other platforms you’d need a dedicated app to get at that data. To access your Spotify playlist you need the Spotify app, to access your LinkedIn data you need the LinkedIn app and so on. Many of the sites and services where my data is held is accessible via an API of some kind. I’d love to see scopes created to surface that data directly to my face when I want it.

Manage Spotify Playlist

I use and love Spotify. One problem I have is that I don’t often add new music to my playlists. I don’t use or value the search function in the app, or the social connected features (I don’t have my Spotify hooked up to Facebook, and don’t have any friends on Facebook anyway). I tend to add new music when I’m having a real life verbal conversation with people, or when listening to the radio.

So what I would like is some quick and easy way to add tracks to my playlist, which I can subsequently play later when I’m not in the pub / driving / listening to the radio during breakfast. This could possibly sign in to Spotify using my credentials, allow me to search for tracks and then use the API to add tracks to playlist

Amazon Wishlist

My family tell me I’m really hard to buy presents for, especially at this time of year. I disagree as I have an Amazon wishlist containing over a hundred items at all price points :) When I visit family they may ask what’s on my wishlist to find out what I’m most interested in.

I’d like to be able to pull out my phone, and with a couple of swipes show them my wishlist. It would also be useful if it had the ability to ‘share’ the wishlist URL over some method (email is one, SMS might be another) so they get their own copy to peruse later.

I’d also like to be able to add things to the wishlist easily. Often when I’m out I think “That’s cool, would love one of those” and that could be achieved with a simple search function, then add to my wishlist.

Location Specific

Satellites Overhead

I (and my kids) like to watch the International Space Station go over. Perhaps I enjoy it more than the kids who are made to stand outside in the cold, but whatever. When I travel it would be nice to have a scope which I can turn to during twilight hours to see when the ISS (or indeed other satellites) are passing overhead. This information appears to be publicly available via well documented APIs.

Upcoming TV Programmes

I frequently forget that my favourite TV programmes are on, or available to stream. It would be awesome to pull together data from somewhere like Trakt and show me which of my most loved programmes are going to be broadcast soon, on what local TV channel.

Events Nearby

When I travel I like to know if there’s any music, social or tech events on locally that I might be interested in going to. There’s quite a few sites where people post their events including Songkick, Meetup and Eventbrite (among many others I’m sure) which have a local event look-up API. One of the cool things about scopes is you can aggregate content from multiple scopes together. So there could be a scope for each of the above mentioned sites, plus a general “Local Events” scope which pulls data from all of those together. Going to one scope and refreshing when I arrive in a new location would be a great quick way to find out what’s on locally.

Some of the above may be impractical or not possible due to API limitations or other technical issues, they’re just some ideas I had when thinking about what I would like to see on my phone. I’m sure others can come up with great ideas too! Let your imagination run wild! :)

Good luck to all those entering the contest!