DON'T YOU KNOW WHO I AM!!

At 20:30 UTC today the Ubuntu UK Podcast goes live with the 4th Season! If you’re around and fancy listening in to us blather on for an hour about Ubuntu...

At 20:30 UTC today the Ubuntu UK Podcast goes live with the 4th Season!

If you’re around and fancy listening in to us blather on for an hour about Ubuntu and the community around it then point your browser at:-
http://podcast.ubuntu-uk.org/live/

The icecast stream URL (which is only active when the show is on) is below, and should work in various media players.
http://podcast.ubuntu-uk.org:8000/uupc.mp3

We’d love to see you drop by the IRC channel #ubuntu-uk-podcast too! There’s a webchat thing on that page, or you can use a ‘proper’ IRC client if you’re so inclined.

One thing we’re having a play with this season is a quiz. We’re going to be asking eachother some taxing Ubuntu related questions, and you’re welcome to join in.

If you can’t join in, don’t worry, we’ll put the show out as a podcast tomorrow.

If you have any feedback we’d love to hear it via any of the methods below!

Phone: +44 (0) 203 298 1600, sip: podcast@sip.ubuntu-uk.org and skype: ubuntuukpodcast
http://twitter.com/uupc
http://identi.ca/uupc
http://www.facebook.com/pages/Ubuntu-UK-Podcast/47581495708
http://uk.ubuntuforums.org/
Note: We’re still fiddling with the site theme, so if it’s a bit janky bear with us!

Time for another in a series of blog posts about the use I’m making of my HP ProLiant MicroServer I previously blogged about. This one covers caching Ubuntu .deb packages...

Time for another in a series of blog posts about the use I’m making of my HP ProLiant MicroServer I previously blogged about. This one covers caching Ubuntu .deb packages to reduce duplicated package downloads.

I have a few Ubuntu machines at home, and like to help out testing the latest release which can involve installing and re-installing Ubuntu multiple times. In the past I have used tools like apt-mirror and debmirror to cache the entire Ubuntu repository locally. This has been useful to speed up installs and upgrades of machines, but it’s quite an overhead to keep many GB of disk space dedicated to the Ubuntu repository. There’s also a lot of packages that I will download that I’ll almost certainly never use (like all of KDE for example) which just wastes bandwidth and disk space.

So I have decided to take a different approach, and only cache what I actually use. This is actually really easy to do with a package called squid-deb-proxy. In theory all I had to do was install squid-deb-proxy and avahi-utils on my server, and on each client machine (and VM) install squid-deb-proxy-client and that should be it.

The server should advertise over avahi/bonjour/zeroconf (call it whatever you will) that there is a squid proxy configured to cache deb files from Ubuntu repositories. The squid proxy runs on port 8000 by default on the server and is configured with sane defaults for caching Ubuntu packages. The clients (via the squid-deb-proxy-client package) are configured to look for such a proxy on the local network, and if found, use it. If none is found they should revert to using whatever repositories are already configured.

The configuration files for squid-deb-proxy are in /etc/squid-deb-proxy/ and are called allowed-networks-src.acl, mirror-dstdomain.acl, and squid-deb-proxy.conf.

Here’s the basic contents of allowed-networks-src.acl:-

# private networks
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16

This lists the networks that squid will accept connections from. These are the usual netblocks used by domestic routers. However my LAN uses 10.10.10.10/24 so I had to add this line to allowed-networks-src.acl

10.10.10.0/24

The file mirror-dstdomain.acl contains:-

# default ubuntu and ubuntu country archive mirrors
.archive.ubuntu.com
ports.ubuntu.com
security.ubuntu.com

Which lists the hosts that squid will cache content for. This is fine for a default install of Ubuntu 10.04 or 10.10 but for Natty (11.04) I had to add:-

extras.ubuntu.com

I also happen to use a bunch of PPAs and third party repositories so I added those to the list as well:-

ppa.launchpad.net
archive.canonical.com
linux.dropbox.com
repository.spotify.com

Restarting the service after changing those files was more traumatic than I’d imagined:-

Note: ‘ubuntuserver’ is my server and ‘wopr’ is the client in the shell captures below.

alan@ubuntuserver:~$ sudo /etc/init.d/squid-deb-proxy restart
[sudo] password for alan:
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service squid-deb-proxy restart
Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the restart(8) utility, e.g. restart squid-deb-proxy
restart: Job failed to restart

Ok, so it wants me to use sudo service squid-deb-proxy restart, no problem.

alan@ubuntuserver:~$ sudo service squid-deb-proxy restart
restart: Unknown instance:

Uh, okay. At this point it’s not running:-

alan@ubuntuserver:~$ ps -ef | grep squid
alan 9177 9025 0 10:15 pts/0 00:00:00 grep --color=auto squid

But I can start it with:-

alan@ubuntuserver:~$ sudo /etc/init.d/squid-deb-proxy start
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service squid-deb-proxy start
Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the start(8) utility, e.g. start squid-deb-proxy
squid-deb-proxy start/running, process 9197

As can be seen:-

alan@ubuntuserver:~$ ps -ef | grep squid
root 9197 1 0 10:15 ? 00:00:00 squid -f /etc/squid-deb-proxy/squid-deb-proxy.conf
proxy 9199 9197 0 10:15 ? 00:00:00 (squid) -f /etc/squid-deb-proxy/squid-deb-proxy.conf
alan 9202 9025 0 10:15 pts/0 00:00:00 grep --color=auto squid

I filed bug 695937 about it.

The next problem I had was that my clients refused to actually use the proxy, mainly because they couldn’t see it being advertised over avahi which is somewhat fundamental. I tested with avahi-browse:-

alan@wopr:~$ avahi-browse -kprt _apt_proxy._tcp
alan@wopr:~$

What should happen is this:-

alan@wopr:~$ avahi-browse -kprt _apt_proxy._tcp
+;eth1;IPv4;Squid\032deb\032proxy;_apt_proxy._tcp;local
=;eth1;IPv4;Squid\032deb\032proxy;_apt_proxy._tcp;local;ubuntuserver.local;10.10.10.124;8000;

To get that working I had to manually start squid-deb-proxy-avahi on the server:-

alan@ubuntuserver:~$ sudo /etc/init.d/squid-deb-proxy-avahi start
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service squid-deb-proxy-avahi start
Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the start(8) utility, e.g. start squid-deb-proxy-avahi
squid-deb-proxy-avahi start/running, process 9412

Once that’s all done the easy test is to perform an update from the client machine:-

alan@wopr:~$ sudo apt-get update
[sudo] password for alan:
Hit http://gb.archive.ubuntu.com maverick Release.gpg
Hit http://security.ubuntu.com maverick-security Release.gpg
Hit http://archive.canonical.com maverick Release.gpg
:
: snip
:
Hit http://gb.archive.ubuntu.com maverick-updates/restricted amd64 Packages
Hit http://gb.archive.ubuntu.com maverick-updates/universe amd64 Packages
Hit http://gb.archive.ubuntu.com maverick-updates/multiverse amd64 Packages
Reading package lists... Done
alan@wopr:~$

Which also looks good on the server side in the proxy server logs:-

alan@ubuntuserver:~$ sudo tail -f /var/log/squid-deb-proxy/access.log
1293791393.226 22 10.10.10.103 TCP_MISS/304 380 GET http://gb.archive.ubuntu.com/ubuntu/dists/maverick/multiverse/binary-amd64/Packages.bz2 - DIRECT/194.169.254.10 -
1293791393.253 16 10.10.10.103 TCP_MISS/304 380 GET http://gb.archive.ubuntu.com/ubuntu/dists/maverick-updates/main/source/Sources.bz2 - DIRECT/194.169.254.10 -
1293791393.280 150 10.10.10.103 TCP_MISS/304 280 GET http://linux.dropbox.com/ubuntu/dists/maverick/main/binary-amd64/Packages.bz2 - DIRECT/174.36.30.74 -
1293791393.290 30 10.10.10.103 TCP_MISS/304 376 GET http://gb.archive.ubuntu.com/ubuntu/dists/maverick-updates/restricted/source/Sources.bz2 - DIRECT/194.169.254.10 -
1293791393.322 17 10.10.10.103 TCP_MISS/304 379 GET http://gb.archive.ubuntu.com/ubuntu/dists/maverick-updates/universe/source/Sources.bz2 - DIRECT/194.169.254.10 -
1293791393.347 17 10.10.10.103 TCP_MISS/304 378 GET http://gb.archive.ubuntu.com/ubuntu/dists/maverick-updates/multiverse/source/Sources.bz2 - DIRECT/194.169.254.10 -
1293791393.375 19 10.10.10.103 TCP_MISS/304 380 GET http://gb.archive.ubuntu.com/ubuntu/dists/maverick-updates/main/binary-amd64/Packages.bz2 - DIRECT/194.169.254.10 -
1293791393.399 15 10.10.10.103 TCP_MISS/304 376 GET http://gb.archive.ubuntu.com/ubuntu/dists/maverick-updates/restricted/binary-amd64/Packages.bz2 - DIRECT/194.169.254.10 -
1293791393.428 22 10.10.10.103 TCP_MISS/304 380 GET http://gb.archive.ubuntu.com/ubuntu/dists/maverick-updates/universe/binary-amd64/Packages.bz2 - DIRECT/194.169.254.10 -
1293791393.456 19 10.10.10.103 TCP_MISS/304 378 GET http://gb.archive.ubuntu.com/ubuntu/dists/maverick-updates/multiverse/binary-amd64/Packages.bz2 - DIRECT/194.169.254.10 -

Now to test the deb file caching by installing a package on the client machine:-

alan@wopr:~$ sudo apt-get install cowsay
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
filters
The following NEW packages will be installed
cowsay
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 20.2kB of archives.
After this operation, 270kB of additional disk space will be used.
Get:1 http://gb.archive.ubuntu.com/ubuntu/ maverick/universe cowsay all 3.03+dfsg1-2 [20.2kB]
Fetched 20.2kB in 1s (12.9kB/s)
Selecting previously deselected package cowsay.
(Reading database ... 233939 files and directories currently installed.)
Unpacking cowsay (from .../cowsay_3.03+dfsg1-2_all.deb) ...
Processing triggers for man-db ...
Setting up cowsay (3.03+dfsg1-2) ...

Over on the server we can see a cache miss because nobody has installed this version of cowsay recently:-

1293791535.157 551 10.10.10.103 TCP_MISS/200 20635 GET http://gb.archive.ubuntu.com/ubuntu/pool/universe/c/cowsay/cowsay_3.03+dfsg1-2_all.deb - DIRECT/194.169.254.10 application/x-debian-package

Back on the client, if I clean up and then reinstall cowsay it should go via the proxy and find the cached content:-

alan@wopr:~$ sudo apt-get clean
alan@wopr:~$ sudo apt-get install --reinstall cowsay
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 20.2kB of archives.
After this operation, 0B of additional disk space will be used.
Get:1 http://gb.archive.ubuntu.com/ubuntu/ maverick/universe cowsay all 3.03+dfsg1-2 [20.2kB]
Fetched 20.2kB in 1s (19.7kB/s)
(Reading database ... 233998 files and directories currently installed.)
Preparing to replace cowsay 3.03+dfsg1-2 (using .../cowsay_3.03+dfsg1-2_all.deb) ...
Unpacking replacement cowsay ...
Processing triggers for man-db ...
Setting up cowsay (3.03+dfsg1-2) ...

..meanwhile over on the server:-

1293791607.984 10 10.10.10.103 TCP_MEM_HIT/200 20642 GET http://gb.archive.ubuntu.com/ubuntu/pool/universe/c/cowsay/cowsay_3.03+dfsg1-2_all.deb - NONE/- application/x-debian-package

TCP_MEM_HIT (A valid copy of the requested object was in the cache, AND it was in memory so it did not have to be read from disk). Success!

That’s great, now all I need to do is make sure all my Ubuntu machines have squid-deb-proxy-client installed and I’ll be caching debs! Fun!

I previously blogged about my HP ProLiant MicroServer which I’m playing with for doing backups of remote data and other “small/home office” tasks. I’ve decided to backup my GMail-based email...

I previously blogged about my HP ProLiant MicroServer which I’m playing with for doing backups of remote data and other “small/home office” tasks. I’ve decided to backup my GMail-based email using OfflineIMAP, which is awesome.

Offlineimap is in the Ubuntu repository and is pretty easy to setup once you know the format of the configuration file ‘~/.offlineimaprc’. Something like this is required at the top where the ‘accounts’ line can specify multiple accounts separated by commas.

Note: All of the sections below are in one file – ~/.offlineimaprc

[general]
accounts = alan_example,clare_example
ui = Curses.Blinkenlights
maxsyncaccount = 2

The “ui” parameter defines the look of offlineimap when it’s running. “Curses.Blinkenlights” gives a funky (but sometimes unstable) interface which is useful to use initially because it clearly shows offlineimap working. There’s a screenshot of this running below.

Next we need a section for each account using the name we used in the ‘general’ section above:-

[Account alan_example]
localrepository = gmailLocal_example_alan
remoterepository = gmailRemote_example_alan
autorefresh = 15

[Account clare_example]
localrepository = gmailLocal_example_clare
remoterepository = gmailRemote_example_clare
autorefresh = 60

Next a section to define where the local mail is stored:-

[Repository gmailLocal_example_alan]
type = Maildir
localfolders = /srv/offlineimap/example/alan

[Repository gmailLocal_example_clare]
type = Maildir
localfolders = /srv/offlineimap/example/clare

Finally the settings for the remote repository (your mail store) which in my case is GMail.

[Repository gmailRemote_example_alan]
type = IMAP
remotehost = imap.gmail.com
ssl = yes
remoteport = 993
remoteuser = alan@example.com
remotepass = password
maxconnections = 1
realdelete = no

[Repository gmailRemote_example_clare]
type = IMAP
remotehost = imap.gmail.com
ssl = yes
remoteport = 993
remoteuser = clare@example.com
remotepass = password
maxconnections = 1
realdelete = no

Of course your settings will differ for remoteuser and remotepass, but the rest should work fine. I then ssh to my server and run offlineimap inside Byobu (which I love) and that’s basically it.

Sometimes offlineimap crashes – especially if my internet connection goes down. When offlineimap crashes – or indeed when I quit it nicely, I sometimes get this:-

I then have to issue the reset command because the terminal session becomes unusable.

To avoid this, and to enable me to run offlineimap in the background (once I’ve proved it works okay) I simply change the setting for “ui” in the ~/.offlineimap file from:-

ui = Curses.Blinkenlights

to

ui = Noninteractive.Quiet

Which will cause offlineimap to run and then quit when it's finished. There's no need for the 'autorefresh' options above now, and I can schedule offlineimap to run via the following crontab entry so if my net connection drops it will restart sometime after it comes back up. It will also respawn a new offlineimap 30 mins after the hour if one isn't already running.

*/30 * * * * /usr/bin/offlineimap >/dev/null 2>&1

Now that's all setup I have a full backup of my gmail accounts on local storage. The really cunning thing is that I could point an email client at that store and access my mail locally, even if my Internet connection or GMail goes down.

Tweet