Pages
Categories
Archives
- July 2009
- June 2009
- May 2009
- April 2009
- March 2009
- December 2008
- November 2008
- September 2008
- August 2008
- July 2008
- June 2008
- May 2008
- April 2008
- March 2008
- February 2008
- November 2007
- October 2007
- September 2007
- August 2007
- July 2007
- June 2007
- May 2007
- April 2007
- March 2007
- February 2007
- January 2007
- December 2006
- November 2006
- October 2006
- September 2006
- August 2006
- July 2006
- June 2006
- May 2006
- April 2006
Creating an Ubuntu repository mirror with apt-mirror
apt-mirror is a handy tool for creating a replica (or ‘mirror’ if you will) of a Debian archive. This can be useful in a number of circumstances if you want to maintain your own archive. Having your own up to date archive means you have fast access to every package in the distro very quickly.
Maybe you need it for installing a Debian-based distro onto lots of machines on a LAN. For example a system builder who installs Linux on many new computers and wants to improve the speed at which the systems are updated after a clean install. There are of course other tools that can achieve this including apt-catcher, apt-proxy and squid. These are in fact arguably better than the apt-mirror approach. But this is more fun
There are other uses for this, and I’m sure I’ll think of them one day.
apt-mirror can be used on any Debian-based repository. So it will work equally well with genuine Debian, Ubuntu or indeed any other .deb based system. Here is the way I setup a local hard disk.
Firstly, prepare some space
Lots of it! The Ubuntu 6.10 (Edgy Eft) repository including main, universe and multiverse currently takes around 30GiB of disk space. Disk space is relatively cheap though, so having a local archive isn’t a big deal space-wise. Bandwidth is a separate consideration though. 30GiB is quite a fair amount of data to pull down over the internet.
I used a 250GiB external USB hard drive. This has the advantage that I can physically attach it to a machine that has no net connection, chuck in an Ubuntu 6.10 (Edgy Eft) CD (or even upgrade from a 6.06 [Dapper] one) and install/upgrade a machine there and then. See – told you there was another use for this!
This is also especially handy if someone has a wierd ethernet card or modem that’s only supported in a very new kernel, or requires something from universe or multiverse to get online. Can’t think of any specific examples of that, but hey, I’m coming up with excuses^Wreasons to do this whole apt-mirror thing all the time.
I have a completely empty USB disk plugged in, lets see how much space there is available.. Ooh lots.
Make the necessary directories
My USB hard disk gets mounted in the usual place (/media/usbdisk) when I plug it in and turn it on. So I just need to make a directory on that disk and a few subdirectories and we’re done with this step.
Install apt-mirror
apt-mirror is in the standard repositories. You can install using your favourite tool like synaptic or aptitude, call me old fashioned, I like apt-get though.
Edit config
The default settings might work for you okay, but I decided to make some changes.
Here’s the default config file:-
Here’s mine. I changed the location where I wanted the repository stored, used the nearest mirror, changed the number of threads and a few extra repositories that I wanted to mirror.
Set aside some time to start the mirroring. It will take “time”. How long? Well, divide ~30GiB by your internet connection speed
The first time I ran apt-mirror it scanned the online archives and wanted to download around 30GiB of packages. Worth noting that I didn’t leave it running but killed it when I wanted to get some bandwidth back and restarted it later. It doesn’t start all over again but continues, and you can see this because the total download size goes down with each execution until you have the full archive. If you have a particularly slow connection, or you stop it quite a lot then you’ll likely never finish, at least not whilst edgy is in development!
Once the distro goes stable you can be sure the number of changes will go down. At the time of writing there’s hundreds of MBs changes per day, so now may not be the best time to do this
The good news is that Edgy is released this week – Yay!
Here’s one of the instances of me running it. Note this isn’t the first time I ran it.
Note the handy-dandy script that gets created at the end to clear out the old files you have in your local repository. So we kick that off and here’s what happens next:-
That was painless!
Using the archive
Ok, so what can we do with this archive. Install software from it! Here’s an example of installing software from the local archive connected to the USB port.
Backup current sources.list
Edit the sources.list
Here’s the sources.list that goes with the apt-mirror config I used above.
Update
Now we update my local machine to let it know what’s in the repository.
Fix the NO_PUBKEY warnings
The warnings occur because I don’t have those keys in my keyring. This fixes that:-
Update again
..to show those keys got imported correctly in the last step.
Yay!
Install stuff
Here’s where we try to install stuff off the local mirror. I’ll just get a small package. Note that it doesn’t copy the package to the local filesystem “Need to get 0B”, but instead unpacks it directly from the repo “Unpacking cowsay (from …/c/cowsay/cowsay_3.03-8_all.deb) …”. All very good stuff. Like it.
Test installed stuff
alan@multivac:~$ cowsay apt-mirror rocks! ___________________ < apt-mirror rocks! > ------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||Test upgrading
Lets just make sure an upgrade works.
alan@multivac:~$ sudo apt-get upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages have been kept back:
python-adns python-clientcookie python-crypto python-egenix-mxproxy python-egenix-mxstack python-egenix-mxtexttools python-htmlgen
python-htmltmpl python-jabber python-kjbuckets python-ldap python-mysqldb python-pam python-pexpect python-pylibacl python-pyopenssl
python-pyxattr python-simpletal python-soappy python-sqlite python-syck python-xmpp
The following packages will be upgraded:
ubuntu-docs
1 upgraded, 0 newly installed, 0 to remove and 22 not upgraded.
Need to get 0B/4984kB of archives.
After unpacking 123kB disk space will be freed.
Do you want to continue [Y/n]?
(Reading database … 156091 files and directories currently installed.)
Preparing to replace ubuntu-docs 6.10.3 (using …/ubuntu-docs_6.10.4_all.deb) …
Unpacking replacement ubuntu-docs …
Setting up ubuntu-docs (6.10.4) …
Future options
What I will likely do is keep this USB hard disk up to date but rsync the contents to a local fileserver so that I have it accessible not only via USB but also via NFS/SMB. That way I can share it out over the network. I don’t really want to share out a USB hard disk, as it’s a transient connection. I’d rather share out a permanent directory on a fileserver.
I have a fileserver called “hal” which I could backup to like this:-
That will copy the entire contents of my repository over the network via SSH to /mnt/media on the server hal.
I could reduce the overall download size by eliminating some stuff. For example I could choose not to download all the source packages by commenting out the lines starting “deb-src”. I could also reduce the number of architetures, for example not downloading the AMD64 binaries because I don’t (currently) have any AMD64 based systems.
My ISP provides me with a higher bandwidth allocation at night than during the day. So it might also be useful to schedule an apt-mirror via cron to run during the nightime hours to take advantage of this.