How to cache openSUSE repositories with Squid
Contents
Summary
How to make your local Squid web cache work with openSUSE repositories and the openSUSE network installation process. In effect, how to run a fully autonomous, local on-demand repository mirror. Even with a high-speed ADSL internet connection, savings of up to 60% are easily achieved.
In 2012, I wrote this for squid 2.7. In April 2016, I updated everything to work with squid 3.4.x.
Before you begin
Unless you already have a working squid web-cache, or you want to set it up anyway, the setup I describe here is probably too complex. It will probably be easier to run a local rsync-mirror.
Background
In my company, we do quite a lot of testing of openSUSE, and over the last three-four years, we have increasingly switched to installing over the network. Prior to that, we would install from DVD images over NFS served by a local server. However, over last couple of years, we've been working a lot more with Factory and the regular snap-shots that lead up to a final/gold release. With those it is much easier to just point the installation process to the right URL and have everything downloaded there and then.
When we're testing installation or new hardware, we often have to repeat the installation process many times on different machines. Not because it doesn't work as such, but because we might be testing or debugging our own add-ons or to collect diagnostics. Sometimes we install on virtual machines, sometimes on desktops, more often on server hardware in our downstairs datacentre. We have a local Squid web cache, but after having switched to doing network installs more frequently, I have often been annoyed by the lack of effectiveness for caching the openSUSE repository. When I've already done one installation, the downloads for a subsequent one should obviously happen a lot faster, in fact at wire speed. Well, they don't and that's annoying when you know they could have been cached.
The immediate alternative would be to run a local mirror of the openSUSE repositories, but it requires a process for keeping a the local mirror up-to-date, plus a bit of manual interaction (adding the right URL when installing. This is all entirely feasible, but I thought using Squid would be a more elegant and (hopefully) fully autonomous solution. so I decided to figure out why our Squid wasn't coping.
Well, Squid and the openSUSE network installation process just don't work together very well. Not out-of-the-box anyway. The repository at download.opensuse.org is served by a load-distribution system combining mirrorbrain and metalinks. I won't go into any further detail, suffice to say that this means packages are downloaded using segmented downloading spread over multiple mirrors, which together makes it impossible for squid to do much caching.
The problem
Well, three problems really:
- the openSUSE repositories are mirrored around the world, clients are served by Mirrorbrain. Mirrorbrain does a good job of picking the most suitable IPv4 mirrors depending on your location, which presumably also means a good distribution so individual mirrors aren't overloaded. However, Squid does not know that multiple mirror sites serve the same file, making caching at best ineffective.
- the segmented download means a package is downloaded in bits from multiple mirrors. This is good for speeding up the download and for maximizing the use of the available downstream bandwidth. The problem is that Squid is only able to cache whole files, not parts of files, rendering caching completely useless.
- at the time of writing, the openSUSE mirror setup had a poor IPv6 geolocation. If you are accessing via IPv6, you are redirected to arbitrary mirrors around the world, some of them very slow.
I have solved all three problems:
- using a Squid url rewriter, I map all the mirror locations on to a single one.
- using a Squid logfile and a custom written daemon, I do complete downloads of all the files that are being fetched with segmented downloading.
Summary
For anyone, an individual or a group of people, doing repeated ad-hoc installations of openSUSE (typically Factory), using this squid setup means
- significantly faster installation due to downloads at wire speed
- significant bandwidth savings due to a working cache
- less load on openSUSE mirrors due to a working cache
- zero local mirror management (assuming a working squid setup).
- no need to worry about where to install from
Others doing e.g. repeated updates or adding software, should enjoy similar benefits (once the packages have been cached).
60% faster at 6Mbit/s downstream
I run this setup primarily to save time on installations. In the office, we have a 6000/600Kbit ADSL connection. It's sufficient for most activities, but when installing openSUSE over the network, it's really a bit slow. For openSUSE 12.1, it takes about an hour to complete phase 1 of the install process - 6-7 minutes for the initial 6 system installation images, then 50 minutes for a vanilla KDE installation.
However, installing at wire speed (our LAN is 100Mbit) from the Squid cache is a lot faster, taking only 22 minutes (15 seconds for the initial 6 installation images, 21 minutes for phase 1 to complete. That is a reduction of more than 60%. With a slower network connection and perhaps slower mirrors too, only more time saved.
Overall/theoretical efficiency
Update 2019/06/16. Since I originally wrote this article, the internet speeds available to consumers have been improving every other month. My measly 6Mbit ADSL line is way outdated by now, but it is a good example. Your potential gains from implementing the scheme I have described here depend entirely on
- your internet downlink speed
- your local network speed
- your usage patterns
With my 2012 example, I had a 6 MBit/s downlink and 100Mbit/s on local ethernet. Theoretically that made for a factor 16 improvement. Today I have 1Gbit/s local ethernet and 1Gbit/s downlink, so theoretically no improvement. However, depending on mirrors available, my 1Gbit downlink might be well below saturation, I might only have an average of 500Mbit/s. Some repositories, e.g. tumbleweed, have a limited selection of mirrors, so your benefits might be even more significant.
For my personal/business use or benefit, the use case has all but disappeared. Our downlink is plenty for infrequent single installs and we maintain a public mirror too. We don't do many regular installs either, so the main saving is on the load on mirrors, which is probably negligible anyway. (unless thousands of users were to dom local caching with this scheme).
Download
For the impatient, I've tar'ed everything into a single download. This contains the daemon code, one sample config files and the scripts for keeping up with the list of openSUSE mirrors. It's not as easy as just plonking another package into your openSUSE system with YaST or zypper, but the following step by step guide will hopefully help.
Current version is 1.3.
Change history
2016/04/19 version 1.3 Updates to work with Squid 3.4.x, added a fetcher206 systemd service file, updated jesred to 1.4. 2016/04/15 version 1.2a Tiny syntax error correction, no change in version#. Thank you, hpj. 2015/05/21 version 1.2 Miscellaneous updates from the last three years. 2012/06/04 version 1.1 Improved parsing of the mirror list 2012/05/18 version 1.0 First public release
Step by step
Squid
The Squid web-proxy is the key element in this setup, so a working Squid installation is prerequisite. Setting up Squid is not as complicated as it may appear, but you'll have to consult squid documentation, it's outside the scope of this article. Whether you prefer directing access using environment variables http_proxy et al, or if you run a transparent proxy (like I do), is not really important.
jesred
jesred is the URL rewriter. It's fairly mature, but fully functional. (original webpage). I had to make a couple of changes to make it fully compatible with squid 2.7, this was version 1.3. In April 2016, I made some minor changes to make it work with IPv6 clients too, version 1.4:
For the moment, it does not come packaged, you'll have to build it from scratch:
tar xzvf <tarball> cd jesred-1.4 make
Installation: when you're done, copy the binary jesred into /usr/local/bin or whatever your preferred location for your own binaries is.
The config file for jesred: /etc/squid/jesred.conf
allow = /etc/squid/redirector.acl rules = /etc/squid/opensuse-redirect.rules redirect_log = /var/log/squid/redirect.log rewrite_log = /var/log/squid/rewrite.log
Using /etc/squid/redirector.acl you can control which clients' requests the rewriter should process, but I find this is actually easier to control with Squid's ACL and storeurl_access directive, so I enable for all clients:
# rewrite all URLs from 0.0.0.0/0
/etc/squid/squid.conf
Configuration: add the following lines to /etc/squid/squid.conf
store_id_program /usr/local/bin/jesred store_id_children 5 acl metalink req_mime_type application/metalink4+xml store_id_access deny metalink store_id_access allow localnet
I also recommend changing "maximum_object_size" to e.g. 128Mb (default is 4Mb).
fetcher206 logfile
Amend /etc/squid/squid.conf as follows:
logformat f206 %{%Y-%m-%dT%H:%M:%S}tl %Ss/%03Hs %rm %ru %mt
access_log /var/log/squid/fetch206.log f206
This log will be read by fetcher206.
To prevent it growing too big, add the following to /etc/logrotate.d/ :
/var/log/squid/fetch206.log {
compress
dateext
maxage 365
rotate 5
size=+4M
notifempty
missingok
create 640 squid root
sharedscripts
postrotate
/etc/init.d/squid reload
endscript
}
squid delay pool
This is an optional step - depending on your available downstream bandwidth, you may want to restrict what is used by fetcher206 for retrieving the repository files. This prevents
- slowing down the current installation and
- abuse of the internet connection
delay_pools 1 delay_class 1 1 delay_access 1 allow localhost delay_parameters 1 1000000/1000000
Add the above to /etc/squid/squid.conf - it defines one delay_pool, only accessible from localhost (which is where fetcher206 will be running wget) with a maximum bandwidth of 1MByte/sec.
If you have other http/proxy traffic originating from localhost, you could just add another 127.0.0.x address, and use that specifically for fetcher206.
mirror database
We need a current list of the available openSUSE mirrors. This can be retrieved from mirrors.opensuse.org. For the time being, I use XSL to parse the HTML page, but I hope to move to a suitably formatted list direct from MirrorBrain.
mkdir -p /var/lib/fetcher206 cp tarball/Makefile.mirrors /var/lib/fetcher206/Makefile cp tarball/extract* /var/lib/fetcher206/ make -C /var/lib/fetcher206 cp tarball/opensuse_mirrors.cron /etc/cron.d/opensuse_mirrors
April 2016: I have not yet found a way of getting a mirror list straight from mirrorbrain.
reload squid
When you've come this far, it's time to reload squid with
squid -k reconfigure
fetcher206
fetcher206 is, for the time being, a PHP script. Install it by simply copying it into /usr/local/bin. It has a few hard-coded options, such as number of wgets to run concurrently, name of logfile etc.
fetcher206 comes with a systemd service unit, see the tar file.
fetcher206 needs a couple of extra PHP modules - php5-pcntl and php5-xsl.
