Setup Package Cache Server on Ubuntu 18.04

If you’re like me and have several Debian/Ubuntu machines on your network there’s going to come a time when you need to upgrade them. Doing so, will use up a lot of bandwidth while every machine will likely be downloading the same packages. This may or may not upset your significant other who’s binge-watching Gilmore Girls on Netflix.

Since you’ve slowed the Internet down to a crawl, this it might  be a good excuse to leave the computer and get outside for some fresh air. HA, who am I kidding, we got stuff to do. Let’s setup a cache!

Here, I’ll be using Ubuntu 18.04 LTS and setting up apt-cacher-ng.

While we could setup Squid to function in the same way, and cache way more than just debian/ubuntu packages, using apt-cacher-ng is a quick win and requires hardly any configuration to get going. Maybew I’ll cover how to setup Squid in a future post.

First, we’ll make sure everything is up to date, then install apt-cacher-ng.

sudo apt-get update
sudo apt-get dist-upgrade -y
sudo apt-get install apt-cacher-ng -y

Let’s go over a few config options. We won’t go over every single one, just the ones that might be relevant. Open /etc/apt-cacher-ng/acng.conf using your favorite text editor and let’s start.

CacheDir. This is where acng will actually do its caching and store packages as they’re downloaded. You may want to change this if you’d like to save packages to a different partition with more space.

CacheDir: /var/cache/apt-cacher-ng

Port. Here, you can change the TCP port apt-cacher will listen on. Note, this should be higher than 1024. Otherwise, you would need to run acng as root.

Port: 3142

BindAddress. If you have a multi-honed server with several IP addresses, you might want acng to only listen on one. Just provide the IP here to do so. By default, it will listen on all interfaces (0.0.0.0).

BindAddress

ReportPage. If you’d like to see some misc statistics about the caching of packages (hit/miss ratio, space usage, etc…) set the page name here. To disable it, just comment out this line.

ReportPage: acng-report.html

Here’s how the page looks from my server:

ExThreshold. The number of days before deleting unreferenced files. You may want to tweak this to cache packages for longer periods of time.

ExTreshhold: 4

MaxDlSpeed. Here, you can limit how much bandwidth acng will use up. Very handy in some environments. Units are KiB/s.

MaxDlSpeed: 250

There are several other options layed out in the config file, feel free to read more on them and tweak as needed. For now, let’s move onto setting up our hosts to point at our new cache server for packages.

Log into one of your Ubuntu/Debian machines and create a new file at /etc/apt/apt.conf.d/21acng. Replace SERVER_IP with the IP address of the cache server you setup. If you specified a BindAddress above, use that one instead.

echo '"Acquire::HTTP::proxy "http://SERVER_IP:3142";' | sudo tee /etc/apt/apt.conf.d/21acng

Now, let’s try it out! From this client machine, run:

sudo apt-get update

Back on the caching server, you can see what’s happening by tailing the log file.

sudo tail -f /var/log/apt-cacher-ng/apt-cacher.log
.......
............
1532529680|O|227|10.137.5.1|ppa.launchpad.net/openjdk-r/ppa/ubuntu/dists/bionic/InRelease
1532529680|O|228|10.137.5.1|packages.cloud.google.com/apt/dists/cloud-sdk-bionic/InRelease
1532529680|O|217|10.137.5.1|dl.google.com/linux/chrome/deb/dists/stable/Release
.........
..............

Don’t forget, you can access the statistics page by opening a web browser to http://SERVER_IP:3142/acng-report.html

The home lab

After reading about others home lab environments, I was inspired to write about mine. It’s nothing too fancy but maybe somebody out there will find it interesting.

I’ve flipped-flopped several times between various operating systems and hypervisors trying to find the solution that best fits my needs. I’ve tried everything from pure Debian (kvm/libvirt, lxc), and Ubuntu (kvm/libvirt, lxd), to FreeBSD 11 (jails, bhyve), and SmartOS (zones, kvm).

At this point, I’ve settled on SmartOS, using project-fifo for my management layer and scheduler. I briefly tried out Joyent’s Triton, but it requires an entire machine be dedicated to running the head node with (at a minimum) 64GB of DRAM. Further, while it does provide an operations portal for configuring the system, the end-user portal requires you to sign-up for a support contract.

Why? SmartOS provides many benefits including running completely from DRAM. This is nice because I don’t have devote entire disk(s) for running the OS. The OS boots from a simple USB key where it loads its configuration and imports the ZFS datasets responsible for running your VM’s (zones/kvm). I won’t dive into the many benefits of ZFS here, but if you’re like me and value your data, you’ll appreciate it.

Because it boots from a USB key and the config is a single file, upgrades are extremely easy, just insert another USB key with an updated image and reboot. Boom.

So let’s get into the physical gear.

Servers:
* 1x Dell R720xd = 64GB RAM, 24x 300G 10K SAS + 1 SSD L2ARC
* 2x Dell R610 = 64GB RAM, 6x 250G SSD

Firewall/Router (not pictured):
* PCEngines APU2 = 2GB RAM, 30GB mSATA, 3 intel (igb) gigabit ports

Switch:
* Quanta LB4M 48port GB switch +2 10GB ports

Networking:
* Intel x520-da1 10Gbe (ixgbe) NIC on seach server
* Broadcom quad port 1gbe nic (bnx)
* Each server has two links to the switch. The first is a 1gbe link for the admin_nic and the second is a 10gbe for the trunk_nic which connects my zones/VMs to their respective VLANS.

I’ll layout the various VM’s I have running and why in a later post, or I’ll simply update this one at a later date.

Nginx and Letsencrypt on SmartOS

acme_tiny is a nice, small utility for creating and renewing your doman SSL/TLS certificates.

It’s less than 200 lines of bash and just works. Here’s how to set it up and have your certificate automatically renewed once a month.

First, let’s get things installed; nginx and the acme-tiny client and setup the necessary directories.

pkgin in nginx py36-acme-tiny
mkdir -p /opt/local/etc/acme /opt/local/www/acme

Add the following stanza towards the top of your nginx config and reload nginx.

vi /opt/local/etc/nginx/nginx.conf
  location ^~ /.well-known/acme-challenge/ {
     alias /opt/local/www/acme/;
     try_files $uri =404;
  }
nginx -s reload

Now, let’s setup the necessary keys so we can request a new certificate for our site.

cd /opt/local/etc/acme
openssl genrsa 4096 > account.key
openssl genrsa 4096 > domain.key
openssl req -new -sha256 -key domain.key -subj "/CN=shaner.life" > domain.csr

Now we can create the initial request for our domain.

acme_tiny   --account-key /opt/local/etc/acme/account.key \
            --csr /opt/local/etc/acme/domain.csr \
            --acme-dir /opt/local/www/acme \
            > /opt/local/etc/acme/signed.crt
curl -s 'https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem' > /opt/local/etc/acme/intermediate.pem
cat /opt/local/etc/acme/signed.crt /opt/local/etc/acme/intermediate.pem > /opt/local/etc/acme/fullchain.pem

If you get an error complaining about the expired terms of service pdf don’t fret, we just need to update the acme script with the name of the updated version.

vi /opt/local/bin/acme_tiny
 # search for 'pdf' and replace the url with the one from the error you recieved during your (failed) certificate request.

If you got an error complaining about not being able to access the acme-challenge token, double check your nginx config and be sure to restart nginx.

One last step before we can setup nginx with our new certs. We need to create a diffie-helman key. This will take a while, go grab some coffee and when you get back it should be done.

openssl dhparam 4096 > /opt/local/etc/nginx/dhparam.pem

Now that we have our dh key, certificate and private key. Let’s setup nginx to use them. Here are the relevant SSL bits that can be added to your server stanza in _opt/local/etc/nginx.conf_ .

server_tokens off;
ssl_certificate /opt/local/etc/acme/fullchain.pem;
ssl_certificate_key /opt/local/etc/acme/domain.key;
ssl_dhparam dhparam.pem;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5';

Let’s automate this thing. Here, we’re creating a script to be called by a cron job for automatic renewal of our certificate. We do this on a somewhat frequent schedule as lets-encrypt only issues certificates that are valid for 3 months.

cat > /opt/local/etc/acme/renew.sh <<EOF
#!/bin/bash
acme_tiny --account-key /opt/local/etc/acme/account.key --csr /opt/local/etc/acme/domain.csr --acme-dir /opt/local/www/acme > /opt/local/etc/acme/signed.crt
curl -s 'https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem' > /opt/local/etc/acme/intermediate.pem
cat /opt/local/etc/acme/signed.crt /opt/local/etc/acme/intermediate.pem > /opt/local/etc/acme/fullchain.pem
cp fullchain.pem /opt/local/etc/nginx/ssl/fullchain.pem 
nginx -s reload
EOF

Okay, with our script in place, let’s create the cron job to run once a month (on the first).

crontab -e 
0 0 1 * * /opt/local/etc/acme/renew.sh >/dev/null 2>&1

All set! Good Job 🙂