The home lab

After reading about others home lab environments, I was inspired to write about mine. It’s nothing too fancy but maybe somebody out there will find it interesting.

I’ve flipped-flopped several times between various operating systems and hypervisors trying to find the solution that best fits my needs. I’ve tried everything from pure Debian (kvm/libvirt, lxc), and Ubuntu (kvm/libvirt, lxd), to FreeBSD 11 (jails, bhyve), and SmartOS (zones, kvm).

At this point, I’ve settled on SmartOS, using project-fifo for my management layer and scheduler. I briefly tried out Joyent’s Triton, but it requires an entire machine be dedicated to running the head node with (at a minimum) 64GB of DRAM. Further, while it does provide an operations portal for configuring the system, the end-user portal requires you to sign-up for a support contract.

Why? SmartOS provides many benefits including running completely from DRAM. This is nice because I don’t have devote entire disk(s) for running the OS. The OS boots from a simple USB key where it loads its configuration and imports the ZFS datasets responsible for running your VM’s (zones/kvm). I won’t dive into the many benefits of ZFS here, but if you’re like me and value your data, you’ll appreciate it.

Because it boots from a USB key and the config is a single file, upgrades are extremely easy, just insert another USB key with an updated image and reboot. Boom.

So let’s get into the physical gear.

Servers:
* 1x Dell R720xd = 64GB RAM, 24x 300G 10K SAS + 1 SSD L2ARC
* 2x Dell R610 = 64GB RAM, 6x 250G SSD

Firewall/Router (not pictured):
* PCEngines APU2 = 2GB RAM, 30GB mSATA, 3 intel (igb) gigabit ports

Switch:
* Quanta LB4M 48port GB switch +2 10GB ports

Networking:
* Intel x520-da1 10Gbe (ixgbe) NIC on seach server
* Broadcom quad port 1gbe nic (bnx)
* Each server has two links to the switch. The first is a 1gbe link for the admin_nic and the second is a 10gbe for the trunk_nic which connects my zones/VMs to their respective VLANS.

I’ll layout the various VM’s I have running and why in a later post, or I’ll simply update this one at a later date.

Nginx and Letsencrypt on SmartOS

acme_tiny is a nice, small utility for creating and renewing your doman SSL/TLS certificates.

It’s less than 200 lines of bash and just works. Here’s how to set it up and have your certificate automatically renewed once a month.

First, let’s get things installed; nginx and the acme-tiny client and setup the necessary directories.

pkgin in nginx py36-acme-tiny
mkdir -p /opt/local/etc/acme /opt/local/www/acme

Add the following stanza towards the top of your nginx config and reload nginx.

vi /opt/local/etc/nginx/nginx.conf
  location ^~ /.well-known/acme-challenge/ {
     alias /opt/local/www/acme/;
     try_files $uri =404;
  }
nginx -s reload

Now, let’s setup the necessary keys so we can request a new certificate for our site.

cd /opt/local/etc/acme
openssl genrsa 4096 > account.key
openssl genrsa 4096 > domain.key
openssl req -new -sha256 -key domain.key -subj "/CN=shaner.life" > domain.csr

Now we can create the initial request for our domain.

acme_tiny   --account-key /opt/local/etc/acme/account.key \
            --csr /opt/local/etc/acme/domain.csr \
            --acme-dir /opt/local/www/acme \
            > /opt/local/etc/acme/signed.crt
curl -s 'https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem' > /opt/local/etc/acme/intermediate.pem
cat /opt/local/etc/acme/signed.crt /opt/local/etc/acme/intermediate.pem > /opt/local/etc/acme/fullchain.pem

If you get an error complaining about the expired terms of service pdf don’t fret, we just need to update the acme script with the name of the updated version.

vi /opt/local/bin/acme_tiny
 # search for 'pdf' and replace the url with the one from the error you recieved during your (failed) certificate request.

If you got an error complaining about not being able to access the acme-challenge token, double check your nginx config and be sure to restart nginx.

One last step before we can setup nginx with our new certs. We need to create a diffie-helman key. This will take a while, go grab some coffee and when you get back it should be done.

openssl dhparam 4096 > /opt/local/etc/nginx/dhparam.pem

Now that we have our dh key, certificate and private key. Let’s setup nginx to use them. Here are the relevant SSL bits that can be added to your server stanza in _opt/local/etc/nginx.conf_ .

server_tokens off;
ssl_certificate /opt/local/etc/acme/fullchain.pem;
ssl_certificate_key /opt/local/etc/acme/domain.key;
ssl_dhparam dhparam.pem;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5';

Let’s automate this thing. Here, we’re creating a script to be called by a cron job for automatic renewal of our certificate. We do this on a somewhat frequent schedule as lets-encrypt only issues certificates that are valid for 3 months.

cat > /opt/local/etc/acme/renew.sh <<EOF
#!/bin/bash
acme_tiny --account-key /opt/local/etc/acme/account.key --csr /opt/local/etc/acme/domain.csr --acme-dir /opt/local/www/acme > /opt/local/etc/acme/signed.crt
curl -s 'https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem' > /opt/local/etc/acme/intermediate.pem
cat /opt/local/etc/acme/signed.crt /opt/local/etc/acme/intermediate.pem > /opt/local/etc/acme/fullchain.pem
cp fullchain.pem /opt/local/etc/nginx/ssl/fullchain.pem 
nginx -s reload
EOF

Okay, with our script in place, let’s create the cron job to run once a month (on the first).

crontab -e 
0 0 1 * * /opt/local/etc/acme/renew.sh >/dev/null 2>&1

All set! Good Job 🙂

Install elasticsearch 5.x on SmartOS

Whenever possible, I like running software in containers instead of kvms. Aside from the obvious performance gains, server density increases significantly since you’re not kidnapping huge chunks of DRAM from the OS and holding it hostage.

I recently had a need to setup an elasticsearch 5.x cluster on a SmartOS machine. It was mostly straight-forward except for a couple gotchas. Here’s how you do it.

1. Create the zone/container. Here, we’re just using a stock ubuntu 16.04 LTS image. Note, the key here is the max_lwps field. Elasticsearch requires at least 2048.

vmadm create << EOF
{
  "brand": "lx",
  "image_uuid": "7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b",
  "autoboot": true,
  "alias": "elastic1",
  "hostname": "es01",
  "dns_domain": "example.com",
  "resolvers": [
    "192.168.1.1",
    "8.8.8.8"
  ],
  "max_physical_memory": 8192,
  "max_swap": 4096,
  "quota": 60,
  "max_lwps": 2048,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "192.168.1.100",
      "netmask": "255.255.255.0",
      "gateway": "192.168.1.1",
      "primary": true
    }
  ]
}
EOF

2. Login to the container and install elasticsearch

zlogin `vmadm list | awk '/elastic/{print $1}'`
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" &amp;gt; /etc/apt/sources.list.d/elastic-5.x.list
apt-get update ; apt-get install elasticsearch

3. Disable system_call_filter install during bootstrap phase

echo "bootstrap.system_call_filter: false" >> /etc/elasticsearch/elasticsearch.yml

4. Enable and fire it up!

systemctl enable elasticsearch ; systemctl start elasticsearch

5. Tail the logs to make sure everything started up OK.

tail -f /var/log/elasticsearch/elasticsearch.log