Install elasticsearch 5.x on SmartOS

Whenever possible, I like running software in containers instead of kvms. Aside from the obvious performance gains, server density increases significantly since you’re not kidnapping huge chunks of DRAM from the OS and holding it hostage.

I recently had a need to setup an elasticsearch 5.x cluster on a SmartOS machine. It was mostly straight-forward except for a couple gotchas. Here’s how you do it.

1. Create the zone/container. Here, we’re just using a stock ubuntu 16.04 LTS image. Note, the key here is the max_lwps field. Elasticsearch requires at least 2048.

vmadm create << EOF
{
  "brand": "lx",
  "image_uuid": "7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b",
  "autoboot": true,
  "alias": "elastic1",
  "hostname": "es01",
  "dns_domain": "example.com",
  "resolvers": [
    "192.168.1.1",
    "8.8.8.8"
  ],
  "max_physical_memory": 8192,
  "max_swap": 4096,
  "quota": 60,
  "max_lwps": 2048,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "192.168.1.100",
      "netmask": "255.255.255.0",
      "gateway": "192.168.1.1",
      "primary": true
    }
  ]
}
EOF

2. Login to the container and install elasticsearch

zlogin `vmadm list | awk '/elastic/{print $1}'`
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" &amp;gt; /etc/apt/sources.list.d/elastic-5.x.list
apt-get update ; apt-get install elasticsearch

3. Disable system_call_filter install during bootstrap phase

echo "bootstrap.system_call_filter: false" >> /etc/elasticsearch/elasticsearch.yml

4. Enable and fire it up!

systemctl enable elasticsearch ; systemctl start elasticsearch

5. Tail the logs to make sure everything started up OK.

tail -f /var/log/elasticsearch/elasticsearch.log

Starting out in IT

I recently received an email from a family friend asking what courses or certifications he should look into for getting started in IT. Without knowing what particularly interests him about IT, I wrote the following email and decided it might be useful for others just starting out.

TL;DR: Learn Linux, a scripting language of some kind (Python, Javascript, Bash, etc.), and basics of networking.

I’d whole-heartedly recommend starting with some sort of Linux certification/course. This will provide a solid foundation for whatever direction you decide to take in the I.T. field, be it developer, networking, operations, sys admin., cyber-security etc. They all take root in Linux/Unix systems and having a solid background in it will definitely get you off the ground running.

After this, I’d recommend picking up some sort of scripting language (Python, Bash, Javascript), it doesn’t really matter which, just get really good with at least one and build some stuff with it (even silly/stupid stuff). As you progress in your career/education, you’ll naturally pick up others along the way.

Things I’ve learned the hard way:

– Be a team player and learn from each other. Nobody makes it on their own and everyone is better than you at *something*, learn from this.

– Don’t get discouraged. Everybody in IT was once where you are right now. You’ll never learn everything there is to know, and any employer or individual that expects you to, isn’t worth your time. IT is a life of learning.

– Leave you ego at the door. This can sometimes be a difficult one and it may just be something you learn over time. I’ve interviewed several individuals that we passed on due to ego. Some of the best technologists I’ve worked with are those that always kept an open mind to different ways of doing things and didn’t succumb to one-up-man ship. It’s okay to be right, but be willing to entertain other points-of-view.

– Use the best tool for the job. There’s no single piece of technology for every task. Don’t pigeon-hole your skill-set. Just because you can make something work, doesn’t mean you should.

Bhyve pfSense 2.4 no console menu

I ran into an annoying issue today while trying to install pfsense 2.4.2 in a bhyve VM using the ISO installer. Everything went swimmingly until post-install when pfsense finished startup and never provided the expected pfSense console. All it would show is bootup complete.

I went through and confirmed /etc/ttys was configured properly and added console=comconsole to /boot/loader.conf. However, it still wouldn’t work. I’d get all the typical startup info but it still wouldn’t drop to the pfSense console.

To fix this, I ended up having to install pfSense using the memstick serial installer.

In case you aren’t already using vm-bhyve, here’s how it went down from start to finish:

1. Install and initial setup:

pkg install -y vm-bhyve grub2-bhyve
zfs create -o mountpoint=/bhyve zroot/bhyve
sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/bhyve"
vm init

I manage network bridges myself, so I’ll just import them into vm-bhyve so it can use them.

vm switch import wan bridge0
vm switch import mgmt bridge1
fetch -o /tmp/pf-memstick-serial.img.gz https://nyifiles.pfsense.org/mirror/downloads/pfSense-CE-memstick-serial-2.4.2-RELEASE-amd64.img.gz
gunzip /tmp/pf-memstick-serial.img.gz

2. Create pfsense VM:

cd /bhyve/.templates
cat > pfsense.conf <<EOF
loader="bhyveload"
cpu=2
memory=512M
network0type="virtio-net"
network0switch="wan"
network1type="virtio-net"
network1switch="mgmt"
disk0type="virtio-blk"
disk0name="disk0.img"
EOF
vm create -t pfsense -s20G pf1

3. Temporarily reconfigure the VM to use the memstick installer.
Basically, we just need to add another disk (the installer image) and make sure it’s first to boot.

cd /bhyve/pf1/
cp /tmp/
cp pf1.conf pf1.orig.conf
cat >pf1.conf<<EOF
loader="bhyveload"
cpu=2
memory=512M
network0_type="virtio-net"
network0_switch="wan"
network1_type="virtio-net"
network1_switch="mgmt"
disk0_type="virtio-blk"
disk0_name="/tmp/pf-memstick-serial.img"
disk1_type="virtio-blk"
disk1_name="disk0.img"
EOF
vm start pf1
vm console pf1

Walk through install process and when finished DON’T reboot. Simply disconnect from the console (~ + ctrl-d) and shut the vm down. If we let it reboot, it’ll just reboot back into the intaller since it’s still configured as the first disk.

vm stop pf1
mv /bhyve/pf1/pf1.orig.conf /bhyve/pf1/pf1.conf
vm start pf1

After it was all said and done, I checked what the memstick installer inserts into /boot/loader.conf to make it work. I’m guessing this is the key, and what I should’ve added to the loader config when I tried using the ISO installer initially.

boot_multicons="YES"
boot_serial="YES"
console="comconsole,vidconsole"
comconsole_speed="115200"

I didn’t try using the ISO installer and adding the above as I just wanted to get up and running, but it’d be interesting to see if it would do the trick.

Hope this helps some of you!