Syncthing on SmartOS

This is a quick tutorial on how to get syncthing running on SmartOS by means of an LX branded zone (ubuntu-16.04-20170403).

I initially tried using Joyent brand, but when starting syncthing, I received ‘Watching is not supported’ which is related to an issue with fsnotify on github . An alternative would be to simply use a bhyve vm which would get around this, but a container has a little less overhead than a full virtual machine, so that’s what I’m going with for now.

The setup is pretty straight forward but there are a few gotchas.

First, we’ll start with the zone definition. Note, this reserves a 100GB dataset for the container which we’ll use for syncing our files to/from.

$ cat > syncthing.json <<EOF
  "brand": "lx",
  "kernel_version": "4.3.0",
  "alias": "",
  "image_uuid": "7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b",
  "quota": 100,
  "delegate_dataset": true,
  "max_physical_memory": 1024,
  "resolvers": [
  "nics": [
        "nic_tag": "external",
        "ip": "",
        "netmask": "",
        "gateway": "",
        "primary": true

Next, we’ll create and login to the zone and change the mountpoint for data.

$ vmadm create -f syncthing.json
Successfully created VM fa986110-8fef-6110-bc37-a27b1d70cd3f
$ zlogin fa986110-8fef-6110-bc37-a27b1d70cd3f

# PATH=/native/usr/sbin:/native/usr/bin:$PATH
# zfs set mountpoint=/data zones/$(zonename)/data
# apt-mark hold systemd-sysv udev

Ok, we should be all set. Let’s add the syncthing repo and install it.

# curl -s | sudo apt-key add -
# echo "deb syncthing stable" > /etc/apt/sources.list.d/syncthing.list
# apt-get update
# apt-get install -y syncthing

When I tried enabling the syncthing service, I get the below error.

# systemctl enable syncthing
Failed to execute operation: No such file or directory

So to fix this I did the following:

# cp /usr/lib/systemd/user/syncthing.service /etc/systemd/system/

If you try to start syncthing up now, it will fail. We need to modify the syncthing service file and comment out the process hardening settings as this is an LX container and not relevant. Because we’re disabling all the hardening, we should setup a non-privileged user to run under.

# useradd -d /data -M syncthing

Another thing we need to do is set a HOME environment variable in the service definition. All said and done, it should look like this:

Description=Syncthing - Open Source Continuous File Synchronization

ExecStart=/usr/bin/syncthing -no-browser -no-restart -logflags=0
SuccessExitStatus=3 4
RestartForceExitStatus=3 4


Now, we should be all set. Let’s start it up and check status.

# systemctl daemon-reload
# systemctl enable syncthing
# systemctl start syncthing
# systemctl status syncthing
● syncthing.service - Syncthing - Open Source Continuous File Synchronization
   Loaded: loaded (/etc/systemd/system/syncthing.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-06-18 20:21:14 UTC; 3s ago
     Docs: man:syncthing(1)
 Main PID: 571357 ((yncthing))
   CGroup: /system.slice/syncthing.service
           └─571357 /usr/bin/syncthing -no-browser -no-restart -logflags=0
           ‣ 571357 /usr/bin/syncthing -no-browser -no-restart -logflags=0

Now, since syncthing admin will listen on localhost, we’ll ssh port-forward so we can access the admin page.

shaner@prec:~$ ssh -N -L 8384:

Using cloud-init with SmartOS

SmartOS provides the ability to inject cloud-init data into a zone/VM. This is extremely useful for automating some of the menial tasks one would normally have to perform manually like setting up users, installing packages, or pulling down a git repo. Basically, anything you can stuff into cloud-init user-data is at your disposal.

However, since SmartOS zone definitions are in JSON and cloud-init data is in yaml, it’s not immediately obvious how to supply this information. What it boils down to is, escape all double-quotes (“) and line-feeds.

Here’s our cloud-init config which creates a new user and import their ssh key from


  - default
  - name: shaner
    ssh_import_id: shaner
    lock_passwd: false
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    shell: /bin/bash

So following the above escape rules above, here’s our full SmartOS zone spec, including the cloud-init data. Note the cloud-init:user-data key.

  "brand": "kvm",
  "alias": "ubuntu-xenial",
  "ram": "2048",
  "vcpus": "2",
  "resolvers": [
  "nics": [
      "nic_tag": "admin",
      "ip": "",
      "netmask": "",
      "gateway": "",
      "model": "virtio",
      "primary": true
  "disks": [
      "image_uuid": "429bf9f2-bb55-4c6f-97eb-046fa905dd03",
      "boot": true,
      "model": "virtio"
  "customer_metadata": {
    "cloud-init:user-data": "#cloud-config\n\nusers:\n  - default\n  - name: shaner\n    ssh_import_id: shaner\n    lock_passwd: false\n    sudo: \"ALL=(ALL) NOPASSWD:ALL\"\n    shell: /bin/bash"

Let’s go ahead and create the zone on our SmartOS box.

[root@vmm01 /opt/templates]# vmadm create < ubuntu-xenial.json
Successfully created VM 0e908925-600a-4365-f161-b3a51467dc08
[root@vmm01 /opt/templates]# vmadm list 
UUID                                  TYPE  RAM      STATE             ALIAS
0e908925-600a-4365-f161-b3a51467dc08  KVM   2048     running           ubuntu-xenial

After a bit of time, we can try logging in as our new user we requested. Recall, we asked cloud-init to pull in our public ssh key from launchpad so, if you get prompted for a password, something is wrong.

shaner@tp25:~$ ssh
The authenticity of host ' (' can't be established.
ECDSA key fingerprint is SHA256:hFPjwUJjd7N/Gb9EE37fTVt2Lk6NVzoLKvhFN7wYw2M.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-116-generic x86_64)
Certified Ubuntu Cloud Image

   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ;  Instance (Ubuntu 16.04.3 LTS 20180222)

 * Documentation:
 * Management:
 * Support:

  Get cloud support with Ubuntu Advantage Cloud Guest:

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

shaner@0b8d7a26-ffe4-e859-eb56-d96d02bf213e:~$ sudo ls
shaner@0b8d7a26-ffe4-e859-eb56-d96d02bf213e:~$ sudo apt-update && sudo apt-upgrade -y

There’s a LOT you can do with cloud-init data. See the below links for more info.

Cloud-init examples:
Joyent Datasource:
Joyent Ubuntu Image documentation:

SmartOS on Alix APU2c2

Over the past few years, I’ve been using SmartOS as my hypervisor of choice coupled with a management layer called Project-Fifo. I have to say, it’s been a joy to work with.

I had a couple Alix APU’s laying around and was curious how well SmartOS would run it.

To begin, I downloaded the SmartOS USB image and pushed onto a USB drive.


bzcat smartos-latest-USB.img.bz2 | dd of=/dev/sdb conv=fdatasync

Now,  we can plug the USB drive into the USB port of our APU then power it up. Be sure to plug in your serial cable first so you can see what’s going on.

Soon as you see the GRUB boot menu, press ‘c’ enter the below to switch the console output to ttya from vga. Then hit enter.

variable os_console ttya

After doing the above, you should see something like this

And eventually this. You can now follow along and configure your new SmartMachine.

After networking is all configured, you’ll be prompted to layout a zpool on your disk (hopefully you have an m-sata installed).

Now, just hit enter and be sure to remove the USB drive so SmartOS will boot from the m-sata drive. Eventually, you’ll be presented with the login screen like the below where you can login as root using the password you provided during setup.

Okay, so let’s import a debian dataset and spin up a zone (read: container)

Define the zone config and use vmadm to create it and login!

Cool! A fully functioning Debian 9 container. Let’s setup wordpress and run a load test for funsies.


Now, let’s use Siege to perform some load testing

Not Bad!