Bro on FreeBSD Using Netmap

NETMAP is a framework for very fast packet I/O from userspace with support for FreeBSD, Linux, and even Windows. Here, we’ll show how to set Bro up to use it.

Bro provides support for monitoring interfaces using netmap. However, as of FreeBSD 11.1 (bro-2.5.1) the binary package doesn’t ship with the needed netmap plugin. Furthermore, the port doesn’t support building any auxiliary plugins. Not to worry, we’ll just install Bro from source. It’s painless, trust me. 🙂

Setup used:
– FreeBSD 11.1-RELEASE
– NIC – Intel (igb driver)

1. First, we’ll need to install the necessary dependencies for compiling bro.

pkg install -y bash git flex bison cmake libpcap python py27-sqlite3 caf swig30`

2. Download the source tarball and extract it.

mkdir /usr/local/src && cd /usr/local/src
fetch https://www.bro.org/downloads/bro-2.5.2.tar.gz
tar xzf bro-2.5.2.tar.gz && rm bro-2.5.2.tar.gz
cd bro-2.5.2

2. Let’s compile bro. Note, if you just need the plugin (for another system with a binary install using pkg), don’t run `make install`. Instead, after performing the build, just grab `./build/dist/Bro_Netmap-0.1.tar.gz` and copy/extract it where you need (with similiar FreeBSD version). Otherwise, proceed with the typical configure, make, and make install.

./configure && make -j2 && make install

3. Now we can compile and install the netmap plugin.

cd aux/plugins/netmap
./configure && make && make install

4. Now, we just need to configure bro to use netmap. Here, we’re instructing bro to create 4 load balancer processes for monitoring the the igb1 interface.

cat >/usr/local/bro/etc/node.cfg <<EOF
[logger]
type=logger
host=localhost
[manager]
type=manager
host=localhost
[proxy-1]
type=proxy
host=localhost
[worker-1]
type=worker
host=localhost
lb_method=custom
lb_procs=4
interface=netmap::igb1

If the interface you’re having Bro monitor is dedicated to Bro and nothing else, enable `promisc` on the interface. Just edit /etc/rc.conf:

ifconfig_igb3="promisc mtu 9000 up"

5. This is all well and good, however packets won’t be balanced across your four `lb_procs` without the help of a utility called `lb`. There’s currently no port for `lb` and you won’t find it with other netmap utilities shipped under `/usr/src/tools/tools/netmap`. We’ll have to compile it manually.

cd /usr/local/src
fetch https://github.com/luigirizzo/netmap/archive/master.zip
unzip master.zip && rm master.zip
cd netmap-master/apps/lb
pkg install gmake
gmake
cp lb /usr/local/bin/
rehash

Now, lets start `lb` in the background and fire up Bro.
Make sure to create the same amount of pipes (`-p`) as `lb_procs` from the Bro config.

lb -i igb1 -p 4&
/usr/local/bro/bin/broctl deploy
cd /usr/local/bro/logs/current

You should now see some logs start to roll in.

6. Okay now, let’s create some init scripts so both `lb` and `bro` start at boot time.

fetch -o /usr/local/etc/rc.d/bro https://gist.githubusercontent.com/shanerman/3988b535e9b7d1ee92afd88e47ba50da/raw/2a3e3feec2a3b9734013736e4dc43513608a9afd/bro
fetch -o /usr/local/etc/rc.d/lb https://gist.githubusercontent.com/shanerman/0efe1f5c4b5c7ab083f274a3d9c14543/raw/acdb2e7d6ae3d0599a0783006de9dd8cc3672d31/lb
chmod 555 /usr/local/etc/rc.d/{lb,bro}

That’s all folks. If you have any questions or tips, you can email me at shaner@shaner.life

Install Bro on pfSense

I’ve been working with Bro a lot lately and thought it’d be worth trying to get Bro running on pfSense. In an ideal situation, you wouldn’t normally run an IDS on your firewall, but for low bandwidth installations or the budget constrained, it’ll work fine.

1. You’ll need to enable ssh access to your pfSense firewall as it’s not enabled by default. To do this, login to pfsense and browse to System > Advanced, then scroll down to the SSH section and check ‘Enable Secure Shell’.

I recommend setting up pub key authentication by adding your public key to the admin user in pfsense. This will allow you to login via ssh without using a password. Just don’t lose your private key!

2. Now open a terminal and ssh into pfsense. Note, we’re using the ‘root’ user instead of the normal ‘admin’ you typically use to login via web interface.

ssh root@192.168.1.1

You’ll then be presented with a text interface. You’ll want to drop to a shell which is option ‘8’.

3. By default, pfSense disables upstream pkg repositories (for good reason). So lets re-enable them albeit, temporarily. There are two files you’ll need to edit.

/usr/local/etc/pkg/repos/FreeBSD.conf
/usr/local/share/pfSense/pkg/repos/pfSense-repo.conf

Make it look like:

FreeBSD: { enabled: yes }

4. Now, we can update the pkg cache and get on with installing and configuring bro.

pkg update && pkg install -y bro

5. Bro should now be installed. **You should now reverse the changes you made in step 3.** You’ll need to pick which interface you’d like Bro to monitor. I’m going to monitor my (LAN) interface which equates to ‘igb1’ for my Intel NIC.

cat > /usr/local/etc/node.cfg <<EOF
[logger]
type=logger
host=localhost
[manager]
type=manager
host=localhost
[proxy-1]
type=proxy
host=localhost
[worker-1]
type=worker
host=localhost
interface=igb1
EOF

6. Next, we’ll disable status emails, and have bro rotate logs once a day instead of the default one hour.

cat > /usr/local/etc/broctl.cfg <<EOF
MailTo = root@localhost
MailConnectionSummary = 0
MinDiskSpace = 0
MailHostUpDown = 0
LogRotationInterval = 86400
LogExpireInterval = 0
StatsLogEnable = 1
StatsLogExpireInterval = 1
StatusCmdShowAll = 0
CrashExpireInterval = 1
SitePolicyScripts = local.bro
LogDir = /usr/local/logs
SpoolDir = /usr/local/spool
CfgDir = /usr/local/etc
EOF

7. Have bro check your configuration and start it up. While the ‘deploy’ command will automatically run ‘check’ for you, it’s good practice to run it by itself after any modifications to Bro before deploying those changes.

broctl check && broctl deploy

8. You should now be able to watch the logs Bro is generating.

tail -f /usr/local/logs/current/*

So there you have it, bro running on pfSense. In upcoming articles, I’ll dive into parsing bro logs using `bro-cut` and also how to setup Bro to push logs into an Apache Kafka pipeline for more fun and profit.

Using zxfer to backup ZFS pools

I was recently looking for an easy way to backup some FreeBSD jails I have running various services. With the jails residing on top of ZFS (using iocage), a quick Google search turned up the usual zfs ‘send’ and ‘receive’ mixed with miscellaneous pipes and redirection. Having wrote several backup scripts in the past, they all felt sort of hack-ish and rushed (which they were). After thinking to myself “surely, someone has dealt with this problem before.” I finally came across zxfer.

I’m unsure of the original author and it was apparently abandoned several years ago around FreeBSD 8.2. Huge thanks to Allan Jude for maintaining the current port.

You can tell a lot of thought went into not just the program itself, but
the supporting documentation as well. I’m typically not one to judge a book by its cover, but with documentation like this, I feel it was a safe bet. It doesn’t just throw command line switches at you and set you on your way. Instead, nearly ever option explains why and when you might use it.

Goal:  Backup iocage jails to remote server (also running zfs).

Solution: Use iocage’s built-in snapshot management and zxfer to send those snapshots off-server and/or off-site.

Note, assume we’ve already got iocage setup and we’re running some jails. Also note, zxfer doesn’t perform any snap-shotting itself. Its up to you to setup a sensible snap-shotting regimen.

On the jail host, take a snapshot of all running jails

for j in $(iocage list | awk '/up/{print $4}'); do iocage snapshot ${j}; done 

Note, zxfer can be used in either a push or pull method, wherein the connection is initiated from the jail host or the backup server respectively. Here, I’ve decided to use the pull method.

On the backup server:

zxfer -dFkPv -g31 -O root@172.16.0.7 -R zroot/iocage/jailszroot/backups 

Assuming you’ve already setup SSH key authentication, from the backup server, we’re recursively sending all dataset snapshots under zroot/iocage/jails on the jail host (172.16.0.7) to our local zfs pool (zroot/backups), keeping the last 30 days of snapshots (on both servers).

After the initial sync, any further runs of the above command will send just the difference between the last two snapshots of the given datasets!

MAAS Custom Partitioning

Currently, MAAS doesn’t support custom partitioning through the MAAS dashboard for CentOS. However, you can utilize curtin to perform this task for you.

Here’s an example /etc/maas/preseeds/curtin_userdata_centos assuming /dev/vda is your disk:

#cloud-config

verbosity: 3
showtrace: true

block-meta:
    format: gpt

partitioning_commands:
    builtin: []
    01_partition_announce: ["echo", "'### Partitioning disk ###'"]
    01_partition_make_label: ["/sbin/parted", "/dev/vda", "-s", "'","mklabel","gpt","'"]
    02_partition_make_part: ["/sbin/parted", "/dev/vda", "-s", "'","mkpart","primary","1M","2M","'"]
    03_partition_set_name: ["/sbin/parted", "/dev/vda", "-s", "'","set","1","bios_grub","on","'"]
    04_partition_make_part: ["/sbin/parted", "/dev/vda", "-s", "'","mkpart","primary","2M","512M","'"]
    05_partition_make_part: ["/sbin/parted", "/dev/vda", "-s", "'","mkpart","primary","512M","5G","'"]
    06_partition_make_part: ["/sbin/parted", "/dev/vda", "-s", "'","mkpart","primary","5G","10G","'"]
    07_partition_make_part: ["/sbin/parted", "/dev/vda", "-s", "'","mkpart","primary","10G","20G","'"]
    08_partition_make_part: ["/sbin/parted", "/dev/vda", "-s", "'","mkpart","primary","20G","3500G","'"]
    09_partition_make_part: ["/sbin/parted", "/dev/vda", "-s", "'","mkpart","primary","3500G","3508G","'"]
    10_partition_announce: ["echo", "'### Creating filesystems ###'"]
    12_partition_make_fs: ["/sbin/mkfs", "-t", "ext4", "/dev/vda2"]
    13_partition_make_fs: ["/sbin/mkfs", "-t", "ext4", "/dev/vda3"]
    14_partition_make_fs: ["/sbin/mkfs", "-t", "ext4", "/dev/vda4"]
    15_partition_make_fs: ["/sbin/mkfs", "-t", "ext4", "/dev/vda5"]
    16_partition_make_fs: ["/sbin/mkfs", "-t", "ext4", "/dev/vda6"]
    17_partition_make_swap: ["sh", "-c", "mkswap /dev/vda7"]
    18_partition_label_fs: ["/sbin/e2label", "/dev/vda2", "maas-boot"]
    19_partition_label_fs: ["/sbin/e2label", "/dev/vda3", "maas-root"]
    20_partition_label_fs: ["/sbin/e2label", "/dev/vda4", "maas-usr"]
    21_partition_label_fs: ["/sbin/e2label", "/dev/vda5", "maas-var"]
    22_partition_label_fs: ["/sbin/e2label", "/dev/vda6", "maas-home"]
    23_partition_mount_fs: ["sh", "-c", "mount /dev/vda3 $TARGET_MOUNT_POINT"]
    24_partition_mkdir: ["sh", "-c", "mkdir $TARGET_MOUNT_POINT/boot"]
    25_partition_mkdir: ["sh", "-c", "mkdir $TARGET_MOUNT_POINT/usr"]
    26_partition_mkdir: ["sh", "-c", "mkdir $TARGET_MOUNT_POINT/var"]
    27_partition_mkdir: ["sh", "-c", "mkdir $TARGET_MOUNT_POINT/home"]
    28_partition_mount_fs: ["sh", "-c", "mount /dev/vda2 $TARGET_MOUNT_POINT/boot"]
    29_partition_mount_fs: ["sh", "-c", "mount /dev/vda4 $TARGET_MOUNT_POINT/usr"]
    30_partition_mount_fs: ["sh", "-c", "mount /dev/vda5 $TARGET_MOUNT_POINT/var"]
    31_partition_mount_fs: ["sh", "-c", "mount /dev/vda6 $TARGET_MOUNT_POINT/home"]
    32_partition_announce: ["echo", "'### Filling /etc/fstab ###'"]
    33_partition_make_fstab: ["sh", "-c", "echo 'LABEL=maas-root / ext4 defaults 0 0' >> $OUTPUT_FSTAB"]
    34_partition_make_fstab: ["sh", "-c", "echo 'LABEL=maas-boot /boot ext4 defaults 0 0' >> $OUTPUT_FSTAB"]
    35_partition_make_fstab: ["sh", "-c", "echo 'LABEL=maas-usr /usr ext4 defaults 0 0' >> $OUTPUT_FSTAB"]
    36_partition_make_fstab: ["sh", "-c", "echo 'LABEL=maas-var /var ext4 defaults 0 0' >> $OUTPUT_FSTAB"]
    37_partition_make_fstab: ["sh", "-c", "echo 'LABEL=maas-home /home ext4 defaults 0 0' >> $OUTPUT_FSTAB"]
    38_partition_make_fstab: ["sh", "-c", "echo '/dev/vda7 none swap sw 0 0' >> $OUTPUT_FSTAB"]


debconf_selections:
 maas: |
  {{for line in str(curtin_preseed).splitlines()}}
  {{line}}
  {{endfor}}

late_commands:
  maas: [wget, '--no-proxy', '{{node_disable_pxe_url}}', '--post-data', '{{node_disable_pxe_data}}', '-O', '/dev/null']

Starting out in IT

I recently received an email from a family friend asking what courses or certifications he should look into for getting started in IT. Without knowing what particularly interests him about IT, I wrote the following email and decided it might be useful for others just starting out.

TL;DR: Learn Linux, a scripting language of some kind (Python, Javascript, Bash, etc.), and basics of networking.

I’d whole-heartedly recommend starting with some sort of Linux certification/course. This will provide a solid foundation for whatever direction you decide to take in the I.T. field, be it developer, networking, operations, sys admin., cyber-security etc. They all take root in Linux/Unix systems and having a solid background in it will definitely get you off the ground running.

After this, I’d recommend picking up some sort of scripting language (Python, Bash, Javascript), it doesn’t really matter which, just get really good with at least one and build some stuff with it (even silly/stupid stuff). As you progress in your career/education, you’ll naturally pick up others along the way.

Things I’ve learned the hard way:

– Be a team player and learn from each other. Nobody makes it on their own and everyone is better than you at *something*, learn from this.

– Don’t get discouraged. Everybody in IT was once where you are right now. You’ll never learn everything there is to know, and any employer or individual that expects you to, isn’t worth your time. IT is a life of learning.

– Leave ego at the door. This can sometimes be a difficult one and it may just be something you learn over time. I’ve interviewed several individuals that we passed on due to ego. Some of the best technologists I’ve worked with are those that always kept an open mind to different ways of doing things and didn’t succumb to one-up-man ship. It’s okay to be right, but be willing to entertain other points-of-view.

– Use the best tool for the job. There’s no single piece of technology for every task. Don’t pigeon-hole your skill-set. Just because you can make something work, doesn’t mean you should.