Saturday, 30. March 2024 Week 13

Puppet updated!

Yay! Successfully updated my Puppet Server setup from 5.3.7 to 8.4.0 🎉

It was quite a step (5.3.7 was released in January 2019) and as expected 3 major version bumps came with a couple changes.

I opted to re-create the PuppetDB and CA stores from scratch (to avoid having to migrate 5 years of data schema changes, and the CA cert is now also valid for a couple more years again).

To make the old manifests and modules work with the new versions, quite some effort was needed. This included rewriting some no longer maintained modules to use newer stdlib and concat libraries, updating a couple modules from the puppet forge (with the bonus that my puppet server runs airgapped and I had to use the download-tar-copy-extract way to install them) and fixing no longer valid syntax here and there in my custom manifests. Overall I spent about 5 hours on it (and have now a recurring reminder to update puppet more often to make this process less painful).

Helpful as usual were the resources from Vox Pupuli, in particular the Puppet Server and PuppetDB Docker images and the CRAFTY repo which contains a fully self-contained Docker Compose setup very similar to what I'm running.

Some commands that came in handy:

puppet config print ssldir --section agent
Returns the path of the TLS config folder on the client. Useful during a CA change (where you rm -rf the whole folder and then request a new TLS certificate).

puppet agent -t --noop
Dry-run the changes on the client (it does request a new TLS cert though!). Shows a nice diff of the changes it would do to files, helpful to validate that a manifest still behaves the same in the new version.

Monday, 25. March 2024 Week 13

Linux Crisis Tools

Brendan Gregg posted the following list of 'crisis tools' which you should install on your Linux servers by default (so they are available when an incident happens).

PackageProvidesNotes
procpsps(1), vmstat(8), uptime(1), top(1)basic stats
util-linuxdmesg(1), lsblk(1), lscpu(1)system log, device info
sysstatiostat(1), mpstat(1), pidstat(1), sar(1)device stats
iproute2ip(8), ss(8), nstat(8), tc(8)preferred net tools
numactlnumastat(8)NUMA stats
tcpdumptcpdump(8)Network sniffer
linux-tools-common
linux-tools-$(uname -r)
perf(1), turbostat(8)profiler and PMU stats
bpfcc-tools (bcc)opensnoop(8), execsnoop(8), runqlat(8), softirqs(8),
hardirqs(8), ext4slower(8), ext4dist(8), biotop(8),
biosnoop(8), biolatency(8), tcptop(8), tcplife(8),
trace(8), argdist(8), funccount(8), profile(8), etc.
canned eBPF tools[1]
bpftracebpftrace, basic versions of opensnoop(8),
execsnoop(8), runqlat(8), biosnoop(8), etc.
eBPF scripting[1]
trace-cmdtrace-cmd(1)Ftrace CLI
nicstatnicstat(1)net device stats
ethtoolethtool(8)net device info
tiptoptiptop(1)PMU/PMC top
cpuidcpuid(1)CPU details
msr-toolsrdmsr(8), wrmsr(8)CPU digging
Sunday, 24. March 2024 Week 12

Installing the AREDN firmware on a MikroTik hAP ac lite

At the recent HB9TF AGM fellow radio amateur HB9GVM gave an introductory presentation about AREDN.

Motivated by this, I ordered a MikroTik hAP ac lite and installed the AREDN firmware on it.
The following are my notes of the installation process.

  1. Download the firmware images for the MikroTik hAP ac lite from http://downloads.arednmesh.org/afs/www/ (both the *kernel.bin and *sysupgrade.bin are needed)
  2. Install Dnsmasq as a PXE server on Mac OS:
    brew install dnsmasq
  3. Setup the *kernel.bin for PXE:
    mkdir tftp-root
    cp $HOME/Downloads/aredn-3.23.12.0-ath79-mikrotik-mikrotik_routerboard-952ui-5ac2nd-initramfs-kernel.bin tftp-root/rb.elf
  4. Connect a Ethernet dongle and configure it with this static IP: 192.168.1.10/24
  5. Run dnsmasq as a PXE server listening on the network interface of the Ethernet dongle:
    ifconfig en6 # use to check that IP is configured
    sudo dnsmasq -i en6 -u $(whoami) --log-dhcp --bootp-dynamic --dhcp-range=192.168.1.100,192.168.1.200 -d -p0 -K --dhcp-boot=rb.elf --enable-tftp --tftp-root=$(pwd)/tftp-root/
  6. Power off the hAP ac lite and connect the Ethernet dongle to port 1 (PXE booting only seems to work on this port!)
  7. Press the reset button on the hAP ac lite, power it on and keep the button pressed for about 20 seconds (there is some output of dnsmask once the PXE booting is in progress)
  8. Wait until the hAP ac lite stops blinking and the LEDs are steady again (it also issues a new DHCP request via port1, but this time no PXE booting).
  9. Move the cable to port 2 (the AREDN default config for the MikroTik hAP ac lite uses port 1 for the Internet uplink, and port 2 for the 'inside' local network).
  10. Test that you can ping the inside IP of the AREDN node:
    ping 192.168.1.1
  11. Open the admin page on http://192.168.1.1/cgi-bin/admin (Username is root and password is hsmm).
  12. In the Firmware Update section, click on 'Upload Firmware' and select the previously downloaded sysupgrade.bin file.
  13. Wait until a reboot has happened twice (it takes a couple minutes!) to complete the installation.
  14. Open http://192.168.1.1 – congratulations, you now have a freshly installed (and not yet configured) AREDN node :-)
Sunday, 17. March 2024 Week 11
Thursday, 7. March 2024 Week 10