Saturday, 30. March 2024 Week 13

Puppet updated!

Yay! Successfully updated my Puppet Server setup from 5.3.7 to 8.4.0 🎉

It was quite a step (5.3.7 was released in January 2019) and as expected 3 major version bumps came with a couple changes.

I opted to re-create the PuppetDB and CA stores from scratch (to avoid having to migrate 5 years of data schema changes, and the CA cert is now also valid for a couple more years again).

To make the old manifests and modules work with the new versions, quite some effort was needed. This included rewriting some no longer maintained modules to use newer stdlib and concat libraries, updating a couple modules from the puppet forge (with the bonus that my puppet server runs airgapped and I had to use the download-tar-copy-extract way to install them) and fixing no longer valid syntax here and there in my custom manifests. Overall I spent about 5 hours on it (and have now a recurring reminder to update puppet more often to make this process less painful).

Helpful as usual were the resources from Vox Pupuli, in particular the Puppet Server and PuppetDB Docker images and the CRAFTY repo which contains a fully self-contained Docker Compose setup very similar to what I'm running.

Some commands that came in handy:

puppet config print ssldir --section agent
Returns the path of the TLS config folder on the client. Useful during a CA change (where you rm -rf the whole folder and then request a new TLS certificate).

puppet agent -t --noop
Dry-run the changes on the client (it does request a new TLS cert though!). Shows a nice diff of the changes it would do to files, helpful to validate that a manifest still behaves the same in the new version.

Monday, 25. March 2024 Week 13

Linux Crisis Tools

Brendan Gregg posted the following list of 'crisis tools' which you should install on your Linux servers by default (so they are available when an incident happens).

PackageProvidesNotes
procpsps(1), vmstat(8), uptime(1), top(1)basic stats
util-linuxdmesg(1), lsblk(1), lscpu(1)system log, device info
sysstatiostat(1), mpstat(1), pidstat(1), sar(1)device stats
iproute2ip(8), ss(8), nstat(8), tc(8)preferred net tools
numactlnumastat(8)NUMA stats
tcpdumptcpdump(8)Network sniffer
linux-tools-common
linux-tools-$(uname -r)
perf(1), turbostat(8)profiler and PMU stats
bpfcc-tools (bcc)opensnoop(8), execsnoop(8), runqlat(8), softirqs(8),
hardirqs(8), ext4slower(8), ext4dist(8), biotop(8),
biosnoop(8), biolatency(8), tcptop(8), tcplife(8),
trace(8), argdist(8), funccount(8), profile(8), etc.
canned eBPF tools[1]
bpftracebpftrace, basic versions of opensnoop(8),
execsnoop(8), runqlat(8), biosnoop(8), etc.
eBPF scripting[1]
trace-cmdtrace-cmd(1)Ftrace CLI
nicstatnicstat(1)net device stats
ethtoolethtool(8)net device info
tiptoptiptop(1)PMU/PMC top
cpuidcpuid(1)CPU details
msr-toolsrdmsr(8), wrmsr(8)CPU digging
Sunday, 24. March 2024 Week 12

Installing the AREDN firmware on a MikroTik hAP ac lite

At the recent HB9TF AGM fellow radio amateur HB9GVM gave an introductory presentation about AREDN.

Motivated by this, I ordered a MikroTik hAP ac lite and installed the AREDN firmware on it.
The following are my notes of the installation process.

  1. Download the firmware images for the MikroTik hAP ac lite from http://downloads.arednmesh.org/afs/www/ (both the *kernel.bin and *sysupgrade.bin are needed)
  2. Install Dnsmasq as a PXE server on Mac OS:
    brew install dnsmasq
  3. Setup the *kernel.bin for PXE:
    mkdir tftp-root
    cp $HOME/Downloads/aredn-3.23.12.0-ath79-mikrotik-mikrotik_routerboard-952ui-5ac2nd-initramfs-kernel.bin tftp-root/rb.elf
  4. Connect a Ethernet dongle and configure it with this static IP: 192.168.1.10/24
  5. Run dnsmasq as a PXE server listening on the network interface of the Ethernet dongle:
    ifconfig en6 # use to check that IP is configured
    sudo dnsmasq -i en6 -u $(whoami) --log-dhcp --bootp-dynamic --dhcp-range=192.168.1.100,192.168.1.200 -d -p0 -K --dhcp-boot=rb.elf --enable-tftp --tftp-root=$(pwd)/tftp-root/
  6. Power off the hAP ac lite and connect the Ethernet dongle to port 1 (PXE booting only seems to work on this port!)
  7. Press the reset button on the hAP ac lite, power it on and keep the button pressed for about 20 seconds (there is some output of dnsmask once the PXE booting is in progress)
  8. Wait until the hAP ac lite stops blinking and the LEDs are steady again (it also issues a new DHCP request via port1, but this time no PXE booting).
  9. Move the cable to port 2 (the AREDN default config for the MikroTik hAP ac lite uses port 1 for the Internet uplink, and port 2 for the 'inside' local network).
  10. Test that you can ping the inside IP of the AREDN node:
    ping 192.168.1.1
  11. Open the admin page on http://192.168.1.1/cgi-bin/admin (Username is root and password is hsmm).
  12. In the Firmware Update section, click on 'Upload Firmware' and select the previously downloaded sysupgrade.bin file.
  13. Wait until a reboot has happened twice (it takes a couple minutes!) to complete the installation.
  14. Open http://192.168.1.1 – congratulations, you now have a freshly installed (and not yet configured) AREDN node :-)
Sunday, 17. March 2024 Week 11
Thursday, 7. March 2024 Week 10
Sunday, 25. February 2024 Week 8

Tunnelbroker to the rescue

After nicely delivering native IPv6 connectivity for 3 years, my Internet provider Solnet made some changes in their backbone config on January 31st which broke their IPv6 setup.
Unfortunately escalating with their support did not bear any fruits so far (current state: they no longer respond on the support ticket…).

Thus change of plans, tunnelbroker.net (Hurricane Electric) to the rescue.
Took about 10 minutes to set everything up and now I'm enjoying IPv6 connectivity again (although with a reduced end-to-end MTU due to the 6in4 encapsulation).

Guess I'll have to look for a different Internet provider again.
Especially annoying is that I renewed the yearly plan with Solnet only a couple weeks ago, so will be stuck without native IPv6 connectivity for the next 11 months :-(

The High-Risk Refactoring

In the The High-Risk Refactoring article there is this concise Addressing Risk checklist to keep in mind when refactoring.
During past refactorings (also low-risk ones) I often used almost the same guidelines to help me and can only recommend you to do the same:

βœ… Define constraints. How far should I go.
βœ… Isolate improvements from features. Do not apply them simultaneously.
βœ… Write extensive tests. Higher level (integration) with fewer implementation details. They should run alongside changes.
βœ… Have a visual confirmation. Open the browser.

❌ Do not skip tests. Don't be lazy.
❌ Do not rely too much on code reviews and QA. Humans make mistakes.
❌ Do not mix expensive cleanups with other changes. But do that for small improvements.

(via)

Please Blog

Please Bloga plea for less Big Web and more Small Web and an encouraging article to write your own blog. It also touches on the part about writing on your own domain (so to keep your content yours and not be at risk of a third-party commercial 'social' service going away).

Don’t wait for the Pulitzer piece. Tell me about your ride to work, about your food, what flavor ice cream you like. Let me be part of happiness and sadness. Show me, that there is a human being out there that, agree or not, I can relate to. Because without it, we are just actors in a sea of actors, marketing, proselytizing, advocating, and threatening towards each other in an always vicious circle of striving for a relevance that only buys us more marketing, more proselytizing, more advocating, and more threats.

(discovered via Thomas Gigold)

Sunday, 18. February 2024 Week 7

ads.txt

Added and ads.txt file to the blog. The idea is to avoid that someone can sell fake advertisment space for this blog.

As I don't use any advertisment here the content of the file is pretty basic:

contact=https://blog.x-way.org/about.html

ldapauth

ldapauth is a Node.js script which I have been using for the last 12+ years mostly unchanged.

It started its life in a LXC container, eventually was moved to a Docker container and recently ended up in its own repository on GitHub.

The functionality it provides is not extraordinary, but helped to bridge a gap where no other product was available.
It talks LDAP one one side (although limited to handle user lookup requests) and on the other side connects to a MongoDB database where the information is stored.

It emerged out of the desire to have an easy way to manage individual user accounts for my home WiFi. I already had MongoDB running for some other personal project and simply added the list of users there (including the UI for managing them).
Thus the missing part was to get the WiFi accesspoint to lookup user accounts in MongoDB.

Of course WiFi accesspoints do not directly talk MongoDB, but rather some other protocol like RADIUS.
A freeradius server was quickly setup, but still couldn't talk to MongoDB at the time. Thus comes in ldapauth, which takes LDAP queries from freeradius and turns them into MongoDB lookups so that in the end the WiFi accesspoint receives the user accounts :-)

Not sure if this is particularly useful for anyone else, but at least here it did provide good services (and continues to do so).
Current score is that it has survived three different WiFi accesspoints and has been running on 5 different servers over the time.