Sunday, 7. June 2020 Week 23

Replace the root disk

Recently the disk holding the root (/) filesystem on one of my linux systems started to report increased SMART raw read error rates, seek error rates and ECC recovered hardware errors.

As these are early indications of a failing disk, it became time to replace the disk.

Normally replacing a disk comes down to plugging in the new one, coyping over the data, umount the old disk, mount the new one in place, unplug the old disk.
But when it is the disk with the root filesystem a couple extra steps are needed.

The steps below worked for my Debian system without problems (even used the opportunity to upgrade to an SSD :-)

(source is this thread on StackExchange)

The following makes some assumptions:

  • All commands ran as root when possible
  • You are on a physical console to the host (need to type in grub commands to boot up the new disk!)
  • You want an ext4 files system
  • You are loosely familiar on a basic level with all commands run
  • You are NOT booting from a RAID device

So here we go.

  1. Physically install new disk into computer and connect to available port leaving old disk in existing position.
  2. Boot computer into old OS.
  3. Prepare and mount new disk; first identify new disk
    fdisk -l
  4. Partition new disk
    fdisk /dev/(newdisk)
    Make partition primary partition with type "83" file system type.
  5. Create filesystem
    mkfs.ext4 /dev/(newpartition)
  6. Mount new filesystem
    mkdir /mnt/(newpartitionmountpoint)
    mount /dev/(newpartition) /mnt/(newpartitionmountpoint)
  7. Copy disk:
    /sbin/init 1 (drop to single user mode)
    rsync -avxHAX / /mnt/(newpartitionmountpoint)
  8. Update FSTAB on newdisk
    blkid (note UUID of new partition)
    vi /mnt/(newpartitionmountpoint)/etc/fstab
    Replace existing UUID of / in FSTAB to new disk UUID
  9. Configure grub and install to new disk boot loader:
    grub-mkconfig
    update-grub
    grub-install /dev/(newdisk)
  10. Copy grub.cfg from old disk to new
    cp -ax /boot/grub/grub.cfg /mnt/(newpartitionmountpoint)/boot/grub/grub.cfg
  11. Open grub.cfg on new disk and replace all UUIDs with new disk
    vi /mnt/(newpartitionmountpoint)/boot/grub/grub.cfg
    Replace all old UUIDs with the UUID of the new disk
  12. Shut down computer
    shutdown
  13. Physically move the new drive to the 1st drive location and remove old drive
  14. Start computer and grub should present:
    error: no such device: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    
    GRUB rescue>
  15. Manually boot new OS from grub; first identify the drive and partition of the boot files
    ls [to identify your drive and partition options]
    ls (hdx,p)/ [to identify which partition has the /boot folder]
  16. Then, you can load the boot menu manually from the drive and partition you found above. Typically this would be (hd0,msdos1).
    set prefix="(hdx,p)/boot/grub"
    set root="(hdx,p)"
    insmod normal
    normal
  17. Login to OS on new drive
  18. Configure grub again
    fdisk -l (note dev of newdisk)
    grub-mkconfig
    update-grub
    grub-install /dev/newdisk

And that should be it!

Sunday, 24. May 2020 Week 21

rkhunter CRLF confusion

On my Linux hosts I'm running rkhunter. On a newly configured host it lately reported the following warning:

Warning: The SSH and rkhunter configuration options should be the same:
        SSH configuration option 'PermitRootLogin': no
	Rkhunter configuration option 'ALLOW_SSH_ROOT_USER': no

On first sight the warning does not seem to make much sense, as both configuration options seem to be set to the same value (no).
But digging further reveals that they are stored slightly different:

# file /etc/rkhunter.conf
/etc/rkhunter.conf: ASCII text
# file /etc/ssh/sshd_config
/etc/ssh/sshd_config: ASCII text, with CRLF line terminators

Turns out that rkhunter is also checking the line terminators as part of the configuration values, and warns because they are different.

Knowing this, the fix is simple: run dos2unix on the CRLF file

Thursday, 21. May 2020 Week 21

ipaddr CLI tool

While doing some maintenance on my server, I got tired of searching through the output of ip addr show to find the IP addresses configured on the interfaces.
Thus I wrote a simple CLI tool to display the information I needed in a concise and human friendly form: ipaddr

$ ipaddr
lo          127.0.0.1/8
ens5        198.51.100.160/24
tun24008    10.123.199.78/32
tun71991639 10.200.123.5/32
tun26724    10.100.100.235/32
tun3883710  10.123.111.7/32

A nice side-effect of writing this in Go is that it works out-of-the-box also on non-Linux systems :-)

Saturday, 18. April 2020 Week 16

Poor man's reboot notification

Sometimes you need to be notified about reboots of a machine without having the luxury of a proper monitoring system.

The following crontab entry triggers an e-mail when the host has been rebooted in the last 5 minutes.

*/5 * * * * [ $(sed -e 's/\..*//' /proc/uptime) -lt 540 ] && echo "Host has been rebooted! Uptime: $(uptime)"
Sunday, 12. April 2020 Week 15
Saturday, 28. March 2020 Week 13
Saturday, 21. March 2020 Week 12

ip_compact and ip_diff

Somehow I always end up working with lists of IP networks and needing to minimize and compare them.

Some of my Perl scripts for this might still be hidden in a corporate source repository, and somewhere in the backups of my old Linux laptop should be even earlier attempts in Bash.

Both of them are not very useful to me where they are, thus I've written yet another version.
This time in Go using the ipaddr package.

Say hello to ip_compact and ip_diff :-)

Tuesday, 17. March 2020 Week 12
Friday, 20. December 2019 Week 51

This Page is Designed to Last

This Page is Designed to Last — a manifesto from Jeff Huang for preserving content on the web, where he advocates to keep content on the web available and pledges to keep his site available for the next 10 years.

Having my content in this weblog online since 2002, I can very much relate to this initiative and additionally would like to point to the efforts of archive.org (aka. The Internet Archive).
The wayback machine of archive.org allows to see old versions of websites, even when the website itself is no longer available.

For me personally this became critically useful when the database of my weblog vanished with no current backup and I then used the archived versions from archive.org to restore the missing content.

Thus I would like to encourage everyone to support the efforts of archive.org with a donation.

Saturday, 2. November 2019 Week 44