Matt White | SWE

Azure's Browser Bias

Wed, 27 Dec 2023 00:00:00 +0000

I’ve been tinkering with the Azure cloud recently. On any cloud provider, I try to configure my hardware keys as a method of multi-factor authentication.

Microsoft’s authentication apparatus is much more abrasive than either Google’s or AWS’s. Anyone with a nonstandard setup is used to companies aggressively locking their accounts - I’m never surprised when I have to jump through additional hoops.

It is always more grating when a company’s security theater comes with unfounded TOS violation accusations:

I plead guilty to the crime of logging in from an unfamiliar browser.

But that’s all uninteresting. Twitter does it. Meta does it. Microsoft does it. What is Microsoft doing that’s unusual?

Microsoft will reject valid credentials depending on which combination of browser/operating system they think you’re using.

I worked through Microsoft’s hardware key enrollment process. They won’t let you use hardware keys as multi-factor - they only let you use it as a password alternative. You can select a security key as your login method.

All my login attempts from Firefox consistently failed.

But if I switched my user agent to say I was using Chrome, access granted.

What Browsers Do Our Benevolent Lord Microsoft Deem Acceptable?

This got me curious, so I went down the list of user agent presets I had - seeing which would succeed and which Microsoft would arbitrarily reject.

I had expected to see a pattern like “always reject linux” or “only accept chromium”, but the actual behavior was a bit more varied.

Observations

Every attempt was conducted on a Linux 6.5.7-arch1-1 x86_64 system from Firefox 118.0.2 (64-bit).

Logging in from OSX or iOS always succeeded - not a shock, given how many companies issue Apple products as standard kit.
Logging in from Windows always succeeded unless the browser was internet explorer.
Firefox’s success depended entirely on the operating system.
Android logins always failed - but Android logins from Chrome produced an error that I didn’t see from any other user agent:

Conclusion

There are a lot of user agent combinations that I haven’t tested (especially around Microsoft Edge), but given the miniscule odds that Microsoft will ever try to mitigate this behavior, being more thorough would be a waste.

I doubt this behavior is intentional, but since I won’t be changing my browser or operating system selection to suit Microsoft, I’ll steer clear of Azure when I have the choice.

Tested User Agents

Failed logins:

Linux / Firefox 120

Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/120.0

Android Phone / Firefox 120

Mozilla/5.0 (Android 14; Mobile; rv:109.0) Gecko/120.0 Firefox/120.0

Android Tablet / Firefox 120

Mozilla/5.0 (Android 14; Tablet; rv:109.0) Gecko/120.0 Firefox/120.0

Windows / IE 11

Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko

Android Phone / Chrome 120

Mozilla/5.0 (Linux; Android 14; Z832 Build/MMB29M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.119 Mobile Safari/537.36

Android Tablet / Chrome 120

Mozilla/5.0 (Linux; Android 14; SAMSUNG-SM-T377A Build/NMF26X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.119 Mobile Safari/537.36

Successful logins:

Windows / Firefox 120

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/120.0

Mac OS X / Safari 17

Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15`

Mac OS X / Firefox 120

Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6) Gecko/20100101 Firefox/120.0

Linux / Chrome 120

Mozilla/5.0 (X11; Linux x86_64; rv:109.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.119 Safari/537.36

Windows / Chrome 120

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.119 Safari/537.36

Windows / Firefox 115 ESR

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0

iPhone / Safari 17

Mozilla/5.0 (iPhone; CPU OS 17_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Mobile/14E304 Safari/605.1.15

iPad / Safari 17

Mozilla/5.0 (iPad; CPU OS 17_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Mobile/15E148 Safari/605.1.15

LUKS-Encrypted Arch with systemd-boot and SecureBoot

Tue, 05 Dec 2023 00:00:00 +0000

When I was getting started with Arch linux, I quickly learned that there were a lot of tricky parts that no amount of studying the installation guide could get me through. I ran into a lot of problems that I could only sort through by reading other people’s system writups.

I just ran through a complete system rebuild today and figured it was time to do my own writup.

Live USB

Create and boot from an Arch live USB.

SSH (Optional)

If you have another system you can SSH in from, it’ll allow you to copy/paste commands.

Create a password for the root user on the live USB

passwd

Enable the ssh service

systemctl start sshd.service

You can find the ip address of the system:

ip addr show

The output can be a bit verbose. You can try to narrow it down by piping to grep.

ip addr show | grep "state UP"

Use that ip address to ssh in from a different system:

ssh root@<ip-address-you-found>

Check the Boot Mode

cat /sys/firmware/efi/fw_platform_size

For my system, the output is 64, which indicates that the system is in UEFI mode in 64-bit. A 32 would indicate UEFI mode in 32-bit.

If the output is anything else, you’ve got to go messing with your motherboard.

Create the Partitions

List Disks and Partitions

To identify the disk(s) you want to use during the installation:

lsblk

Example output:

NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sdb             8:0    0 465.8G  0 disk  
sda             8:16   0   9.1T  0 disk

For our purposes, let’s use sdb as the target disk.

Create the Partitions

Using gdisk for partitioning.

gdisk /dev/sdb

Create 3 partitions:

Partition 1 - Code EA00 - XBOOTLDR partition - 512MB
Partition 2 - Code EF00 - EFI System Partition - 256MB
Partition 3 - Code 8309 - Linux LUKS - All the remaining space

Since gdisk is interactive, I have to show this awkward collection of key presses. Each use of [Enter] below indicates that I’m taking the default value.

# Create a new GUID Partition Table (GPT)
o
Y

# Create the XBOOTLDR partition
n
[Enter]
0
+512M
ea00

# Create the EFI partition
n
[Enter]
[Enter]
+256M
ef00

# Create the LUKS partition
n
[Enter]
[Enter]
[Enter]
8309

# Persist the changes
w
Y

Setup LUKS

Use luks to encrypt and open the 3rd partition:

cryptsetup luksFormat \
    --use-random \
    -S 1 \
    -s 512 \
    -h sha512 \
    -i 5000 \
    /dev/sdb3
cryptsetup open /dev/sdb3 cryptlvm

Setup Volumes

First, create the physical volume. I use the name cryptlvm, but the name is arbitrary.

pvcreate /dev/mapper/cryptlvm

Create a volume group:

vgcreate vg /dev/mapper/cryptlvm

Now create logical volumes for swap, root, and home.

Use the appropriate amount of space for root, which is where installed packages will live by default. 128G is enough for me right now, but for you it may be too small.

lvcreate -L 8G vg -n swap
lvcreate -L 128G vg -n root
lvcreate -l 100%FREE vg -n home

NOTE the case-sensitivity of arguments. The ‘-l’ is not a typo for ‘-L’.

Format the Partitions

Using FAT32 for the efi and boot partitions - they need to be readable by the firmware.

mkfs.fat -F32 /dev/sdb1
mkfs.fat -F32 /dev/sdb2

Use the old reliable ext4 for root and home.

mkfs.ext4 /dev/vg/root
mkfs.ext4 /dev/vg/home
mkswap /dev/vg/swap

Mounting

Mount the root partition.

mount /dev/vg/root /mnt

Create the nested mount points.

mkdir /mnt/home
mkdir /mnt/boot
mkdir /mnt/efi

And mounting

mount /dev/vg/home /mnt/home
mount /dev/sdb1 /mnt/boot
mount /dev/sdb2 /mnt/efi
swapon /dev/vg/swap

At this point, lsblk should produce output that looks similar to this:

NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sdb             8:0    0 465.8G  0 disk  
├─sdb1          8:1    0   512M  0 part  /boot
├─sdb2          8:2    0   256M  0 part  /efi
└─sdb3          8:3    0 465.2G  0 part  
  └─cryptlvm  254:0    0 465.2G  0 crypt 
    ├─vg-swap 254:1    0     8G  0 lvm   [SWAP]
    ├─vg-root 254:2    0   128G  0 lvm   /
    └─vg-home 254:3    0 329.2G  0 lvm   /home

Setup Initial Installation

Packages

Alter the mirror list, if you have any preferences.

vim /etc/pacman.d/mirrorlist

Install any essential programs.

pacman -Sy archlinux-keyring
pacstrap /mnt \
    base \
    linux \
    linux-firmware \
    base-devel \
    git \
    sudo \
    archlinux-keyring \
    lvm2 \
    mkinitcpio \
    vim \
    dhcpcd \
    openssh

Configure fstab

This will copy your current setup for all the mounted volumes/partitions/etc and save it to fstab so that it can be setup automatically on boot.

genfstab -U /mnt >> /mnt/etc/fstab

Chroot Into the Installation

arch-chroot /mnt

Update Root Password

passwd

Create User and Basic Configuration

Create a user and set the password.

useradd -m matt
passwd matt

Add the user to the wheel group, which will determine who has access to sudo.

gpasswd -a matt wheel

Run visudo, which will open a file where you can add wheel users to sudoers. Uncomment this relevant section:

## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL) ALL

Setting the Clock

timedatectl set-ntp true

Adjust to your own time zone as-needed.

ln -sf /usr/share/zoneinfo/America/Denver /etc/localtime
hwclock --systohc

Locale

Uncomment en_US.UTF-8 UTF-8 in /etc/locale.gen (or whatever line applies to you) and generate locale:

locale-gen

And similar for locale configuration:

echo "LANG=en_US.UTF-8" >> /etc/locale.conf

Networking

echo "myhostname" >> /etc/hostname

Update /etc/hosts

127.0.0.1	localhost
::1		ip6-localhost
127.0.1.1	myhostname.localdomain	myhostname

Enable dhcpcd and ssh so everything works on the system’s first reboot

systemctl enable dhcpcd
systemctl enable sshd

.local Hostname Resolution

I have a ton of machines on my local network that I need to access on a regular basis. Local hostname resolution helps avoid hardcoding IP addresses everywhere.

Setup local hostname resolution with Avahi.

pacman -Sy avahi nss-mdns

Update hosts in /etc/nsswitch.conf to add mdns. The main thing is to make sure mdns_minimal [NOTFOUND=return] gets added somewhere between files and dns. Resolution will check sources in order. We don’t want it querying dns for local requests.

hosts: mymachines resolve [!UNAVAIL=return] files myhostname mdns_minimal [NOTFOUND=return] dns

note that myhostname and mymachines are actual values, not placeholders

Enable avahi

systemctl enable avahi-daemon

Initramfs

Create a Keyfile

A keyfile lets us bypass unnecessary passwords on boot. Create one and add it as a LUKS key.

mkdir /root/secrets
chmod 700 /root/secrets
head -c 64 /dev/urandom > /root/secrets/crypto_keyfile.bin
chmod 600 /root/secrets/crypto_keyfile.bin
cryptsetup -v luksAddKey -i 1 /dev/sdb3 /root/secrets/crypto_keyfile.bin

Find the partition’s UUID value (and keep it on hand, we need it a few times). It will be one of these:

blkid | grep crypto_LUKS

Add this info to /etc/crypttab:

cryptlvm       UUID=<YOUR UUID> /root/secrets/crypto_keyfile.bin luks

Update mkinitcpio

We need to modify /etc/mkinitcpio.conf and add some hooks. lvm2 is required since this enables the initramfs to properly manage LVM volumes during the boot process. encrypt is required since we’re using LUKS. filesystems is required for ext4, but should already be there.

There’s a section for HOOKS, which should end up looking similar to this:

HOOKS=(base udev autodetect keyboard modconf block encrypt lvm2 filesystems fsck)

Create the initramfs Image

mkinitcpio -p linux

Add Microcode Support

You can check your processor with

lscpu

Based on the results, install either amd-ucode or intel-ucode

pacman -S <your-ucode-type>

Note this value, which is used in the bootctl configuration, below.

Configure the Bootloader

So far I prefer systemd-boot to GRUB, which I was never able to get to cooperate with SecureBoot or LUKS2.

Create an entry at /boot/loader/entries/arch.conf:

title Arch Linux
linux /vmlinuz-linux
initrd /<YOUR-UCODE-TYPE>.img
initrd /initramfs-linux.img
options cryptdevice=UUID=<UUID-OF-ROOT-PARTITION>:cryptlvm root=/dev/vg/root rw cryptkey=rootfs:/root/secrets/crypto_keyfile.bin

Configure /efi/loader/loader.conf:

default  arch.conf
timeout  5
console-mode max
editor   no

You may want a lower timeout value; it dictates how many seconds the loader selection screen shows on boot.

Sanity Check

Run bootctl with no params. It should tell you if your configuration is obviously broken.

Restart the System

Exit chroot and reboot.

exit
umount -a /mnt
reboot

The new installation should be able to boot via systemd-boot without the live usb.

Secure Boot

Boot up normally without the live USB.

Run as root: sudo su

Check the Current Status

Install sbctl:

pacman -S sbctl

Check if everything is ready by running:

sbctl status

These are the states we’re looking for. Setup mode, no secure boot, sbctl installed.

Installed:	    ✓ sbctl is installed
Setup Mode:	    ✗ Enabled
Secure Boot:	✗ Disabled

This can still go fine if sbctl doesn’t realize that its installed.

Disable Secure Boot and Enter Setup Mode

If the status isn’t what we want, we need to update the firmware settings.

systemctl reboot --firmware

When your system boots into the firmware, disable secure boot and enable setup mode/remove existing keys.

Unfortunately some motherboards make this quite hard to find.

You’ll want to set your UEFI firmware supervisor (administrator) password in the firmware settings; that way an attacker can’t simply boot into UEFI setup utility and turn off SecureBoot.

Don’t use the same UEFI firmware supervisor password as your encryption password for LUKS. On some old systems, the supervisor password can be recovered as plaintext from the EEPROM chip.

Restart.

Enroll the Keys

When you’re booted back into the new system:

sbctl create-keys

Enroll the keys

sbctl enroll-keys -m

Assuming that went well, we need to sign certain EFI files. To identify the files we need to sign, we use sbctl verify.

The lazy way I use:

sbctl verify | grep '✗' | cut -d ' ' -f 2 | xargs -n 1 sbctl sign -s

Need to sign these as well.

sbctl sign -s /boot/vmlinuz-linux

sbctl sign -s \
    -o /usr/lib/systemd/boot/efi/systemd-bootx64.efi.signed \
    /usr/lib/systemd/boot/efi/systemd-bootx64.efi

Run sbctl verify again to ensure everything is kosher.

$ sbctl verify
Verifying file database and EFI images in /efi...
✓ /boot/vmlinuz-linux is signed
✓ /efi/EFI/BOOT/BOOTX64.EFI is signed
✓ /efi/EFI/systemd/systemd-bootx64.efi is signed
✓ /usr/lib/systemd/boot/efi/systemd-bootx64.efi.signed is signed

Turn on SecureBoot

Back to firmware settings.

systemctl reboot --firmware

Turn on Secure Boot in the settings and save. If all is well, your system should boot successfully.

Check if Secure Boot Worked

Now to check if everything worked out. sbctl status can help:

$ sbctl status
Installed:	✓ sbctl is installed
Owner GUID:	<some-guid>
Setup Mode:	✓ Disabled
Secure Boot:	✓ Enabled
Vendor Keys:	microsoft

And we can check the efi variable:

od -An -t u1 /sys/firmware/efi/efivars/SecureBoot-<some-long-string-of-numbers>

If it worked, the last number should be 1. Example output: 6 0 0 0 1

If this process didn’t work and you didn’t miss anything, the wiki is the only hope.

Backup and Duplicate Raspberry Pi Images

Thu, 23 Mar 2023 00:00:00 +0000

Setting up a Raspberry Pi from scratch can quickly become tedious - especially if you need to configure a lot of devices in sequence.

If you create backup snapshots of your Raspberry Pi images, you can quickly restore or duplicate the setup across other devices.

For this example, I’ll use a Raspberry Pi that I configured with IP camera software.

The process is very straightforward. You’ll need enough free space to backup the entire card (but we’ll fix the backup image size at the end).

Setup the Micro SD Card

Remove the micro SD card from the device you want to snapshot. Insert it into your system.

Identify the Card

Identify the SD card using lsblk or fdisk -l and find the storage that matches the size of your SD card.

~ $ lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                 8:0    0 931.5G  0 disk  
├─sda1              8:1    0   512M  0 part  
└─sda2              8:2    0   931G  0 part  
nvme1n1           259:0    0  27.3G  0 disk  
...
sdg                 8:96   1  58.2G  0 disk  
├─sdg1              8:97   1   256M  0 part  
└─sdg2              8:98   1    58G  0 part  
...

My SD card is 64GB, which matches the sdg device listed above. Remember that the advertized size uses 1000 to represent a GB and the system uses 1024 to represent a GiB, which accounts for most of the 64GB/58.2GiB size discrepancy.

Unmount the Card, if Necessary

If your system auto-mounted the card, it’s best to unmount it. This prevents data from changing on the card in the middle of backing it up, which could result in corrupted data.

Unmount the device you identified:

sudo umount /dev/sdg*

The asterisk wildcard should catch the individual partitions of the card.

Back up the Image

Run the command below, replacing /dev/sdg with the device you identified and /home/$USER/Documents/rpi-backup.img with your preferred location.

sudo dd bs=4M \
    if=/dev/sdg \
    of=/home/$USER/Documents/rpi-backup.img \
    conv=fsync \
    status=progress

Backing up the image typically takes a long time. The entire disk gets backed up, despite the fact that most of it is likely empty.

After the process completes, I always chown the result

sudo chown $USER: /home/$USER/Documents/rpi-backup.img

Shrink the Image

Download PiShrink

Now to use a handy repository called PiShrink, which abstracts away the complexity of resizing a Pi image behind a shell script.

You can either pull the repository with git:

git clone git@github.com:Drewsif/PiShrink.git
cd PiShrink

Or if you’re more daring, wget the thing and place it in your path somewhere.

wget https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh
chmod +x pishrink.sh
sudo mv pishrink.sh /usr/local/bin

Shrink Ray the Day Away

sudo pishrink.sh /home/$USER/Documents/rpi-backup.img

The space savings depend on how much empty space the image has. For me, PiShrink typically takes my Pi Images from 60 GiB down to 3 or 4 GiB, which adds up quickly if you’re saving a lot of snapshots over time.

Applying the Image

These images can be applied the same way as any other.

Option 1: Terminal

In the terminal, you can apply our backup process in reverse:

sudo dd bs=4M \
    if=/home/$USER/Documents/rpi-backup.img \
    of=/dev/sdg \
    conv=fsync \
    status=progress

Option 2: Etcher

As I’ve mentioned elsewhere, I like to use Etcher to apply images:

Your browser doesn't support this video.

This post has a more detailed explanation on using Etcher, if you need it.

Raspberry Pi Webcams

Tue, 22 Nov 2022 00:00:00 +0000

Dust not included

It’s pretty easy to set up a Raspberry Pi as an independent IP camera. It’s even easier to create a plug-and-play USB camera. You might be able to make one too, if you can manage to find a Raspberry Pi anywhere.

Supplies

A Raspberry Pi

I’ve only ever done this on Pi Zero v1.3’s, Pi Zero W’s, and Pi 4’s, so I don’t know about other models.

Pi Zero W (recommended)
- Nominal cost: $15
- wifi-capable
- easier to find than the Pi Zero W2
Pi Zero v1.3
- Nominal cost: $10
- no onboard wifi
- fine if you’re ok with always using USB for connectivity
Pi 4
- Nominal cost: $45
- wifi-capable
- probably overkill for a webcam
Pi Zero W 2 (untested)
- Nominal cost: $15
- wifi-capable

Raspberry Pi boards are perpetually out of stock, so you may need to search across multiple suppliers using something like rpilocator. Raspberry Pi lists suppliers per-region on their website, but I’ve had little luck with it.

A Camera Module

You’ll need an actual camera. The 12MP Raspberry Pi High Quality Camera ($50) is my go-to, but you have to buy a lens to go with it.

Lens-wise, I usually go for the 6mm 3MP Wide Angle Lens for Raspberry Pi HQ Camera - 3MP ($25) if I don’t need a super wide-view, but more and more I’m using these ultra-wide Arducam lenses ($35) so that a single camera can cover more area.

Or go with budget-conscious camera modules

You can go with an 8 megapixel camera ($30), the NoIR version of same ($30) for low-light footage, or Amazon has even cheaper options. Whatever works for your use-case.

Heads up - these cheaper camera options will require some creativity to mount, since the mount I use is designed with the HQ camera module in mind.

A Compatible Camera Cable

Pi Zeros use a different camera cable that’s tapered on one end.

Make sure you get the right one for your Pi. Here is the standard Raspberry Pi’s cable.

Mounting

The HQ camera has a 1/4" threaded hole in its base - which is compatible with most camera mounts.

To mount the camera and Raspberry Pi together, I like the Zero Pro Plate for HQ Camera. There’s also a version for the Pi 4.

(Physical) Port Access

Depending on the stand you’re using, you might need a spacer so there’s room to access the USB slot on the Pi Zero. I’ve been using this $10 camera adapter kit for years and it did the job fine for me.

Putting it together

Assembly varies based on the components you selected, but is generally straightforward.

Once assembled, you can easily throw everything on a little tripod stand and call it a day.

Software setup completely depends on what type of camera you want to use. Let’s start with the simpler of the 2 options.

Setup Option 1 - Plug-and-Play USB Camera

There are many different methods floating around, but the only software that worked for me is showmewebcam.

Download the image

Download the appropriate image from https://github.com/showmewebcam/showmewebcam/releases. The release files are named based on the Raspberry Pi they’re intented to work with. At time of writing, that means:

sdcard-raspberrypi0-8e9f9ac.img.zip for the Pi Zero v1.3
sdcard-raspberrypi0w-8e9f9ac.img.zip for the Pi Zero W
sdcard-raspberrypi4-8e9f9ac.img.zip for the Pi 4

Unzip the image for the next step.

Image Burning Software

I prefer balenaEtcher to actually burn images.

Download balenaEtcher from the official site. Windows and OSX have their own installers. On linux, you can just unzip and run the AppImage.

unzip balena-etcher-electron-1.7.9-linux-x64.zip
./balenaEtcher-1.7.9-x64.AppImage

insert a micro SD card
launch Etcher
select the .img file you downloaded
select the micro SD card you inserted
burn the image.

Etcher is usually able to guess pretty well which is the SD card you probably want, but be careful not to pick the wrong volume.

Your browser doesn't support this video.

With everything burned onto the micro SD card, you just have to insert the card into the Pi camera.

Plug and Play

Now the camera should be completely ready. Just plug it into your system using a USB cable. The Pi’s build-in LED should blink 3 times to indicate it’s ready. It should come up as a typical web camera.

The project’s README has you covered if you need to debug or to change the camera settings.

Setup Option 2 - IP Camera

The best IP interface I’ve found (and admittedly, the only one I’ve used much) is RPi-Cam-Web-Interface. This is how most of my cameras are configured.

Your browser doesn't support this video.

With this setup, the camera runs as its own host on your network. You can connect to it from your browser to take videos and images. You can even do time lapses.

I find the ability to capture video based on motion detection invaluable.

Install Raspberry Pi OS on a Micro SD Card

Follow the steps on how to setup headless Raspberry PI here.

After Raspberry Pi OS is installed and you have SSH access to the system, continue below.

Install RPi-Cam-Web-Interface

Boot up the Raspberry Pi and connect over SSH. For me:

ssh matt@raspberrypi.local

If you haven’t already, run yer updates:

sudo apt-get update
sudo apt-get dist-upgrade

Install git, which doesn’t come standard on Raspberry Pi OS Lite:

sudo apt-get install git

It’s critical to enable the legacy camera interface for the software to work.

sudo raspi-config

Navigate to Interface Options > Legacy Camera and select “Yes” to enable the interface.

Your browser doesn't support this video.

Now we can download and install the repo:

git clone https://github.com/silvanmelchior/RPi_Cam_Web_Interface.git
cd RPi_Cam_Web_Interface
./install.sh

I typically just use the defaults, but there are different options in terms of webservers and authentication, if you want it.

The installation takes 10-15 minutes on a Pi Zero W.

Your browser doesn't support this video.

The scripts in the RPi_Cam_Web_Interface folder do pretty much what you’d expect.

cam.sh
debug.sh
install.sh
remove.sh
start.sh
stop.sh
update.sh

Use ./start.sh if it’s not started already.

Now you can navigate to http://raspberrypi.local/html/ to see the interface. If you have a different host set for your raspberrypi, use that host or the IP address, instead.

Your browser doesn't support this video.

the purple artifacts are from my hand nudging the camera’s ribbon cable

Now go crazy. The wiki has a lot more in-depth information on configuration and different setups, if you want more: https://elinux.org/RPi-Cam-Web-Interface

Raspberry Pi Headless on Bullseye

Tue, 23 Aug 2022 00:00:00 +0000

I’ve been on a crusade to upgrade my half dozen Raspberry Pi’s to the most recent Raspberry Pi OS version. It’s more of a pain to setup headless than it used to be. Here’s my notes on the process.

All this assumes you’ll be using either wifi or ethernet with your Raspberry Pi.

What to Download

Raspberry Pi OS Lite

Download one of the Raspberry Pi OS Lite images on the official downloads page. Only download the 64-bit version if your Raspberry Pi is compatible (e.g. Pi Zero W’s and Pi Zero v1.3’s don’t work with 64 bit).

You may need to decompress the OS image with xz. If you don’t already have xz, you can install it via:

# Arch
sudo pacman -Sy xz

# Debian
sudo apt install xz-utils

Decompressing:

xz -d 2022-04-04-raspios-bullseye-armhf-lite.img.xz

Image Burning Software

I prefer balenaEtcher to actually burn the image.

Download balenaEtcher from the official site. Windows and OSX have their own installers. On linux, you can just unzip and run the AppImage.

unzip balena-etcher-electron-1.7.9-linux-x64.zip
./balenaEtcher-1.7.9-x64.AppImage

With Etcher and the Pi OS image at hand, insert a micro SD card and burn the image.

Your browser doesn't support this video.

It’s not generally this fast.

Remember which device it is so you can mount it later; for me, it’s /dev/sdf. Etcher is usually able to guess pretty well which devices you probably want, but be careful not to pick the wrong one.

Mount the Micro SD Card

If your system automatically mounts devices, you can probably just unplug and replug the card after flashing. You’ll need to update the paths in the rest of this guide if you go with that.

You can use lsblk to look at the currently mounted devices, where they’re mounted, and other info. This can help you narrow down which device or path you’re looking for.

Your browser doesn't support this video.

I’m using /dev/sdf, as mentioned above. To manually mount:

# Make the mount point
sudo mkdir -p /media/microsd/boot

# Mount the partition
sudo mount /dev/sdf1 /media/microsd/boot/

We’re only mounting boot because it’s the only partition we need.

Setup Wifi

This is only necessary if you aren’t using ethernet for your Pi.

Create a wpa_supplicant.conf file in the boot drive. Use sudo and your favorite editor to create /media/microsd/boot/wpa_supplicant.conf:

country=US
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
       ssid="YOUR WIFI'S NAME HERE"
       psk="YOUR WIFI'S PASSWORD HERE"
       key_mgmt=WPA-PSK
}

Update your network info and country as-needed.

Add a User

Since Bullseye, the default pi user is no longer available. For Raspberry Pi OS desktop, it’ll force you to make a user on first boot; with headless that’s not an option.

To setup a user that you can login with, we need to add one to boot/userconf. Note the spots for your desired password and desired username.

sudo sh -c 'echo "YOUR-PASSWORD" | \
openssl passwd -6 -stdin | \
xargs -I {} echo "YOUR-USERNAME:{}" > /media/microsd/boot/userconf'

Enable SSH Access

sudo touch /media/microsd/boot/ssh

SSH Over USB Configuration (Optional)

Source: Getting WIFI and USB serial port to work on Pi0W

Open the command line config file /media/microsd/boot/cmdline.txt and add: modules-load=dwc2,g_cdc after rootwait.

Append to the config file:

sudo sh -c 'echo "dtoverlay=dwc2,dr_mode=peripheral" >> /media/microsd/boot/config.txt'

The USB connection won’t work until the gadget service is running. After your first boot, you’ll need to enable login via the gadget serial port (covered below).

Unmount

sudo umount /media/microsd/boot

First Boot

Now you can remove the micro SD card from your system and place it in the Raspberry Pi. If you wanted to use ethernet, plug that in. Turn on the system and wait for it to boot.

SSH into the pi using the credentials you setup in userconf. Most of the time you can use raspberrypi.local to access the Pi.

# Clear out any old host information
ssh-keygen -R raspberrypi.local

# Connect to the default hostname
ssh YOUR-USERNAME@raspberrypi.local

With a successful connection, we can run updates on the Pi, itself.

sudo apt update
sudo apt full-upgrade

And you can run through raspi-config.

sudo raspi-config

The configurations I always update are:

System > Hostname
Localisation > Locale
Localisation > Timezone

Your browser doesn't support this video.

Loading times have been cut out - locale takes awhile to generate.

After reboot, you should be in a pretty good state. Most people can stop here.

If you updated the hostname, you’ll need to use your-hostname.local instead of raspberrypi.local for the remainder of ssh commands.

Enable SSH Over USB Service (Optional)

If you configured SSH over USB, the connection won’t work until the gadget service is running. After your first boot, enable login via the gadget serial port on the Raspberry Pi:

sudo systemctl enable --now getty@ttyGS0.service

When you connect the Raspberry Pi over USB, usually there’s a specific USB port you have to use.

SSH Hardening and Uploading Keys (Optional)

Unless you really care about security, you can skip this. If you lose your key, you lose your access to the Pi’s SSH interface.

It’s best to avoid password authentication over SSH and opt for public key authentication, instead.

If you don’t have an existing key, you can generate one (on your local system) as follows:

# If you choose to enter a password, you'll be prompted for it on each connection
ssh-keygen -t rsa -b 4096 -C "your-email" -f ~/.ssh/your-key-name.pem

Uploading an SSH key to the Raspberry Pi:

ssh-copy-id -i ~/.ssh/your-key-name.pem YOUR-USERNAME@raspberrypi.local

Don’t be afraid to use ssh-copy-id -f to force it if ssh-copy-id thinks the key is already there.

Test that your key works:

ssh -i ~/.ssh/your-key-name.pem YOUR-USERNAME@raspberrypi.local

You should NOT be prompted for the Raspberry Pi’s password unless there’s a problem with the key setup. If you set an SSH key password, you will be prompted for that.

Only proceed if the key is working. It’s better to use password authentication than to lock yourself out.

If the key’s working, you can make SSH-ing easier by adding your Pi’s info to ~/.ssh/config:

Host your-hostname
  IdentityFile ~/.ssh/your-key-name.pem
  User your-username
  Hostname your-hostname.local

Now you should be able to connect just by running ssh your-hostname.

Basic Hardening

Open the sshd config with some terminal editor:

sudo nano /etc/ssh/sshd_config

Most of the settings we’re looking for are already in the sshd config; many of them are commented out. Here are the key settings I usually update/add:

AllowUsers YOUR-USERNAME
LoginGraceTime 1m
PermitRootLogin no
StrictModes yes
PasswordAuthentication no
PermitEmptyPasswords no
UsePAM no
Protocol 2

Now restart sshd on the Raspberry Pi to pick up the new settings.

sudo systemctl restart sshd

Leave your existing SSH session open until you’ve tested that you can still connect. If something goes wrong, you can roll back the changes with the existing session.

You should no longer be able to login to the Raspberry Pi using a password; the key you generated should be the only way to do it.

Debugging Issues With Setup

The Raspberry Pi gives a repeating LED blink sequence.

Check out the LED blink error codes for Raspberry Pi’s to help debug the issue.

For example, my Pi Zero W was giving 7 short blinks over and over; it couldn’t detect the kernel because I’d accidentally used the incompatible 64-bit version of Raspberry Pi OS Lite.

ssh: Could not resolve hostname: Name or service not known

1st check: Is your Raspberry Pi properly booted or is it exhibiting a blink sequence?

2nd check: Is your Raspberry Pi connected to your network? It’s worth double checking your ethernet cable or the wpa_supplicant.conf file from earlier.

Scanning for the Raspberry Pi

If your Pi is booted and connected, your system may be unable to resolve raspberrypi.local. You can use nmap to find it manually.

This will look for devices with port 22 open and provide the SSH version. As of writing, Raspberry Pi OS should have Raspbian in the SSH version name, so it’s easy to filter on.

# Scan for open SSH ports
nmap 192.168.1.0/24 -p22 --open -sV | \
# Filter it down to 'Raspbian' results
grep Raspbian -B 4

Example output:

You probably won’t have this many results; ideally you’ll just have 1.

Try out these IP addresses with SSH. In my case, I needed 192.168.1.134:

ssh matt@192.168.1.134

You can use the IP address in place of the hostname.

CAD With Code: First 3D Models in OpenSCAD

Thu, 28 Jul 2022 00:00:00 +0000

CAD modeling never felt very intuitive to me. It still doesn’t. Reducing complex geometries to a set of basic shapes and transformations is hard. Finding a good workflow can help a lot.

OpenSCAD

Autodesk’s Fusion 360 is the de facto standard that seemingly every ‘maker’ uses. Most alternatives work in more or less the same way. But then there’s OpenSCAD, which ditches the UI elements in favor of raw code. The syntax is basically Go and Python thrown into a blender.

Despite knowing about OpenSCAD for awhile, I kept putting off working with it. I wanted to have the conveniences that Fusion 360 gives you out of the box. Niceties like chamfers, bevels, and smoothing can make a huge difference in a model.

My fears were misguided.

Belfry OpenSCAD Library

The Belfry OpenSCAD Library (BOSL2) has support for most of the features I thought I would miss from Fusion 360. Plus, the quality of its wiki is extraordinary - stuffed with great examples.

So I’ll be relying on BOSL2 for the foreseeable future.

Your browser doesn't support this video.

First Steps

The official tutorial for OpenSCAD is quite good. I went through about half of it before getting bored and bailing.

I want to learn OpenSCAD and reduce the pain of future modeling at the same time. How? Most of the models I’ve made involved screw holes - and screw holes are about as easy as CAD can get, so that’s where I started.

I want to make a jig so that I can consistently create holes that standard metric screws can cut their threads into.

As a starting point for screw sizes, I went off the specs. m2 screws are supposed to be 2 millimeters; m3 are supposed to be 3mm and so on. (But if you’re going to do 3d modeling, a set of calipers is an absolute must.)

Here’s what I worked out for a very basic test model.

include <BOSL2/std.scad>;

// for metric screws to self-cut threads in the 3d prints
module screwTest() {

    // All of these are in mm.
    // I just fiddled with them until it looked alright.
    depth = 5;
    height = 7;
    width = 55;

    // subtract the screw holes from the base
    difference() {

        // the base jig to contain all of the screw holes
        cuboid([width,height,depth]);

        // m2
        left(width * 0.35)
            cyl(d=2, h=depth);
        // m2.5
        left(width * 0.12)
            cyl(d=2.5, h=depth);
        // m3
        right(width * 0.05)
            cyl(d=3, h=depth);
        // m4
        right(width * 0.23)
            cyl(d=4, h=depth);
        // m5
        right(width * 0.43)
            cyl(d=5, h=depth);
    }
}

screwTest();

units are all in millimeters

Your browser doesn't support this video.

This gave me a decent jig, but we can make it look a lot cleaner and add labels.

include <BOSL2/std.scad>;

// for metric screws to self-cut threads
module screwTest() {
    depth = 5;
    height = 7;
    width = 55;

    difference() {
        cuboid(
	    [width,height,depth],
	    rounding=3, // smooths out the edges
	    edges="Z"   // leaves the top and bottom flat (for cleaner 3d printing)
	);

        // m2
        left(width * 0.35)union(){
            cyl(d=2, h=depth);

            up(depth * 0.5 - 1)
            fwd(1)
            left(7)
            text3d(
                "m2",
                size=3
            );
        }
        // m2.5
        left(width * 0.12)union(){
            cyl(d=2.5, h=depth);

            up(depth * 0.5 - 1)
            fwd(1)
            left(10.5)
            text3d(
                "m2.5",
                size=3
            );
        }
        // m3
        right(width * 0.05)union(){
            cyl(d=3, h=depth);

            up(depth * 0.5 - 1)
            fwd(1)
            left(7.5)
            text3d(
                "m3",
                size=3
            );
        }
        // m4
        right(width * 0.23)union(){
            cyl(d=4, h=depth);

            up(depth * 0.5 - 1)
            fwd(1)
            left(8)
            text3d(
                "m4",
                size=3
            );
        }
        // m5
        right(width * 0.43)union(){
            cyl(d=5, h=depth);

            up(depth * 0.5 - 1)
            fwd(1)
            left(8.5)
            text3d(
                "m5",
                size=3
            );
        }
    }
}

screwTest();

So much better.

Your browser doesn't support this video.

With that as a base, I created additional jigs. One to size out how to make slots for captive nuts. One to size out these press-fit inserts I bought and never used.

Captive nuts are when you put a nut inside the plastic itself and screw into it. Since nuts are usually much stronger than 3d printed plastic, using a captive nut can really increase the strength of a model.

Your browser doesn't support this video.

Press-fit inserts are female-threaded pieces of metal that you heat up and push into the plastic. The idea is that the plastic will melt around the insert and hold it tight after cooling.

Your browser doesn't support this video.

Iteration

The current versions of any of these: https://github.com/matt-fff/scadlib

With these basic jigs set up, I could actually test them on the 3d printer.

So comes the iteration loop:

Print the jig.
Test the hole sizes.
Tweak the SCAD file.
Repeat.

All of the jig holes need to be large enough that they’re easy to use, but none of the holes can be too large.

Press-fit insert jig: The holes need to be small enough that the plastic will fully conform to the insert after it cools.

Your browser doesn't support this video.

Captive nut jig: The holes need to be small enough to hold the nut steady as a screw is driven into it.

Your browser doesn't support this video.

Self-cutting screw jig: The holes need to be small enough that the screws cut their threads in the plastic.

It took a lot of iterations, but I finally settled on values that can reliably reproduce ideal screw sizes for my workflow. I created “official” screw modules so that I can reuse them in new SCAD files.

screws.scad

include <BOSL2/std.scad>;

module m2_hole(height, self_cut=true, anchor=CENTER) {
    diam = (self_cut) ? 2.2 : 3;
    cyl(
        d=diam,
        h=height,
        anchor=anchor
    );
}
...
module m5_hole(height, self_cut=true, anchor=CENTER) {
    diam = (self_cut) ? 5 : 6;
    cyl(
        d=diam,
        h=height,
        anchor=anchor
    );
}

module m2_captive_nut(height, inset=2, anchor=CENTER) {
    width = 4.5;
    cuboid(
        [width, width + inset + 1.5, height],
        anchor=anchor
    );
}
...
module m5_captive_nut(height, inset=3, anchor=CENTER) {
    width = 9;
    cuboid(
        [width, width + inset + 1.5, height],
        anchor=anchor
    );
}

Reusing the screw things

I now have a library of screw holes that I know will work with my screws and 3D printing process. Now what?

I have four brushless motors that I plan on using soon, so it makes sense to start there. Each motor has four m3 screws arranged around its core.

As you can see, the screw holes are somewhat asymmetric. There’s an additional 3mm between the wider set of screws.

A tip for finding the distance between two screw holes:

Measure the distance between the outer edges of the holes
Measure the distance between the inner edges of the holes
Add the distances together and divide by two.

motor.scad

include <BOSL2/std.scad>;
use <scadlib/screws.scad>;

module screw_holes(height, anchor=CENTER) {
    width1 = (13 + 18.5) / 2;
    width2 = (16 + 21.5) / 2;
    fwd(width1/2)
        m3_hole(height, self_cut=false, anchor=anchor);
    back(width1/2)
        m3_hole(height, self_cut=false, anchor=anchor);
    zrot(90)union() {
        fwd(width2/2)
            m3_hole(height, self_cut=false, anchor=anchor);
        back(width2/2)w
            m3_hole(height, self_cut=false, anchor=anchor);
    }
}

module air_hole(height) {
    cuboid(
        // idk why the height needs bumped
        [8, 4, height * 1.01],
        anchor=BOTTOM,
        edges="Z",
        rounding=2
    );
}

module air_holes(height) {
    width = (13 + 22.5) / 2;
    fwd(width/2)air_hole(height);
    back(width/2)air_hole(height);
    zrot(90)union() {
        fwd(width/2)air_hole(height);
        back(width/2)air_hole(height);
    }
}


module motor_mount(
    motor_diam=28,
    shell_thickness=1.5,
    shell_lip=5,
    wire_width=7,
){
    double_thick = shell_thickness * 2;
    difference() {
        // shell
        cyl(
            d=motor_diam + double_thick,
            h=shell_lip + shell_thickness,
            anchor=BOTTOM,
            chamfer2=1
        );
        up(shell_thickness)union() {
            // motor negative space
            cyl(
                d=motor_diam,
                h=shell_lip + shell_thickness,
                anchor=BOTTOM,
                chamfer=1
            );
            // wire negative space
            fwd(shell_thickness)cuboid(
            [
                wire_width,
                motor_diam,
                shell_lip + shell_thickness
            ], anchor=BOTTOM);

            // holes
            down(shell_thickness)union() {
                // m3 screw holes
                zrot(-45)
                    screw_holes(shell_thickness, anchor=BOTTOM);
                // air holes
                air_holes(shell_thickness);
            }
        }
    }
}

motor_mount();

The resulting model:

Your browser doesn't support this video.

I printed out the model to make sure it actually matched the motors. Perfect fit.

Your browser doesn't support this video.

Chardet: Easy Codec Detection in Python

Sun, 19 Jun 2022 00:00:00 +0000

If you have to deal with uploads from real-life users, you run into encoding issues from time to time. This is especially pernicious with CSV files, which Excel loves to export with strange and Windows-specific latin codecs.

What’s with the garbled characters? What the heck darn kind of outfit are you running here?

^ Feedback from a client, probably

Enter chardet

chardet is a character encoding detector from the good folks at Mozilla. While originally written for C++, we’ll be using the Python port.

pip install chardet

chardet looks at the data you feed it and makes an educated guess about the codec. Here’s a very basic codec guessing function:

from chardet.universaldetector import UniversalDetector

DEFAULT_ENCODING = "utf-8"

def guess_codec(file_name: str) -> str:
    codec_detector = UniversalDetector()
    with open(file_name, mode="rb") as file:
        while True:
            line = file.readline()
            if not line:
                break

            codec_detector.feed(line)
            if codec_detector.done:
                break

    result = codec_detector.close()
    encoding = result.get("encoding")
    return encoding or DEFAULT_ENCODING

I understand that if you’re dealing with user uploads, there’s a pretty tight ceiling on your happiness, but I hope this helps in a marginal way.

7 Deadly API Sins

Tue, 22 Mar 2022 00:00:00 +0000

Maximum suffering - minimal effort

This guide and the descriptions below are fictitious. No identification with actual APIs (living or dead) is intended or should be inferred - especially not seller, vendor, marketplace, or advertising APIs from companies named after notable rainforests.

1. Keep your documentation accuracy as close to 66.6% as possible

Obviously 66.6% looks rather insidious, but that’s purely coincidental.

Found through extensive trial and error, 66.6% is the “golden ratio” for API documentation. Your docs have to be accurate enough to keep API users referencing them, but inaccurate enough to cause untold suffering.

The most fun part of this is redirecting users to the docs when they inevitably open support tickets about everything else in this guide.

2. Use completely different ID’s for your UI and your API

Using different ID’s is only half the battle; you need to make sure there’s literally no way for API users to map between the IDs in the UI and the IDs in the API.

Make integrators try to guess which resource is which based on fields that aren’t unique in any way. That’s half the fun.

Bonus points if you have a third layer of completely separate object IDs for internal use.

3. Rotate HTTP codes and messages for the exact same errors

API integrators get complacent if you leave their error handling intact for too long.

As a rule, make your errors as vague as possible, but when you can’t avoid specificity, make it a point to reword and reclassify errors at regular (but not too regular) intervals.

You can rephrase “this region is not supported” in many ways - “support is not available in this region”, “support for this region is not yet implemented”, “this operation is not available in this marketplace”, etc.

And don’t forget to change your response codes. Change some 429 Too Many Requests to 401 Unauthorized’s. Change some 401 Unauthorized’s to 403 Forbidden’s. Hell, change some 400 Bad Request’s to 200 OK’s - you get the idea.

The important thing is to make sure your errors change all the time. Let loose and don’t be afraid to experiment!

4. Use PATCH and PUT interchangeably

HTTP verbs are more what we’d call “guidelines”, after all. Use PUT to mean PATCH on 70-90% of endpoints. While tempting, don’t make the mistake of doing it too consistently - your users will figure it out.

For the 30-10% of the time when you really mean PUT, it’s quite fun to require fields that the user has no way of retrieving - never return all the required data in your GET endpoints.

If API users ask about these mystery fields, pretend to miss the point and lecture them about how PUT requests work. Direct them to the documentation.

5. By frugal - reuse old ID’s for new objects

When API objects can finally be retired, it would be wasteful to retire the unique identifier with them. With rising support for renewables across virtually all demographics, the only thing better than a unique ID is a recycled ID.

Sustainable practices and invalidating countless amounts of user data at the same time? What’s not to love?

Like with #2, it’s critical to ensure that API users have no mechanism to determine which ID maps to which object. It’s pointless to invalidate swaths of data when your users can simply “undo it” by remapping millions and millions of foreign keys.

6. Consistency is the enemy of curiosity

In a vacuum, different teams will naturally produce very different APIs. Inertia is on your side - the hard part is maintaining that vacuum.

The sad truth is - many engineers will naturally want to provide a consistent experience for users. The best defence is to ensure that your devs have no concept of what ‘consistent’ would even look like.

Block teams off from existing work and documention. Cross-team coordination is completely out. If you find your team building consistent APIs - even accidentally, fabricate “user feedback” to guide them in a different direction.

7. Throttling and rate limits are best communicated by choosing error responses at random

More and more, API statistics are showing a sharp drop in API usage immediately following HTTP 429 Too Many Requests responses. This kind of user disloyalty will not reflect well on the team when placed in the quarterly powerpoint presentation.

How do you keep your users sending requests when you’re throttling them? Simple - you make sure they have no idea they’re being throttled.

Send 429 Too Many Requests responses just often enough to make API users think they can rely on it. Now rotate in 400 Bad Request, 401 Unauthorized, 403 Forbidden, 500 Internal Service Error, 502 Bad Gateway, 504 Gateway Timeout - all seemingly at random.

As a bonus, you can place the user authentication tokens in the error response body. This way, API consumers who save unexpected errors might accidentally log the auth tokens in plaintext.

Ignore any support tickets that suggest exposing credentials in response bodies may be inadvisable. Direct them to the documentation.

A Terminal Case of Workspace Management

Sun, 30 Aug 2020 00:00:00 +0000

Who is this for?

You work in the terminal a lot
You have to jump between projects
You think easy context switching might make you curse less
(Bonus) you use terminal-based editors

Lately I’ve found myself with an D-tier superpower - near-painless workspace switching.

Your browser doesn't support this video.

Normalizing your workflow

I regularly work with a dozen different projects - none of which have consistent command usage. For example, to run the unit test suite for any given project:

# These are all equivalent, depending on the project
pytest tests
poetry run pytest tests
docker-compose run whatever-project pytest tests
docker-compose run whatever-project python manage.py test
docker-compose -f ./compose/docker-compose-test.yml run whatever-project pytest tests
...

This is cognitive overhead - and usually involves a few screwups each time I context switch. I wanted a setup where I could simply type test in each project and not have to think about it.

Enter desk

desk is a lightweight workspace manager - essentially just a wrapper for shell scripts that define each workspace.

I maintain a deskfile (really just a .sh file) for each project I work with regularly. By default, deskfiles live at ~/.desk/desks/ - a great folder to keep under version control.

So what goes into a desk workspace? Whatever fits your workflow. For me, the vast majority falls under:

run - to run the project
lint - for any linting tools
test - for the test runner
build - to build the project and/or container
format - for code formatting
interact - to enter a project’s docker container
typecheck - to run static type analysis

These are simply aliases in shell files; you can do whatever makes your workflow easier - as sparse or comprehensive as you want. Here’s how a couple of mine look.

A project based on Python Poetry

#!/bin/bash
#
# Description: video editing library
#

basedir=~/Workspaces/matt/ergot

cd $basedir || exit

# code formatting
alias format="poetry run black ."

# run unit tests
alias test="poetry run pytest tests/"

# build the project
alias build="poetry build"

# static type checking
alias typecheck="poetry run mypy ."

# run flake8
alias flake8="poetry run flake8 --max-complexity 10 ."

# run pylint
alias pylint="poetry run pylint tests/ ergot/"

# run linting
alias lint="flake8 && pylint"

# run bandit security check
alias bandit="poetry run bandit -r ergot/"

# run code quality
alias quality="test; format; typecheck; bandit; lint"

# run the ergot command line
alias run="poetry run python main.py"

A project with the development flow built in docker compose

#!/bin/bash
#
# Description: scheduled and queue driven jobs
#

basedir=~/Workspaces/matt/JobSlob

export COMPOSEPREFIX="docker-compose -f ${basedir}/docker-compose.yaml"
export PYTHONPATH=$basedir
cd $basedir || exit

# close running instances
alias down="$(echo "${COMPOSEPREFIX}") down --remove-orphans"

# code formatting
alias format="$(echo "${COMPOSEPREFIX}") run jobslob formatting"

# run unit tests
alias test="$(echo "${COMPOSEPREFIX}") run jobslob test"

# build the project
alias build="$(echo "${COMPOSEPREFIX}") build"

# static type checking
alias typecheck="$(echo "${COMPOSEPREFIX}") run jobslob typecheck"

# run linting
alias lint="$(echo "${COMPOSEPREFIX}") run jobslob lint"

# run code quality
alias quality="test; format; typecheck; lint"

# run bash inside the app container
alias interact="$(echo "${COMPOSEPREFIX}") run --entrypoint bash jobslob"

I’d welcome a better solution than my weird echo "${COMPOSEPREFIX}" nonsense. Email me, shell experts.

Obviously some projects will need more or fewer aliases, but these workspaces are much more uniform than before. It doesn’t matter which workspace I’m in - test runs the test suite. Whatever you want.

If you need to know what’s available, desk will give you a nice rundown of commands.

Your browser doesn't support this video.

Persisting and switching between your workspaces

Desk is great for rapidly switching between terminal workspaces, but we can still do better. This is where terminal multiplexers come in.

To borrow from tmux’s wiki (which says this far better than I could), terminal multiplexers:

[let] you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal.

With tmux, we can easily keep many workspaces running simultaneously and rapidly switch between them.

I’ll focus on integrating desk with tmux, but I’m sure the same thing can be accomplished with other multiplexers like screen.

New to tmux?

If you need a quickstart, tmux has a very solid getting started guide and there are countless tutorials out there that I can’t beat.

tmux cheat sheet

For our purposes, the following should be enough.

tmux uses ctrl+b as its prefix hotkey by default. After entering ctrl+b, you can either type a hotkey sequence or use : and enter a command.
to detach from a running tmux session: ctrl+b, d
to list and switch between running sessions: ctrl+b, s
(Optional) to create a new named session from within tmux: ctrl+b, :new -s new-session-name

Integrating with desk

The real magic happens when you add the following to your .bashrc/.zshrc/.whatevershellrc file:

tsession=$([[ -n "${TMUX+set}" ]] && tmux display-message -p "#S")

# strip suffixes so that you can make
# multiple instances of the same desk
# by using deskname-1, deskname-2, etc
desk=$(echo "$tsession" | sed -E "s/-[0-9]+$//g")

if [ "x$tsession" != "x" ] && [ "$tsession" != "$WITHIN_DESK" ]
then
	export WITHIN_DESK="$tsession"
	desk . "$desk"
fi;

This snippet sets up your shell to detect whether you’re running in a tmux session. If you are, the shell will attempt to activate a desk with the same name as the session.

At this stage, if you have a desk called whatevs, you can run tmux new-session -s whatevs to create a new tmux session that automatically activates your whatevs desk. If it’s already running, you could use tmux attach -t whatevs to attach to it.

But again, we can make it easier.

t () {
  tmux attach -t $1 || tmux new-session -s $1
}

Adding this to your .bashrc/.zshrc/.whatevershellrc file, you can just run t whatevs and not worry about its state. tmux will drop you into your workspace.

Take advantage of tmux session switching

This is how you switch between open workspaces quickly. By default, tmux will give you a list of sessions when you enter ctrl+b, s. You can navigate the list with your arrow keys or use the numbers displayed alongside each session.

Your browser doesn't support this video.

It’s so easy to jump between projects - and even more useful if you rely on terminal-based editors. I’ve never had such a fluid workflow.

Deleted scenes

Alternatives

direnv

If desk doesn’t suite your purposes (to be fair, it hasn’t had much development lately), I’ve found that direnv can meet many of the same needs.

For me, the advantage goes to desk for the following reasons:

desk stores workspaces in a single folder that’s trivial to put in version control
direnv would require a completely different method to integrate with tmux in a similar way

tmuxinator

tmuxinator seems like it could be quite useful for workspace management and customization, but I’ve never personally used it.

Desired future improvements

Cleanly integrate pass or some other credential manager with this setup
Get rid of the not found warning starting tmux with a name that has no matching desk

Python 3.9 Features I'm Excited to See

Thu, 13 Aug 2020 00:00:00 +0000

As of 2020-08-11, the first release candidate of Python 3.9 has been released - officially making it Christmas in August; let’s open presents.

Exciting

Vaguely ordered by ascending excitement.

Native Dict Unions

PEP 584: Add union operators to dict

At the moment, if you want to merge a couple of dicts, the primary methods are to use either {**first_dict, **second_dict} or first_dict.update(second_dict) (which modifies first_dict in-place):

a = {0: 0, 1: 0, 2: 0}
b = {1: 1, 2: 2, 3: 3}

self.assertEqual({**a, **b}, {0: 0, 1: 1, 2: 2, 3: 3})

# `b` is modified in-place
b.update(a) 

self.assertEqual(b, {0: 0, 1: 0, 2: 0, 3: 3})

The syntax introduced in Python 3.9 is a lot more straightforward, allowing you to merge dicts with a pipe, e.g. first_dict | second_dict or first_dict |= second_dict (which, again, modifies first_dict in-place).

The order of the dicts still matters, i.e. whichever dict comes last will take precedence when there are key collisions.

a = {0: 0, 1: 0, 2: 0}
b = {1: 1, 2: 2, 3: 3}

# `b` colliding keys `1` and `2` still take precedence over `a`
self.assertEqual(a | b, {0: 0, 1: 1, 2: 2, 3: 3})

# `b` is modified in-place
b |= a

# `a` colliding keys `1` and `2` still take precedence over `b`
self.assertEqual(b, {0: 0, 1: 0, 2: 0, 3: 3})

Native String (Pre|Suf)fix Removal

PEP 616: String methods to remove prefixes and suffixes

It’s hard to overstate how often I need to strip prefixes or suffixes - and doing it in Python has been pretty verbose up to now.

# Stripping a prefix
if some_str.startswith(some_prefix):
    some_str = some_str[len(some_prefix):]

# Stripping a suffix
if some_str.endswith(some_suffix):
    some_str = some_str[:-len(some_suffix)]

Yeah, you can collapse these to one liners, but it’s still ugly as hell - and God forbid you need to do both at the same time.

I’m thrilled that removeprefix and removesuffix are finally available in Python 3.9:

# Stripping a prefix
some_str = some_str.removeprefix(some_prefix)

# Stripping a suffix
some_str = some_str.removesuffix(some_suffix)

# What the hell? Let's do both!
some_str = some_str.removeprefix(some_prefix).removesuffix(some_suffix)

Standard Collection Type Annotations

PEP 585: Type hinting generics in standard collections

This is a feature I’ll be using every day. If you use type annotations to a serious extent, you’ll probably be familiar with some version of the following:

from typing import Dict, List, Set

str_dict: Dict[str, str] = {
   "only": "strings",
   "nothing": "else",
}

float_list: List[float] = [1.1, 2.2, 3.3]

int_set: Set[int] = {1, 2, 3}

To get the benefits of type checking, you either have to import from typing (like above) or import annotations from __future__.

It’s just an import, but it can add up in a large project - especially if you find yourself copying any type-annotated code from one file to another (or, say, from StackOverflow).

With 3.9, things will be a bit cleaner:

str_dict: dict[str, str] = {
   "only": "strings",
   "nothing": "else",
}

float_list: list[float] = [1.1, 2.2, 3.3]

int_set: set[int] = {1, 2, 3}

Honorable Mention

I’m not not excited.

Type Annotation.. Annotations?

PEP 593: Flexible function and variable annotations

As more Python libraries leverage type annotations, there’s value in giving a flexible way to add metadata alongside particular types. This PEP introduces the Annotated construct which takes a type and an arbitrary list of arguments that serve as metadata, e.g.

Annotated[float, ValueRange(42, 420)]
Annotated[int, IsDivisibleBy(4), IsAfraidOfSeven(False)]
Annotated[tuple, MinLen(1), MaxLen(7)]

To support these annotations, typing.get_type_hints is changing to provide an include_extras kwarg (defaulting to False for backwards compatibility) that indicates whether to return this additional metadata.

I see how this could be a pretty useful PEP for libraries, but I don’t see a use for it in my own work, yet.

Good to Know

Not throwing shade or anything. Just don’t see these changing my life very much.

PEP 615: Support for the IANA time zone database in the standard library

This is a PEP that I need to spend more time with to really understand. My ridiculously dumbed-down (due to me, not you) summary is that it introduces a new zoneinfo API that replaces a lot of the functionality that you’d normally want to use pytz for, e.g.

from zoneinfo import ZoneInfo
from datetime import datetime

zone = ZoneInfo("America/Denver")
dt = datetime(2020, 12, 9, 6, tzinfo=zone)

PEP 692: Annual release cycle for Python

Python releases once a year now.

PEP 617: New PEG parser for CPython

Probably only relevant if you work on CPython or have an interest in compilers.

Inexact Number Comparisons in Python

Mon, 22 Jun 2020 00:00:00 +0000

It becomes tricky to write correct comparison logic when your numbers may not be exactly the same. Working with anything more than basic metrics and contrived data will often result in the need to go beyond a basic comparison:

# python
a == b

# unittest
self.assertEqual(a, b)

# pytest
assert a == b

Tolerant Comparisons in Tests

If you’re focused on tests and using pytest, you’re golden - pytest.approx should have you covered:

import pytest
assert a == pytest.approx(b)

Unfortunately, most of my active projects use unittest, so it’s natural to try to turn to assertAlmostEqual to compare:

# unittest: make sure that a and b are equal up to 2 decimal places.
self.assertAlmostEqual(a, b, 2)

This works well enough for confined number ranges, but test cases often involve a large variety of different metrics running through the same assertions - assertAlmostEqual falls pretty flat when you’re using it to compare number ranges from 0.000001 up to 100000000000.

# This evaluates as True despite the values being quite different on a relative basis
self.assertAlmostEqual(0.000001, 0.009, 2)

# This evaluates as False despite the values being virtually identical
self.assertAlmostEqual(100000000000, 100000000000.01, 2)

math.isclose for Relative Tolerances

Obviously you can tailor every assertion to the data, but if you’re looking for something closer to pytest’s approx (and available without importing a testing framework), I’ve taken to using math.isclose.

isclose allows you to compare values using a relative tolerance rather than a fixed number of decimal places:

import math

# will return True if a and b are within 0.1% of each other
math.isclose(a, b, rel_tol=0.001)

You can use the function directly or drop it into a test case helper function for your unittest or Django test classes:

import unittest
import math

class TestWhatever(unittest.TestCase):
    def assert_is_close(self, a, b, rel_tol=0.001):
        self.assertTrue(
            math.isclose(a, b, rel_tol=rel_tol),
            # custom message, as the default failure message won't be informative
            f"'{a}' is not within {rel_tol:.2%} of '{b}'"
        )

Unchecked Speculation on Proposed Python Ideas

The day this post first went out, I saw some discussion on the python-ideas mailing list about introducing a kind of plus-or-minus operator into Python’s grammar, e.g.

a == b +- 0.05

This would evaluate to true if b was within 0.05 above or below a.

While I think the syntax is pretty intuitive, I’m not sure how useful it is. As discussed above, absolute value comparisons aren’t very useful for numbers that aren’t within a confined range. Maybe if there was a clean way to indicate if it’s a relative comparison, e.g.

# for absolute comparisons
a == b +- 0.05

# for relative comparisons
a == b +- 0.01%

But it seems like a needless complication when math.isclose is simple enough to use. We don’t need another walrus operator debacle.

Work Around Poor Access Control: Limit GitHub Access Tokens to Specific Repos

Mon, 08 Jun 2020 00:00:00 +0000

Irrelevant Context

I recently decided to migrate my JetBrains IDE settings from their automatic settings sync over to storing the settings in a repository - with far-off hopes that it’ll be less annoying and inconsistent.

I created blank repos to store settings for GoLand, PyCharm, WebStorm, and DataGrip.

The frustrating part is that the JetBrains repo sync doesn’t seem to support SSH, so my usual auth method is out; for my IDEs to use these repos, I have to set up a GitHub personal access token for JetBrains to use - and there’s currently no built-in way to limit access tokens to specific repositories. I only want the JetBrains access token to work for the settings repositories, specifically.

The Workaround: Blank GitHub Collaborator Accounts

It’s pretty straightforward to create a GitHub account. Tada. And it may be wise to enable 2FA for good measure.

Send the invites on your primary account

Now on each of the desired repositories, I go to Settings > Manage Access > Invite a collaborator.

Accept the invites on your collaborator account

Actually finding these invitations in the collaborator account is far harder than it should be. Invitations are supposed to show up in GitHub notifications, but this isn’t true for me. The 2 ways that worked for me include 1) via the invitation email GitHub sends or 2) navigating to https://github.com/<your_account>/<your_invite_repo>/invitations directly. Obviously you need to be logged into the collaborator account.

Generate a personal access token on your collaborator account

Heading on over to Settings > Developer Settings > Personal Access Tokens, the new token is ripe for the generating.

We’re still forced to use an overly broad scope (because GitHub), but at least the token only has access to the specific repositories we set up.

Copy the plaintext token which, like us, is transient and won’t be accessible later.

And now you have your personal access token with limited repository access.

Print Tracebacks From Python Exception Instances

Mon, 25 May 2020 00:00:00 +0000

The solution:

import traceback; traceback.print_tb(exception_instance.__traceback__)

The rest:

In my daily work, I use pdb pretty heavily for debugging. There are some limitations to the pdb shell that are constant annoyances to me - one of which applies directly to my habit of sticking breakpoints in try/except blocks. For example:

class ExistentialCrisis(Exception):
    pass

def think_about_existence():
    raise ExistentialCrisis("oblivion is inevitable")

try:
    think_about_existence()
except ExistentialCrisis as exc:
    import pdb; pdb.set_trace()
    # This is where we end up in pdb after running this code.
    raise

If we weren’t in pdb, we could solve this pretty easily:

...
try:
    think_about_existence()
except ExistentialCrisis as exc:
    import traceback
    traceback.print_exc()
    raise

And you can do the same traceback.print_exc() in pdb if you use the interact command to drop into the current scope - but I inexplicably hate doing that.

How can you get the traceback from the exception object itself?

import traceback; traceback.print_tb(exc.__traceback__)

Cloning a Repository with Submodules

Mon, 11 May 2020 00:00:00 +0000

Every couple months, I’ll clone a repo with submodules and have to go re-learn how to clone the submodules from this guide.

For future me, who’s forgotten again - how to clone a submodule in a repository you’ve already pulled:

git submodule init
git submodule update

If you have enough foresight - how to download submodules at the same time you clone the main repository:

git clone --recurse-submodules <your-repository>

Hijacking Default Django 'Through' Tables

Thu, 12 Dec 2019 00:00:00 +0000

A few times in the last year, I’ve run into the need to add some metadata to a Django many-to-many relationship. By default, there’s no explicit model to add fields to, but - if you’re working on an active project - you probably have existing data in the default ’through’ table that you don’t want to lose.

So what are you to do if you don’t want to have to create a completely new table and migrate the data over? Let’s hijack the existing one.

Model Setup

The existing models:

from django.db import models

class ModelA(models.Model):
    b_models = models.ManyToManyField("app.ModelB", related_name="a_models", blank=True)

    class Meta:
        db_table = "app_model_a"


class ModelB(models.Model):
    class Meta:
        db_table = "app_model_b"

Create a model that matches the existing ’through’ table exactly - and make sure to specify the existing table name using Meta.db_table:

from django.db import models


class ModelAModelB(models.Model):

    modela = models.ForeignKey("app.ModelA", on_delete=models.CASCADE)
    modelb = models.ForeignKey("app.ModelB", on_delete=models.CASCADE)

    class Meta:
        db_table = "app_model_a_model_b"

Update the many-to-many relationship to use the new model:

class ModelA(models.Model):
    b_models = models.ManyToManyField("app.ModelB", related_name="a_models", through="app.ModelAModelB", blank=True)

    class Meta:
        db_table = "app_model_a"

Migrations

Now generate a new migration with python manage.py makemigrations; it’ll need some editing. The initial migration:

import datetime
from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('api_app', '69420_whatever_your_last_migration_was'),
    ]

    operations = [
        migrations.CreateModel(
            name='ModelAModelB',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
            ],
            options={
                'db_table': 'app_model_a_model_b',
            },
        ),
        migrations.AlterField(
            model_name='modela',
            name='b_models',
            field=models.ManyToManyField(blank=True, related_name='a_models', through='app.ModelAModelB', to='app.ModelB'),
        ),
        migrations.AddField(
            model_name='modelamodelb',
            name='modela',
            field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='app.ModelA'),
        ),
        migrations.AddField(
            model_name='modelamodelb',
            name='modelb',
            field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='app.ModelB'),
        ),
    ]

Wrap the Migration

We just need to wrap these operations in migrations.SeparateDatabaseAndState to get the models in sync without screwing up the existing database setup. All of the changes above represent state changes:

import datetime
from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('api_app', '69420_whatever_your_last_migration_was'),
    ]

    state_operations = [
        migrations.CreateModel(
            name='ModelAModelB',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
            ],
            options={
                'db_table': 'app_model_a_model_b',
            },
        ),
        migrations.AlterField(
            model_name='modela',
            name='b_models',
            field=models.ManyToManyField(blank=True, related_name='a_models', through='app.ModelAModelB', to='app.ModelB'),
        ),
        migrations.AddField(
            model_name='modelamodelb',
            name='modela',
            field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='app.ModelA'),
        ),
        migrations.AddField(
            model_name='modelamodelb',
            name='modelb',
            field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='app.ModelB'),
        ),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            state_operations=state_operations,
        )
    ]

And that should do it. That migration should successfully tie the existing ’through’ table to your new model. Now you can add fields and create additional migrations as normal.

For my notes - filtering on ’through’ fields

I want to make one additional note - since it wasn’t immediately obvious to me. With Django 2.2, you should be able to filter on the ’through’ table fields by using the lowercase name of the ’through’ model.

You can add something like state to the ’through’ model:

class ModelAModelB(models.Model):

    modela = models.ForeignKey("app.ModelA", on_delete=models.CASCADE)
    modelb = models.ForeignKey("app.ModelB", on_delete=models.CASCADE)
    state = models.CharField(
      max_length=16,
      null=True,
      blank=True,
      choices=[("good", "good"), ("ungood", "ungood")]
    )

    class Meta:
        db_table = "app_model_a_model_b"

Then filtering on this field looks a lot like this:

# querysets
ModelA.objects.filter(modelamodelb__state="good")
ModelB.objects.filter(modelamodelb__state="ungood")
ModelAModelB.objects.filter(state="good")

# related manager on instances
a_instance = ModelA.objects.first()
a_instance.b_models.filter(modelamodelb__state="ungood")

b_instance = ModelB.objects.first()
b_instance.a_models.filter(modelamodelb__state="good")

Now hopefully I can just come back to this post next time I need to do this.

Git Hooks: Autoformat before commit

Sat, 05 Jan 2019 00:00:00 +0000

Autoformatters can be great, keeping diffs small and a code base readable across teams and engineers. Black formatting is a standard in my Python repositories.

But what isn’t great is forgetting to run the formatter, having the CI build fail, and having to add a “Formatting” commit. That’s where Git Hooks come in.

Git hooks are stored in .git/hooks folder of any given Git project. There should be some samples pre-populated. For our purposes, pre-commit is the hook of interest.

Lets focus on the Black formatter for Python as an example. For the example, it needs to be installed locally (pip install black). Naively, we can add formatting on pre-commit by adding a file called pre-commit to .git/hooks with these contents:

black .

(For Javascript, you could use Prettier and prettier --write **/*.js)

Don’t forget to make the file executable (use chmod +x .git/hooks/pre-commit), or the hook may not fire.

It’s nice and simple, but there are a couple issues with this approach. First off, Black will be running over all the source files rather than just the ones that changed (which can be quite slow, depending on the size of your project). Secondly, the modified source files won’t actually be added to the commit, which means you need to re-commit the reformatted files.

Here comes the convoluted piping. To get the names and statuses of files that are staged for commit, we can use git diff --cached --name-status to yield output like this:

> $ git diff --cached --name-status

M       someproject/src/whatevs.py
A       someproject/tests/whatevs_test.py
D       someproject/tests/lasers.py
M       someproject/README.md

We want to ignore deleted files, since we can’t format them, so let’s filter them out:

> $ git diff --cached --name-status | \
grep -v '^D'

M       someproject/src/whatevs.py
A       someproject/tests/whatevs_test.py
M       someproject/README.md

We want to ignore non-Python files (or Javascript or Golang or whatever you’re using), so let’s filter on file extension:

> $ git diff --cached --name-status | \
grep -v '^D' | \
grep '.py'

M       someproject/src/whatevs.py
A       someproject/tests/whatevs_test.py

Now that we have the files we want, we just want to focus on the names:

> $ git diff --cached --name-status | \
grep -v '^D' | \
grep '.py' | \
sed 's/[A-Z][ \t]*//'

someproject/src/whatevs.py
someproject/tests/whatevs_test.py

And we can pass that right into Black (or whatever formatter you’re using):

> $ git diff --cached --name-status | \
grep -v '^D' | \
grep '.py' | \
sed 's/[A-Z][ \t]*//' | \
xargs black

reformatted someproject/src/whatevs.py
reformatted someproject/tests/whatevs_test.py
All done! ✨ 🍰 ✨
2 files reformatted.

So that’s all great, but the changes still aren’t staged. Let’s fix that by processing the output from Black into just the files that were reformatted:

> $ git diff --cached --name-status | \
grep -v '^D' | \
grep '.py' | \
sed 's/[A-Z][ \t]*//' | \
xargs black 2>&1 | \
grep '^reformatted' | \
sed 's/reformatted[ \t]//'

someproject/src/whatevs.py
someproject/tests/whatevs_test.py

And now we can finally stage the formatted files with xargs git add.

So the final .git/hooks/pre-commit file should look something like this:

#!/bin/bash

git diff --cached --name-status | \
grep -v '^D' | grep '.py' | \
sed 's/[A-Z][ \t]*//' | \
xargs black 2>&1 | \
grep '^reformatted' | \
sed 's/reformatted[ \t]//' | \
xargs git add

Then you need to make sure it’s runnable:

> $ chmod +x .git/hooks/pre-commit

That should do it.

SQLAlchemy + FactoryBoy: Passing arbitrary sessions to factories

Mon, 02 Jul 2018 00:00:00 +0000

In work projects in the past, my team would try to avoid dealing with the complexity of SQLAlchemy database sessions by making one global session that every module referenced. It made things easy and straightforward and it played nice with FactoryBoy - whose factories seem to work quite well under those conditions.

Going into building lobit.io, I tried to do things the same way, but as the code base and the number of unit tests grew, I kept facing what seemed to be an unquashable avalanche of OperationalErrors griping about “Too many connections”.

I implemented an overhaul of the codebase, wrapping every piece of ORM code in a separate session and creating a DbUtils class to handle sessions consistently without too much boilerplate:

from contextlib import contextmanager
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm.session import Session
from config import config


class DbUtils:
    Session = sessionmaker(bind=create_engine(config.db_connection_url))

    @classmethod
    @contextmanager
    def session_scope(cls) -> Session:
        """Provide a transactional scope around a series of operations."""
        session = cls.Session()
        try:
            yield session
            session.commit()
        except:
            session.rollback()
            raise
        finally:
            session.close()

Fixing the test factories

When it came to unit tests, I was at a loss for how to best handle all of the factories I was using. They were all tied into the global session. Furthermore, factories took in the session at the time of class creation, not instantiation, e.g.

import factory
from models.post import Post
from tests.factories.post_thread_factory import PostThreadFactory


class PostFactory(factory.alchemy.SQLAlchemyModelFactory):
    class Meta:
        model = Post
        sqlalchemy_session_persistence = "commit"
        sqlalchemy_session = session

    text = factory.Faker("text", max_nb_chars=100)
    external_id = factory.Faker("text", max_nb_chars=50)
    post_thread = factory.SubFactory(PostThreadFactory)

Wrapping factory class creation in a function did the trick:

import factory
from models.post import Post
from tests.factories.post_thread_factory import PostThreadFactory


def PostFactory(session):
    class _PostFactory(factory.alchemy.SQLAlchemyModelFactory):
        class Meta:
            model = Post
            sqlalchemy_session_persistence = "commit"
            sqlalchemy_session = session

        text = factory.Faker("text", max_nb_chars=100)
        external_id = factory.Faker("text", max_nb_chars=50)
        post_thread = factory.SubFactory(PostThreadFactory(session))

    return _PostFactory

While simple wrapping worked, it made my tests look God-awful. Manually managing sessions had already added boilerplate overhead to every unit test that hit the database.

Before refactoring, I could create a post using PostFactory.create(). Now it was PostFactory(session).create(). Most of the unit tests hit multiple factories several times each, so it was also less efficient to keep creating the same classes over and over.

I took to creating a single instance of each required factory at the top of each session:

    def test_post_stuff(self):
        with DbUtils.session_scope() as session:
            post_factory = PostFactory(session)
            post_thread_factory = PostThreadFactory(session)
            ...

Cleaning it up

A given test could have nearly a dozen factories defined at the top - which looked terrible even if it didn’t trigger a too-many-locals warning from Pylint (which it almost always did).

So I made one more change and hired a manager to keep all the factories running smoothly.

from tests.factories.post_factory import PostFactory
from tests.factories.post_thread_factory import PostThreadFactory
...

class FactoryManager:
    known_factories = [
        PostFactory,
        PostThreadFactory,
        ...
    ]

    def __init__(self, session):
        self.session = session

        for factory_func in self.known_factories:
            setattr(self, factory_func.__name__, factory_func(self.session))

The FactoryManager didn’t require a lot of code, but let me leave my tests largely unchanged. Instead of initializing several objects at the top of each test, I could initialize one.

    def test_post_stuff(self):
        with DbUtils.session_scope() as session:
            fm = FactoryManager(session)

Now I could prefix all factory calls with fm. and get something almost as clean as before. At least, fm.PostFactory.create() looks better to me than PostFactory(session).create() or littering each test with a bunch of locals.

Matt White | SWE

Azure's Browser Bias

User Agent Determines Login Success

What Browsers Do Our Benevolent Lord Microsoft Deem Acceptable?

Conclusion

Tested User Agents

Failed logins:

Successful logins:

LUKS-Encrypted Arch with systemd-boot and SecureBoot

Live USB

SSH (Optional)

Check the Boot Mode

Create the Partitions

List Disks and Partitions

Create the Partitions

Setup LUKS

Setup Volumes

Format the Partitions

Mounting

Setup Initial Installation

Packages

Configure fstab

Chroot Into the Installation

Update Root Password

Create User and Basic Configuration

Setting the Clock

Locale

Networking

.local Hostname Resolution

Initramfs

Create a Keyfile

Update mkinitcpio

Create the initramfs Image

Add Microcode Support

Configure the Bootloader

Sanity Check

Restart the System

Secure Boot

Check the Current Status

Disable Secure Boot and Enter Setup Mode

Enroll the Keys

Turn on SecureBoot

Check if Secure Boot Worked

Backup and Duplicate Raspberry Pi Images

Setup the Micro SD Card

Identify the Card

Unmount the Card, if Necessary

Back up the Image

Shrink the Image

Download PiShrink

Shrink Ray the Day Away

Applying the Image

Option 1: Terminal

Option 2: Etcher

Raspberry Pi Webcams

Supplies

A Raspberry Pi

A Camera Module

Or go with budget-conscious camera modules

A Compatible Camera Cable

Mounting

(Physical) Port Access

Putting it together

Setup Option 1 - Plug-and-Play USB Camera

Download the image

Image Burning Software

Plug and Play

Setup Option 2 - IP Camera

Install Raspberry Pi OS on a Micro SD Card

Install RPi-Cam-Web-Interface

Raspberry Pi Headless on Bullseye

What to Download

Raspberry Pi OS Lite

Image Burning Software

Mount the Micro SD Card

Setup Wifi

Add a User

Enable SSH Access

SSH Over USB Configuration (Optional)

Unmount