I've changed browser. Although I used Firefox for years, I've switched to Waterfox. As well as talking about how I migrated, this blog lays out _why_ I moved and some of the issues that I ran into along the way. The installation section is Linux specific, but the rest should be cross-platform. * * * ### Contents * Contents * Why? * Terms Of Service Change * Jumping On The LLM Bandwagon * Looking to 2026 * Moving On * Migrating * Installing Waterfox * First Launch * What Did It Import? * Using Sync * Configuring Extensions * Multi-Account Containers Disaster * Migrating localStorage data * Dynamic Overwrites * Attempting to Identify Sites * Using Waterfox * Enabling Vertical Tabs * Conclusion * * * ### Why? Note: if you're not interested in why, skip down to Migrating. 2025 wasn't a particularly great year for Firefox in my opinion. Although a lot of features were delivered, there was also a bit to be concerned about (_especially_ coming on the heels of 2024's announcement of an entry into advertising). * * * #### Terms Of Service Change At the end of February 2025, Mozilla issued an update to the Firefox Terms of Use which seemed to suggest that Mozilla were claiming ownership of anything that a user did. At the same time, they updated the Firefox FAQs and removed references to **not** selling data from the following: > The Firefox Browser is the only major browser backed by a not-for-profit that doesn’t sell your personal data to advertisers while helping you protect your personal information. After the (inevitable) community backlash, Mozilla walked the changes back but effectively told users that they must simply have been confused by the new wording. I don't think that Mozilla actually intended any harm with the changes, but it was pretty tone deaf and came across about as well as pissing on the community. * * * #### Jumping On The LLM Bandwagon In recent years, Mozilla has rarely been afraid of jumping onto resource sapping bandwagons, including cryptocurrency (they walked that one back a week later). It should come as no surprise then, that they've also been swept up in the AI goldrush and leapt onto the LLM bandwagon: * Adding Perplexity.AI as a search option * Embedding chatbots into the sidebar * AI powered local translation1 * Adding "AI enhanced" tab groups * LLM powered link previews * Plans to add AI agentic mode windows Unfortunately, it's not just the browser that they've been busily crow-barring AI into. Mozilla decided to start using LLMs to automatically translate knowledge-base content. This automation tramples edits and corrections made by human translators, which ultimately led to Japanese localisation community resignations. Searching Mozilla's support site for "AI" shows that others have raised concerns too: Mozilla is aware of the objections that are being raised by it's community, but has chosen a somewhat chiding tone: > standing still while technology moves forward doesn’t benefit the web or the people who use it. That’s why we see it as our responsibility to shape how AI integrates into the web, in ways that promote openness, transparency, and choice. That way, users and developers can use it freely, help shape it, and truly benefit from it. This isn't just an isolated position, but one that Mozilla have chosen to repeat on the product page for AI Window: > **The web is changing, and sitting it out doesn’t help anyone.** Although I've some sympathy for the argument that participation is necessary to (try to) shape the way that these technologies are deployed, it involves an inherent assumption that LLMs are the future of the web. Despite the current level of hype, that's very far from certain. It's probably just as likely that large portions of the web will become inaccessible to LLMs as publishers increasingly put up walls to protect their content, revenue and bandwidth bills. * * * #### Looking to 2026 Unfortunately, things in Firefox world are probably going to get worse before they get better. In December, Mozilla Corporation appointed a new CEO, who quickly re-affirmed their commitment to AI: > Firefox will grow from a browser into a broader ecosystem of trusted software. Firefox will remain our anchor. It will evolve into a modern AI browser Now, in fairness, he was also very clear that AI should always be opt-in (although there seems to be some confusion about what actually _counts_ as opt-in). Sadly, it seems clear that, for the forseeable future, we can expect a continuation of this: Whether the AI is opt-in or not, its very inclusion means that Mozilla are putting some of their finite resources into a featureset that I (and by the looks of it, a good chunk of their userbase) have no intention of _ever_ using. * * * ### Moving On Although I briefly moved away, I've used Firefox for years (and, before that, used the Mozilla browser). The existence of Firefox as an alternative is important, especially with Google increasingly flexing it's monopolistic muscles and interfering with (unfortunate) web necessities like ad blocking. There are also Firefox features that I _actively_ like and didn't want to lose. Although Vivaldi looked appealing, I didn't want to move to a Chromium fork if it could be avoided2. I had a play around with LibreWolf, but ultimately felt that I got on with Waterfox a little bit better3. Note: Both LibreWolf and Firefox have clearly stated that they won't add AI. * * * ### Migrating #### Installing Waterfox Waterfox's download page offers two options: a flatpak or a tarball. Flatpaks are convenient, but I've previously had issues with running browsers in Canonical's flavour of them (`snap`), so opted for the tarball instead. # Download and extract curl https://cdn.waterfox.com/waterfox/releases/6.6.7/Linux_x86_64/waterfox-6.6.7.tar.bz2 | tar jx # Install system wide sudo mv waterfox/ /opt/ # Link into $PATH sudo ln -s /opt/waterfox/waterfox /usr/local/bin/ With that, Waterfox was installed and available. However, it wasn't yet present in my window manager's menu. For that, I needed to create a `.desktop` file: cat << EOM > waterfox.desktop [Desktop Entry] Version=1.0 Name=Waterfox Web Browser Comment=Browse the web GenericName=Web Browser Keywords=Internet;WWW;Browser;Web;Explorer Exec=/usr/local/bin/waterfox %u Terminal=false X-MultipleArgs=false Type=Application Icon=/opt/waterfox/browser/chrome/icons/default/default128.png Categories=Network;WebBrowser; MimeType=text/html;text/xml;application/xhtml+xml;application/xml;application/rss+xml;application/rdf+xml;image/gif;image/jpeg;image/png;x-scheme-handler/http;x-scheme-handler/https;x-scheme-handler/ftp;x-scheme-handler/chrome;video/webm;application/x-xpinstall; StartupNotify=true Actions=new-window;new-private-window; [Desktop Action new-window] Name=Open a New Window Exec=/usr/local/bin/waterfox -new-window [Desktop Action new-private-window] Name=Open a New Private Window Exec=/usr/local/bin/waterfox -private-window EOM # Install it sudo cp waterfox.desktop /usr/local/share/applications/ Moments later, an icon appeared within my launcher menu * * * #### First Launch Launching Waterfox for the first time fired up an onboarding wizard: The next step offered to import data from other browsers but, in my case, was displaying an error: The message suggested that it was looking at the wrong profile, so I clicked the drop-down to select a different one. However, not only were multiple profiles listed, it looked like there were also duplicates: Which profile was the correct one? I switched back to Firefox and visited `about:profiles`, which displayed a list, only one of which was marked as being in use: Back in Waterfox, I selected that profile and a list of data types was displayed: I hit the `Import` button and the wizard moved on. A couple of steps later it displayed a list of recommended extensions: This was pleasing because it confirmed that I could continue to use Mult-Account Containers. The final step offered to make Waterfox the default browser before dropping me into an empty Waterfox session: * * * ##### What _Did_ It Import? There was, however, an issue. Although my (limited collection of) bookmarks had imported, nothing else seemed to have done so: there was no browsing history and no extensions had been added4. I went into settings and used the importer again (even trying different profiles), but with no luck. This posed a bit of a problem. Because history searching is so good nowadays, I've fallen out of the habit of using bookmarks and instead rely on suggestions when typing into the omnibox. Without my browser history, though, that simply doesn't work. * * * #### Using Sync One option might have been to copy Firefox's profile files over to Waterfox. However, I didn't _particularly_ want to mess around with this: although it might _just_ work, it might also break things. So, instead, I decided to use Firefox sync to bring things across (I actually used a self-hosted instance but that's not strictly necessary). First, I connected my original Firefox browser and made sure that sync was enabled for all data types: I then opened Waterfox and started to configure that. At this stage, I ran into a minor niggle: when setting Waterfox up I had set its theme to "automatic" (so that it follows my window manager). It was showing up in dark mode but the text boxes used to sign into a Mozilla account end up being white text on a white background, making my entry invisible unless highlighted: This became an issue again later when the process prompted me to connect a mobile browser: This isn't a Waterfox specific issue and happens on Firefox too - the underlying problem is that CSS for the parent `input` explicitly sets its background to white. While I was grumbling and poking at this, Waterfox installed the extensions that I'd had in Firefox, opening post-install tabs for a couple of them * * * ##### Configuring Extensions Unfortunately, although the extension _list_ was synced, Firefox Sync does not sync extension configuration. Some of my add-ons didn't need additional configuration5, but I had to manually configure: * 1Password * Cookie Auto Delete (though this was easy because it has import/export functions) * Stylus (I just had to copy and paste styles across) This was only really a small annoyance, but it'd have been nice to have had them come across already configured and working. * * * ##### Multi-Account Containers Disaster There's a certain amount of custom configuration involved with a Multi Account Container setup: * Creating custom containers * Container config (should it use a specific proxy etc?) * Defining which domains should always load in a given container I had noticed that the Multi Account Containers extension has an (off by default) option to enable synchronisation in it's preferences: Unfortunately, it turns out that the extension is quite fragile. When I enabled this in Firefox, the add-on silently crashed. Having not noticed, I turned the same setting on in Waterfox, which obediently synced it's config to the sync server. Of course, it's config was the extension's _default, unconfigured state_. I noticed that Waterfox hadn't pulled down the expected containers and so went to check Firefox. When I clicked the extension's icon, I got an empty white dialog and had to disable and re-enable the Add-On to get it working again. The extension leapt to life and pulled config down from the sync server. In an instant, my Multi-account containers configuration was irrevocably overwritten. The only option that I had left was to reconfigure the bits that I could remember. There should be a video here, if it isn't displaying you may need to visit in a browser In this case it's Fred McAuley's iconic "Fucking.... boo" scene from Mock The Week * * * #### Migrating `localStorage` data The javascript `localStorage` property provides key value storage that web applications can use to store data on your computer. This is an _incredibly_ useful way to have user data persist between site visits. However, Firefox sync **does not sync it** and so that data hadn't come across. This, for example, meant that my Excalidraw doodlings weren't available in Waterfox. I'd also lost state in some other webapps. Searching for a solution led me to an extension called NotSoLocalStorage. Unfortunately, at some point the extension broke and hasn't been fixed since. I played around troubleshooting it for a little while, but didn't really get anywhere (and, of course, didn't want to risk accidentally blowing away the very data that I was trying to save). In the end, I decided that I'd have to migrate the data manually. I wrote a short javascript snippet to dump `localStorage` data for a site items = { ...localStorage } console.log(btoa(JSON.stringify(items))) (I base64 encoded the output because Developer Tools kept escaping things rather than printing the raw JSON) As a test, I * Visited Excalidraw in Firefox * Hit `F12` to open Developer Tools * Pasted the snippet I then just needed to expand and copy the output line. In order to restore the data into Waterfox, I wrote a snippet to take the base64'd string, decode and write the data into `localStorage`: function load(j){ obj = JSON.parse(j); keys = Object.keys(obj) i = keys.length; while (i--){ console.log("Creating " + keys[i]); localStorage.setItem(keys[i], obj[keys[i]]); } } // Replace with the base64d string json_str = <string printed by dumping> load(atob(json_str)) * * * ##### Dynamic Overwrites In Waterfox, I browsed to Excalidraw and * Hit `F12` to open Developer Tools * Pasted the snippet * Clicked `Storage` and confirmed the items appeared * Refreshed the tab Unfortunately, although the `localStorage` items had appeared after I pasted the snippet they disappeared again when the tab was refreshed. This is because, on exit, Excalidraw saves its current state back to `localStorage` and so was overwriting the values that I'd added. To sidestep this, I needed to follow a slightly different process: * Hit `F12` to open Developer Tools * Pasted the snippet * Clicked `Storage` and confirmed the items appeared * Clicked `Debugger` * Pressed `F8` to pause javascript execution * Refreshed the tab This time Excalidraw loaded with all of my doodling intact! Thankfully, I didn't have this issue with any other sites - for those, pasting the snippet was enough. * * * ##### Attempting to Identify Sites One of the headaches here is that I needed to actively identify sites that were using `localStorage`6 and was concerned that I'd wind up missing something. Fortunately, Firefox stores `localStorage` data in a `sqlite` database: sqlite3 ~/.mozilla/firefox/che0auei.default-release/webappsstore.sqlite Querying a list of websites using `localStorage` is easy: select distinct originKey from webappsstore2; However, this returned well over a thousand sites - far more than I was willing to check through manually. I was able to, roughly, halve this by filtering the list to only include localStorage entries which contained a JSON value (that being a rough proxy for sites that set more complex config): select distinct originKey from webappsstore2 where value like '{%}' and value != '{}'; Although this was an improvement, it was still a **lot** of entries, something that also wasn't particularly helped by the fact that Firefox writes the origin key backwards: moc.elgoog.www.:https:443 Ultimately, I decided not to bother. Although I could have copied all of the data across, given how many sites seemed to be using `localStorage` to track visits, it seemed more beneficial to start with a clean slate. I used my javascript snippet to migrate `localStorage` for the sites that I could think of. Realistically, my Firefox profile will likely persist on disk for some time, so I can always backfill if I find something important that I've missed. * * * ### Using Waterfox I was ready to close Firefox and cut over to using Waterfox full time. As I went along, though, I did find that I needed a few more tweaks: * Set the theme to light mode to avoid the invisible text issue * Set search to Ecosia (though I might try Waterfox private search at some point) * Disabled DNS over HTTPS (I have split horizon DNS, so DoH causes headaches) * * * #### Enabling Vertical Tabs The process of enabling vertical tabs was a little unintuitive, though easy once you know where to look. Next to the back button is a small icon to reveal the sidebar At the bottom of the sidebar is a Settings cog, which reveals an option to use Vertical tabs Once vertical tabs are enabled, clicking out and then back into the settings will reveal an option for `Tree Vertical Tabs`. I'm still getting used to going to the side of the window to open a new tab (though, strangely, don't have the same issue when looking for an existing one), but so far I quite like it. If nothing else, it's certainly given me the feeling of using something new and shiny. * * * ### Conclusion I really _want_ Firefox to succeed: As small as it's market share has become, it's _incredibly_ important that non-Chrome based browsers continue to exist. From their various public statements, execs at Mozilla clearly believe that their current direction is a way to achieve sustainability. What they seem to be missing, though, is that their actions are increasingly alienating the users that they **do** have, in pursuit of users who are statistically unlikely to switch away from Chrome or Edge without some kind of _unique_ proposition. Perhaps I'm wrong and throwing more and more resources into the LLM firepit will pay off but, in the meantime, I don't want to have to continue to turn new stuff off every time there's a release. I want to be _excited_ about new features, not groaning because a context menu has been re-ordered to wedge in LLM related menu items. Although it's quite comfortable to use, moving to Waterfox wasn't quite as straightforward as I had been hoping. Even with Sync enabled, extension configuration and `localStorage` data doesn't come across. Even without changing browser, the apparent shortcomings of Sync are something that I'd love Mozilla to put resources into instead of AI. Now that I've crossed that hurdle, though, it _does_ make it even less likely that I'd move back to Firefox. I'm certainly less than keen to go through all that hassle again, so I'd need to be having quite significant issues. Note: If you're also using Waterfox, you can (and should) help support it's developer on BuyMeACoffee. * * * 1. When this launched, I quite liked the concept. In practice, though, the LLM translation is often unbearably slow and can be quite unreliable in terms of accuracy. ↩ 2. and, no, Brave was never a contender, even before they went all in on AI ↩ 3. Though this is entirely subjective - I can't really say _why_ I ended up preferring one over the other ↩ 4. To be fair, the wizard never suggested that extensions would come over ↩ 5. Either because I stick with the default settings or because they don't need configuring in the first place ↩ 6. Interestingly, this would also have been an issue with the extension if it had worked - it triggers on changes in tab, so I'd have still had to enumerate and visit sites ↩

In one form or another, Firefox Sync has been around for years (in fact, it's not _that_ far off being decade**s**...). It allows you to share tabs, history and bookmarks between browsers on different devices (though, frustratingly, it doesn't sync settings or extension config). Although sending that data via a 3rd party server might sound concerning, Mozilla are unable to see the content that's being synced because the data is end-to-end encrypted. All the same, I like to self host where possible (and, frankly, needed something to keep me out of trouble), so decided to look at the feasibility of self-hosting a sync server. This post talks about the process of setting the sync server up: it's been tested as working with Firefox, Waterfox and Firefox Mobile. It should work with other fireforks too. * * * ### Using an Upstream Project I didn't have to look far to find an implementation of the sync server, because Mozilla publish theirs. Unfortunately, they _don't_ publish much in the way of documentation on how to run it (to be fair, `syncserver-rs` is a Rust re implementation of their original which was pretty well documented). Luckily for me, though, someone had already done the hard work and developed a `docker-compose.yml` to stand up the service and it's dependencies. * * * ### First Deployment There _were_ a few things in the `docker-compose` that I didn't like (particularly the creation of two separate MariaDB instances) but as the repo hadn't been touched in nearly a year I decided to start by deploying without modification to ensure that it still actually worked: git clone https://github.com/porelli/firefox-sync.git cd firefox-sync ./prepare_environment.sh Enter FQDN for your Firefox sync server [firefox-sync.example.com]: fs.<my domain> Enter full path for the docker-compose file [/home/ben/docker_files/config/firefox-sync]: Listening port for syncstorage-rs [5000]: 8101 Max allowed users [1]: Docker user [ben]: I _did_ , however, change the volumes in `docker-compose` so that bind mounts were used instead of named volumes: diff --git a/docker-compose.yml b/docker-compose.yml index f9aa22f..e995788 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -45,7 +45,7 @@ services: timeout: 5s retries: 3 volumes: - - syncstorage-db:/var/lib/mysql + - /home/ben/docker_files/files/firefoxsync/syncstorage-db:/var/lib/mysql restart: unless-stopped tokenserver_db: @@ -64,7 +64,7 @@ services: timeout: 5s retries: 3 volumes: - - tokenserver-db:/var/lib/mysql + - /home/ben/docker_files/files/firefoxsync/tokenserver-db:/var/lib/mysql restart: unless-stopped tokenserver_db_init: @@ -84,7 +84,3 @@ services: MAX_USERS: ${MAX_USERS} DOMAIN: ${SYNCSTORAGE_DOMAIN} entrypoint: /db_init.sh - -volumes: - syncstorage-db: - tokenserver-db: I started the containers: docker compose up -d They all came up without complaint. * * * ### DNS and Reverse Proxy Next, I needed to set up DNS and configure my reverse proxy. I pointed `fs.<my domain>` at the box hosting the reverse proxy and then added some Nginx config: server { listen 80; root /usr/share/nginx/letsencryptbase; index index.php index.html index.htm; server_name fs.example.com; location / { return 301 https://$host$request_uri; add_header X-Clacks-Overhead "GNU Terry Pratchett"; } location /.well-known/ { try_files $uri 404; } } I invoked `certbot` to acquire a SSL certificate: certbot certonly \ --preferred-challenges http \ --agree-tos \ --email $MY_EMAIL \ --webroot \ -w /usr/share/nginx/letsencryptbase \ --rsa-key-size 4096 \ -n \ -d "fs.example.com" (actually that's only partly true - I have a wrapper which invokes `certbot` for me). Once `certbot` had acquired a certificate, I added some more Nginx config: server { listen 443; root /usr/share/nginx/letsencryptbase; index index.php index.html index.htm; server_name fs.example.com; ssl on; ssl_certificate /etc/letsencrypt/live/fs.example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/fs.example.com/privkey.pem; ssl_session_timeout 5m; # Handle ACME challenges etc location /.well-known/ { try_files $uri 404; } location /{ proxy_pass http://192.168.13.125:8101; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect off; proxy_read_timeout 120; proxy_connect_timeout 10; gzip off; add_header X-Clacks-Overhead "GNU Terry Pratchett"; client_max_body_size 10M; # Let's not open this to the world include '/etc/nginx/conf.d/auth.inc'; } } * * * ### Configuring Browsers #### Desktop Configuring a desktop browser to use a custom sync server is, on the face of it1, quite simple: * If you're already signed in, sign out * Go to `about:config` * Search for `identity.sync.tokenserver.uri` * Update from `https://token.services.mozilla.com/1.0/sync/1.5` to `https://<your domain>/1.0/sync/1.5` * Open the main menu * Click option to sign in * Sign in with existing Mozilla sync credentials If you tail the sync container's logs you should see something like this Jan 03 12:39:17.275 INFO {"ua.os.family":"Linux","ua.name":"Firefox","ua":"140.0","ua.browser.family":"Firefox","metrics_uid":"59319a658b89f87d5120c4a0f559be2c","ua.browser.ver":"140.0","ua.os.ver":"UNKNOWN","uri.path":"/1.0/sync/1.5","token_type":"OAuth","uid":"<redacted>","uri.method":"GET","first_seen_at":"1767443957267"} * * * #### Android Firefox Mobile (or Waterfox on Android) requires a slightly different approach * If you're already signed in, sign out * `Menu` -> `Settings` -> `About Firefox` * Tap the logo 5 times, you should get a notification `debug menu enabled` * Go back to `Settings` * At the top should be `Sync Debug`. Tap it * Tap `Custom Sync Server` * Enter `https://<your domain>/1.0/sync/1.5` * Tap `OK` * An option to stop Firefox will appear, tap it * Re-open Firefox * Sign back in to sync * * * ### Problems It _mostly_ seemed to work: when I signed into a new browser install, it started installing extensions etc. However, `Tabs from other devices` stayed resolutely empty. After poking around a bit and getting nowhere, I searched the web and found this ticket on the original project - sync broke back in June and has (apparently) stayed that way since. So, I updated my `docker-compose.yml` to pin to the tag published prior to the first broken one: services: syncstorage: image: ghcr.io/porelli/firefox-sync:syncstorage-rs-mysql-0.18.3-20250526T064429-linux-amd64 The container restarted cleanly, but my browsers looked unhappy: the sync wheel span and span with no logs showing up in the container. I decided to reconnect them, so I signed out of all of the browsers and then signed back in. Tab syncing started working between my laptops! * * * #### Which Server? Tab syncing was working, but shouldn't have been. Although the two browser installs were _clearly_ communicating, my sync server wasn't generating any logs. On one of the browsers, I went back to `about:config` and found that the setting had reverted: I was initially a little confused and concerned by this. Had Mozilla screwed something up which meant that the setting wouldn't persist? The cause turned out to be simpler than that: when you sign out of Sync, it resets this value to the default. It turns out that that's not the only time that it can get overridden. * * * #### Firefox Mobile: But You Said... At this point, I was almost ready to give up. I decided to leave the desktop browsers using Mozilla's sync service and opened Firefox mobile in order to cut it back over. I went through the same process as before: * Signed out of sync * Tapped the logo 5 times * Went to `Sync Debug` and tapped `Custom Sync Server`. It had originally been blank, so I blanked it and hit `OK` before restarting Firefox. When it came back up, I signed back into sync but my sync server's logs showed Firefox Mobile continuing to connect to it. I went back in and updated the custom URL to `https://token.services.mozilla.com/1.0/sync/1.5` before restarting again. Signing back in, once again, led to logs on my sync server. I rebooted the phone, no change. Whatever I did, the phone continued to try and use my sync server. Amongst the logs, though, I noticed that it was now successfully submitting tab information: { "ua":"145.0.2", "ua.name":"Firefox", "ua.os.family":"Linux", "ua.browser.ver":"145.0.2", "uri.path":"/1.5/4/storage/tabs?batch=true&commit=true", "uri.method":"POST", "ua.os.ver":"UNKNOWN", "ua.browser.family":"Firefox" } So I decided to give things one more shot and point the laptops back at my sync server. It worked! * * * ### Additional Notes Although this moves sync history onto infra that I control, it's not a complete solution: authentication is still performed using a Mozilla account. It _is_ possible to self-host Mozilla's auth servers and there _are_ projects which simplify that process, but it's really a couple of steps further than I needed. Speaking of authentication, by default the sync server is quite promiscuous: it'll handle syncing for anyone with a valid Mozilla account. The `firefox-sync` project worked around that by introducing the `MAX_USER` environment variable. It's set to 1 in the `docker-compose.yml` so, once you've registered, it shouldn't allow anyone else to do so. All the same, though, it would be prudent to enforce some kind of access control at the reverse proxy (for mine, access is limited to the LAN and my tailnet). * * * ### Backing Up The Data The data that's synced is encrypted, so backups are only _really_ useful for restoring the sync server if something happens to it. We can pull credentials from the environment file and then use `mariadb-dump` to dump a backup of the databases: # Load creds etc source .env # Backup the sync server docker exec -it \ firefox-sync-syncstorage_db-1 \ mariadb-dump \ -u "$MARIADB_SYNCSTORAGE_USER" \ --password="$MARIADB_SYNCSTORAGE_PASSWORD" \ "$MARIADB_SYNCSTORAGE_DATABASE" | gzip > sync_storage-`date +'%Y%M%d'`.sql.gz # Backup the token server docker exec -it \ firefox-sync-tokenserver_db-1 \ mariadb-dump \ -u "$MARIADB_TOKENSERVER_USER" \ --password="$MARIADB_TOKENSERVER_PASSWORD" \ "$MARIADB_TOKENSERVER_DATABASE" | gzip > tokenserver_storage-`date +'%Y%M%d'`.sql.gz We can also, out of idle curiousity, confirm that data does seem to be encrypted. First, we connect to the sync server database: docker exec \ -it firefox-sync-syncstorage_db-1 \ mariadb \ -u "$MARIADB_SYNCSTORAGE_USER" \ --password="$MARIADB_SYNCSTORAGE_PASSWORD" \ "$MARIADB_SYNCSTORAGE_DATABASE" Then we query an item select batch,payload from batch_upload_items limit 1; Which results in something like this As you'd hope, base64 decoding the ciphertext leads to a seemingly meaningless binary blob. * * * ### Conclusion Although this isn't something that the average Firefox user is going to need (or want) to do, the sync server is relatively easy to get up and running. There are some _definite_ oddities in Firefox's config behaviour, so moving away from default settings does bring increased potential for confusing issues down the line. But, having the data sync onto a system that I control means that I can do mundane things like _back it up_. It also, for what little it seems worth, means that my browser history isn't sat in a database which _might_ later be subject to Harvest Now, Decrypt Later. Most of all, though, it's given me something fun to fiddle about with this afternoon. At some point, I'll likely look at consolidating the two MariaDB servers into a single instance and maybe lift shift the entire stack into my K8S cluster. * * * 1. Should have known it's never that simple! ↩

I recently deployed Woodpecker CI. My original post talks about using CI jobs to periodically rebuild container images, but that's not the only thing that I've been using Woodpecker for. Almost all of my sites are built using static site generators (SSG): mostly Nikola with some sites using Hugo (Note: I also want to get around to playing about with Stefano Marinelli's BSSG). Deploying a static site isn't a particularly novel use of CI, but it's still something that I wanted to get up and running. This post describes using Woodpecker CI to build and deploy static sites before flushing the BunnyCDN cache. * * * ### Why? Deploying and managing via git is known as Gitops and brings a number of benefits (including auditability). In my professional life, I've used GitOps a **lot**. _Most_ of the benefits, though, aren't all that relevant to a blog that's only ever updated by a single person. The primary benefit, for me, lies in reducing the opportunity for a class of mistake that I didn't think I'd make... until I very nearly did. * * * #### Ooops I've talked, in various places, about my psuedo-blog. It's a private space where I blog for an audience of none (though I do sometimes promote posts up to this site). I've found that doing so can help to work through thoughts, decisions and (sometimes) feelings. It's not entirely unfiltered1, but writing in that private space helps with things that I either can't or don't want to discuss publicly. Late last year, I had quite a significant rant into it - letting off steam whilst working through something that had _really_ pissed me off. A day or two later, I drafted a post on a _different topic_ for `www.bentasker.co.uk` and started to manually publish: * Opened the Obsidian window * `Ctrl-A`, `Ctrl-C` * Switched to my terminal window, SSH'd into my host * Truncated the post file (which had the previous draft in) * Opened the post file, `Ctrl-v` * Saved and exited * Triggered `nikola build` As the build output flew by, there were a surprising number of warnings about Nikola not being able to resolve some magic links: You've probably guessed what had happened: I'd clicked on the _wrong Obsidian window_ and had pasted in the content of a post that was never meant for public consumption. I realised quickly enough to interrupt the process before it had updated my RSS feed (thus preventing my POSSE scripts from advertising my mistake across the internet). Advertised or not, though, the post itself had still been published. I had to _very carefully_ work through Nikola's output directory to make sure that I'd removed every trace of it (because the slug was different, it had been written out to a different URL than the post that I _meant_ to publish, so simply pasting the correct content in wasn't enough). All that prevented this from being a _very_ public fuck-up was luck. If I hadn't noticed the warnings, I'd only have realised when Telegram buzzed my phone. Moving to gitops makes this class of mistake less likely: there's no longer any copy & pasting of posts to screw up, I simply commit and push2. * * * ### Building A Hugo site One of the sites that I've hooked up to CI is built using Hugo. The workflow (saved in `.woodpecker/publish.yaml`) is configured to run on any push to the `main` branch: when: - event: push branch: main It's not **just** Hugo that CI needs to invoke, I also have a BASH script to scale images down to multiple sizes so that they can be injected into a `srcset` attribute: <img src="{{ .url }}" alt="{{ .alt | safeHTML }}" srcset="{{ .srcset }}" sizes="{{ .size }}" /> The script itself is pretty simple: #!/bin/bash set -e widths="600 480 250" mkdir -p static/images/scaled for img in static/images/*.jpg do fname=`basename "$img"` for width in $widths do new_fname=`echo "$fname" | sed "s/\./.$width./"` convert "$img" \ -sampling-factor 4:2:0 \ -resize ${width}x \ -quality 85% \ -interlace JPEG \ -colorspace RGB \ "static/images/scaled/$new_fname" done done In woodpecker, I have a pipeline step to install the necessary dependencies and then invoke my script: steps: - name: scale images image: cgr.dev/chainguard/wolfi-base commands: - apk -Uuv add imagemagick bash - tools/img_scale.sh Next, is a step to build the site by invoking Hugo: - name: build site image: hugomods/hugo:exts-0.134.2 commands: - hugo The pipeline then moves onto publishing and flushing * * * ### Building a Nikola Site My Nikola sites follow a similar pattern. Unlike the Hugo sites, the custom script that they invoke doesn't scale images and instead works around Obsidian not adding a leading slash to absolute paths: #!/bin/sh echo "Fixing image embeds" # The embeds we care about will look something like #  # We just need to inject a / grep -Rl '(images/' posts/ | while read -r post do sed -i 's~(images/~(/images/~g' "$post" done Because it's quite a simple task, the `Nikola` container image already has the necessary utilities, so doesn't require a separate pipeline step: when: - event: push branch: main steps: - name: build image: dragas/nikola:alpine commands: - ./scripts/fix_image_embeds.sh - nikola build * * * #### Publishing and Flushing CDN Caches Once a site has been built, it needs to be pushed to the hosting server. My first implementation of the publishing workflow was for my psuedo-blog, which doesn't sit behind a CDN and just needs to `rsync` the generated files to the relevant host. In Woodpecker I created a secret called `ssh_key` and pasted a (freshly generated) private key into it. The pipeline step pulls a base image, installs the necessary packages and then adds the SSH key before attempting to rsync the `output` directory up to the hosting server: - name: upload image: cgr.dev/chainguard/wolfi-base environment: REMOTE: www@internal-host:/mnt/nginx/psuedo_blog SOURCE: output SSH_KEY: from_secret: ssh_key commands: - apk -Uuv add rsync openssh - mkdir -p /root/.ssh - echo "$${SSH_KEY}" > "/root/.ssh/id_rsa" - chmod 0600 /root/.ssh/id_rsa - rsync -racv --delete -e 'ssh -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o StrictHostKeyChecking=no' $${SOURCE} $${REMOTE} I added the associated public key to the hosting server and things just worked. However, most of my sites sit behind BunnyCDN. Ages ago, I wrote a python script to purge a URL from Bunny's cache, so I decided to turn it into a Woodpecker CI Plugin allowing me to add a simple pipeline step for cache purges. The plugin just needs to be passed an API key and a list of the URLs to flush3: - name: Flush CDN Cache image: codeberg.org/bentasker/woodpecker-ci-bunnycdn-cache-flush settings: BUNNY_API_KEY: from_secret: bunny_api_key FLUSH_URLS: "https://www.bentasker.co.uk/posts/* https://www.bentasker.co.uk/" Having created this plugin, I decided to expand it so that it could also be used to rsync published files up3: - name: Publish image: codeberg.org/bentasker/woodpecker-ci-bunnycdn-cache-flush settings: LOAD_KEY: "y" SSH_KEY: from_secret: ssh_key DO_RSYNC: "y" RSYNC_REMOTE: sites@ext-host:/var/sites/bt RSYNC_PORT: 1322 RSYNC_SOURCE: public DEBUG: "n" DO_FLUSH: "n" - name: Flush CDN Cache image: codeberg.org/bentasker/woodpecker-ci-bunnycdn-cache-flush settings: BUNNY_API_KEY: from_secret: bunny_api_key FLUSH_URLS: "https://www.bentasker.co.uk/posts/* https://www.bentasker.co.uk/" You can find the plugin source on Codeberg. * * * ### Putting It All Together For a Nikola site, my YAML looks something like this when: - event: push branch: main steps: - name: build image: dragas/nikola:alpine commands: - ./scripts/fix_image_embeds.sh - nikola build - name: Publish image: codeberg.org/bentasker/woodpecker-ci-bunnycdn-cache-flush settings: LOAD_KEY: "y" SSH_KEY: from_secret: ssh_key DO_RSYNC: "y" RSYNC_REMOTE: sites@ext-host:/var/sites/bt RSYNC_PORT: 1322 RSYNC_SOURCE: public DEBUG: "n" DO_FLUSH: "n" - name: Flush CDN Cache image: codeberg.org/bentasker/woodpecker-ci-bunnycdn-cache-flush settings: BUNNY_API_KEY: from_secret: bunny_api_key FLUSH_URLS: "https://www.bentasker.co.uk/posts/* https://www.bentasker.co.uk/" - name: Flush CDN Cache image: codeberg.org/bentasker/woodpecker-ci-bunnycdn-cache-flush settings: BUNNY_API_KEY: from_secret: bunny_api_key FLUSH_URLS: "https://www.bentasker.co.uk/posts/* https://www.bentasker.co.uk/" * * * ### Conclusion I haven't moved everything over yet, but most of my sites can now be updated with no more effort than a `git push`. As well as being more convenient for me, the new flows also significantly reduce the likelihood of publishing content onto _the wrong site_. It also means that I have a meaningful audit history of changes to my site - if I do ever significantly screw up, putting things right should just be a `git revert` away. * * * 1. Even though I consider it a private space, I still end up self-censoring out of concern it might one day leak/be compromised ↩ 2. I could, of course, still write them in the wrong repo to begin with, but that's less likely ↩ 3. The idea being that it'll eventually support a mode where it figures out which URLs to flush for itself ↩↩

My PCP air rifle is a Weihrauch HW100 T. The rifle developed a slow leak: I could fill and use it, but over the course of a couple of days any stored pressure would be gone. This was problematic because leaving the gun empty for long periods can allow moisture to find its way into the cylinder. My leak appeared after I'd removed the cylinder for a depressurisation. I refilled it before re-attaching, so the seals will have gone from 1 to 200 bar in an instant (which is fine... until it isn't). This post describes dismantling the gun to replace various o-rings. It walks through checking the most likely culprits up to replacing the regulator seals and then resetting the regulator pressure. * * * ### Contents * Contents * First Check: Cylinder * Leaking Gun * Identifying The Leak Location * Disclaimer * Tools * Stock Removal * Brass Valve Seals * Regulator Seals Replacement * Pre-warning * Process * Adjusting The Regulator * Attaching a gauge * Dismantling to Adjust * Adjusting * Checking Shot Consistency * Re-Assembly * Monitoring * Contents * First Check: Cylinder * Leaking Gun * Identifying The Leak Location * Disclaimer * Tools * Stock Removal * Brass Valve Removal * Regulator Seals Replacement * Pre-warning * Process * Adjusting The Regulator * Attaching a gauge * Dismantling to Adjust * Adjusting * Checking Shot Consistency * Re-Assembly * Monitoring * * * ### First Check: Cylinder The _very_ first thing to identify was whether the leak was in the gun or the cylinder. I unscrewed the cylinder from the gun and charged it with air. There were now two options: * Leave the cylinder out for a couple of days and see whether the pressure still dropped * For a quicker check: tie a rubber glove (condoms work in a pinch) over each end and see whether they started to inflate. If the cylinder's air pressure gauge had dropped (or the rubber attachments started to inflate) it would have indicated that the leak was in the air cylinder (requiring different steps to those described here). * * * ### Leaking Gun My cylinder didn't lose pressure while it was off, indicating that the issue was one of the seals within the gun itself. The most likely (and helpfully, easiest to access) seal is the small rubber o-ring on the brass valve where the gun meets the cylinder: Sometimes it can be enough to add a little bit of (airgun safe!) grease to this. My hands were a little too big to get in there, so I dipped the end of a bamboo skewer in the grease and then carefully spread it around the o-ring. I screwed the cylinder back on and hoped the pressure would read the same the next day. * * * ### Identifying The Leak Location Unfortunately, I wasn't _quite_ that lucky and the pressure dropped just as it had before. It was time to do a small amount of disassembly to identify where the leak was coming from. You'll want the air cylinder pressurised and screwed in for this. * * * #### Disclaimer Before we start, a quick safety chat. PCP air rifles contain high pressure air - the recommended fill level for my HW100 is 200 bar (2900 PSI). That's _more than enough_ to cause significant harm. * Obstructing the flow with body parts can lead to air embolism * Obstructing something _propelled by_ that air can lead to puncture injuries. Either can have frighteningly severe consequences (up to and including death). The repairs described in this post are simple to perform but, if you're doing similar, the ultimate responsibility for your safety rests with **you**. I can't accept any liability for anything that happens. If you don't feel absolutely confident performing repairs, pay a qualified specialist instead. * * * #### Tools It's time to start talking tools and supplies. * Gas Leak detector spray (or soapy water) * 10mm socket and wrench * Allen Keys * Replacement Seal Kit * Moly Grease If things get as far as removing the regulator, you'll also want * An Airgun Chronograph * Regulator Setting Gauge * A pair of needle nosed pliers * * * #### Stock Removal The stock is help on by two allen headed screws, remove them: The stock slides down and off, leaving you with the action. On the underside of the block there is a line of three holes Facing these upwards and spraying with leak detector fluid can help identify where the leak is coming from: In my case, you can see that the hole on the left ended up blowing a big bubble. Air coming out from each of the holes indicates a different thing: Left | Middle | Right ---|---|--- Regulator Seals | Regulator test port seal | Brass Valve Seal If bubbles come out of the right, it indicates that the seal that we greased earlier is the issue. If the air isn't leaking out of any of these, the leak is further up the block - unfortunately, this post doesn't cover dealing with that (though we will be accessing some of those seals). The regulator seals consist of three o-rings, one of which is accessible by removing the brass valve, so I (optimistically) decided to try and get away with only replacing that. As a result, the first steps in this post are much the same as if the leak had showed on the righter-most hole hole. If you've got a regulator adjustment gauge, attach it and take a reading so that you know roughly what the set pressure was (remember that it may read a little lower as the result of the leak). * * * ### Brass Valve Seals The brass valve is made of, err, brass. In other words, it's very soft and you need to be careful not to scratch it with ill-fitting tools. The valve has a 10mm head on it, so use a socket and wrench to unscrew it until it comes out If you look down the hole again, you'll now see the head of the regulator and, around the edge, an o-ring I used a pointy ended spudger to hook the o-ring out. Bamboo skewers can also work, you just don't want anything too likely to scratch anything. Once you've got it out, find the replacement o-ring in your set, lightly grease and then drop it down the hole before (carefully) pushing into place. Next, use your finger nail to remove the o-ring from the front of the valve. If you need a tool, use a toothpick or similar to avoid scratching the brass: Push the new o-ring onto the end of the valve. Some HW100s apparently have an o-ring on the back of the valve too so, if you have one, replace that as well: Apply a thin layer of grease to both and then carefully screw the valve back into the block: Screw down until it's tight, but don't try and overtighten it: the block is made of aluminium (ergo, is soft) and the valve is brass, you don't want to be stripping threads. Screw the cylinder back into the block and repeat the gas leak fluid test. If you're lucky, there now won't be a leak. * * * ### Regulator Seals Replacement I _still wasn't_ that lucky. * * * #### Pre-warning Stripping down to do the regulator seals involves disturbing seals that are single use only: you'll definitely want to have a full seals kit on hand. We're also (obviously) going to be disturbing the regulator, which has a couple of consequences: * Your rifle could cross the legal power limit * Your shots could become less consistent, impacting accuracy * You might reduce the number of shots you can get between fills You _might_ not care about the first1 but will almost definitely care about the second and third. To proceed with this, you're going to want to make sure that you've got a chronograph (so you can check power output) and (ideally) a regulator servicing gauge. If you don't mind a bit (ok, _a lot_) of trial and error, you _can_ do without the regulator servicing gauge - it just means that you have to re-assemble, test with a chrono and then dismantle again to adjust the regulator. * * * #### Process Remove the air cylinder and then use your socket to loosen (but not yet remove) the brass valve. If, like me, you'd tried to get away without disturbing scope mounts you're now out of luck: they're going to need to come off. The trigger mechanism is held in place by two screws. There's one at the rear: And one at the front: Once these are removed, the entire trigger mechanism lifts up and out Next, we need to split the rear and front actions. These are held together by the two horizontal screws in the photo above. Slacken both off: Slide them both carefully out. You should now be able to slide the front block away from the rear action. Now we need to remove the breech block which is held in by two allen headed screws. There's a spring loaded valve under it, so cover the block with your thumb as you slacken the screws to prevent anything pinging off: Once it's removed you should be able to remove the breech block, the exhaust valve and the spring: If you look at the breech block you'll see an o-ring. This o-ring is single use so **must** be replaced - do that now while you remember If your leak didn't come out of any of the holes in the bottom of the block, there's also a white PTFE o-ring down that hole in the breech block that you should replace. We now have unimpeded access to the regulator adjustment nut. This requires a larger (5mm) allen key than the ones you've used so far: Unscrew and remove the nut, being careful to keep the nut facing upwards. If you look down the hole, you'll now see a pile of washers and (possibly) a little plastic cover. These are Belleville washers which function as a strong spring behind the regulator piston. Pull the plastic cover out and then put an allen key down the middle of the belleville washers to preserve position/order (although they may look flat, they're slightly cupped & order matters3). Very carefully tip them out and (leaving on the allen key) set safely aside. Now, unscrew and remove the brass valve (again :( ). If you look down the hole, you'll see the top of the regulator piston. It has a little spike on it - you should be able to grasp this with a pair of needle nose pliers and pull it out (if it doesn't want to come, to avoid damage, push it out backwards instead) Replace the o-ring (remembering to apply some grease) The piston sits within a sleeve. If you look in the back of the block you should see the back of that sleeve. Carefully press an allen key against it to push it forwards into the block. You should then be able to reach in and hook it out It should, hopefully, bring the other o-ring that we replaced earlier with it, meaning that you'll see two o-rings: You can re-use the o-ring that we replaced earlier (set it to one side for a minute) but replace the other and apply some grease. Carefully push the regulator piston back into the sleeve (making sure not to nick the o-ring on the piston): Note: when refitting, it won't matter which way the hole in the side is pointing - the block has a groove cut out so it can point in any direction. Put the sleeve, seal first, back in the front of the block and carefully press into place. Then put the second o-ring back in: Screw the brass valve back in place: Flip the block back over and put the washers back in, remembering to position the little plastic cover (if you had one) over the hole: Screw the adjuster back on top of the washers. Try to make it about as tight as it was when you first undid it (we'll adjust it later). Drop the exhaust valve and spring back into place: Put the breech block back on and screw it down. Screw the air cylinder back on, put the block upside down and use leak fluid to check whether you still have a leak. Happily, _finally_... I did not! If you've got an adjuster gauge, now's the time to adjust the regulator. If not, jump down to Re-Assembly. * * * ### Adjusting The Regulator To do this properly, you'll need a regulator gauge (like this one). _Ideally_ you want a digital one, but they're pretty expensive. The regulator's role is to limit the pressure of air in the plenum (the section between the regulator and the exhaust valve) so that each shot is propelled by the same amount of air resulting in much more consistent shots than achievable by an unregulated rifle (where output pressure is, more or less, equal to whatever's left in the bottle). Although the air cylinder is charged up to 200bar, the regulator is likely to be set at more like 90 bar. What this section describes isn't _technically_ tuning: to tune the gun you'd normally set the regulator and then adjust the hammer strike so that the two are in balance. However, the HW100 has an anti-tamper on the hammer, so what we're seeking to do here is to get the regulator set so that it's back in balance with the hammer, ensuring that: * Shots are consistent * The power remains consistently below the legal limit * We're not wasting air (and therefore shot count) for no benefit * * * #### Attaching a gauge First, unscrew the air cylinder to ensure that the block is depressurised. Remember the three holes on the bottom of the block? The middle one is the regulator test port. It's capped with a small torx grub screw and a ball-bearing. Unscrew and remove the grub screw. Carefully turn the block upside down to tip out the ball bearing. Sometimes you need to press it up and down slightly with a finger tip to work it out of the hole. The ball bearings _are_ known for getting stuck, so you might also find that it won't come out. This is **far** from the recommended approach (stripping the regulator back out so that you can push the ball bearing through), but I sat my block on top of a pile of rags and carefully screwed a partially charged cylinder in to blow the stuck ball bearing out into the rags. Your gauge probably came with a washer, pop that on the end and then screw it into the test port, nipping it up slightly with a spanner Carefully screw the air cylinder back in. If it pisses out air, you may need to tighten the gauge. However, hold your hand in the stream to check where the air is coming from because it might also be the front hole: I accidentally chewed up the brass valve o-ring when screwing the cylinder in and so had to replace it. If it's air tight, the gauge should start showing the post regulator pressure: The reading in this photo is the result of my eye-balling when re-assembling the gun and is set too high. * * * #### Dismantling to Adjust You now have a gauge attached to the bottom of your block: To adjust pressure you're going to need to be able to get at the regulator adjustment screw. We can, however, take a bit of a shortcut * Unscrew the scope mount screws on the front block (if you've a scope fitted) * Unscrew the long block joining screws. They won't come all the way out because the trigger assembly is in the way, but they _will_ undo fully You can then slide the two parts apart to access the regulator screw. * * * #### Adjusting The regulator pressure is adjusted by turning the regulator adjustment screw (there's no need to remove the breech block this time): Directions are * Tighten (right): increase pressure * Loosen (left): reduce pressure - never loosen it all the way (remember that there's 200bar of air on the other side!) However, loosening the screw **will not** lead to an immediate drop in pressure because the plenum will already be at the original pressure. You'll need to depressurise the block if you want to adjust down, so it's best to depressurise, adjust right down and then work back up whilst under pressure. Desirable regulator pressures are apparently between 80 - 90 bar. However, be aware that the setting has a direct impact on output power, the following are the muzzle velocities that I measured with 15.89 grain pellets whilst trying different settings: It's also worth remembering that different velocities will be achieved by different weights of pellet. Although it might be tempting to try and get the rifle to sit right on the legal limit, doing so risks coming in over if the Police were ever to run a test with their own choice of pellet. Once you've adjusted the regulator to where you think it needs to be, slide the two halves together again and tighten down the screws. Cock and fire the gun several times2 to check that the gauge reliably returns to where you've set it even after the regulator has settled. Leaving the gauge attached for now, go somewhere that you can safely fire the gun, attach a Chrono and put some pellets through, noting pellet weight and output velocities as you go. You can use an online calculator to convert feet per second to ft-lbs, or you can use the following equation (FPS)^2 x (weight-in-grains / 450240) So, for example: 578 fps ^2 = 334084 15.89gr / 450240 = 0.000035292 334084 * 0.000035292 = 11.7905 ft/lb You need to ensure that your highest observed measurement is below 12 ft/lb. If you're not happy with the output that your getting (or if it's wildly inconsistent), head back to the bench and adjust the regulator some more. Once you're happy * Unscrew the air cylinder to depressurise * Remove the gauge * Pop the ball bearing back in * Screw the grub screw back in (don't torque it down too much, otherwise you may get the ball bearing stuck - just beyond finger tight is fine) * Re-attach the stock * Screw the air cylinder back in * * * #### Checking Shot Consistency The final thing that you'll want to check is that the rifle is giving consistent shots: changes in output power affect the point of impact, so you want to make sure that the adjustment hasn't led to too high a rate of fluctuations (there will always be _some_). Take the gun somewhere that you can safely shoot and put some holes through a target. Your scope will almost certainly need re-zeroing, but the primary aim here is to make sure that pellets land in more or less the same place - any wild flyers could be a sign that your regulator pressure might be out of balance with the hammer strike4. If you're happy, move onto Monitoring. * * * ### Re-Assembly For safety's sake, remove the air cylinder again. Press the front and rear action back together and screw in the two long screws: Sit the trigger mechanism back in place and then tighten down it's screws: Slide the stock back into place and put the retaining screws back in: Screw the air cylinder back in and re-attach your scope. Your rifle is re-assembled! You'll want to attach a chrono and fire some pellets to verify that: * Output power is consistent * Output power is below the legal limit (you can use a calculator like this one to convert to lb/ft) If either isn't true, it points to a need to adjust the regulator. * * * ### Monitoring Once you're happy, fill the cylinder back up and make a note of the gauge reading: Leave the gun for a couple of days and t.. title: Weihrauch HW100 Gas Leak .. slug: weihrauch-hw100-air-leak .. date: 2025-12-26 14:57:00 UTC .. tags: documentation, howto, shooting, hw100, weihrauch .. category: documentation/shooting .. link: .. previewimage: /images/Documentation/hw100-regulator-seals/pressure_gauge.jpg .. description: Post describing an air leak that I had on my HW100 air rifle and how I resolved it. Walks through the process of changing the regulator seals and then using a gauge to help adjust the regulator pressure. .. type: text .. author: Ben Tasker .. status: draft My PCP air rifle is a Weihrauch HW100 T. The rifle developed a slow leak: I could fill and use it, but over the course of a couple of days any stored pressure would be gone. This was problematic because leaving the gun empty for long periods can allow moisture to find its way into the cylinder. My leak appeared after I'd removed the cylinder for a depressurisation. I refilled it before re-attaching, so the seals will have gone from 1 to 200 bar in an instant (which is fine... until it isn't). This post describes dismantling the gun to replace various o-rings. It walks through checking the most likely culprits up to replacing the regulator seals and then resetting the regulator pressure. * * * ### Contents * Contents * First Check: Cylinder * Leaking Gun * Identifying The Leak Location * Disclaimer * Tools * Stock Removal * Brass Valve Seals * Regulator Seals Replacement * Pre-warning * Process * Adjusting The Regulator * Attaching a gauge * Dismantling to Adjust * Adjusting * Checking Shot Consistency * Re-Assembly * Monitoring * Contents * First Check: Cylinder * Leaking Gun * Identifying The Leak Location * Disclaimer * Tools * Stock Removal * Brass Valve Removal * Regulator Seals Replacement * Pre-warning * Process * Adjusting The Regulator * Attaching a gauge * Dismantling to Adjust * Adjusting * Checking Shot Consistency * Re-Assembly * Monitoring * * * ### First Check: Cylinder The _very_ first thing to identify was whether the leak was in the gun or the cylinder. I unscrewed the cylinder from the gun and charged it with air. There were now two options: * Leave the cylinder out for a couple of days and see whether the pressure still dropped * For a quicker check: tie a rubber glove (condoms work in a pinch) over each end and see whether they started to inflate. If the cylinder's air pressure gauge had dropped (or the rubber attachments started to inflate) it would have indicated that the leak was in the air cylinder (requiring different steps to those described here). * * * ### Leaking Gun My cylinder didn't lose pressure while it was off, indicating that the issue was one of the seals within the gun itself. The most likely (and helpfully, easiest to access) seal is the small rubber o-ring on the brass valve where the gun meets the cylinder: Sometimes it can be enough to add a little bit of (airgun safe!) grease to this. My hands were a little too big to get in there, so I dipped the end of a bamboo skewer in the grease and then carefully spread it around the o-ring. I screwed the cylinder back on and hoped the pressure would read the same the next day. * * * ### Identifying The Leak Location Unfortunately, I wasn't _quite_ that lucky and the pressure dropped just as it had before. It was time to do a small amount of disassembly to identify where the leak was coming from. You'll want the air cylinder pressurised and screwed in for this. * * * #### Disclaimer Before we start, a quick safety chat. PCP air rifles contain high pressure air - the recommended fill level for my HW100 is 200 bar (2900 PSI). That's _more than enough_ to cause significant harm. * Obstructing the flow with body parts can lead to air embolism * Obstructing something _propelled by_ that air can lead to puncture injuries. Either can have frighteningly severe consequences (up to and including death). The repairs described in this post are simple to perform but, if you're doing similar, the ultimate responsibility for your safety rests with **you**. I can't accept any liability for anything that happens. If you don't feel absolutely confident performing repairs, pay a qualified specialist instead. * * * #### Tools It's time to start talking tools and supplies. * Gas Leak detector spray (or soapy water) * 10mm socket and wrench * Allen Keys * Replacement Seal Kit * Moly Grease If things get as far as removing the regulator, you'll also want * An Airgun Chronograph * Regulator Setting Gauge * A pair of needle nosed pliers * * * #### Stock Removal The stock is help on by two allen headed screws, remove them: The stock slides down and off, leaving you with the action. On the underside of the block there is a line of three holes Facing these upwards and spraying with leak detector fluid can help identify where the leak is coming from: In my case, you can see that the hole on the left ended up blowing a big bubble. Air coming out from each of the holes indicates a different thing: Left | Middle | Right ---|---|--- Regulator Seals | Regulator test port seal | Brass Valve Seal If bubbles come out of the right, it indicates that the seal that we greased earlier is the issue. If the air isn't leaking out of any of these, the leak is further up the block - unfortunately, this post doesn't cover dealing with that (though we will be accessing some of those seals). The regulator seals consist of three o-rings, one of which is accessible by removing the brass valve, so I (optimistically) decided to try and get away with only replacing that. As a result, the first steps in this post are much the same as if the leak had showed on the righter-most hole hole. If you've got a regulator adjustment gauge, attach it and take a reading so that you know roughly what the set pressure was (remember that it may read a little lower as the result of the leak). * * * ### Brass Valve Removal The brass valve is made of, err, brass. In other words, it's very soft and you need to be careful not to scratch it with ill-fitting tools. The valve has a 10mm head on it, so use a socket and wrench to unscrew it until it comes out If you look down the hole again, you'll now see the head of the regulator and, around the edge, an o-ring I used a pointy ended spudger to hook the o-ring out. Bamboo skewers can also work, you just don't want anything too likely to scratch anything. Once you've got it out, find the replacement o-ring in your set, lightly grease and then drop it down the hole before (carefully) pushing into place. Next, use your finger nail to remove the o-ring from the front of the valve. If you need a tool, use a toothpick or similar to avoid scratching the brass: Push the new o-ring onto the end of the valve. Some HW100s apparently have an o-ring on the back of the valve too so, if you have one, replace that as well: Apply a thin layer of grease to both and then carefully screw the valve back into the block: Screw down until it's tight, but don't try and overtighten it: the block is made of aluminium (ergo, is soft) and the valve is brass, you don't want to be stripping threads. Screw the cylinder back into the block and repeat the gas leak fluid test. If you're lucky, there now won't be a leak. * * * ### Regulator Seals Replacement I _still wasn't_ that lucky. * * * #### Pre-warning Stripping down to do the regulator seals involves disturbing seals that are single use only: you'll definitely want to have a full seals kit on hand. We're also (obviously) going to be disturbing the regulator, which has a couple of consequences: * Your rifle could cross the legal power limit * Your shots could become less consistent, impacting accuracy * You might reduce the number of shots you can get between fills You _might_ not care about the first1 but will almost definitely care about the second and third. To proceed with this, you're going to want to make sure that you've got a chronograph (so you can check power output) and (ideally) a regulator servicing gauge. If you don't mind a bit (ok, _a lot_) of trial and error, you _can_ do without the regulator servicing gauge - it just means that you have to re-assemble, test with a chrono and then dismantle again to adjust the regulator. * * * #### Process Remove the air cylinder and then use your socket to loosen (but not yet remove) the brass valve. If, like me, you'd tried to get away without disturbing scope mounts you're now out of luck: they're going to need to come off. The trigger mechanism is held in place by two screws. There's one at the rear: And one at the front: Once these are removed, the entire trigger mechanism lifts up and out Next, we need to split the rear and front actions. These are held together by the two horizontal screws in the photo above. Slacken both off: Slide them both carefully out. You should now be able to slide the front block away from the rear action. Now we need to remove the breech block which is held in by two allen headed screws. There's a spring loaded valve under it, so cover the block with your thumb as you slacken the screws to prevent anything pinging off: Once it's removed you should be able to remove the breech block, the exhaust valve and the spring: If you look at the breech block you'll see an o-ring. This o-ring is single use so **must** be replaced - do that now while you remember If your leak didn't come out of any of the holes in the bottom of the block, there's also a white PTFE o-ring down that hole in the breech block that you should replace. We now have unimpeded access to the regulator adjustment nut. This requires a larger (5mm) allen key than the ones you've used so far: Unscrew and remove the nut, being careful to keep the nut facing upwards. If you look down the hole, you'll now see a pile of washers and (possibly) a little plastic cover. These are Belleville washers which function as a strong spring behind the regulator piston. Pull the plastic cover out and then put an allen key down the middle of the belleville washers to preserve position/order (although they may look flat, they're slightly cupped & order matters3). Very carefully tip them out and (leaving on the allen key) set safely aside. Now, unscrew and remove the brass valve (again :( ). If you look down the hole, you'll see the top of the regulator piston. It has a little spike on it - you should be able to grasp this with a pair of needle nose pliers and pull it out (if it doesn't want to come, to avoid damage, push it out backwards instead) Replace the o-ring (remembering to apply some grease) The piston sits within a sleeve. If you look in the back of the block you should see the back of that sleeve. Carefully press an allen key against it to push it forwards into the block. You should then be able to reach in and hook it out It should, hopefully, bring the other o-ring that we replaced earlier with it, meaning that you'll see two o-rings: You can re-use the o-ring that we replaced earlier (set it to one side for a minute) but replace the other and apply some grease. Carefully push the regulator piston back into the sleeve (making sure not to nick the o-ring on the piston): Note: when refitting, it won't matter which way the hole in the side is pointing - the block has a groove cut out so it can point in any direction. Put the sleeve, seal first, back in the front of the block and carefully press into place. Then put the second o-ring back in: Screw the brass valve back in place: Flip the block back over and put the washers back in, remembering to position the little plastic cover (if you had one) over the hole: Screw the adjuster back on top of the washers. Try to make it about as tight as it was when you first undid it (we'll adjust it later). Drop the exhaust valve and spring back into place: Put the breech block back on and screw it down. Screw the air cylinder back on, put the block upside down and use leak fluid to check whether you still have a leak. Happily, _finally_... I did not! If you've got an adjuster gauge, now's the time to adjust the regulator. If not, jump down to Re-Assembly. * * * ### Adjusting The Regulator To do this properly, you'll need a regulator gauge (like this one). _Ideally_ you want a digital one, but they're pretty expensive. The regulator's role is to limit the pressure of air in the plenum (the section between the regulator and the exhaust valve) so that each shot is propelled by the same amount of air resulting in much more consistent shots than achievable by an unregulated rifle (where output pressure is, more or less, equal to whatever's left in the bottle). Although the air cylinder is charged up to 200bar, the regulator is likely to be set at more like 90 bar. What this section describes isn't _technically_ tuning: to tune the gun you'd normally set the regulator and then adjust the hammer strike so that the two are in balance. However, the HW100 has an anti-tamper on the hammer, so what we're seeking to do here is to get the regulator set so that it's back in balance with the hammer, ensuring that: * Shots are consistent * The power remains consistently below the legal limit * We're not wasting air (and therefore shot count) for no benefit * * * #### Attaching a gauge First, unscrew the air cylinder to ensure that the block is depressurised. Remember the three holes on the bottom of the block? The middle one is the regulator test port. It's capped with a small torx grub screw and a ball-bearing. Unscrew and remove the grub screw. Carefully turn the block upside down to tip out the ball bearing. Sometimes you need to press it up and down slightly with a finger tip to work it out of the hole. The ball bearings _are_ known for getting stuck, so you might also find that it won't come out. This is **far** from the recommended approach (stripping the regulator back out so that you can push the ball bearing through), but I sat my block on top of a pile of rags and carefully screwed a partially charged cylinder in to blow the stuck ball bearing out into the rags. Your gauge probably came with a washer, pop that on the end and then screw it into the test port, nipping it up slightly with a spanner Carefully screw the air cylinder back in. If it pisses out air, you may need to tighten the gauge. However, hold your hand in the stream to check where the air is coming from because it might also be the front hole: I accidentally chewed up the brass valve o-ring when screwing the cylinder in and so had to replace it. If it's air tight, the gauge should start showing the post regulator pressure: The reading in this photo is the result of my eye-balling when re-assembling the gun and is set too high. * * * #### Dismantling to Adjust You now have a gauge attached to the bottom of your block: To adjust pressure you're going to need to be able to get at the regulator adjustment screw. We can, however, take a bit of a shortcut * Unscrew the scope mount screws on the front block (if you've a scope fitted) * Unscrew the long block joining screws. They won't come all the way out because the trigger assembly is in the way, but they _will_ undo fully You can then slide the two parts apart to access the regulator screw. * * * #### Adjusting The regulator pressure is adjusted by turning the regulator adjustment screw (there's no need to remove the breech block this time): Directions are * Tighten (right): increase pressure * Loosen (left): reduce pressure - never loosen it all the way (remember that there's 200bar of air on the other side!) However, loosening the screw **will not** lead to an immediate drop in pressure because the plenum will already be at the original pressure. You'll need to depressurise the block if you want to adjust down, so it's best to depressurise, adjust right down and then work back up whilst under pressure. Desirable regulator pressures are apparently between 80 - 90 bar. However, be aware that the setting has a direct impact on output power, the following are the muzzle velocities that I measured with 15.89 grain pellets whilst trying different settings: It's also worth remembering that different velocities will be achieved by different weights of pellet. Although it might be tempting to try and get the rifle to sit right on the legal limit, doing so risks coming in over if the Police were ever to run a test with their own choice of pellet. Once you've adjusted the regulator to where you think it needs to be, slide the two halves together again and tighten down the screws. Cock and fire the gun several times2 to check that the gauge reliably returns to where you've set it even after the regulator has settled. Leaving the gauge attached for now, go somewhere that you can safely fire the gun, attach a Chrono and put some pellets through, noting pellet weight and output velocities as you go. You can use an online calculator to convert feet per second to ft-lbs, or you can use the following equation (FPS)^2 x (weight-in-grains / 450240) So, for example: 578 fps ^2 = 334084 15.89gr / 450240 = 0.000035292 334084 * 0.000035292 = 11.7905 ft/lb You need to ensure that your highest observed measurement is below 12 ft/lb. If you're not happy with the output that your getting (or if it's wildly inconsistent), head back to the bench and adjust the regulator some more. Once you're happy * Unscrew the air cylinder to depressurise * Remove the gauge * Pop the ball bearing back in * Screw the grub screw back in (don't torque it down too much, otherwise you may get the ball bearing stuck - just beyond finger tight is fine) * Re-attach the stock * Screw the air cylinder back in * * * ##### Checking Shot Consistency The final thing that you'll want to check is that the rifle is giving consistent shots: changes in output power affect the point of impact, so you want to make sure that the adjustment hasn't led to too high a rate of fluctuations (there will always be _some_). Take the gun somewhere that you can safely shoot and put some holes through a target. Your scope will almost certainly need re-zeroing, but the primary aim here is to make sure that pellets land in more or less the same place - any wild flyers could be a sign that your regulator pressure might be out of balance with the hammer strike4. If you're happy, move onto Monitoring. * * * ### Re-Assembly For safety's sake, remove the air cylinder again. Press the front and rear action back together and screw in the two long screws: Sit the trigger mechanism back in place and then tighten down it's screws: Slide the stock back into place and put the retaining screws back in: Screw the air cylinder back in and re-attach your scope. Your rifle is re-assembled! You'll want to attach a chrono and fire some pellets to verify that: * Output power is consistent * Output power is below the legal limit (you can use a calculator like this one to convert to lb/ft) If either isn't true, it points to a need to adjust the regulator. * * * ### Monitoring Once you're happy, fill the cylinder back up and make a note of the gauge reading: Leave the gun for a couple of days and then check back to verify that the reading is the same. * * * 1. Though you should, however low the likelihood of getting caught, a firearms charge isn't something you want to be facing ↩↩ 2. Although in a springer or Co2 gun this would be considered "dry firing", it isn't on a PCP: what matters on a PCP is that the valve is under pressure - you should never fire a PCP that isn't pressurised. ↩↩ 3. If you do accidentally spill the washers, they need to be reassembled with two washers facing in the same direction, faced by two in the opposite (i.e. `(())(())`) ↩↩ 4. Obviously you'll also need to factor in the wind and your own accuracy etc. ↩↩

Last weekend, I decided to stand up Woodpecker CI so that I could have a play around with it. In my working life, I've been exposed to a (horrifying) range of CI/CD systems but (perhaps as a consequence) have never really felt much desire to run anything similar at home. But, I was in the mood to play around with something and this crossed my mind first. Most of my projects live in a self-hosted Gitlab instance, so I needed to hook Woodpecker up to that. This post talks about deployment and experimentation, including automating rebuild of some container images. * * * ### Install #### Gitlab Pre-Config Woodpecker doesn't have it's own authentication system, instead relying on the VCS system that it's connected to (forges in Woodpecker parlance). As a result, it needs to be provided with an OAuth application secret generated within the forge. To create those, I logged into Gitlab and went `Settings` -> `Applications` -> `Add new Application`. I also needed to login as admin to permit Gitlab to make requests to Woodpecker's LAN IP (`Admin` -> `Settings` -> `Network`): * * * #### Deployment Woodpecker CI pipeline steps run within Docker containers so, although it's possible to do a native install of Woodpecker, it's really much easier to spin it up with docker compose. I only wanted Woodpecker to be reachable from within the LAN, so I didn't bother allocating a subdomain to it1. The deployment consists of two components * Woodpecker server: the thing that API requests are made against (as well as serving the web interface) * The agent: a container which spins up worker containers. This can run on a different machine (and there can be multiple of them) Communications between the agent and the server are authenticated with a shared secret. To generate a suitable secret, I ran: openssl rand -hex 32 My `docker-compose` entries looked like this: woodpecker-server: image: woodpeckerci/woodpecker-server:v3 container_name: woodpecker-server ports: - 8000:8000 # Make the RPC port available to agents deployed # on other boxes - 9000:9000 volumes: - /srv/files/woodpecker/data:/var/lib/woodpecker/ environment: # Allow registration (users have to be a user in # the forge anyway) - WOODPECKER_OPEN=true # I used an IP because I don't care about external # access. - WOODPECKER_HOST=http://192.168.13.25:8000 # Use Gitlab - WOODPECKER_GITLAB=true # Provide the client id and secret # generated for the application in Gitlab - WOODPECKER_GITLAB_CLIENT=<redacted> - WOODPECKER_GITLAB_SECRET=<redacted> # Use your real gitlab url here - WOODPECKER_GITLAB_URL=https://gl.example.com # Provide the secret generated with # openssl rand -hex 32 # # Note: the same secret must be provided to # the agent - WOODPECKER_AGENT_SECRET=<redacted> woodpecker-agent: image: woodpeckerci/woodpecker-agent:v3 container_name: woodpecker-agent command: agent restart: always depends_on: - woodpecker-server volumes: - /srv/files/woodpecker/agent:/etc/woodpecker - /var/run/docker.sock:/var/run/docker.sock environment: # Because they're in the same docker network # we can use container name # # If this were on another box, we'd need to # use the system name/IP - WOODPECKER_SERVER=woodpecker-server:9000 # This must be the same secret as provided to # the server - WOODPECKER_AGENT_SECRET=<redacted> After saving the file, I brought the containers up: docker compose up -d The server instance immediately crashed out: > {"level":"warn","time":"2025-12-14T17:26:37Z","message":"no sqlite3 file found, will create one at '/var/lib/woodpecker/woodpecker.sqlite'"} > > {"level":"error","error":"unable to open database file: no such file or directory","time":"2025-12-14T17:26:37Z","message":"error running server"} It's supposed to create a database at startup but seemingly wasn't able to. I checked the upstream dockerfile and noted that it ran under a user with UID of 1000, so I updated permissions on the local directory sudo chown -R 1000:1000 /srv/files/woodpecker/data/ docker compose up -d The container came up cleanly this time. Hitting the Web interface (port 8000) prompted me to log in with Gitlab: It was a little slow, but logging in worked. * * * ### What Now? * * * #### An Image Build Pipeline I recently moved our music to Gonic and, as part of that, built a Wolfi based container image to sync stars between Gonic and Last.FM. Wolfi is a rolling-release (un)distro benefiting from regular CVE remediation2, so there's tangible benefit to periodically rebuilding container images that are based upon it. This seemed like a good place to start. I added the Repo to Woodpecker My git repo _literally_ consisted of a `Dockerfile` and a README, so my intention was to use the docker-buildx plugin to build the image. In the repo, I created `.woodpecker/build.yaml`: when: - event: push branch: main - event: cron cron: Daily steps: - name: publish image: woodpeckerci/plugin-docker-buildx settings: platforms: linux/amd64 repo: registry.example.com/utilities/gonic-last-fm-sync-docker registry: registry.example.com tags: latest Committing and pushing this triggered CI, which _immediately_ failed: The error message is pretty self explanatory but, previous versions of Woodpecker automatically considered the plugin privileged. That changed relatively recently. The solution was to add an environment variable to `woodpecker-server` and restart it: - WOODPECKER_PLUGINS_PRIVILEGED=woodpeckerci/plugin-docker-buildx I went to Woodpecker's web interface and hit `Restart` on the failed job. It failed again... I thought I'd broken something, but a bit of searching around suggested that this was a bit of a nonsense error and that the "fix" was actually just to push a new commit. So I did that. The job got further, this time, but failed again + git fetch --no-tags --depth=1 --filter=tree:0 origin +cb7487b3d3b07c4e77392522c1a97512a9a5a8d7: fatal: could not read Username for 'https://gl.example.com': No such device or address exit status 128 The error made it clear that there was some authentication required, but I _thought_ that that was all supposed to be magically handled in the background. The issue turned out to be the gitlab project's visibility setting: The result of this was that Woodpecker could see the repo in API calls but didn't know that it needed to authenticate. If the visibility had been `Private` or `Public`, it would have just worked. I also needed to turn on support for HTTPS cloning3 (`Admin` -> `Settings` -> `General`): To ensure it could handle `Internal` repos, I added a new env variable to `woodpecker-server`: - WOODPECKER_AUTHENTICATE_PUBLIC_REPOS=true After restarting the container and pushing a new commit, CI built a new image and pushed it to my custom registry. Finally, I logged into Woodpecker's UI and created a cron called `Daily`: Every night since, the image has been rebuilt and pushed. * * * ##### Other Registries I won't go into as much depth here because most of it is the same regardless of the registry that you're pushing to. I decided to enable periodic rebuilds for some of my other images, but those are published on external registries: * Docker Hub * Github Container Registry * Codeberg Registry The thing that each of these has in common, of course, is that they require authentication to push. Unsurprisingly, the build plugin has support for auth. I created secrets in Woodpecker (the option is under the settings cog in each repo): Secrets can then be referenced under the `settings` attribute: steps: - name: publish image: woodpeckerci/plugin-docker-buildx settings: repo: ghcr.io/bentasker/soliscloud-inverter-control registry: ghcr.io platforms: linux/amd64,linux/arm64/v8 tags: latest username: from_secret: github_username password: from_secret: github_token The only real difference for Docker Hub is that you store a password rather than a token. * * * ##### Additional Settings There are a couple of additional settings that I've since used which aren't mentioned above * `auto_tag`: enables automatic tag calculation. If I push a (git) tag of `v1.1.1`, CI will automatically tag the container with `v1.1.1`, `v1.1`, `v1` along with whatever I've specified in `tags` * `when`: Allows you to filter events at a per-pipeline step level. I've used this to have one build definition but change the destination tags based on whether the build is the result of a Cronjob or a push. There's an example of using both in my soliscloud inverter control repo (that repo automatically mirrors into my Gitlab, allowing Woodpecker to act upon it). * * * ##### Automatically Rolling Containers Although Woodpecker was periodically rebuilding images, the systems _consuming_ those images still needed to be configured to periodically pull new versions. `docker compose` _does_ have support for time based pull policies: * `daily`: Compose checks the registry for image updates if the last pull took place more than 24 hours ago. * `weekly`: Compose checks the registry for image updates if the last pull took place more than 7 days ago. * `every_<duration>`: Compose checks the registry for image updates if the last pull took place before `<duration>`. Duration can be expressed in weeks (`w`), days (`d`), hours (`h`), minutes (`m`), seconds (`s`) or a combination of these. However, although these _sound_ ideal, it appears that they only apply when containers are restarted or recreated, so extra orchestration would still be required. Instead, I looked at using Watchtower. However, my initial attempts to use that failed because the embedded docker client was too old: Error response from daemon: client version 1.25 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version" That message led me to a Github issue full of people complaining of the same thing and noting that the project seemed to be unmaintained (since then, the project has actually been archived and a goodbye note posted - I assume someone went and bugged the maintainer about the breakage). Although the Github issue contained a workaround, I decided that I wanted something that's actively maintained and so settled on Nick Fedor's fork. The box that runs `gonic-lastfm-sync` also runs some other containers that I didn't want automatically restarting. Because I also run Rancher's k3d on that system, some of those containers can't easily have custom labels added to them. Although it's possible to pass `watchtower` an explicit list of containers to monitor, I decided to rely on labels instead (the idea being that I _should_ be less likely to forget when making changes). I added a label to `gonic_sync`: gonic_sync: restart: always image: registry.example.com/utilities/gonic-last-fm-sync-docker:latest container_name: gonic-lastfm-sync labels: - "com.centurylinklabs.watchtower.enable=true" environment: - GONIC_GONIC_USERNAME=ben volumes: - /home/ben/docker_files/files/gonic/data:/data I then stood `watchtower` up with an environment variable to tell it to only restart appropriately labelled containers: watchtower: image: nickfedor/watchtower container_name: watchtower environment: # Only restart containers with enable set # to true - WATCHTOWER_LABEL_ENABLE=true # This is actually the default # but I don't want to have to remember that - WATCHTOWER_POLL_INTERVAL=86400 volumes: - /var/run/docker.sock:/var/run/docker.sock * * * ### Telegraf Everything seemed to be up and running, but I realised that there _was_ a small issue (in fact, more _a risk_ , really). I use Telegraf and InfluxDB to monitor my kit. Although InfluxDB 3 significantly reduces cardinality concerns, I'm still running the older versions which can suffer more with extremely high cardinality datasets. Woodpecker CI creates a dedicated container for each pipeline step, using a UUID as part of the container name. So that I can monitor resource usage, I have Telegraf's docker input plugin configured to collect metrics: [[inputs.docker]] endpoint = "unix:///var/run/docker.sock" timeout = "5s" interval = "5m" This resulted in series appearing in InfluxDB with quite meaningless container names: The metrics themselves _weren't_ meaningless: I **wanted** to be able to see what resources CI was consuming, but I also didn't want (or need) an individual series per pipeline step invocation. I _definitely_ didn't need any cardinality concerns that might follow as a result. So, I decided to use the Starlark processor plugin to squash those container names down into a placeholder. For those who aren't familiar with Starlark, it's essentially a dialect of Python: def apply(metric): # Rewrite worker container names if "container_name" in metric.tags and metric.tags["container_name"].startswith("wp_"): metric.tags["container_name"] = "woodpecker_worker_container" # Also strip the wp_uuid tag if "wp_uuid" in metric.tags: metric.tags.pop("wp_uuid") return metric The entry in Telegraf's config file looked like this: [[processors.starlark]] # jira-projects/LAN#248 # Don't let woodpecker ephemeral containers # blow up cardinality namepass = ["docker*"] source = ''' def apply(metric): if "container_name" in metric.tags and metric.tags["container_name"].startswith("wp_"): metric.tags["container_name"] = "woodpecker_worker_container" # Also strip the wp_uuid tag if "wp_uuid" in metric.tags: metric.tags.pop("wp_uuid") return metric ''' The next time that a CI job ran, the new placeholder name appeared in InfluxDB: As a solution, this really isn't perfect: InfluxDB uses tag values to build the series key and multiple writes into the same series with the same timestamp will overwrite one another. This change removed a unique identifier. While that reduces cardinality growth, doing so also increases the potential for collision if two CI jobs run at the same time. However, there are probably still enough distinguishing tags left, including: * `container_image`: the image being used by that step (e.g. `woodpeckerci/plugin-docker-buildx`) * `wp_step`: The name given to the step in YAML (e.g. `publish`) For a collision to occur, CI would need to be running exactly the same container image, with exactly the same step name in two jobs at the same time. That's not impossible but (at the scale that I'm using it) _reasonably_ improbable. If I ever reached the scale that it was more likely, distributing jobs across hardware would help (because the `hostname` tag would then differ). They're really not the most interesting of graphs, but the metrics allow me to visualise the RAM, CPU and Network that pipeline steps are consuming: * * * ### Conclusion Going into this, I think I overestimated how complex hooking Woodpecker CI up would be. Although I ran into some (quite minor) headaches, it really didn't take long at all, leaving me to figure out what I was actually _going to do_ with the system. Since then, I've set up automated (and scheduled) rebuilds of some container images, pushing the result to a range of container registries. I've also set up git-ops for one of my Static Sites but, as this post is already quite long, I'll probably write about that separately. One of the other benefits that I haven't mentioned so far, is that it's allowed me to (easily) start cross building container images. With no extra hassle for me, images like soliscloud-inverter-control can now be run on ARM as well as on x86_64. So far, the system seems to be reliable and dependable4 although I'm not _exactly_ pushing it hard. * * * 1. I've since decided that this was probably a mistake, so may well change it over the Xmas period ↩ 2. Disclosure: I work for Chainguard but I'm not trying to sell you anything and any views expressed here are my own and not necessarily that of my employer ↩ 3. I'm not actually sure if this was off by default or if I've previously disabled it ↩ 4. I _may_ have joked to someone this week that Github's 39% reduction in Actions prices was as a result of them tying pricing to reliability and performance. ↩

We have _almost_ always self-hosted our music collection and, for a little over a decade now, have been streaming music from a self-hosted instance of Subsonic. However, that install is now _positively ancient_ : the most recent release of Subsonic was cut in November 2019 and is susceptible to things like the log4j vulnerability (I mitigated by locking down access at the reverse proxy). I've periodically looked for alternatives but (until recently) have never quite found anything that we were able to make the jump to. Ever since Subsonic went closed source, there have been various forks (Airsonic _et al_), but few seem to have stayed in development for very long (and are all, of course, heavy Java apps). Recently, though, I stumbled across Gonic. Rather than being a fork, Gonic is a Subsonic _compatible_ server, which meant that I could still (more or less) just point our existing players at it. This post talks about migrating our music collection from Subsonic to Gonic. * * * #### History of Our Music The timeline of our digital music collection looks something like this: I think that there was also a brief period where I was using Realplayer (RealOne at the time) instead of the original Windows Media Player. Essentially, our collection started off as a bunch of locally stored audio files, courtesy of ~~Kazaa~~ ~~Limewire~~ ripping CDs. Our short stint of having to use (the _awful_) SonicStage for our MP3 players is best left buried in the sands of time. As time went by, we moved from playing files stored on the same system to streaming those same files over a network - first from Google Play Music and then from a self-hosted Subsonic instance. I never _quite_ got onto the Spotify bandwagon: I tried (and liked) it when they first came to the UK, but was put off by them requiring a paid subscription to be able to use it on Linux (and/or my phone)1. * * * ### Alternatives There are a _bunch_ of self-hosted music solutions out there, but the following came particularly close to being chosen. * * * #### Navidrome I first looked at Navidrome about a year ago. Navidrome doesn't use a folders heirachy and, instead, organises the collection by looking at how tracks are tagged (there are no plans to change this). This seemed like a reasonable position to take, but after experimenting, I found that our collection needed more work than I was willing or able to invest to straighten all of the tags out (I kept running into annoying edge cases, like some tracks having `Album Artist` set to the name of the production company). It's a pity, because I really like the interface (have a look at the demo site, it's nice!). * * * #### Funkwhale Funkwhale is Fediverse enabled, meaning that users can also explore music that others have shared. I _really_ like the idea and, so, was quite drawn to it. There are Funkwhale instances with open registration, but I didn't want to be tied to someone else's infra: the whole point is that our collection should always be under _our_ control. Although it's possible to self-host Funkwhale, it has some infra level dependencies - at a minimum it needs Postgres and Redis. I've been working to get Postgres _back out_ of my LAN, so didn't really want to add another dependency on it. * * * ### Gonic After much exploration, I settled on Gonic. Unlike some of the other options its web interface is quite simple and does not let you explore or play media: I've a vague feeling that, when I was looking last year, I dismissed Gonic because it felt like a big step back from the functionality provided by Subsonic's interface: However, it turned out that the only person who actually uses that interface is _me_. Even then, I only really use it to trigger library updates (which Gonic's interface _does_ support). Actual playback primarily occurs via the following: * Jamstash * Dsub * My music kiosk Most importantly, Gonic exposes a Subsonic compatible API so we could continue to use the same players. * * * #### Deployment Gonic is written in Go, so can easily be compiled and run natively. However, I now tend to deploy containers (primarily because it makes it easier to lift & shift between hardware). I started by creating directories to use as persistent storage mkdir gonic mkdir gonic/playlists mkdir gonic/cache mkdir gonic/podcasts mkdir gonic/data I _didn't_ create a directory for music: that lives on an NFS share which was already mounted elsewhere on the host (the `subsonic` container lived on the same box). I added the following to my `docker-compose.yml`: gonic: restart: always image: sentriz/gonic:latest container_name: gonic ports: - 4747:80 environment: - GONIC_SCAN_AT_START_ENABLED=true - GONIC_SCAN_INTERVAL=1440 - GONIC_MUSIC_PATH=/mnt/Music-NAS/Albums_Sorted volumes: - /mnt/Music-NAS/:/mnt/Music-NAS:ro - /home/ben/docker_files/gonic/data:/data - /home/ben/docker_files/gonic/playlists:/playlists - /home/ben/docker_files/gonic/cache:/cache - /home/ben/docker_files/gonic/podcasts:/podcasts There's quite an important note here: by default, Gonic expects to find music under `/music`, but I've overridden that. I _could_ have bound `/mnt/Music-NAS` to `/music` (in fact, I originally _did_) but the prefix became quite important when importing playlists from Subsonic (more on that below). I started the container: docker compose up -d Gonic's web interface came up almost immediately and, in the background, `gonic` began scanning our music library. While waiting for the scan to finish, I changed the admin creds (the default is `admin`/`admin`) and created an unprivileged user. After logging in as my user, I hooked Gonic up to Last.fm and ListenBrainz: This means that the server will automatically scrobble - something that, apparently, I haven't done since 2010!2 In order to provide external players with access, I configured my reverse proxy to point to Gonic and acquired a SSL cert from LetsEncrypt: upstream gonic { server 192.168.11.145:4747 weight=1; keepalive 5; } server { listen 443 ssl; server_name gonic.example.com; root /usr/share/nginx/letsencryptbase; index index.php index.html index.htm; ssl_certificate /etc/letsencrypt/live/gonic.example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/gonic.example.com/privkey.pem; ssl_session_timeout 5m; location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; proxy_pass http://gonic; # Force use of upstream keepalives proxy_http_version 1.1; proxy_set_header Connection ""; proxy_redirect http:// https://; proxy_buffers 16 16k; proxy_buffer_size 16k; add_header X-Clacks-Overhead "GNU Terry Pratchett"; add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload" ; } } Once the library index had finished (it took about 10 minutes), I reconfigured Dsub on my phone to point to the Gonic server and attempted to play some music - it worked! * * * #### Playlist Import Although our music collection was there, our playlists still needed copying across. Subsonic doesn't expose a way to bulk export playlists, however the web interface does provide a way to do them one by one. If you click into a playlist there's an `Export` option at the top: Clicking this downloads a m3u8 format playlist. I had worried that playlists might rely on Subsonic specific track IDs, but they turned out to be _much_ simpler than that: #EXTM3U /mnt/Music-NAS/Albums_Sorted/Semblant/Obscura (2020)/03 - Dethrone the Gods, Control the Masters (Legacy of Blood, Pt. IV).mp3 /mnt/Music-NAS/Albums_Sorted/Semblant/Lunar Manifesto/01 Incinerate.mp3 /mnt/Music-NAS/Albums_Sorted/Semblant/Lunar Manifesto/02 Dark of the Day.mp3 /mnt/Music-NAS/Albums_Sorted/The Agonist/Prisoners (2012)/05. Panophobia.mp3 /mnt/Music-NAS/Albums_Sorted/Alien Weaponry/Tu/05 Kai Tangata.mp3 /mnt/Music-NAS/Albums_Sorted/Arch Enemy/Will To Power (2017)/06. Reason to Believe.mp3 /mnt/Music-NAS/Albums_Sorted/Arch Enemy/Will To Power (2017)/04. The World Is Yours.mp3 **This** is why I needed to recreate my `gonic` container with music mounted under `/mnt/Music-NAS`. Technically, I could have used `sed` (or similar) to replace the path prefix, but I was migrating enough playlists that I didn't want to _have_ to process them if it could be avoided. To import the playlists into `gonic`, I needed to know `gonic`'s ID for the user that would own them. There are two ways to figure this out 1. Count on your fingers - the default `admin` user is `1`, the next is `2` etc 2. Query gonic's database The first is easier, but doesn't _quite_ scratch the geek itch: sqlite3 gonic.db 'select id, name from users' 1|admin 2|ben Note: if you're setting up `gonic` yourself, don't copy this database to random systems - `gonic` stores credentials in plaintext :( Once I knew the user ID, I just needed to create a directory with the same name and move the playlists into it: mkdir gonic/playlists/2 mv ~/playlists/*.m3u8 gonic/playlists/2 I didn't even need to restart `gonic`, the playlists appeared in my player almost immediately and took their names from the playlists's filename. There was, however, one last thing to do. Some of my Subsonic playlists were made available to other users on the server: `gonic` _also_ supports this, but it wasn't immediately clear how to enable it. After fiddling around in DSub, I found an `Update Information` option which allowed me to mark the playlist as shared with others. After I'd done so, I opened the playlist file in `less` - new metadata had been added to the top: #EXTM3U #GONIC-NAME:"Heavy mix" #GONIC-COMMENT:"" #GONIC-IS-PUBLIC:"true" /mnt/Music-NAS/Albums_Sorted/Semblant/Obscura (2020)/03 - Dethrone the Gods, Control the Masters (Legacy of Blood, Pt. IV).mp3 /mnt/Music-NAS/Albums_Sorted/Semblant/Lunar Manifesto/01 Incinerate.mp3 Although I could have edited the relevant playlists, it seemed quicker (and less error prone) to use Dsub, so I worked through the (relative few) playlists that needed sharing and updated the setting. * * * ### Star Syncing In hindsight, I'm not overly convinced that I _needed_ this service. The author of `gonic` has a second repo: `gonic-lastfm-sync`. This contains a small codebase which provides bi-directional syncing of "favourites" (or stars, depending on the player) between `gonic` and `last.fm`. So, if I previously favourited something in `last.fm`, that state could be pulled down. The repo contains a `Dockerfile`, but I wanted something a bit lighter and so wrote my own: FROM cgr.dev/chainguard/wolfi-base AS builder RUN apk add go git \ && mkdir /build \ && cd /build \ && git clone --depth=1 https://github.com/sentriz/gonic-lastfm-sync.git \ && cd gonic-lastfm-sync \ && go build -o lastfm-gonic-sync . FROM cgr.dev/chainguard/wolfi-base COPY --from=builder /build/gonic-lastfm-sync/lastfm-gonic-sync /bin/ ENV GONIC_DB_PATH /data/gonic.db CMD ["sh", "-c", "while true; do lastfm-gonic-sync; sleep 3600; done"] Once I'd built the image, I added a section to my `docker-compose.yml` to run it, passing it the path to the `gonic` container's database: gonic_sync: restart: always image: gonic-lastfm-sync:b0d860 container_name: gonic-lastfm-sync environment: - GONIC_GONIC_USERNAME=ben volumes: - /home/ben/docker_files/gonic/data:/data I brought the container up: docker compose up -d The software went to work. In practice, though, it turned out that I'd only actually previously starred a few things anyway: $ docker logs -f gonic-lastfm-sync 2025/12/01 17:36:12 no match for "dreamtheaterthisdyingsoul" 2025/12/01 17:36:12 saved lastfm->gonic stars, 7 of 8 matched 2025/12/01 17:36:12 saved gonic->lastfm stars, 0 new Ah well * * * ### Cutover The original plan was for _me_ to use `gonic` for a while and then cut the rest of the family over once I was happy that things were working. But, everything seemed to be OK. In Dsub, I deleted my temporary profile and then updated the URL of my existing Subsonic profile to see whether profile reuse caused any issues: it didn't. I couldn't _really_ think of anything else that needed checking, so, I decided to take a risk and just cut everyone over. To prepare for the change, I need to create user accounts in `gonic` with the same credentials as those used for Subsonic. I didn't have those credentials, but Subsonic _did_ : within it's data directory is a file called `subsonic.script` - this is essentially a bunch of SQL statements that Subsonic uses to recreate it's database at startup. I used `grep` to extract the statements used to populate user records: grep "INSERT INTO USER VALUES" subsonic.script This returned lines like this: INSERT INTO USER VALUES('admin','enc:6e65766572676f6e6e6167697665796f757570',149828415,0,0,FALSE,NULL) From this, I needed two things: the username and a decoded password. Although not in ASCII, Subsonic _does_ store credentials in the clear - they're only hex encoded and so can be decoded using `xxd`: grep "INSERT INTO USER VALUES" subsonic.script | cut -d"(" -f2 | cut -d, -f1,2 | while read -r line do username=`echo "$line" | cut -d, -f1 | tr -d "'"` password=`echo "$line" | cut -d, -f2 | tr -d "'" | cut -d: -f2 | xxd -r -p` echo "${username}:${password}" done With this information in hand, I: * Created corresponding users in `gonic` with the same passwords as in Subsonic * Logged into their subsonic accounts and exported playlists * Imported playlists into `gonic` Once I was ready, I updated my reverse proxy for the Subsonic domain so that it would proxy onto `gonic` instead. proxy_pass http://gonic; One Nginx reload later, nothing broke. * * * ### Bonus: A New Desktop Player Although the migration was transparent to the _other_ users of my Subsonic instance, it did change things for me. It was no longer possible to log into a web interface to manage our music collection and I now needed to do so through a front-end app. DSub is perfectly capable, but I'm not a fan of **any** app-only workflow (_looks pointedly towards the finance sector_), if I don't have the option of doing it from a desktop, I'm not doing it _at all_. Jamstash is great for _playing_ music but isn't really suited to _managing_ it. Whatever I landed on would, obviously, need to be able to talk to a Subsonic compatible API. After a bit of looking around, I stumbled across Feishin: It has _pretty much_ everything that I'm likely to need and, after years of keeping a browser tab open, there's something quite refreshing about going back to having a dedicated music app (even if, being an electron app, it is actually still just a wrapper around a browser). Although I haven't played around with it yet, it also seems that Amarok is back in development!3 * * * ### Resource Usage Back when first deploying Subsonic, I added the following comment to my ticket: It should come as no surprise that a Go program is more RAM efficient than an aged Java beast, but _just look at the difference_ : Even idle, Subsonic still requires _ten times_ the memory demanded by `gonic` (admittedly, I'm not sure that half a gig of RAM constitutes "RAM hungry" nowadays). CPU usage graphs tell a similar story: That brief spike in usage appears to be `gonic` scanning our music library for updates. * * * ### Conclusion Because of my experiences with Navidrome last year, I expected that this was going to be a big, long running project. That hasn't proven to be the case though - in fact, it's taken me substantially longer to write the first draft of this blog post than it did to perform the full migration. The move was entirely transparent to my Subsonic users and our music is now served by an _extremely_ lightweight container without any of the overhead associated with Java apps. I've now been able to tear the Subsonic container down, greatly reducing the average age of the software running on our LAN, let alone that which is exposed to the internet. * * * 1. We were skint, so I couldn't really afford/justify a subscription. But, they also lost the future me that _could_ afford to pay for subscriptions. ↩ 2. Which, I think, _probably_ puts a date on when we moved from Amarok to Google Play Music ↩ 3. Though it's not a given that the Subrok plugin will still work with it - the plugin was a bit long in the tooth a decade ago and seems to have been abandoned since. ↩ 4. This was actually a drop from when I first looked - it had been closer to 800MiB the day before ↩

Not long after we moved into this house, we installed a "smart" thermostat: a Google Nest Gen3. I've disliked it ever since. Lowlights include "True Radiant" turning the boiler on 5 hours early and Home/Away assist turning the heating off while we watched a movie (we hadn't moved from the sofa, so _obviously_ it decided that no-one was home). Once I'd turned off all of its "smarts", my dislike for the thermostat simmered down to something closer to disdain: what we now had was a massively overpriced schedulable thermostat. Except, it was actually slightly _worse_ : normal thermostats don't come with a dependency on Google's cloud platform (a dependency that only increased once I hooked it up to Home Assistant so that I could restore some level of smartness). Google recently end-of-lifed the Gen 1 and Gen 2 Nest thermostats leaving owners with a dumb (but at least working) thermostat. This served as the kick up the arse that I needed: the writing would probably soon be on the wall for my Gen 3 too and, if I wanted to maintain my current capabilities, I'd need to find _something_ to replace it. Four years ago, I wrote the following: > Given the choice between NEST and Hive, I think I'd now go HIVE, but realistically the next time I'm in the market, my choice will probably be "neither". Although I did (briefly) look at Hive, I decided that it was best to stick with my original instinct and look for local-only hardware. This post describes replacing our Nest thermostat with locally controllable hardware, hooked up to Home Assistant for scheduling, control and automation. * * * ### Contents * Contents * Topology * Hardware * Boiler Control Thermostat * Room Thermostat * Installation * Matter Server * Room Thermostat * Boiler Control Thermostat * Matter Pairing * Helpers * Automation * Linking The Devices * Scheduled Temperature Changes * Disaster Recovery Plan * Graphs * Matter Over Wifi * Conclusion * * * ### Topology The layout of our house made this a little more difficult than it might otherwise have been. The boiler lives in an unheated single story extension on the back of the house, while the thermostat lives in the middle. The Nest thermostat connected wirelessly to the boiler via the Heat Link. Because the extension is unheated (and somewhat under insulated), the thermostat needs to be in the house. However, my investment in this project didn't really extend to pulling up floors and running wires, so the replacement thermostat would also need to be wireless. * * * ### Hardware I searched around for a _good_ wireless thermostat and control box, but didn't really find much - especially given that I needed something which was local only and automatable. Eventually, I concluded that I wasn't going to be able to get exactly what I needed and so would need to put multiple products together. * * * #### Boiler Control Thermostat After a lot of searching around, I settled on a thermostat made by Meross, the MTS215B: Although I'd originally hoped to find a Zigbee thermostat, the Meross had the best reviews and was a reasonable price. It'd be my first Matter device, but I wasn't _too_ concerned about that1. * * * #### Room Thermostat I did, briefly, consider using a simple and cheap zigbee temperature sensor: it could stick unobtrusively to the wall in the hall and report back the temperature in the most central part of the house. However, that wouldn't be a like-for-like replacement for the Nest thermostat. I decided that I wanted something that would allow us to adjust the target temperature level without having to whip out a phone. That requirement narrowed the market quite considerably, only really leaving the Ecobay Master Controller: This zigbee temperature control unit seems to have been created with radiator TRVs in mind: you join them all to the Tuya app and then group them. But, crucially, it had all of the features that I needed: temperature monitoring, target temperature input and was wireless (though, disappointingly it turned out to be battery only - I'd assumed the slot on the side was for a USB power supply). Tuya kit can be a _little_ funny, so there wasn't any guarantee that it'd be able to work with Home Assistant, but for £25 it seemed worth a go. * * * ### Installation #### Matter Server Not previously having had any Matter devices, I needed a Matter server for Home Assistant to communicate with. So, whilst waiting for the thermostats to turn up, I stood up a python-matter-server container: mkdir -p docker_files/matter/data docker run -d \ --name matter-server \ --restart=unless-stopped \ --security-opt apparmor=unconfined \ -v /home/ben/docker_files/matter/data:/data \ --network=host \ ghcr.io/matter-js/python-matter-server:8.1.1 (I don't particularly like running containers in host networking mode, but apparently it's necessary in this case). With this up and running, I logged into Home Assistant and went `Settings` -> `Devices and Integrations` -> `Add Integration` -> `Matter` When prompted, I provided the address that the Matter server was listening on: With that, it was done. * * * #### Room Thermostat The temperature unit was quite easy: install batteries and join it to the Zigbee network. That said, I did briefly think that I'd wasted my money - zigbee2mqtt reported that the device was unsupported and did not display any readings from it. However, my installation was quite old - I updated it and the device was then correctly recognised: The device appeared in Home Assistant as a `climate` device: It also exposed a `boolean_sensor` to indicate whether it was currently calling for heat: * * * #### Boiler Control Thermostat The Meross thermostat was, understandably, a bit more complex to install. To begin with, I turned off power to the boiler. There was a small captive screw on the underside of the heat link, I slackened that off and removed the cover: Rather unhelpfully, the terminals were only labelled with numbers. So, to find out which wire was which, I had to find the Nest install doc: So, in the photo above * Terminal 2 (brown) is common * Terminal 3 (blue) is call for heat (or "normally open"/NO) I screwed a new (deep) pattress to the wall, passed the wires through and then screwed them into the back of the meross thermostat (the diagram is offset, I'm using L, N, NO and COM): I screwed the thermostat into the back box and then clipped its screen back on. After restoring power, the thermostat came to life I turned the temperature up to make sure that the boiler kicked in, then turned it back down to make sure that it shut off. * * * ##### Matter Pairing With the thermostat working, the next step was to connect it to the Matter network (known as a Fabric). Part of the setup process uses bluetooth: Home Assistant relies on the mobile app to handle this. I got my phone out of my pocket and, to my surprise, it was already prompting me to add the newly detected Matter device. After hitting `continue`, I was prompted to scan the device's QR code. Android then asked me if I wanted to add the device to Google Home. There wasn't a "fuck no" button, so I had to settle for "use another service instead" - this displayed a couple of options, including Home Assistant. Moments later, the thermostat was enrolled into and could be controlled by Home Assistant, again appearing as a `climate` device: * * * #### Helpers I don't like hardcoding values into automations (I tend to forget where to look), so I created a set of helpers in Home Assistant to store important values * `boiler_control_on_temp`: the temperature to set the thermostat to when we want the boiler on (set to 29) * `boiler_control_off_temp`: the temperature to set the thermostat to when we want the boiler off (set to 10) * `heating-low`: the temperature to set to during "low" periods * `heating-high`: the temperature to set to during "high" periods To allow for easy tweaking in future, I added these to my Heating dashboard * * * ### Automation Both of the devices now existed in Home Assistant but weren't, in any way, linked. Leaving them like this would have been problematic - the extension that the boiler lives in is unheated, so the boiler control thermostat would have had a _completely_ unrealistic view of the house's temperature (likely bankrupting us in its attempts to sufficiently warm an unheated cupboard). * * * #### Linking The Devices The Meross thermostat doesn't have a simple on/off toggle, so to control the boiler's state I needed to set it to a temperature which would cause it to start (or stop) the boiler. The room stat would push readings into Home Assistant which could then distil the reading down to a binary choice (on or off) and then set the boiler control thermostat to the corresponding temperature: Originally, my plan had been to create a `Generic Thermostat` and link both ends to it. However, I ran into some issues with the way that the Zigbee panel appeared in Home Assistant and was unable to get it to feed its temperature readings into the generic stat. This wasn't a major issue, though, because the panel also exposes a boolean sensor to indicate whether it's calling for heat. When I ordered it, I'd expected a simple input/output and had assumed that the thermostat logic was within Tuya. But, the device itself is actually a thermostat and so is able to tell Home Assistant when it'd like the boiler to turn on. I created an automation to turn the boiler on when the panel calls for heat: alias: Turn Boiler On description: "" triggers: - trigger: state entity_id: - binary_sensor.temperature_panel_heating_demand to: - "on" conditions: [] actions: - action: climate.turn_on metadata: {} data: {} target: entity_id: climate.smart_thermostat - action: climate.set_temperature metadata: {} data: temperature: "{{ float(states('input_number.boiler_control_on_temp')) }}" target: entity_id: climate.smart_thermostat mode: single I created a second automation to handle turning the boiler back off: alias: Turn Boiler Off description: "" triggers: - trigger: state entity_id: - binary_sensor.temperature_panel_heating_demand to: - "off" conditions: [] actions: - action: climate.set_temperature metadata: {} data: temperature: "{{ float(states('input_number.boiler_control_off_temp')) }}" target: entity_id: climate.smart_thermostat mode: single Technically these could have been combined into a single automation using an `if`, but having them separate makes it easier for me to see when each state last fired. I turned the target temperature up on the Zigbee panel and, sure enough, the boiler kicked in. * * * #### Scheduled Temperature Changes Nest **do** have quite a nice scheduling interface: This schedule is a big part of how we reduced our heating bills: I'm at home during the day, but I'm not as prone to feeling cold (**and** I'm sat in a particularly warm room anyway). Ideally, I wanted to be able to maintain a similar level of control. Unfortunately, Home Assistant's scheduling functionality isn't quite as polished. The `Schedule` helper _does_ allow you to provide arbitrary data (which could be used to defined the desired temperature) in scheduled blocks, but there isn't a way to have that data display as nicely as in Nest. Some of the temperatures that I'd scheduled in Nest were very similar though, so I decided to try shifting to a two-tier schedule * Low: 15C * High: 19C (I rounded values down because I'd rather be slightly too cool than slightly too warm) I stored these values in the number helpers `heating-low` and `heating-high` and then created a `Schedule` helper to control when we transition between each: `on` periods are `High` and `off` is `Low`: I then created a couple of automations to respond to changes in the schedule state by setting the zigbee panel's target temperature to the appropriate value: alias: Switch Heating To High description: "" triggers: - trigger: state entity_id: - schedule.heating_high_schedule to: - "on" conditions: [] actions: - action: climate.set_temperature metadata: {} data: hvac_mode: heat temperature: "{{ float(states('input_number.heating_high')) }}" target: entity_id: climate.temperature_panel - action: climate.turn_on metadata: {} data: {} target: entity_id: climate.temperature_panel mode: single --- alias: "Turn heating down" description: "" mode: single triggers: - trigger: state entity_id: - schedule.heating_high_schedule to: - "off" conditions: [] actions: - action: climate.set_temperature metadata: {} data: hvac_mode: heat temperature: "{{ float(states('input_number.heating_low')) }}" target: entity_id: climate.temperature_panel - action: climate.turn_on metadata: {} data: {} target: entity_id: climate.temperature_panel Setting these automations live meant that I'd reached feature parity with the Nest that I'd replaced. Finally, I worked through my existing automations (including the overnight boost script) and updated them to point to the new `climate` device instead of the one associated with the Nest thermostat: * * * ### Disaster Recovery Plan Although the Zigbee temperature panel is a nice bit of kit, I did have some reservations about the setup being _quite_ so tightly tied to it: zigbee devices don't always have the best build quality and, if it were to die, we could be left without working heating. So, I decided to define a DR procedure in the event that the panel failed: * Create a boolean helper to act as the heater switch * Create a new `Generic Thermostat` * Bind the thermostat to some other temperature sensor (probably the one in the living room) * Update automation to turn boiler on/off when the new boolean helper changes Unfortunately, there isn't quite so much that could be done if the Meross thermostat were to die - I'd probably have to reconnect the Nest while I waited for a replacement to arrive. Losing Home Assistant would also be pretty bad (though not just because of the heating). However the Meross also supports use of an external temperature probe. It seems quite likely that there's a preexisting cable run from the boiler to somewhere in the main house (there _must_ have been a thermostat at some point). The next time that I've got flooring up, I'll try to find it so that I can install a temperature probe in the house, giving the Meross the ability to run entirely on it's own if required. * * * ### Graphs I have Home Assistant set up to write metrics into InfluxDB and so already had relevant Grafana dashboards. Because the Zigbee panel exposes a binary sensor, it was easy to write a query to display its reported heating state: from(bucket: "home_assistant/autogen") |> range(start: v.timeRangeStart, stop: v.timeRangeStop) |> filter(fn: (r) => r._measurement == "binary_sensor.temperature_panel_heating_demand") |> filter(fn: (r) => r._field == "value") |> aggregateWindow(every: v.windowPeriod, fn: max) To allow comparison, I added this query to a cell which shows Nest's reported status: The new thermostat is a bit more prone to flapping than the Nest, turning on and off in quite short time periods. Unfortunately, the device doesn't have a setting to control this (if I'd been able to use a `Generic Thermostat`, I could have set `cold_tolerance` to address this). I'm not sure that it's too big of an issue, but if it continues to bug me I can build a delay in to debounce the automations. I also added a series to my temperatures graph. It seems that the zigbee panel tends to report a lower temperature than Nest (the initial spike is because the panel was sat in my office): This was a useful data point as it indicated that I should probably _reduce_ the target temperature to account for the panels (apparent) under-read2 * * * ### Matter Over Wifi Although I'm impressed with the ease of integration that Matter brings, I _do_ have some initial concerns with it * It's an IP capable device: I more or less have to trust that the firmware isn't configured to do unwanted stuff * Devices need to be in the same network as Home Assistant: Matter uses link-local IPv6 addresses and IPv6 Multicast, so I can't isolate devices onto my dedicated IoT wifi network (at least, not without multi-homing Home Assistant onto it) But, in practice, these concerns aren't all that different to those I have about things like the Kasa/Tapo wifi smart sockets. Essentially, I've been spoilt by Zigbee. * * * ### Conclusion This project seems to have been a success: Automation of our heating is no longer reliant on the reachability of GCP's pub/sub system and our thermostat is no longer reporting into Google. Although the hardware is a little more piece-meal than I'd originally planned for it to be, our heating is now _entirely_ locally controlled (though the Home Assistant mobile app means that I _can_ still control it when out and about). Because we were originally using Home Assistant for other things, it's only _slightly_ more load bearing than it had been before3. The boiler control thermostat is also my first Matter device: so far I've been pleasantly surprised with the ease of integration. I do recall seeing quite a few complaints about Matter, but I _think_ that was often more in the context of Matter-over-Thread than about Matter itself. Either way, the Nest is _finally_ gone. * * * 1. There **are** some caveats with using Matter though. The biggest is that you will need to ensure your LAN supports IPv6 (you don't need external support, but you do need to not have turned it off everywhere that you could find) ↩ 2. I haven't yet checked whether it's the panel or Nest that's reading incorrectly - it doesn't _overly_ matter as long as I know what _relative_ value to set targets to. ↩ 3. Though the change is enough that I've put a task on my list to move it to dedicated hardware and think more about redundancy ↩

It's a little over a month since I was prescribed medicinal cannabis and have been taking half to a full gummy each evening (aside from not wanting to risk being impaired during the day, higher levels of daily use are associated with lower levels of pain relief). For my prescriptions to continue, I need to (virtually) attend a check up appointment in order to discuss whether it's proving effective (as well as ensure that there haven't been any issues etc). Conveniently (unfortunately?), I have just had a demonstration of efficacy. Last week, I attended a work on-site in Playa Del Carmen, Cancun. Unfortunately, I was unable to carry medicinal cannabis into the country (I asked the Mexican embassy in advance and they said "as far as we are concerned, it is prohibited"). That's _not great_ , but did lead to me having a better understanding of how much it's been helping ("You never realise the value of something until it's gone" and all that). This post talks about that experience. * * * ### Talking About Comparative Pain Levels Ever since starting on the gummies, I've had the sense that they were helping but it was hard to say by _how much_. Pain really is quite weird in that respect: * When you're _in_ pain you know about it (although meaningfully communicating about pain is a known problem area). * When you're suffering less pain, you're aware that it's reduced but it's very hard to say by how much (being _out_ of pain is easier, because you can say that it's gone... I live in hope) This makes it quite difficult to say _how effective_ a given form of pain relief is (particularly when improvements have been gradual). Although it'd be nice to be able to talk comparatively, you can only really talk about pain levels _now_ (the trick, for the record, is to focus on externalities rather than the pain itself - is it helping enough that you're comfortable, not miserable, do stairs make you want to cry etc) But... if you go back from an improved state to where you were, it suddenly becomes _uncomfortably_ easy to see how big the difference is. * * * ### Going A Week Without The relief that I had been getting lasted for about 2 days after my last gummy. After that, I was back to being extremely uncomfortable (even though I was still taking the medications that I _could_ carry into Mexico). Stairs, once again, became a serious obstacle and I strained my shoulder a little by having to support my weight on the handrail (something that I hadn't consciously noticed that I'd stopped needing to do). Swimming was also problematic. I only really felt able to do it on the first morning (whilst still benefiting a little from that previous gummy). Rather than going downstairs and socialising with others over breakfast, I ordered room service, sat on the balcony and tried to mentally prepare for a day of repeatedly getting up and moving around (although the hotel had a lift, there were also areas that were only accessible via stairs). My flying experience was quite different too: despite the seats being physically identical each way, my leg was _much_ more uncomfortable on the journey home. One thing that I'm **not** able to compare, is sleep: * Any sleep disturbances could equally be explained by jet lag and/or sleeping in an unfamiliar environment * I don't _usually_ go to bed full of beer and margaritas (alcohol at volume is also great at pain relief, but the associated loss of inhibition can lead to you making things worse1) Across the trip, there were days that I'd probably have preferred to be laid up (and, arguably perhaps should have been). There were a ton of other smaller differences too (like having to really work to regain focus in meetings after a pang throws your mind off). Because I had my daily tablets, it wasn't entirely unbearable, but could _definitely_ have been more pleasant. The experience made me think of a passage in the open letter that I wrote nearly eight years ago: > I spend every day of my life in pain that I know could be mitigated Pain is awful, but knowing that it could be alleviated and that you're suffering because of _someone else's_ misguided principles only serves to worsen the torment. I _loved_ Mexico, but I did not love how it made me feel. * * * ### Getting Home I've only been home a day, so have only had a single gummy. Much like the very first night of taking them, I had a little bit of relief directly after, but haven't got any today. Clearly, time is needed for it to build back up in my system (although going from ~28C to -1 also can't have helped). I was also a little unsure of dosage when I got home, not knowing whether I should: * pick up where I left off (and potentially find it was too much) * start small and work up (extending the amount of time before meaningful relief) In the end, I chose to pick up where I left off - it didn't end up blowing my head off, so this was presumably the right choice. * * * ### Judging Pain By Stopping It seems worth discussing one of the problems with judging the efficacy of pain relief by withdrawing it. Certain drugs are known to _cause_ pain. Codeine (and other opiates), for example, can cause headaches. The user will then treat that pain with more codeine, potentially leading to a self-fuelling addiction but certainly generating more frequent headaches. Some drugs don't cause pain while _in use_ , but can cause pain during withdrawal. Although it's uncommon, cannabis can be one of these. So, even aside from having to deal with pain, there _is_ a methodological issue with stopping medication to judge pain levels. In this particular case, I know that that's not been an issue: * It's unlikely that I've been using it long enough to develop a dependency * I've not had any other symptoms of withdrawal (I'd certainly recognise those from my previous experiences with Tramadol) * Because of the focus on externalities, it's possible to directly compare to my previous experiences and say that this was not a _new_ form of pain And, of course, I didn't stop solely to judge the pain - it was an unfortunate requirement of me wanting to travel to a country that doesn't currently recognise the validity of my prescription. Hopefully, those will be fewer and fewer over time. * * * ### Conclusion I'm sure that the doctor will have quite specific questions at my upcoming checkup, but last week has shown me that the cannabis gummies are actually more effective than I'd previously realised. Things aren't perfect and I do still have to be a little mindful of what I'm doing, but last week reminded me of various nuisances and pain that I've previously just had to accept as part of living with chronic pain. While writing this post, I also remembered just _how distracting_ it is - barely a paragraph has passed between being distracted by a painful leg twitch, a need to shift in my seat or a yelp as I move uncomfortably. It's now quite clear to me that my life _is_ improved by the availability of medicinal cannabis. It's not just the differences that I've described here, but the fact that it does something that my previous prescriptions never could: it allows me to achieve these benefits without a cost to clarity of mind during the day. * * * 1. One night, after _a lot_ of drinks (even by my standards), I was successfully peer pressured into dancing. I had fun, but the next day was **baaad** ↩

My work laptop is made by Framework and apparently has a fingerprint sensor built into the power button. Once I found out, I wanted to set it up so that I had a more convenient means of authenticating when working on the go (it's a little less useful in my office as I tend to dock with the laptop lid closed). PAM integration allows me to use the fingerprint reader to login to, and unlock my laptop. _Technically_ it can also be used for SSH authentication, though I wouldn't normally have the necessary proximity to actually use it. This post describes how to enable fingerprint authentication on Linux. Although I wrote it because of my Framework, it also works with fingerprint enabled Thinkpads (you can find a list of supported chips here). This post details the commands needed for both Debian and Red-hat based systems. * * * ### Checking The Reader The first thing to do, is to ensure that your fingerprint reader is actually visible to the system: GUID=$(fwupdmgr get-devices | grep -A 6 -E "fingerprint" | grep -i "GUID" | awk -F ': ' '{print $2}' | awk '{print $1}') fwupdmgr get-devices $GUID This should result in output like the following Selected device: Fingerprint Sensor Framework Laptop 16 (AMD Ryzen 7040 Series) │ └─Fingerprint Sensor: Device ID: 6dc14cd3f5a7e031a01b561fa9619f84c5fb8b07 Summary: Match-On-Chip fingerprint sensor Current version: 01000334 Vendor: Goodix (USB:0x27C6) Install Duration: 10 seconds Serial Number: UID212F61A6_XXXX_MOC_B0 GUID: a11b1337-569e-521d-52ed-16cafa32a8c1 ← USB\VID_27C6&PID_609C Device Flags: • Updatable • Supported on remote server • Device stages updates • Device can recover flash failures • Signed Payload * * * ### Enable Reader Fingerprint reader support is provided by fprint, so we need to install that and the associated PAM module: # Debian based distros (inc Ubuntu) sudo apt install libpam-fprintd fprintd # Fedora and Red-hat alikes sudo dnf install fprintd fprintd-pam Tell the system that it can use the reader for authentication: # Debian based distros sudo pam-auth-update # Fedora and Red-hat alikes sudo authselect enable-feature with-fingerprint sudo authselect apply-changes For Ubuntu users, this will display an ncurses based menu, use space to enable `Fingerprint authentication` and then hit Ok: * * * ### Enrolling a Fingerprint The system now supports using a fingerprint for PAM operations, so the next step is to enroll one: fprintd-enroll -f right-index-finger Put your right index finger on the reader, remove it and then put back on. Repeat this until the program exits: $ fprintd-enroll -f right-index-finger Using device /net/reactivated/Fprint/Device/0 Enrolling right-index-finger finger. Enroll result: enroll-stage-passed Enroll result: enroll-stage-passed Enroll result: enroll-stage-passed Enroll result: enroll-stage-passed Enroll result: enroll-retry-scan Enroll result: enroll-stage-passed Enroll result: enroll-stage-passed Enroll result: enroll-stage-passed Enroll result: enroll-retry-scan Enroll result: enroll-stage-passed Enroll result: enroll-stage-passed Enroll result: enroll-stage-passed Enroll result: enroll-stage-passed Enroll result: enroll-remove-and-retry Enroll result: enroll-stage-passed Enroll result: enroll-completed Your fingerprint is now enrolled. If you lock your screen, the unlock dialog should allow you to present a fingerprint. On XFCE, you're actively prompted to provide one: If it does not, you may need to restart your system. * * * ### Maintenance Once you're set up, there isn't really anything more that you _need_ to do. However, there are commands that you might useful in future. As ever, the Arch wiki also has some more advanced information. * * * #### List Stored Fingerprints You can fetch a list of the fingerprints that are known about for a given user: $ fprintd-list $USER found 1 devices Device at /net/reactivated/Fprint/Device/0 Using device /net/reactivated/Fprint/Device/0 Fingerprints for user ben on Goodix MOC Fingerprint Sensor (press): - #0: right-index-finger * * * #### Delete Fingerprint If ~~someone has stolen your finger~~ you are having issues with a finger not reading, you might want to delete the stored fingerprint $ fprintd-delete Usage: fprintd-delete [OPTION…] Delete fingerprints <username> [-f finger-name [usernames [-f finger-name ]...] Help Options: -h, --help Show help options * * * #### Test Reading You can also verify that your fingerprint reads correctly: fprintd-verify

About 6 months ago, I imported our photo collection into Ente Photos. Before that, our memories lived in Nextcloud. Although it's great for file syncing, Nextcloud didn't really work all that well for us as a photo manager (in fairness, that's probably largely because I'd pulled them in via a Shared Storage rather than putting any _real_ effort in). Like Nextcloud, Ente is open source. Originally, I'd intended to use Ente's SaaS offering for a little while and the migrate to self-hosting once I was happy that it was the right solution. That migration never happened (I've been happy enough, and the cost is low enough, that I've not really felt the urge to move). It does mean, though, that the availability of our (ever growing) photo collection is reliant on Ente's disaster recovery posture. Ente have not given me any reason to doubt them (in fact, their approach to reliability is well documented), but our photos are _utterly irreplaceable_ and using _any hosted offering_ comes with some risk of disappearance (or of being acquired by a company which goes on to ruin it), often without any prior warning. To their credit, this is something that Ente explicity recognised when first introducing the CLI. This blog post talks about using the `ente` CLI to automate a periodic incremental backup of the photos that we store in Ente. It's primarily focused on deploying into Kubernetes but also details how to do so using Docker or a native install. * * * ### Contents * Contents * Containerising * Storage * Running In Kubernetes * First Time Setup * Scheduling * TL:DR * Manual Run * Running Without Kubernetes * With Docker * Without Containers * Caveats * Conclusion * * * ### Containerising There wasn't any _particular_ need for the backup to run within Kubernetes, other than that I've had a cluster that I could run it in. The CLI is actually pretty simple, so setting things up without using Docker or Kubernetes isn't too much different (details of that are below). To keep things lightweight, I based my container on Wolfi: FROM cgr.dev/chainguard/wolfi-base AS builder Ente is written in Go, so I installed `go` and `git` before cloning Ente's source down and compiling the CLI: RUN apk add go git \ && mkdir /build \ && cd /build \ && git clone --depth=1 --branch=$ENTE_RELEASE https://github.com/ente-io/ente.git \ && cd ente/cli \ && go build -o "bin/ente" main.go This produced a standalone binary, so I copied it into a fresh image, created the directories that it needed and configured the container to run as a non-privileged user: FROM cgr.dev/chainguard/wolfi-base # Copy the built binary over # Make sure we also ship the license file COPY --from=builder /build/ente/cli/bin/ente /usr/bin COPY --from=builder /build/ente/LICENSE /LICENSE RUN mkdir /cli-data/ /cli-export/ \ && chown -R nonroot:nonroot /cli-data/ \ && chown -R nonroot:nonroot /cli-export/ USER nonroot ENTRYPOINT ["/usr/bin/ente"] The full Dockerfile can be found in Codeberg and my build of the image can be pulled from `codeberg.org/bentasker/ente-cli-docker`. * * * ### Storage The CLI requires a couple of storage volumes: * `/cli-data` this is where the CLI will maintain a database of image metadata (along with the creds it uses to talk to Ente). * `/cli-export` this is where photos will be exported to The CLI data path can be overridden via env variable `ENTE_CLI_CONFIG_DIR`. The export path can be any arbitrary path, but has to be provided when adding an account to the CLI's config. * * * ### Running In Kubernetes #### First Time Setup The CLI isn't of _much_ use until it's linked to an account. Unfortunately, there isn't a programmatic way to pre-configure it, so I needed to spin up a pod so that I could login to the CLI. As it seemed _possible_ that I might need to manually interact with the CLI again in future, rather than manually creating a pod, I defined a deployment but set it to be scaled to 0 pods: apiVersion: v1 kind: Namespace metadata: name: ente-backup --- apiVersion: apps/v1 kind: Deployment metadata: name: ente-backup-cli namespace: ente-backup spec: selector: matchLabels: app: ente-backup replicas: 0 template: metadata: labels: app: ente-backup spec: containers: - name: ente-backup image: codeberg.org/bentasker/ente-cli-docker:v0.1 env: - name: ENTE_CLI_CONFIG_DIR value: "/cli-data/" - name: ENTE_CLI_SECRETS_PATH value: "/cli-data/.secrets" command: [ "/bin/sh", "-c", "while true; do sleep 3600; done" ] resources: requests: cpu: 150m memory: 64Mi volumeMounts: - mountPath: /cli-data name: kubestorage subPath: ente-backup/config - mountPath: /cli-export name: kubestorage subPath: ente-backup/export restartPolicy: Always volumes: - name: kubestorage nfs: server: 192.168.3.233 path: "/volume1/kubernetes_misc_mounts" readOnly: false The important thing here is that the pod needs to use the same storage volumes as our cronjob will. Scaling to 0 means that the necessary configuration will be present in the cluster when I need it, but won't waste resources by running pods unnecessarily. I scaled the deployment up to `1` so that a pod would come online: kubectl -n ente-backup scale --replicas=1 deployment/ente-backup-cli I exec'd into the new pod and triggered the account addition flow: kubectl -n ente-backup exec -it ente-backup-cli-669dff58f4-vzbsv -- /usr/bin/ente account add When prompted, I set the export directory to `/cli-export/` (you can enter whatever you want, but be aware that the path needs to exist - the setup flow won't create it for you if it doesn't). Once the account had been added, I scaled the deployment back down to 0: kubectl -n ente-backup scale --replicas=0 deployment/ente-backup-cli * * * #### Scheduling `ente` was now configured to work with my account. The next step was to configure an automated run, using a `CronJob`. The podspec is, more or less, identical to the spec used for the deployment above. The only _real_ change is the command (which invokes `ente export`): --- apiVersion: batch/v1 kind: CronJob metadata: name: ente-backup namespace: ente-backup spec: schedule: "0 4 * * *" failedJobsHistoryLimit: 5 successfulJobsHistoryLimit: 5 concurrencyPolicy: "Forbid" jobTemplate: spec: template: spec: restartPolicy: Never containers: - name: ente-backup image: codeberg.org/bentasker/ente-cli-docker:v0.1 imagePullPolicy: IfNotPresent env: - name: ENTE_CLI_CONFIG_DIR value: "/cli-data/" - name: ENTE_CLI_SECRETS_PATH value: "/cli-data/.secrets" command: ["/usr/bin/ente", "export"] volumeMounts: - mountPath: /cli-data name: kubestorage subPath: ente-backup/config - mountPath: /cli-export name: kubestorage subPath: ente-backup/export volumes: - name: kubestorage nfs: server: 192.168.3.233 path: "/volume1/kubernetes_misc_mounts" readOnly: false This schedules the job to trigger at 4am each day. * * * #### TL:DR You can grab a copy of the above config from Codeberg. You'll need to update the storage volumes so that they are applicable to your cluster, but once that's done, you just need to apply: kubectl apply -f ente-backup.yml Scale up the deployment so that you can login to the CLI: kubectl -n ente-backup scale --replicas=1 deployment/ente-backup-cli kubectl -n ente-backup get pods kubectl -n ente-backup exec -it ente-backup-cli-669dff58f4-vzbsv -- /usr/bin/ente account add Once the flow's completed, scale back down: kubectl -n ente-backup scale --replicas=0 deployment/ente-backup-cli Wait for the cron to trigger (or move onto the next section to trigger it manually). * * * #### Manual Run I didn't want to have to wait for the next day to find out whether the backup had run, so I manually created a job from the `CronJob`: kubectl -n ente-backup create job ente-backup-manual --from=cronjob/ente-backup I then tailed the logs kubectl -n ente-backup logs job/ente-backup-manual It took some time to work through all our photos, but eventually it logged completion: Starting a new job resulted in a quick exit, as there was nothing new to do: * * * ### Running Without Kubernetes #### With Docker For those without a cluster to hand, the container can also be run using Docker. Just as with Kubernetes, the important thing here is that volumes persist between manual invocations and cron'd runs: ENTE_BACKUP_DIR=/path/to/backups # Set up a storage location mkdir -p ${ENTE_BACKUP_DIR}/ente/data ${ENTE_BACKUP_DIR}/ente/config # Do the first time setup docker run --rm \ -it \ -v $ENTE_BACKUP_DIR/ente/data:/cli-export \ -v $ENTE_BACKUP_DIR/ente/config:/cli-data \ codeberg.org/bentasker/ente-cli-docker account add A backup wrapper would then look something like this: #!/bin/bash # # Trigger the export container ENTE_BACKUP_DIR=/path/to/backups cd "$ENTE_BACKUP_DIR" docker run --rm \ -it \ -v $PWD/ente/data:/cli-export \ -v $PWD/ente/config:/cli-data \ codeberg.org/bentasker/ente-cli-docker The backup wrapper then just needs adding to a crontab 0 4 * * * /path/to/wrapper.sh * * * #### Without Containers `ente` is a standalone binary, so can also be run without using containers _at all_. If you want to build it from source, you'll need `go` installed - see the dockerfile steps above for an indicator of how to build it. If you're happy fetching a pre-built binary, though, you can grab one from Github: curl https://github.com/ente-io/ente/releases/download/cli-v0.2.3/ente-cli-v0.2.3-linux-amd64.tar.gz | tar xvz Setup is: ente account add and the command that you need to add to cron is: ente export * * * ### Caveats There are a couple of caveats here: **Exports are per _user_ account**: although Ente allows sharing of albums between users, it's very much set up as an individual user thing1. If you've got multiple Ente users all uploading their own photos (particularly automatically), you'll need to export from each of these (you can run `account add` multiple times to add them). **There's limited overlap protection** : The `CronJob` is configured to try and prevent overlapping runs, however there's nothing to prevent manually triggering a job while another is running. I don't know exactly what the outcome of an overlapping run will be, but it's unlikely to be anything good. * * * ### Conclusion There are additional docs on the CLI available here, but the setup above provides for a scheduled incremental backup of an Ente account. Using this with their hosted service provides the low maintenance associated with using SaaS offerings2, but maintains some of the access to data that self-hosting would provide. The export separates images into albums (mirroring the organisation that's performed in-app), so if something _were_ to happen to Ente, the backup of our photos is already sorted for convenient import into something else. * * * 1. This is one of the things that I like _least_ about Ente - shared albums currently give something of a sub-par experience because they appear in a totally different place within the interface. ↩ 2. Well... good ones anyway ↩

Our power supply is normally pretty reliable, but last week we had a an outage. Although we've got solar, we don't (currently) have an islanding switch, so when the grid goes down, so do we. This power outage only lasted about 45 minutes, but came at a _really_ bad time: I was due to be interviewing someone, so had to try and get signal so that I could _at least_ send a SMS and tell them that we'd need to re-schedule. I _used_ to have a UPS, but didn't replace it after the battery reached end-of-life - at the time we had a young child in the house, so having something be persistently energised seemed like quite a bad idea. That's no longer a concern though, so I decided that it was time to plug important things (laptop, switch router etc) into a UPS - partly to protect them from damage, but also so that there's something that I can _do_ during an outage (this week, I couldn't do much more than sit and work my way through a Toblerone). This post details the process of installing Network UPS Tools (NUT) and configuring Telegraf to collect metrics from it, allowing graphing and alerting in Grafana. * * * ### The UPS It doesn't matter _too much_ what model of UPS you have, NUT supports a wide range of kit. Mine has a USB connection, so we're using NUT's `usbhid` support. My UPS is a Powerwalker VI Series UPS and shows up in `lsusb` like this Bus 006 Device 015: ID 0764:0601 Cyber Power System, Inc. PR1500LCDRT2U UPS The UPS has 4 mains plug sockets on the back, so I've got a few things plugged in: * My router/firewall (our fiber ONT is in a different room and has it's own battery backup) * My main switch * My NAS * An external HDD array * The extension lead which runs my desk Running my desk means that it has to power a couple of monitors **and** a couple of laptops. This isn't _quite_ as bad as it sounds though: * If I'm not at my desk, the monitors will be off and the laptops will be (relatively) idle * If _I am_ at my desk, the plan is to unplug the laptops and have them run off battery so that they're not using the UPS's capacity * * * ### NUT setup #### Installing NUT is in the Ubuntu repos, so: sudo apt update sudo apt install nut nut-client nut-server Next we confirm that NUT can actually see the UPS: sudo nut-scanner -U If all is well, this'll write out a config block: [nutdev1] driver = "usbhid-ups" port = "auto" vendorid = "0764" productid = "0601" product = "2200" serial = "11111111111111111111" vendor = "1" bus = "006" We need to write that into NUT's config, so invoke again but redirect: sudo nut-scanner -UNq 2>/dev/null | sudo tee -a /etc/nut/ups.conf The name `nutdev1` isn't _particularly_ informative, though, so we can also hand edit `ups.conf` to change it (and add a `desc` attribute to provide a description of the UPS): sudo nano /etc/nut/ups.conf I set mine like this: [deskups] desc = "Cyber Power System UPS" driver = "usbhid-ups" port = "auto" vendorid = "0764" productid = "0601" product = "2200" serial = "11111111111111111111" vendor = "1" bus = "006" Make a note of the name (the bit in square brackets), we'll need it shortly. * * * #### Setting Up For Monitoring Next we want to set up credentials for NUT server I used my `gen_passwd` utility to generate a random password, but use whatever method suits you: NUT_PW=`gen_passwd 24 nc` Create the user: echo -e "\n[monitor]\n\tpassword = ${NUT_PW}\n\tupsmon master\n" | sudo tee -a /etc/nut/upsd.users Now provide the credentials to `upsmon`, change the value of `UPS_NAME` to match the one that you set for the UPS in `ups.conf` earlier: # Change to match the name in ups.conf UPS_NAME="deskups" echo -e "\nMONITOR $UPS_NAME@localhost 1 monitor $NUT_PW master\n" | sudo tee -a /etc/nut/upsmon.conf Keep a note of the UPS name and password, we'll need it again when configuring `telegraf`. Configure NUT to run as a netserver (so that Telegraf can talk to it): sudo sed -e 's/MODE=none/MODE=netserver/' -i /etc/nut/nut.conf Restart services: for i in nut-server nut-client nut-driver nut-monitor do sudo systemctl restart $i done Confirm that nutserver is listening: $ sudo netstat -lnp | grep 3493 tcp 0 0 127.0.0.1:3493 0.0.0.0:* LISTEN 3854210/upsd tcp6 0 0 ::1:3493 :::* LISTEN 3854210/upsd Check that we get data back about the UPS: upsc $(upsc -l 2>/dev/null) 2>/dev/null If all is well, we're ready to move onto collecting data. * * * ### Collection and Visualisation With NUT now able to report on the UPS, the next step is to have that data collected so that we can visualise it and (optionally) alert based upon it. * * * #### Telegraf We're going to use the upsd input plugin to talk to NUT. This was introduced in Telegraf v1.24.0 so, if you're using an existing install, make sure that your `telegraf` is recent enough: telegraf version If you don't have Telegraf, there are install instructions here (note: you're also going to want an InfluxDB instance or free cloud account because the Dashboard that we'll use for visualisation uses Flux). The input plugin is pretty simple to configure, append the following to `/etc/telegraf/telegraf.conf`: [[inputs.upsd]] ## A running NUT server to connect to. ## IPv6 addresses must be enclosed in brackets (e.g. "[::1]") server = "127.0.0.1" port = 3493 # The values for these are found in /etc/nut/upsmon.conf username = "deskups@localhost" password = "[redacted]" additional_fields = ["*"] # Map enum values according to given table. ## ## UPS beeper status (enabled, disabled or muted) ## Convert 'enabled' and 'disabled' values back to string from boolean [[processors.enum]] [[processors.enum.mapping]] field = "ups_beeper_status" [processors.enum.mapping.value_mappings] true = "enabled" false = "disabled" After restarting (or reloading) `telegraf`, you should start to see metrics appearing in InfluxDB: * * * #### Visualisation I use Grafana for visualisation and, conveniently, there was already a community dashboard (the source for which can be found on Github). On the community page Click `Download JSON`. Then, in Grafana * `New Dashboard` * `Import JSON` * Drag the JSON file over You'll be presented with a set of options for the Dashboard - choose the relevant InfluxDB datasource to query against: You'll then be taken to the dashboard itself. It's quite likely that the dashboard will be broken - by default it looks for a bucket called `upsd-Telegraf` (I write into a bucket called `telegraf`). To fix it * `Settings` * `Variables` * `bucket` Scroll down to find `Values seperated by comma` and change it to contain the name of your bucket Click `Back to Dashboard` and the dashboard should now load: I already track electricity costs, plus we're on a 30 minute tariff, so I also edited the dashboard to remove the cost related row (and then the associated variables). * * * #### Alerting The `upsd` measurement contains a field called `ups_status` which will normally be `OL` (online). If the mains cuts out (or someone unplugs it to test behaviour...) the value will change to report that the UPS is running from battery: Note: The new state `OB DISCHRG` isn't actually a single status, it's reporting two (closely related) status flags. After power is restored, the UPS reports itself back online _but_ also notes that the battery is now charging: This means that creating an alert is **not** as simple as `if r.ups_status != "OL"`. I also only _really_ wanted an email notification to warn me of the following status symbols: * We're running from battery (flag: `OB`) * The UPS is reporting an alarm (flag: `ALARM`) * The UPS is reporting that the battery charge is too low (flag: `LB`) * The UPS is reporting overload (flag: `OVER`) * The UPS requires battery replacement (flag: `RB`) RFC 9271 is quite well designed in that no defined symbol exists as a sub-string of another, so we can safely do something like: for flag in ["OB", "ALARM", "LB", "OVER", "RB"]: if flag in ups.status: alarm() Of course, to do that with Grafana's alerting we need to translate the logic into a Flux query: // Define the regex to use when checking for alertable states alarm_regex = /(OB|LB|OVER|RB|ALARM)/ // Extract reported status from(bucket: "telegraf") |> range(start: v.timeRangeStart, stop: v.timeRangeStop) |> filter(fn: (r) => r["_measurement"] == "upsd") |> filter(fn: (r) => r["_field"] == "ups_status") |> group(columns: ["ups_name", "_field"]) |> keep(columns: ["_time", "_value", "_field", "ups_name"]) |> aggregateWindow(every: 1m, fn: last, createEmpty: false) // Identify whether the status contains any flags of concern // Grafana alerting requires the main column to be numeric // so we need to shuffle things around |> map(fn: (r) => ({ _time: r._time, //flags: r._value, ups_name: r.ups_name, _value: if r._value =~ alarm_regex then 1 else 0 })) |> group(columns: ["ups_name"]) The return values of this query are based on whether any of the problematic flags exist - if they don't, it'll return 0, if they do the value will be 1. This allows use of a simple threshold in the grafana alerting config: With the alert saved, I unplugged the UPS and waited: A minute later, the alert was escalated to Pagerduty: A couple of minutes after plugging the UPS back in, the alert recovered. * * * ### Conclusion Setting up monitoring of the UPS was pretty easy - NUT supports a wide range of devices and exposes status in a standardised way. NUT is well supported by Telegraf and there was _already_ a community dashboard available to visualise UPS status. This means that, in practice, the hardest part of all of this was fishing the relevant power leads out of the rack to plug into the back of the UPS. Now, if the power fails, I _should_ (depending on whether our fiber connection is still lit up) get a page to warn me. Either way, the UPS will provide some coverage for small outages.

I'm **not** going to start this post with the words "I don't like LLMs" because, at this point, it'd seem a little redundant. What I _really_ don't like, though, is paying to support the business model of companies like OpenAI. It's bad enough that their business is built off the back of broad scale plagiarism, but, on top of that their activities _continue_ to cost website operators money and resources. Although it's understandable that it might come to mind, I'm _not_ referring to the repeated crawling of their scrapers (along with the activity of all the other crawlers trying to get in on this latest gold rush). ChatGPT's web search mode is able to search the web and then summarise results (not unlike the psuedo-browsing experience that they now promise with Atlas). When the LLM's response includes images, they are hotlinked directly from the original website. Despite their _ridiculous_ valuation, OpenAI have apparently been too cheap to build an image cache into their product (or perhaps, are trying to sidestep copyright concerns). This means that, every time ChatGPT includes one of my images in its answer, **I** pay for the bandwidth necessary to serve it to ChatGPT's customer (who will very likely never visit my site or read an _accurate_ representation of what I've written). Whether or not we're talking about a small or a large amount of money, this is a **500 billion dollar** AI company freeloading on other people's bills (it _is_ somewhat fitting that a company which acts as a leech on the arse of creativity would _also_ be a bandwidth leech). I've been feeling somewhat grumpy this weekend anyway, so I decided to mitigate this by catching the requests and redirecting to a smaller (and therefore _cheaper to serve_) image. * * * ### Identifying Requests `chatgpt.com` sets a `Referrer-Policy` of `strict-origin-when-cross-origin`: This tells the user's browser that, for cross-origin requests (i.e. those that go to another domain), it should include a `referer` header specifying scheme and domain only. So, when `ChatGPT.com` embeds one of my images the request headers look like this: The existence of that `Referer` header means that it's easily possible to identify requests which originated from `chatgpt.com`. * * * ### The Block Page Twenty odd years ago, it was quite common for anti-hotlinking protections to serve up a different image to the one that the user's browser had requested1. Although outright dropping the request is possible, doing so can lead to support overhead: well-intentioned people will helpfully tell you that your images aren't working in ChatGPT. To avoid this, I wanted it to be quite clear that the request was blocked - the easiest way to achieve this was to serve an image which indicated the objection. Bing's AI **really** didn't want to generate the image that I wanted - it seems that Microsoft have configured the filters to try and avoid showing Trump in any kind of satirical or mocking context2, even if what's being asked for is a depiction of something that exists in real life: Still, this is not my first rodeo, so I eventually got Bing to generate the imagery[^2] that I wanted (though I did add the text by hand): The image is hosted on a dedicated subdomain, which should allow me to more easily see how often it's used. * * * ### The Ruleset Although ChatGPT referrals make up an insignificant proportion of my real traffic, I didn't want to interfere with the few users who were actually visiting a page from there: the aim was to **only** impact hotlinking. The ruleset therefore needed to consider what was being requested: IF request is for an image AND referrer contains chatgpt.com OR referrer contains perplexity.ai THEN redirect to blockimage I use BunnyCDN, so the constructed ruleset looks like this: If these rules match, the CDN serves up a temporary redirect (a HTTP 302) to send the user's browser to the block image. * * * ##### Including The App The ruleset above only accounts for people who visit `chatgpt.com` in their browser. Although there are _obviously_ some who do that (otherwise they wouldn't have appeared in my logs in the first place), it's quite likely that they're in the minority. We also need to account for embeds within the app, which (rightfully) doesn't set a `Referer` header. We can, however, identify the app by its user-agent > ChatGPT/1.2025.287 (Android 13; FP4; build 2528715) This is _different_ to the user-agent that ChatGPT uses when fetching something (like a web page) to feed into the LLM for summarisation. A second ruleset catches the app's embeds: * * * ### Testing My logs indicate a particular bias towards hotlinking of images included in Vauxhall repair posts (I've no idea why, it's not like they're uncommon cars). So, I went to `chatgpt.com`, toggled the `search` lozenge and asked it to provide me with images showing how to replace the oil pressure sensor on a Corsa D. The result was even better than I'd expected: I hadn't considered that `chatgpt.com` would crop the image, but the effect is all the better. If the user taps the image, ChatGPT opens a modal displaying the full image: Because the CDN serves a temporary redirect (a HTTP 302), the correct images **are** displayed if the user actually clicks the link to visit my site (and will continue to display correctly while the images are in their cache). I couldn't test the mechanism with Perplexity because they actually seem to have stopped hotlinking my images. Although I'm not complaining, it's a little odd: they still hotlink images from other sites _and_ Perplexity is perfectly willing to regurgitate my content. I've no idea whether that's just luck or whether it might be related to my previous anti-hotlink setup for Perplexity. * * * ### Robustness Anti-hotlinking protections haven't been _particularly_ robust _for years_. They used to be a "good enough" measure because browsers sent a `referer` header by default and most users wouldn't know how to (or wouldn't bother) changing that. However, that changed with the introduction of the Referrer-Policy header, which allows sites to instruct their vis1761491468884itor's browsers to send a more limited `referer` header (or not to send one _at all_). This means that `chatgpt.com` _could_ trivially side-step this mechanism by updating their site to set `Referrer-Policy` to `no-referrer`. Of course, it'd be an obvious bad faith move when they could also do what they should have done from the outset: set up a cache so that it's _them_ carrying the bandwidth bill3 rather than the people who's content they're (mis)using. There are a variety of more robust approaches (including tokenisation), but as long as `referer` headers are available, it's probably not yet worth the additional effort. * * * ### Conclusion I appreciate that, for some, it might come across as petty to be complaining about what _should_ quite be small costs. However, they're still costs that I incur _entirely_ for someone else's benefit: if I _wanted_ to support OpenAI, I'd be paying a monthly subscription. Aside from this being **another** example of AI companies outsourcing what should be their own costs, it's also a matter of freedom. If, as some contend, AI companies are free to consume the _entire public commons_ and regurgitate error-prone facsimilies of it, I am _just_ as free to serve up whatever I see fit in response to requests for my content. It _is_ true that I _could_ have served a simple "request blocked" JPG but, in a political context where Trump is issuing executive orders that will censor AI, it's **much** more amusing to ensure that the product of one of his ~~minions~~ supporters serves something more pertinent to the situation. * * * 1. They tended to be quite explicit (or worse, Goatse) ↩ 2. Which is quite fitting, really, considering the fact that I wanted the image to show billionaire CEO's as being Trump lackies ↩ 3. This is far from a niche idea and is what, Google, whose activities _actually bring my site traffic/benefit_ have done for years. ↩

Although I had previously played around creating various mini-sites, today marks the 20th Anniversary of the initial posts onto my first _durable_ site (`benscomputer.no-ip.org`), which would eventually evolve into this blog. I've always primarily written about tech, but have also never sought to limit myself to it. As a result, I've written about a range of topics including privacy, dogs, health, legal dealings, solar, vehicle maintenance and even creating furniture. Life comes with highs and lows and, while I haven't always written _about_ the lows, I have attempted to continue to write _through_ most of them. The aim of this post is to take a quick look back over my first twenty years of publishing. By it's very nature, this kind of post can be a _little_ self-indulgent, so I've made a conscious effort to try and keep that to a minimum. Admittedly, I **did** get a little carried away with some of the statistics. * * * ### Contents * Contents * The Early Days * Then & Now * Posting Stats * Post Tagging * Words * Highlights * Conversations * Conclusion * * * ### The Early Days Much like my current site, benscomputer.no-ip.org was self-hosted. The site lived on an old Fujitsu-Siemens Scenic P300 connected to the (_extremely_ modest) broadband connection that I'd arranged for the HMO that I was living out of at the time. The PC wasn't dedicated to hosting - it was also my day-to-day driver (something that didn't change until I got over-excited after misreading a pricing label in Cash Converters. It wasn't until I got home that I noticed I'd paid 10x more for the second hand Thinkpad than I'd meant to. They refused to refund and I spent the rest of the month hungry as a result). Although I was _interested_ in tech, I wasn't yet working in it - I was a sales assistant in a small Threshers branch. I'd created websites by hand before, but I remember wanting my focus to be on _what I was writing about_ and so chose to use OpenOffice to create the first posts. The traces of that are visible in the page metadata: <meta name="GENERATOR" content="OpenOffice.org 1.1.4 (Linux)"> <meta name="AUTHOR" content="Ben Tasker"> <meta name="CREATED" content="20051007;20080000"> Everything was implemented using inline HTML, I didn't start using CSS until nearly a year later (and even then, only after someone on a forum had convinced me of it's potential). On the first day, I actually published **two** posts, with one following directly on from the other (in hindsight, it does feel a bit "_and another thing..._ "). * DRM is evil * DRM has to be stopped I later changed the site from static HTML to a self-written CMS which used Server Side Includes to populate content by calling BASH scripts. One odd consequence of that transition is that those first two posts appear in the wrong order on the archived copy of the site's index (something that, apparently, has taken twenty years for me to notice). * * * #### Then & Now I can't say that those early posts are something that I feel _particularly_ proud of: My early writing style was _very_ different to the one that I've grown into. A contemporary example of that can be seen by comparing my recent post on ID cards to one on the same topic in 2007. They are both quite opinionated and I now write at quite a bit more length, but the newer post (IMO) makes for much more comfortable and convincing reading. For example, in the earlier post, I wrote: > Quite frankly I believe Mr Blair has his head buried in one of two places, either the sand or his own, probably best not to go there. This is quite an awkward passage to read. I can _hear_ my younger self saying it, but it still doesn't read that way. The newer post contains significantly more reference to stats and metrics, but also makes a statement of authority: > There is, though, a simple truth which underpins all of this: there are **no** simple solutions to systemic issues. Whether it's illegal immigration, political corruption, terrorism or inequality, anyone who promises that "this one thing" will address it is either mistaken or lying. Read in context, both passages convey that the Government are full of shit, but the more recent one does so in a _much_ more accessible manner. Though they may not be to my (current) taste, my early posts **are** still an important part of what led me to where I am now. Although my previous writing style now feels a little awkward and unfamiliar, the posts _do_ also include details that are still part of my life today: > Secondly completely boycott ANY DRM CDs, non DRM versions are usually availbale [sic], so contact the vendor and ask for a DRM free version. I _still_ look for options without Digital Rights Management and contact publishers/authors to ask if un-encumbered options exist. Admittedly, because of the spread that consumer hostile technologies have achieved, I _do_ now allow myself a few more options than I did before: 1. Find unencumbered option 2. Can I get by without it? 3. Take the encumbered option, strip the DRM and send a grumpy email 4. Go without Of course, my writing style wasn't the _only_ thing to change with time - the site did too and, in June 2011, I launched `bentasker.co.uk`. Since then, the site has been through multiple iterations, most of which are captured in a post that I wrote when migrating to Joomla! 3 (before later moving to Nikola). The sitehistory tag also contains screenshots of each era of the site. * * * ### Posting Stats I don't routinely collect stats about my posting activity. They're are also a bit harder to come by with static sites because there isn't a convenient database to run queries against. So, I hacked together a script to scrape posting information from `www.bentasker.co.uk` and the `benscomputer.no-ip.org` archive. The script was then able to write simple stats into a time series database for querying and visualisation: Note: the overlap between `bentasker.co.uk` and `benscomputer.no-ip.org` exists because (in an effort to ensure that it was preserved) I started republishing some old content. However, I eventually found a _meaningful_ backup of the original site and published the archive instead. Over the last 20 years, there, quite visibly, have been _some_ gaps in posting. Some of those gaps are larger than others, but if we chart out the number of months, per year, with at least one posting we can see that some years _really did_ fare worse than others: 2017 did really poorly, with me neglecting the site for _8 months_ of the year. Most of the years with big misses, though, seem to be because I was focusing on something else. For example, if we factor in posting to `snippets.bentasker.co.uk`, I only actually missed 3 months of 2017. There are still exceptions to this, though, and I've written in the past about sometimes suffering from writers block. Breaking my posts down by category leads to a _messy_ chart, but if we look at the proportional makeup of the 10 biggest categories, we can see quite a significant bias towards Linux documentation1 `documentation:linux` and `blog:general` are both quite broad categories, so their dominance isn't too surprising. What _is_ quite interesting, though, is seeing `blog:house stuff` in third place: I only launched that category in 2021, so it's only taken four years for it to start to break into the top three (our solar probably has a lot to do with that). If we chart out the top 3 categories per year, we can also start to see how what I've chosen to write about most has shifted over the years: In many ways, this is a reflection of how _my_ life has changed and, in particular, how I spend my free time. Some of it is borne out of necessity rather than choice, but I now spend a lot more time doing things around the house than I do fiddling with new software. * * * #### Post Tagging Tag based stats only represent more recent posts - the concept of attaching tags to posts didn't really exist when I first started and wasn't available in Joomla! until 2013. My site's JSONFeed includes details of which tags are applied to each post: { "id": "https://www.bentasker.co.uk/posts/blog/privacy/id-cards-come-around-again.html", "url": "/posts/blog/privacy/id-cards-come-around-again.html", "title": "ID Cards Have Come Around Again", "date_published": "2025-10-03T21:21:00+00:00", "tags": [ "blog", "id cards", "identity", "privacy", "united kingdom" ] This makes calculating stats on tagging frequency quite easy: import requests from collections import Counter tag_counters = {} r = requests.get("https://www.bentasker.co.uk/feed.json") for entry in r.json()["items"]: for t in entry["tags"]: if t not in tag_counters: tag_counters[t] = 0 tag_counters[t] += 1 # Turn into a counter c = Counter(tag_counters) # Exclude top-level category names # unsurprisingly, they dominate excludes = [ "documentation", "blog", "republished", "video" ] # Maintain a counter - we're going # to be excluding entries but still # want to show the top 10 x = 0 for t in c.most_common(30): if t[0] in excludes: continue print(f"{t[0]}: {t[1]} posts") x += 1 if x >= 10: break This indicates that the top 10 most used tags on `www.bentasker.co.uk` are: Tag | Uses ---|--- linux | 99 howto | 92 security | 81 privacy | 73 house stuff | 67 release notes | 48 infosec | 45 vehicles | 43 software | 39 opinion | 37 There _is_ a limiting factor here, though: I've used a _huge_ number of tags over the years, which (until recently) left my tags listing looking somewhat ridiculous: The density of this made it quite difficult for me to skim read existing tags when publishing new posts, so there are likely to be more than a few posts which are missing at least one relevant tag. * * * #### Words Given that I'm talking about writing, it'd be remiss of me not to take a peek at word count. | Words | Profanities | Profanity Ratio | Avg Words Per Post ---|---|---|---|--- `benscomputer.no-ip.org` | 220,586 | 102 | 1:2163 | 585.1 `www.bentasker.co.uk` | 865,570 | 244 | 1:3548 | 1059.4 **Total** | **1,086,085** | **346** | **1:3139** | **909.7** The total word count is actually quite a bit lower than I expected. I suspect, too, that the rate of profanity is quite a bit lower than _people who know me_ would expect. Python's `collections.Counter` allows us to check which words I'm most likely to use: c = Counter(profanity_counter) for t in c.most_common(20): print(f"{t[0]}: {t[1]}") The top place makes sense, because I used to have a software project called BUGGER. The second place also makes sense because, well, I **do** use that word quite a lot in day to day life. Which leads me to something quite interesting. Earlier in this post, I observed that my original writing style was quite conversational: I wrote things, more or less, as I would _say_ them. Combining that with `benscomputer.no-ip.org` having a higher proportion of profanity, I would _expect_ that it also included a _wider_ range of that vocabulary. But... that's **not** the case - in fact, it's not even possible to populate a top 10: bugger: 74 fuck: 15 shit: 5 cock: 4 bollocks: 2 knob: 2 If we assume that every use of `bugger` was in relation to the software project, the rate of profanity on `benscomputer.no-ip.org` drops considerably: once every 7876 words. I've always sworn a lot, a condition that was exacerbated by working with the military, but the increase can't be driven by that because those years passed while I was still posting to `benscomputer.no-ip.org`. So... when did it start? There was a fairly low frequency of posts with profanity, but the rate spiked in 2021, recovered and then remained in an elevated position from late 2022 onwards. That increase _broadly_ aligns with me coming off of Twitter: had I still been on Twitter, it's quite possible that some of the swearier posts would have instead been Twitter threads. Curiously, we don't see the same trend when looking at the _number_ of profanities per month: So, I've written _more posts_ containing profanity, but the overall level of swearing isn't much higher than it was in 2018. It's not _entirely_ clear why that is. * * * ### Highlights Over the last 20 years, I've published quite a _lot_ of content - there are 819 posts on `bentasker.co.uk`, with another 700 on `benscomputer.no-ip.org`. It feels a little strange to use the word "adventure" in relation to sitting at a computer and writing, but over the years I've had and documented a range of experiences. Distilling those down to a small set of highlights is, inevitably, quite difficult, but the following are the experiences that have stuck in my mind: * Taking a (now defunct) ISP through arbitration * Convincing a local council to make their permit parking system more inclusive * Creating a children's book * Being part of the fight against the introduction of Phorm in the UK * Messing around with Powerline adapters (and later discovering a vulnerability in them) * Writing about health and medication issues and living with chronic pain * Having my research into the "unhackable" Bitfi cryptowallet referenced at Defcon by a researcher _dressed as Pikachu_ * Catching a digital intelligence outfit in the course of making a mistake * Giving my fishtank an IP address * Mistreating Bing's AI Chat, Perplexity and Deepseek (warning: there isn't enough mind bleach _in the world_) * Hyperfocusing on packet captures and the behaviour of Golang's net/http * Spending an afternoon in the Sizewell Nuclear Power Station control room simulator * Creating a TV station * Somehow getting mistaken for a qualified battery storage engineer _by the supplier_ * "Dating" a romance scammer Of course, it's not _all_ been sweetness and honey - I've also had legal threats, abusive emails, nuisance abuse complaints and sometimes quite concerted efforts to compromise the site. But, the positives definitely outweigh the negatives. * * * #### Conversations Blogging _sounds_ like a one-way interaction: I post something and then people (hopefully) read what I've written. But, I've never found that to truly be the case. Over the years, I've had a lot of wonderful conversations with people who've read _something_ that I've written and felt motivated enough to contact me about it (whether via email or social media). Some made contact to provide feedback, others had questions that my post didn't answer and _occasionally_ it was to let me know that motivated them to do or create something (DNSChat seems to have been _particularly_ good at that). I've always written _for me_ , but there's something quite gratifying about finding out that it's helped someone else. * * * ### Conclusion I've never been _particularly_ prone to looking to the future, but I don't think that I'd ever have expected to still be _enjoying_ writing 20 years later. One of the things that has really stood out to me while preparing this post, is just how different my writing style has become. My writing used to be _extremely_ conversational and _very_ opinionated. Although that change is a good thing, it does make me wonder what my writing style might be in another 20 years time. Over the years, I've been extremely fortunate in having the time and opportunity to explore and write about quite a range of things. It's been **a lot** of fun and, hopefully, the next few decades will be too. For me, Microsoft ~~Bing AI Chat~~ CoPilot summed it up best: * * * 1. BTW, I use Arch ↩

I've got an SMK Synergy XS26 Air Pistol. It's a pretty satisfying gun to shoot, but ever since I got it, it's been intermittently dieseling (when lubricant gets from the cylinder into the barrel and ignites under pressure). It dieseled whilst attached to my chronograph and was measured throwing a 15.89 grain pellet out at 430 foot/s. That works out to around 6.53 ft/lbs, which is _over_ the UK legal limit for an air pistol! A louder bang and bit of smoke might be a bit of fun, but getting prosecuted for a firearms offence would _definitely_ not be (however much bad luck it might take to reach that stage). On top of that, dieseling isn't great for the seals or shot consistency, so I decided to strip the pistol down and clean it out. This post details the process of disassembling and re-assembling a Sportsmarketing Synergy XS26 Air Pistol * * * ### Strip Down Before you start: be aware that there are a number of small (and easy to lose) parts. Do this on a clear workbench and move slowly so that nothing can spring loose. Ensure that the breech is empty and that the pistol is not cocked. Turn the safety _off_. There's a screw on either side of the gun, remove both: There's also a screw on the underside: Keeping the gun upside down, grip the barrel with one hand and carefully pull the stock up and away. Just behind the trigger, there should be a small plastic pin, backed by a spring. The pin is loose, so if it's not there, it's either still in the stock or has fallen out onto your workbench. Take them out (you can safely tip the gun to get the spring out) and set safely to one side: Now firmly grip the bit of plastic which carries the rear sight and slide it back off the cylinder: Just ahead of the trigger mechanism, there's a pin holding the trigger assembly onto the cylinder. Knock it through and remove it: Towards the back of the assembly is pin with a much bigger diameter - **DO NOT REMOVE THIS**. It doesn't hold the assembly onto the cylinder - it's the pin that the sear pivots on. You also need to take care not to lose it, mine was extremely loose and fell out. If yours also falls out, be aware that there is also a small spring in there - mine went AWOL so I had to replace it (more on that later). Just behind the sear, there are two plastic hooks which sit inside the cylinder. So, slide the entire trigger assembly backwards and you should then be able to lift it up and away: Next, we're going to remove the barrel. Where it meets the cylinder there's a large flat head screw - loosen and remove this, but take care not to lose the nut from the other side. This should leave a sleeve visible: Drive this through. Unhook the cocking lever from the cylinder Pivot the barrel out and away. You should now be left with just the cylinder The next step is removing the end plug and spring guide. However, the spring is under tension, so we need to use a clamp to take up some of the tension before it's possible to drive the retaining pin through. The cap has a dip in it, so I put a socket against it to help the clamp reach. Note: this is far better and safer with a G cramp (or any other type that screws/unscrews), but mine was too big. With the clamp providing opposing tension, it should now be easy to drive the retaining pin through (if it isn't, you've either over-tensioned the clamp or not provided enough): Now we need to release the tension - **this** is where you _really_ want to have used a screw type clamp. I put a heavy blanket over it to stop anything shooting across the workshop and then used a piece of wood to knock the clamp release. Once released, the end plug and spring guide stick a little way out of the cylinder: There's an important thing to note here. In my photo, there's a big hole in the top - there _should_ be a little silver nut in there, but my poorly controlled spring release led to it flying off. I realised later and fished it out of the blanket, but it's missing in a couple of the photos below The piston can now be slid out of the cylinder leaving you with a dismantled pistol: * * * ### Cleaning The reason that the gun had been dieseling so frequently was that it _absolutely lousy_ with oil, which had obviously been making it's way ahead of the piston seal. I sprayed some carb cleaner on to clean the spring, piston and cylinder. After drying everything off, I gave the piston and spring a pretty liberal application of silicone airgun grease (being careful not to apply any ahead of the seal): Note: it's important to use a grease specifically formulated for airguns - using a random grease could introduce ingredients which are more prone to ignition, giving a _much_ bigger bang than intended. * * * ### Re-Assembly Once the piston and spring have been slid back in, we need to compress the spring enough to be able to push the retaining pin back through. First, put the nut back into the hole in the end cap. I couldn't safely put a socket between the end cap and the clamp this time - it kept trying to bend off to an angle and was pretty clearly at risk of pinging off. So, I put the clamp directly against the end-cap and closed it down as far as it would go. Unfortunately this wasn't _quite_ far enough, but it **was** far enough for me to be able to push a screwdriver through. So, I used a screwdriver to hold the clamp in place before loosening the clamp slight and sliding a single penny in between Tightening the clamp again allowed the hole to line up so that I could easily push the retaining pin through: I applied a small amount of grease to the sides of the barrel and pushed it back in place, before sliding the sleeve through I pushed the bolt through and tightened the nut onto the end of it I slid the cocking lever back into place. Next it was time to reinstall the trigger assembly. Hopefully yours didn't come apart during dissassembly. If it **did** then skip to the trigger assembly section below before continuing. I found that it was easiest to re-install with the sear engaged: If yours isn't engaged, run a flathead backwards along it and it should click into place. Carefully line the hooks up with the slots at the back of the cylinder and then slide the entire assembly forwards. Push the front retaining pin back into place (note: I found this was one of the most awkward bits of the whole process, you might need to use a screwdriver as a lever to carefully help line the holes up) Slide the rear sight housing back into place - you may need to carefully lift the sight itself as you do this: the windage adjuster is a screw which pushes against the cylinder to raise the sight up: Flip the gun upside down and put the small spring and plastic probe back in place (rounded end facing out) Carefully slide the stock back over before reinstalling the side screws Install the screw on the underside of the butt (if you _did_ forget to put the nut back into the end-cap, you're about to find out!) It's re-assembled: * * * #### Safety Checks However, you're not quite done as you also need to perform some safety checks. Go somewhere safe and then work through the following checks, stopping if any fail: 1. Ensure that the safety catch operates (it should not be possible to pull the trigger with it engaged) 2. Shake the pistol, you should not hear any rattling 3. Ensure that the pistol cocks reliably 4. Ensure that the pistol fires reliably (make sure there's a pellet in there, springers shouldn't be fired empty) 5. Repeat 3 and 4 a few times to ensure that it's consistent 6. Cock the pistol but _do not_ load a pellet. Holding the pistol out and away from you, use your other hand to hit and knock it - ensure that knocks don't lead to an accidental dishcharge. 7. If you've somewhere you can safely do so, drop it onto some grass: again, it should not fire 8. Finally, load a pellet and fire * * * #### Trigger Assembly Rebuild If, like me, your trigger assembly came apart, it'll need re-assembling. It should go without saying, but it's _extremely_ important that the assembly is put together properly and works reliably. If it's re-assembled incorrectly, the best case scenario is that the pistol won't cock, but the worst case is that the gun fires without you touching the trigger! If you've still got your user manual, there should be an exploded parts diagram on page 10. The trigger assembly looks like this **Hopefully** it's only your sear that has come out. The right hook (number 36 in the diagram) sits at the bottom of the assembly. It's the bit visible on the top of this picture: The sear spring sits over the knobble on the end of the right hook so that it can push up to the bottom of the sear. As I mentioned earlier, my sear spring went missing1. After searching the workshop thoroughly, I still hadn't found it, so I took a spring out of a clicky pen and cut it down to the right length. Although it's tempting to put the sear in first and then work on the assembly upside down, doing this doesn't really work - without the right hook in place, the spring pushes it too far round. Once the right hook and sear spring are installed, the sear should go in with its hook at the front and the slanted section at the back: With both installed, you should be able to look into the inspection hole and see the sear engaging onto the corner of the hook It should not be possible to push the sear forward. The idea is that, when it's locked like this, it'll be holding the piston back. When you pull the trigger, though, the right hook should move and the sear should click forwards Using a flat implement, you should be able to reliably click the sear back and then pull the trigger to release it: There should be a video here, you may need to visit in a browser Now that your trigger assembly is back together, go back to installing it. * * * ### Conclusion SMK air rifles and pistols are pretty popular, but they're also built to a cost. That _can_ make working on them a little frustrating as they're made up of quite a few loose little bits. Stripping the XS26 down and giving it a good clean _does_ seem to have resolved the dieseling issues. The pistol is still a little down on power compared to the manufacturer's claimed spec, but has more than enough for my needs. * * * 1. I actually have some doubts about whether it was ever installed in the first place - the gun hasn't always cocked reliably, something that's ceased to be an issue since this rebuild ↩

Back in 2006, Tony Blair was _extremely_ keen on the idea of introducing mandatory ID cards. Fortunately, they were scrapped by the coalition government following Labour's 2010 general election drubbing (for the historians, though, it is worth noting the cards were originally a Tory idea, and that Blair originally campaigned _against them_). 19 years later, Tony Blair (via the Tony Blair Institute) continues to hawk the idea of a "super identity card". Unfortunately, the current government have announced plans for a mandatory "BritCard", claiming that the scheme will > help combat illegal working while making it easier for the vast majority of people to use vital government services. Digital ID will be mandatory for Right to Work checks by the end of the Parliament. Unfortunately, rather than questioning whether they should be taking advice from a yet-to-be convicted war criminal in the first place, the Government have decided that Digital ID should form a key part of their response to the Reform wave which is currently (and quite regrettably) sweeping the nation. ID cards, particularly those envisaged by Blair, have significant issues and are very unlikely to do anything to address illegal immigration. The purpose of this post is to explain (some of) why. * * * ### Preventing Illegal Migration Let's start with the core of the Government's claim. When announcing the scheme, Kier Starmer said: > I know working people are worried about the level of illegal migration into this country. A secure border and controlled migration are reasonable demands, and this government is listening and delivering. > > Digital ID is an enormous opportunity for the UK. It will make it tougher to work illegally in this country, making our borders more secure. Other material released by the Government notes that this is because Right To Work checks will be amended to require that the worker/applicant provide their digital ID. The _theory_ is simple: an illegal immigrant _won't_ have digital ID and so won't be able to pass the checks (eventually putting immigrants off coming here _at all_). However, Right To Work checks are _already_ pretty strict, with only a small set of documents considered suitable proof of ID: * A passport (showing the holder is a British citizen, or has the right of abode) * A passport (showing the holder is an Irish citizen, or has the right of abode) * A passport endorsed to show the holder is allowed to stay in the UK indefinitely * A document issued by Jersey, Guernsey or the Isle of Man _which has been verified as valid by the UK Home Office_ * A current Immigration Status Document (issued by the Home Office) noting an indefinite right to stay in the UK **accompanied by** an official document providing a National Insurance Number * A UK, Channel Islands, Isle of Man or Irish birth (or adoption) certificate **accompnaied by** an official document providing National insurance number * A certificate of naturalisation as a British Citizen **accompnaied by** an official document providing National insurance number Although the Prime Minister's statement made reference to "finding an old utility bill", those aren't actually of any use during Right To Work checks. To summarise the list above, the only forms of ID _currently acceptable_ for Right To Work checks are documents which have been issued or verified by the UK Government. * * * #### Current Workers It's obviously somewhat difficult to get accurate numbers of those working having arrived here via illegal means. The clue to the reason for that is in the name "undocumented migrant". However, the Government's 2025 report notes that, in the prior year, there were 7130 arrests for illegal working and 2105 civil penalties were issued to employers. We don't know the detection rate, so it isn't possible to extrapolate to a total number of workers, but we also probably don't need to. What really matters (and should surprise no-one) is that there _are_ people working illegally, despite the existence of the system that Digital ID will _have_ to rely on. But, how did that come to be? Maybe _some_ of those workers presented fake ID (the Government certainly suggests that forgeries can be a problem), but it seems far more likely that the majority were working for an employer who _didn't conduct Right To Work checks in the first place_. The number of penalties issued supports that theory too - it seems relatively unlikely that a penalty would have been levied if an employer could show that they'd checked ID in good faith. If Right To Work checks aren't actually being conducted, it doesn't matter what additional ID the Government introduces. In fact, they could introduce _mandatory tap classes_ and have about the same level of impact. * * * #### Identifying Approved Migrants Preventing undocumented workers from working might be quite difficult, but perhaps the aim is to provide legal/documented migrants with the means to prove that they **are** allowed to work? The problem is, **we already do that**. Until the end of last year, migrants were issued with Biometric Residence Permits: The Government stopped issuing these last year and, instead, replaced them with eVisas. eVisas are _very similar_ in function to the Right To Work system that the Government seems to envisage: a smartphone app allowing migrants to prove that they have the right to work in the UK. So, as a country, we **already issue a digital ID to approved migrants**. * * * #### What Is Going To Change? Let's lay out before and after: Group | Before | After ---|---|--- **Undocumented, working illegally** | No Digital ID reqd | No Digital ID reqd **Documented, working legally** | Digital ID reqd | Digital ID reqd **UK National** | No Digital ID reqd | Digital ID reqd The only group who see a meaningful change in requirements, is UK Nationals. It will, effectively, become mandatory1 for UK Nationals to have a digital ID. Approved Migrant workers already have one and black market workers will continue to not have or need one. It _might_ also become a little easier for employers to conduct Right To Work checks (of course, being a Government procured system, it _could_ also get more difficult and unreliable). Unless the Government's position is that the _Home Office are failing_ to properly control or vet official documents, it seems likely that the introduction of digital ID cards will do little to _nothing_ to prevent illegal working. Even that has it's own precedent: in 2005, the minister in charge of the original ID card scheme admitted that: > perhaps in the past the government, in its enthusiasm, oversold the advantages of identity cards [and] did suggest, or at least implied, that they might well be a panacea for identity fraud, for benefit fraud, terrorism, entitlement and access to public services It seems _almost inevitable_ that we're being subjected to a similar level of hyperbole now. * * * ### Learning From History Each of us _already_ exists in multiple Government databases (Driving License, HMRC & Passports etc etc). It might seem odd, then, to be concerned about adding one more. The answer to why lies, partly, in the past. Although the details of _this_ scheme have yet to be determined, looking at the _previous_ ID card scheme can help us see what it might eventually look like (especially given Blair's involvement in both). Under Blair, ID cards started out as just another form of ID (and a voluntary one at that). However, the scheme suffered from **significant** scope creep: * The underlying database (The National Identity Registry) could contain 50 different pieces of information about individuals * It was decided that Passports would be tied to the NIR - those applying/renewing would be entered into the database even if they didn't opt into an ID card * In order to reduce costs, the Government shifted from creating a completely new system to bastardising the DWP's Customer Information System, creating concerns around governance and accountability in the process * From November 2008, Identity Cards became compulsory for those applying to stay as students, or based on marriage * In order to address cost concerns, the Government intended to charge _other Government departments_ , driving up the cost that individuals and businesses would need to pay for other services Although they started out as voluntary and limited in scope, that scope grew quite quickly. As a result of limited uptake, the Government sought to find new ways to drive adoption. In January 2008 a leaked document was published in The Times, noting that > Various forms of coercion, such as designation of the application process for identity documents issued by UK ministers (eg, passports) are an option to stimulate applications in a manageable way. The Identity Minister at the time noted that ID cards presented a "convenient way for young people to prove their age when going to bars" (something, incidentally, that the current Government has _also_ referred to when responding to a petition against Britcard). **A month later** , while on a trip to promote ID cards, that same minister had to admit that even she had forgotten to bring hers with her. Ultimately, there was such a pervasively low level of demand/sign-ups that Civil Servants were _even_ encouraged to email friends and relatives in order to try and convince them to sign up. That Government spent _significant_ amounts of taxpayer money on a scheme that the public didn't want, use or even ultimately benefit from. Unfortunately, the current scheme _seems_ to be being driven by some of the same minds. * * * ### Digital Sovereignty Be Damned This is a topic that many non-techies are unlikely to care about... _for now_. The Government's stated intention is that the new Digital ID will "sit on people's phones" (they have similar plans for a digital Driving License, both of which will live in a Digital Wallet). This aspect of the scheme hasn't been well defined yet, but we can look at similar schemes in order to build expectations around how it's likely to work. Earlier this year, the EU's Age Verification App came into focus as a result of a stated intention to rely on the Google Play Integrity API (and, on iOS devices, Apple App Attestation). These APIs help apps to verify the device that they are running on. However, a hard dependency on them _also_ prevents apps from being used on devices which do not run Google or Apple's services. Essentially, it means that, not only do you need a smartphone, it needs to be running specific software. User running things Lineage and GrapheneOS are, obviously, very much a minority use-case. However, user population sizes are only _one small part_ of this2. The much more crucial aspect of this dependency is that it will tie _every single British citizen_ to the services of these US tech giants. * * * #### International Trends This is happening at a time when the US is still coming to terms with having installed a mad king, whose unpredictability has delivered repeated shocks to global markets and international relationships. Rather than forming part of the resistance, the US tech giants have been bending the knee in various ways. Earlier this year, Microsoft suspended services to a member of the International Criminal Court after Trump sanctioned ICC staff for investigating the role of Israeli politicians played in atrocities perpetrated against Palestinians in Gaza3 . That shot across the bows has not gone unnoticed and various Governments are responding by moving away from US providers: * Austria's Armed Forces are ditching Microsoft Office and will use Libre Office instead "to achieve greater digital sovereignty" * Denmark is also on a path towards moving away from Microsoft * A German state (Schleswig-Holstein) had already started its switch * Dutch parliamentarians passed measures to encourage the Government to move * The European Commission is being lobbied to take action (there's even talk of public sector orgs standardising onto an EU OS) These aren't simply cost saving measures: in every one of these examples, Digital Sovereignty has been listed as a motivating factor. It helps that moving to non-US providers is _far_ more viable than it's ever previously been. When I looked earlier this year, I found that there were a wide range of possible offerings. * * * #### Backend Services In the context of all this upheaval and uncertainty, it seems insane that the UK Government could consider a green-field project which ties citizens to providers that other Governments are increasingly seeking to escape from4. Unfortunately, we've _only_ talked about the app that users will need: there will be back-end services too. According to The Mail (ewww, I know), Ministers have been meeting with the Ellison Institute of Technology, suggesting that Oracle might well be in a position to win the contract to build and operate the Right To Work system. For the non-techies out there: Oracle are renowned for squeezing every penny possible out of their ~~victims~~ customers. They are an extremely aggressive company who've a reputation for deceptive licensing terms, contract lock in and expensive litigation. There are also serious issues of trust: Just this year, Oracle were accused of trying to cover up a cybersecurity incident which impacted customer data. Large databases, such as the one that would need to underpin the Digital ID, _do_ tend to make for a juicy target - the very least that taxpayers should be able to expect is providers who can be trusted to do the right thing when they screw up. Oracle's billionaire owner (Larry Ellison) is quite something too, having once said: > Citizens will be on their best behavior[sic] because we are constantly recording and reporting everything that’s going on. Oracle already have a _lot_ of fingers in the public sector pie, but it really is in no-one's interest to give them more opportunity to (contractually) pick our pockets. What all of this means is that there is a possible future where, not only will UK citizens need a phone made by a US provider, but their data will be held in a system controlled by another US provider (and potentially in reach of the US Government via the CLOUD act). All funded by our taxes, of course. * * * ### Conclusion The "Britcard"5 isn't much more than a proposal at the moment, and there is still **a lot** to be defined. However, the same was true of the scheme at the turn of the century, which grew from its early proposals into an expensive and privacy-hostile mistake. It should be more than a little concerning _for everyone_ that Tony Blair has apparently had a hand in **both** , especially given his close ties to Oracle's Larry Ellison. A quick skim of history suggests that the current Government is following a very similar playbook to Blair's government: both Governments have claimed that ID cards will save us from the _en vogue_ existential threat. Back then, it was terrorism, today it's illegal immigration. What history makes clear, though, is that ID cards do nothing to address these problems on their own. The only way that they can ever have even a tiny chance of doing so, is by being tied to onerous (and sometimes outright dystopian) systems. The previous Labour government's scheme floundered and their response was to try and require further adoption. This Government's statement makes it clear that they intend coercion from the start, by tying ID cards to the Right To Work scheme (even if also making weasely claims that it won't be illegal not to have one). There's is, though, a simple truth which underpins all of this: there are **no** simple solutions to systemic issues. Whether it's illegal immigration, political corruption, terrorism or inequality, anyone who promises that "this one thing" will address it is either mistaken or lying. If digital ID _is_ introduced, there's a significant possibility that it will tie UK citizens to providers that other Governments are currently working to _escape_. Whatever Trump's intentions for the future of the US might prove to be, we too will be tied to them and, as contracts unfold, taxpayer money will go not to local causes but to filling the coffers of US billionaires. * * * 1. The Government insists that it won't be illegal to not have one. They have had to concede though that it'll be pretty hard to exist without one, given that you won't be able to work (or potentially, rent) without using one to pass ID checks. So, not mandatory, but only as long as you're not opposed to starving out in the cold. ↩ 2. I do, though, have a lot of sympathy for the argument that introducing something mandatory means your solution **must** cover everyone ↩ 3. Trump's not done yet, either - he's reportedly considering further sanctions ↩ 4. Yes, I know... the UK and the US enjoy a "special relationship". What happens, though, when he unilaterally slaps tariffs onto lube? ↩ 5. Another lesson they haven't learnt from the past. The original ID cards were first branded as "entitlement" cards, but the name was dropped after focus groups said the name was weaselly. ↩

I've been getting back into shooting recently. There's a fantastic outdoor centre just up the road, boasting a number of woodland ranges. But, although trips there are good fun, it's not exactly the most cost-effective way to shoot an air-rifle. We're pretty fortunate in that we have quite a long and wide garden so I decided to build a small backstop to allow me toshoot safely in the garden. This post describes the process of building a semi-portable backstop for air-rifle shooting. Note: the UK legal power limit for an unlicensed air rifle is 12ft/lbs - if you're elsewhere, with something like a 26ft/lbs rifle you _might_ want to double up a bit. * * * ### Pellet Catcher vs Backstop First, let's be specific about what it is that I'm building. A pellet catcher (or pellet trap) sits behind the target but **only** the target: it'll stop pellets that hit the target card, but not any that are off (mistakes can and do happen). I've got a couple, which look like this: These traps provide a convenient way to hold the target card **and** stop the pellet afterwards. However, they don't provide any security in situations where the pellet has gone wide. Just as importantly, these pellet traps do nothing to protect the target card from rain - despite having a sheltered area to shoot _from_ I can only really use them when the British weather deigns to allow it1. A backstop is wider than the target and can also be used with targets that aren't so easily mounted on a small metal box (like spinners). Although a backstop doesn't **have** to provide protection against the weather, mine is going to. * * * ### The Design I wanted a small wooden unit that targets could sit inside so that, when not in use, we'd have a small unobtrusive wooden cabinet in the garden. After a bit of browsing around, I settled on one of these: I selected this unit based on a number of criteria * Price: I'm likely to put holes in it, so I _really_ didn't want to spend a fortune * I couldn't go _too_ cheap: cheaper items tend to be made of chipboard which doesn't do well in the wet * It's made of Fir, which is pretty environmentally sustainable * It's got a strip of roofing felt on top to help keep the contents dry The last one might sound a bit "duh", but there was another fir unit at around the same price with a flat slatted roof - the reviews were full of people complaining about it not being waterproof. * * * #### Stopping Pellets No cabinet on the market, on it's own, is going to reliably stop air rifle pellets (particularly over time), so the plan was to build a new multi-layered back into the unit. That back would be mounted at an angle behind heavy curtain strips, with the middle shelf left in to create two compartments: The false back is mounted at an angle for a couple of reasons: * it helps to redirect any ricochets downwards * for pellets which do penetrate it, the distance needed to pass all the way through is increased The backboard is made up of 4 layers: The pellet first hits the rubber, before passing in the cork. Although the rubber is tough, both layers compress easily, sapping some of the pellet's energy. If the pellet manages to pass through those, it then reaches the much harder 9mm plywood. A sheet of steel could achieve a similar effect, but the aim of the backboard isn't _just_ to stop the pellet, it's designed to do so as quietly as possible: I have neighbours, so flinging lumps of lead at a noisy backboard is likely to be unwelcome. Because the backboard is mounted at an angle, there's space to add more material behind it to ensure that any pellets that make it through the plywood do not exit through the back of the cabinet. * * * ### The Build First things first, I built the cabinet (which came as a flatpack): I left the shelf out in order to see whether it would be feasible to have a single backboard running the height of the cabinet - I ultimately decided that it was a little too shallow for that and re-fitted the shelf. I drilled a couple of holes on either side of the unit and screwed a bit of timber across the top. To help reduce the noise made by vibrations I stuck (and then stapled) some adhesive backed foam across the top and back of it: I ran another piece of foam across the bottom of the compartment too, so that the backboard would only ever make contact with foam. It was time to make the backboard itself. I took a piece of 9mm plywood and cut it down to size (annoyingly, the damp seems to have got to my wood-stock) Having sanded the mould off, I sprayed the surface with spray glue and then stuck down a layer of cork tiles: Then, I added a second layer, shifting the tiles up slightly to help make sure that the tiles met in different places within each layer. Finally, I wrapped the whole thing in rubber flooring The rubber flooring comes on a big roll and is only about 3mm thick. Had it been much thicker, though, I'd have struggled to get it to wrap around the edges of the board. * * * #### First Test I was _fairly_ sure that my design was sound, but I wanted to be certain: I didn't want to waste my time finishing the build only to find that the backboard didn't stop pellets as well as intended. So, I slotted the backboard into place and put out two of the pellets that I commonly use: one spiked, the other a diablo: For the first test, I used my air pistol. The manufacturer _claims_ that it sits just below the UK legal limit for air pistols (6 ft/lb). However, my (admittedly cheap) chronograph suggests that it's delivering more like 4.5 ft/lbs. From 5 metres away, I shot the backboard with the diablo. I didn't penetrate the rubber and instead left a little impression of itself This wasn't necessarily a good thing: although the backstop went undamaged, the pellet had _bounced_ (albeit in this case downwards). Next, I loaded the spiked pellet, which also left a little impression in the rubber. I dispensed two more pellets and set the air-rifle up at the 5 metre mark. This is **much** closer than I'd ever normally shoot, but the aim was to ensure that the backstop did actually trap and stop the pellets. UK law allows air rifles to have more power than pistols. Again, the manufacturer claims that it's just under the limit (12 ft/lbs3), but my chronograph puts it at more like 10.5ft/lbs. Both pellets went through the rubber and stopped inside the cork layer * * * #### Making a Curtain Happy that the backboard was able to effectively trap pellets, the next step was to make a curtain to sit in front of it. I'm sure that this isn't the _actual_ name for it, but I know the style as a butcher's curtain (I guess that's probably where I first saw one): a curtain made up of thin strips that you walk straight through. The curtain serves two main purposes. Firstly, it helps trap rebounds: we want to ensure that bouncing pellets do not find their way out of the cabinet because they may ultimately end up hurting somebody. Secondly, it steals energy from pellets as they come in. They're not capable of stopping a pellet on their own, but as the pellet passes through it, the curtain strands will move, converting some of the pellet's inertia into curtain movement. To make the curtain, I ordered a cheap rubber doormat: The fabric attached to it is intended for wiping boots, but helps by adding weight. I turned the mat length-ways and cut strips into it, before stapling it along the piece of wood at the top of the unit. As an idea of scale, the pellet trap in this photo holds a 17x17cm target card: * * * ### Testing It was time to give the unit a proper test and see whether any pellets escaped through the backboard into the gap at the back. Obviously, there's not a **huge** amount of value in testing by shooting at a metal pellet trap, so I stood a wood block with some spinners on the shelf too, and then shot from 20 metres away. I used both the rifle and the pistol and shot _a lot_. When I checked it, the backboard was pretty well peppered The photo shows a number of marks where pellets have bounced rather than penetrating. The pile of spent pellets at the bottom suggests that the curtain did it's job. But... did any pellets make it into the back? One did, although it seemed to have taken an unusual route: There was no visible hole in the backboard's plywood. However, on the right hand side, there's a chip in one of the wood panels which make up the cabinet's side. The pellet in question appears to have managed to squeeze between the backboard and the side. That should be a pretty rare occurrence, though, and the cabinet's walls are thick enough that a pellet's unlikely to make it through them that way. The curtain, for it's part, was riddled with holes but holding together just fine * * * ### Conclusion The backboard won't last forever (nor is it designed to), but I now have a cabinet which can be used to provide an extra layer of safety should a shot veer off course (or, of course, be aimed at a backless target like the spinners). Our garden isn't nearly as tranquil2 and idyllic as the woodland range down the road: But, it _is_ a lot closer and cheaper to use. Currently, only the top of the cabinet has a backstop installed. My intention _is_ to build one into the lower half, but I thought it best to wait and see how I get on with this build before duplicating it: the curtain already sits a little close to the front for the spinners, so it **might** be that I try a different design for the bottom. The only thing that's sure, is that it'll probably involve rubber flooring: I've still got more than half a roll left! * * * 1. In fairness, so far this year the weather seems to have been focusing on trying to melt rather than drown us ↩ 2. OK, maybe tranquil isn't quite the right word: it's not exactly _quiet_ with shotguns going off and bits of clay dropping out of the sky. But it's a _different type_ of background noise and no-one is going to interrupt you. ↩ 3. Air rifles with higher power output can be obtained with a firearms license, but then you end up with all of the drawbacks of shooting rimfire but without the benefit of the cordite smell. ↩

Now that the Online Safety act has come into effect and the _inevitable consequences_ are starting to play out, I decided to email my MP to try and find out what the Government intends to do to unpick the mess. Doing this felt particularly important in light of the divisive comments made by Peter Kyle MP (I feel _extremely_ dirty agreeing with anything that Farage has said, but even a stopped clock is right twice a day). Although I didn't want my email to focus on Kyle, it _is_ a topic which merits some discussion. Kyle pointed to the cases of Nicholas Hawkes and Tyler Webb as examples of the Act's benefit. With an act as broad as the Online Safety Act, there will always be _some_ successes. However, both are **also** examples of cases that were pursued without needing _any_ of the act's more draconian measures: Both Hawkes and Webb were caught because of the courage of their victims, who took screenshots and reported the behaviour to the police. Requiring adults across the country to share sensitive personal information had _absolutely_ no bearing on either case - the only relevance of the Online Safety Act is that it defined some offences which did not previously exist. As a society, we tend to understand that the ends do not always justify the means - some limited measure of success does not (and cannot) automatically outweigh consequences. Even if it does _some_ good, the Online Safety Act is _still_ bad law. * * * #### Email Hi, As one of your constituents impacted by some of the (entirely predictable) consequences of the Online Safety Act, I thought I'd reach out with some concerns and questions. **Background** I'll start by saying this upfront: The Online Safety Act (OSA) does **not** protect children and was never going to be able to. Instead, it pushes them away from regulated spaces towards entirely unregulated ones, whilst also exposing adults to increased levels of harm. This is largely Parliament's fault: Ofcom had to work with the cards that they were dealt, and the lack of _meaningful_ definition within the OSA provided no clarity on **who** is affected by **what**. The inevitable result of pairing a lack of clarity with strong penalties is overcompensation: operators have had no choice but to assume "worst case" scenarios, leading to the chilling effects that are currently playing out across the UK web. As you probably recall, the Online Safety Act was preceded by the Online Harms bill. At the time, the (then Tory) Government's own analysis noted that it's age-restriction measures may well push minors onto using VPNs (and more concerningly) the dark web, removing the benefit of any protections which had previously existed and exposing minors to more harmful content. That original attempt to impose an age verification (AV) regime eventually faltered and failed, largely as a consequence of its "ideals" coming into contact with reality. However, important lessons were apparently not learnt, and the OSA eventually came into being. As well intentioned as the OSA's introduction might have been, it has had exactly the effects that were predicted years before. As a few examples: * UK VPN usage has rocketed (ProtonVPN reported over 1000% increase in UK signups * Content in grey areas now requires AV - depriving teens of access to sexual health advice and (unbelievably) requiring sexual assualt survivors to provide ID to continue access to support groups (you can find an incomplete list of censored Subreddits here) * The OSA was supposed to target "irresponsible" providers carrying harmful content who, being irresponsible, have continued to serve content. With "responsible" providers now requiring AV, the "irresponsible" ones are now the only outlets available to minors (see Ofcom's reporting etc) The consequence that we haven't yet experienced (but inevitably will) is a breach of one or more of the Age Verification Providers systems. We don't need to look far, though, to see what that will look like because a US dating app (Tea) has just had a breach of data collected for "safety" purposes: https://www.nytimes.com/2025/07/26/us/tea-safety-dating-app-hack.html / https://www.bbc.co.uk/news/articles/c7vl57n74pqo As a blogger and fediverse server operator, I have also been personally affected by the vagueries of the Online Safety Act. I wrote about performing an OSA assessment (as well as some of the questions that the OSA raises). You can see that it was difficult to reach any hard conclusions - I did try contacting Ofcom for advice, but they were unable to help. I'm also aware of impact on a number of other bloggers. One of those bloggers maintains a popular blog on the topic of their sex life. The stipulations of the OSA have put them in a truly ridiculous position: * Their writing is not covered: the OSA doesn't apply to text only content * They had previously been providing audio recordings of someone reading their posts. But the OSA _does_ apply to audio. The audio contains _exactly_ the same content, but cannot be provided without introducing (expensive and invasive) age verification. The result is that the Online Safety Act has done nothing to protect children (assuming that you believe this was content that needed protecting from in the first place) and instead made the web less accessible to users with accessibility needs. In fact, without going off on too much of a tangent, the OSA is _weirdly_ prescriptive about formats: Its stipulations around pornographic content call out specific image formats as being covered, leaving open the possibility that a provider may not be covered if they've used a WebP instead of a GIF. I'm sorry, but the introduction of that kind of uncertainty really is just a sign of bad lawmaking. Parliament seems to have taken the view that operators "just" need to add age verification. Unfortunately, this is ignorant of the realities on the ground: * AV solutions present significant privacy challenges for users * AV solutions tend to be extremely expensive for small operators * There's significant potential social harm in getting users accustomed to random third parties requesting sensitive details "for age verification purposes" For many operators, the simplest option has been to no longer consider the UK a viable market - geoblocking is cheap, easy and removes all of the overhead associated with assessing and maintaining OSA compliance. Even I, primarily a technical blogger, have ultimately had to take the decision not to serve some content to UK users. You can read the OSA assessment that led to that decision at https://www.bentasker.co.uk/pages/legal/osa/osa-assessment-photosarchive-bentasker-co-uk.html but it came as a direct consequence of Parliament (and Ofcom) being unable to define what is and is not covered. I am **far** from alone. I expect you'll have heard the news that Wikipedia are considering blocking the UK, but here's an (incomplete) list of smaller providers who have shut down or geoblocked the UK, citing the Online Safety Act in the process: * AWSCommunity: A Mastodon instance for discussing AWS technologies * Carlbury debate forum * Dads with Kids: a forum for single and seperated Dads * Eastleigh online discussion forum: a forum for locals to discuss local issues * Furry Energy: A server for UK furry and LGBTQA+ communities * Gaming on Linux discussion forum * The Green living forum: had over half a million posts about sustainable living * The Hamster forum: a community about rodent care * Hexus: A gaming and discussion forum * Lemmy.zip: an IT centric discussion forum * London Fixed Gear and Single Speed: A forum for cycling enthuisiasts * Ready to Go: A discussion forum for Sunderland fans * Red Passion: A discussion forum for Wrexham fans * Renault EV Club: A discussion forum about Renault Electric Vehicles * Sinclair QL: A discussion forum for fans of Sinclair QL computers * Urban Dead: A MMORPG based on a zombie apocalypse As you can see, what the Online Safety Act appears to have "protected" the population from is people seeking out others who are passionate about (sometimes important) subjects. Freedom of expression has been taking a bit of a kicking lately. **Questions** The background above will almost certainly be outdated by the time that you get to read (let alone respond) to this, but with it in mind, my questions are: * What does the Government intend to do to address this mess? * At what point will the Government recognise the need to repeal (or partially repeal) the OSA? Are they waiting for generational change, or are we capable of recognising the negative impacts here and now? * Does the Government instead intend to double down and attempt a crackdown on VPNs, or to require (the now mooted) digital ID? * Has the Government yet commissioned any independent analysis into the effects (harmful and otherwise) of the OSA? Do they intend to? * What measures does the Government intend to pursue to ensure that minors are actually prevented from harm? * As a small site operator, can I expect that the Government will recognise the harm involved in destroying small communities and driving them onto platforms like Facebook (which, as we know, are also something of a fertile hunting ground for predators)? * As a small site operator, can I expect the introduction of more meaningful definitions (and/or small site exemptions)? * What further reassurances can the Government offer small site operators that they actually have this in hand? * What measures does the Government intend to enact in order to protect and encourage freedom of expression? I am both a parent and _extremely_ technical: I fully recognise the challenges involved in trying to make the web safer for kids. My concern is (and has always been) that the Government seems to lack the technical aptitude necessary to understand the impact of these changes. With the resources available, that lack of aptitude and insight cannot have been anything but a _deliberate choice_ , made whilst pursuing idealism over reality. The UK population has started to pay the price, and will likely continue to do so. The Ashley Madison breach was a decade ago this month and, yet, the Government has still _actively chosen_ to create new silos which link ordinary people's identities to some of their most sensitive and personal behaviours. In the process, the Government have deprived minors of access to important advice and support on sexuality, sexual health and even current affairs, pushing them towards much more unsavoury sources instead. Whatever the purity of intention behind it, the Online Safety Act is an actively harmful failure, exposing web users (young and old) to increased levels of harm. It should never have been introduced and should now be repealed. Unfortunately, those are not the noises that the Government is currently making - instead they seem more inclined to try and tighten the grip (for example: https://www.theguardian.com/society/2025/aug/03/uk-pornography-taskforce-to-propose-banning-barely-legal-content-after-channel-4-documentary-airs). The OSA really is an embarrassment on the world stage. Thank you for your time.

The events of last week presented quite a challenging question. **Should** you attempt to address the issue of friends consuming and accepting misinformation, or do you simply accept it as a sign that it's time to break contact? Humans are _very_ good at ignoring points of contention, particularly if addressing them might be inconvenient, but there are also things which cannot easily be brushed over. Last week presented one of those, inaction wasn't a (realistic) option and we _had_ to make a conscious choice on direction. This post was originally written in order to help work through my thoughts and arrive at a decision. The intention of this post _is not_ to lambast the friend, so (even though no-one reading this blog would know them), I'm going to be quite deliberately vague at points. * * * ### The Situation We regularly meet up with some other dog owners so that our dogs can have a run-around and play with one another. It's a good energy outlet for them and helps to keep them all well socialised (which is, in no small part, how we've reduced some of Bonnie's anxiety). It does, of course, means that they're regularly in close contact with each other, even swapping balls (and... _gag_... saliva) as they play. My partner works with dogs and was asked to look after one the play dogs (Bee1) while the owners are on holiday. **However** , while preparing for this, it came to light that Bee was a few years behind on his vaccination boosters. That was problematic for my partner, who can't work with an unvaccinated dog because of the risk of transferring something onwards, impacting other customer's pets2 and causing unnecessary suffering. Aside from the _severe_ emotional impact that communicating a disease would have on someone who _actively chooses_ to work with dogs, word getting out about such an incident would likely **destroy** the business. On top of all of that, we might _even_ face personal liability: it could be argued (and the insurance would certainly try) that **knowingly** interacting with an unvaccinated dog was an _act of negligence_. Vet bills are expensive, **other peoples** vet bills all the more so. However, Bee's owner said that they don't intend to bring the jabs up to date (because older dogs don't need them3) and that they'll make other sitting arrangements instead. * * * ### The Problem The pet-sit was only really the tip of the iceberg that our faces were now pressed flat up against. Bonnie is out playing with Bee regularly and (because Bee's a very cuddly and kissy dog), **we** are in regular contact with him too. Although it's something that we were working on, it's not _particularly_ unusual for Bee to try and mount Bonnie - he's somewhat lacking in aim (ends up thrusting into the air), but if we aren't quick enough to move him, he grabs her by the hips and pulls her back towards him. Between that and the swapping of saliva covered balls, there's more than a bit of potential for contamination and transmission. The risk would be much the same even if my partner stopped attending these meetups. Even if _I_ avoided physical contact with Bee6, any bacteria or viruses that he communicated onto Bonnie would soon be all over the inside of _our_ house. * * * ### Options There only really seemed to be a couple of realistic options. #### Stop Going? I had more than a life's fill of confrontation when I was young, so I have a strong impulse to avoid it if possible. But, it didn't _feel_ right that we should have to stop going because of someone else buying into disinformation. I also wasn't particularly comfortable with the idea of ghosting. Even though they'd be able to work out why, it still felt wrong to just go incommunicado. The _logistics_ of avoiding them were also a little complex: the owner doesn't reliably turn up and sometimes comes out at a different time, so it'd be quite difficult to fully avoid them without some form of prior communication. We also couldn't rely on leaving if we saw them arriving - Bon can still be anxious on a lead around other dogs, so getting her hooked up whilst all the others are still running free could only end in trouble. The only way that unannounced avoidance could work would be for us to forsake the convenience of the field entirely and start routinely walking the dogs somewhere else. * * * #### Say Something? I was not altogether convinced that saying something would achieve much. But, as well as being the _adult_ thing to do, it was the only other viable option. If it went wrong and we ended up parting ways, the reason behind it would at least be clear, saving us from any awkward "oh, haven't seen you in _ages_ " moments in future. * * * ### The Facts I didn't want to rant in my communications with the owner, because it could only be counter-productive and increase the likelihood of them feeling attacked and refusing to act out of stubbornness. But, at the same time, I just **do not** fucking understand how we've all ended up here. It's not _just_ dogs that are running around unvaccinated, there are _kids_ in the community without them too. Somehow we've gone from acting as a semi-cohesive society to one which tolerates parents putting their children (and others) at risk under the guise of personal choice. The result is a record high number of measles cases in the US, with the UK also trending upwards. This is what happens when you delegate medical decisions with a society-level impact to the individuals in a society which includes parents who spike their autistic kids with bleach. Vaccinations don't guarantee that you won't ever _catch_ the disease, but they do tend to make the symptoms less unpleasant and reduce the chances of it being fatal. Getting back to dogs, though, let's lay out exactly what we routinely immunise against: * `Distemper`: a **highly** contagious viral disease with a mortality rate of **50%** in adult dogs. In puppies it's **eighty fucking percent** * `Hepatitis`: A highly contagious disease which damages spleen, kidneys and eyes. Symptoms include bloody vomit, diarrhoea and eye problems. **There is no cure** * `Parvovirus`: So highly contagious, that unvaccinated puppies aren't even allowed to _walk outside_. Incredibly hard on the pup4, even assuming that they manage to survive it * `Leptospirosis`: Bacterial disease that can be mild or life threatening. It's zoonotic and can infect humans (Weil's disease). It's also carried by rodents who tend to leave trails of infected urine or swim through water. Initial symptoms look quite a bit like any other illness, so it's easy to miss * `Kennel Cough`: Not actually a single illness, but a group of them. It's basically bronchitis and some of the bacteria and viruses that cause it also like _human_ bronchi. Despite the name, can be caught and spread anywhere. Basically, we're not talking about minor ailments here. We're talking about lucky-if-it-only-fucks-standard-of-life stuff and, worse, we're talking about a human gambling that they won't be inflicted on **a dog**. _Fuck that noise_. * * * ### Crafting a Message After I'd let off a bit of steam by ranting into my psuedo-blog, I started to draft the message that I'd send. It needed to be short enough that they'd actually read it, but long enough to convey the **important** points. They're not necessarily all essential, but the points that I came up with were: * We'll have to stop coming out * Partner works with puppies and dogs with health complications * Partner works with elderly owners, so zoonotic infections like Lepto have an additional concern (apparently Kennel cough can be zoonotic too) * If went wrong, it could fuck the business and our finances * Parvo is infectious before symptoms appear and for weeks after (in fact Cornell note that it's almost impossible to disinfect an area contaminated by an infected animal) * Kennel cough is infectious before symptoms appear I then distilled them down into a SMS: > Hey, have you thought any more about Bee's jabs? > > Ultimately its your choice, but we have to stop coming out otherwise > > [Partner] visits puppies & dogs with health conditions, if he ever did pick somethin up she'd transmit it on. > > Some customers are old too & lepto/kennel cough can infect humans. Lots of it is infectious before symptoms so there'd be 0 warning > > Not only would she be devo'd but it'd destroy the business. It'd double-fuck us financially because insurance would say she knew. > > Even if I could somehow ignore his face & not give him a cuddle, he'd still get stuff on Bon who'd then spread it round the house Ultimately, I decided that it was better not to overthink it and so hit `send`. * * * #### The Response It took nearly 24 hours for the reply to arrive and, unfortunately, it seemed that my message hadn't had the desired effect: > I'll be working, so we won't be out so you can have mornings with the others and I'll stick to evening time I struggled a little to decide how to respond to this. On the one hand, it was a little jarring and felt like I was being told when we could and could not go out. On the other hand, it didn't really seem worth arguing about because there wasn't anything of substance to engage with: no excuses, no attempt at justification, just a blunt suggestion of how we could best avoid one another. I debated replying with a link that suggested that, although immunity to parvo etc _can_ persist in older dogs, the vaccination needs to be replaced with tests and Lepto boosters are still required yearly. But, I realised that, even in the unlikely event that they listened, it wouldn't really help with the concerns about it affecting my partner's work. So, I replied to say that, if that was the only route they saw for moving forward, it's the one that we'd have to take. * * * ### A Clean Break Although not the outcome that I hoped for5, the response did at least result in a clean break. Although I've tried to describe it as neutrally as I can, there's undoubtedly some hurt on both sides - we were hurt to learn that _a friend_ knowingly put us at risk for years and they're (presumably) hurt that we'd walk away over something that they clearly view as being trivial. If the outcome had been different, we could _perhaps_ have convinced ourselves that they hadn't realised the potential impact, but refusing to change course after the impact has been described makes it hard to be quite so generous. I imagine, though, that it's a little easier for them to rationalise: they believe that Bee's age gives him immunity (removing the need for a jab), so it follows that there's no real risk to us in the first place. I mentioned personal choice earlier, and that's where this gets quite complicated. Rightly or wrongly, _it is_ their choice to make. However, in not telling us, they took away **our** right to choose (as well as that of others in the group). * * * ### When To End The Relationship This post started with a question: > **Should** you attempt to address the issue of friends consuming and accepting misinformation, or do you simply accept it as a sign that it's time to break contact? Even after this experience, I still believe that the answer is to _attempt_ to address it. However, you need to do so prepared for the possibility that your words may not land, and that the friendship may dissolve (or spontaneously combust!) as a result. The death of the relationship could come about because they over-react, or because they refuse to budge on something that you yourself cannot (or will not) compromise on. Beyond a certain point, though, the _why doesn't matter_. What matters is that the relationship is no longer viable and it's time for both sides to move on. Finally, however puerile and uninformed the view that you're responding to might seem, it's important to remember that we're all adults and to communicate with respect. If nothing else, it deprives them of an opportunity to portray you badly ~~if~~ when they start showing your messages to other people. * * * ### Conclusion It's been a sudden and quite emotionally charged change in routine, but I feel like we made the right decision: we engaged calmly and reasonably. We also tried to ensure that our other friends didn't feel like they were caught in the middle. I'd have preferred a solution which didn't involve burning a friendship to the ground, but the "good" outcome wasn't really ours to deliver. Hopefully, Bee's lack of vaccination will never affect his health, but _if_ it does, we won't unknowingly become a part of others suffering as a result. We were brought together by a common interest (dogs), so it hurt all the more to have it fall apart _because_ of a choice which increases the risk of a dog coming to harm. Once messages had been exchanged, the choice that we faced was binary: walk away, or rely on the hope that nothing ever went wrong. It wasn't even _really_ a choice. I'm glad that I said something, even if the outcome was ultimately the same. * * * 1. Not his real name ↩ 2. This isn't a particularly unusual requirement, I've yet to encounter any doggy daycare or kennels that'll accept unvaccinated pups. ↩ 3. As far as I can make out, their advice came from _the son_ of a (now deceased) vet. ↩ 4. Someone I once worked with bought a puppy only to find it had Parvo. It was a long, drawn out (and expensive) recovery involving multiple points at which he (and the vet) thought the poor thing was going to die. I absolutely do not recommend that experience. ↩ 5. The poor bugger, after all, is still running around unvaccinated ↩ 6. I did try, early on, but it was heart-breaking to have to try and ignore him as he grinned up at me wanting cuddles ↩

I've made no _particular_ secret of the fact that I am **not** a fan of large language models: they're overhyped, under deliver and were built off the back of a ripping off of others work at a scale that (for anyone else) would have resulted in jail and/or bankruptcy (even before the more incontrovertible acts of copyright infringement are considered). But, I also work in a industry which is going through a phase of significant AI use (even if a lot of companies do come to regret being over-eager). As a generalist, still decades from retirement, I _have_ to keep my hand in. If nothing else, doing so means that I'm better positioned to explain _why_ and _when_ they seem to be ill-suited. Moving on, I've finally got around to reading the book that I mentioned while procrastinating recently. In one section, the author talks about reviewing how their time is spent: > Every quarter, I spend a few hours categorizing my calendar from the past three months to figure out how I’ve invested my time. This is useful for me to reflect on the major projects I’ve done, and also to get a sense of my general allocation of time. This struck me as a _particularly_ sensible thing to start doing. However, I needed a slightly different approach: although it plays a part, my calendar is not the main organiser of my day because I work from a daily todo list in Obsidian. Although small/quick tasks might sometimes get omitted, they're still far more likely to appear in my TODO list than in my calendar. The trade off, of course, was that I would only able to collect the numbers of task completed rather than a measure of time spent. Pivoting back to AI, I decided that categorising TODO list items **should** be a piece of cake for any reasoning model. This post discusses how I developed metrics about my todo list along with where (and why) I found the LLM unreliable. * * * ### Requirements I had a simple (but strict) set of requirements 1. **The LLM must be local** : I'm a manager and my todo list sometimes contains references to team members. I'm not comfortable sending _any_ of that to one of the hosted providers1 2. **The script should be cron-able** : the whole point is that it should just run without me needing to do anything 3. **It must use my existing format** : Shuffling the TODO list structure around is OK, but changing the way that I write TODOs is likely to be unrealistic - it's supposed to be somewhere to quickly note a need to do something, rather than a document which requires a linter 4. **Tasks should be sorted into a finite set of categories** : LLMs are quite inconsistent in their output, so it'd be unwise to rely on the LLM to also define the categories: I'd end up with tasks spread over "discussions", "email" and "correspondence" 5. **The output should be simple counts** : The aim here wasn't to create a second copy of my TODO list, but instead to express it in terms of simple metrics * * * ### TODO List Structure My list uses a simple markdown template: # Daily Notes: {{date}} ### TODO - [ ] Backlog item ### Today - [ ] Incomplete task - [x] Complete task - [ ] Morning Sync (09:00) **Calls** **Morning Routine** - [ ] Slack - [ ] Email .. etc .. **Follow Up Tomorrow** The `TODO` section contains my backlog which is rolled into the current days notes by the Daily todos rollover plugin (along with any incomplete items under `Today`). As I work through my morning routine, I populate the `Today` section with calls broken out under their own subheading. * * * ### Processing Because they're stored as markdown, the lists are easy to process and the headings make it easy to see which section an item falls under. I've used a similar format in previous roles and previously wrote a simple burndown stats script which built metrics by walking through line by line: for line in fh.readlines(): # Process the line if "### TODO" in line: in_section = "main" print("Started") continue if "**Calls**" in line: in_section = "calls" continue if "**Morning Routine**" in line: in_section = "morning-routine" continue elif in_section and line.startswith('- [ ]'): counters["items"] += 1 elif in_section and line.startswith('- [x]'): counters["items"] += 1 counters["complete"] += 1 For this project, I added a function call to the conditional block for completed items: category = categorise_item(line, in_section, ai, categories) * * * ### Adding AI I span up an ollama docker container and loaded Deepseek-r1 into it ollama run deepseek-r1:7b Use of the 7 billion token model was a compromise: the solution had to run on a laptop (I actually started with the much smaller `phi3:mini-4k` but it _really_ struggled to give accurate responses). I have a python wrapper that I previously wrote to provide a command-line REPL for interacting with Ollama3, so I imported that. Although there's a bit of class initialisation to do, it makes sending a message simple: response = ai.chat_with_model("Hello, are you there?") Next, I needed to write a prompt to tell the LLM what to do with the messages that would be sent to it: You are a task categorisation specialist and are being employed in order to identify what categories of work each of the user's completed todo list items falls into. The user will provide you with a file in markdown format which consists of a list of checkboxes with todo list items next to them. Using the rules below, you are to categorise any task items that the user passes you. **File Structure** * An item starting with - [ ] indicates a task that has not yet been completed, ignore these. * An item starting with - [x] indicates that the task was completed, you should categorise the item * Items that follow the text Calls are calls - these need further classification (see the calls selection below) * Items under Morning Routine are routine items, these should be classified as "daily admin" You may encounter a heading with the text Defunct. If so, ignore everything that follows it. **Calls** There are two types of call classificiations: call-meetings and call-11 If the todo item includes the text 1:1, it's a call-11, otherwise it should be classified as call-meeting. **Categories** ALL completed items must be categorised but must ONLY be categorised into one of the following categories * call-meeting * call-11 * daily admin * incident response * PR work * documentation and planning * other To test, I passed it a single item2 - [x] Swear at AI The response and the reasoning seemed in line with what I wanted > Okay, let's tackle this problem step by step. The user has provided a task list with several items marked as completed (indicated by - [x]). My job is to categorize each of these items into one of the specified categories based on the given rules. > > First, I need to go through each item and determine its category. Let's look at each item: > > 1. "- [x] Swear at AI" This task clearly states a negative action but doesn't provide any specific context like calls or routine tasks. Since it's under "Other" as per the categories list, this should be classified as "other". > > > I don't see any other items provided in the example besides this one. Each item must be categorized without inventing new ones that aren't present in the user's message. > > So, the only category needed here is "other". > > { "category": "other" } I threw it a few more items and all came back correctly. * * * #### Problems However, issues soon began to creep in. Despite having been passed a list of acceptable categories, the model invented a new one: security. Expanding my prompt to include explicit rules (you MUST NOT return a category not listed above) helped a little, but not much. To improve the chances of getting a successful response, I adjusted my script to add the ability to trigger a reprompt: def get_ai_category(item, ai, section, ffs=False): ''' Ask the AI to categorise We often need to strip the response though, because many reasoning models include their reasoning too ''' msg = [] if ffs: msg.append("Reminder: you MUST stick to the categories provided in the prompt. You're a very naughty LLM, your previous response made changes - please try again but stick specifically to the rules in the prompt") msg.append(item) response = ai.chat_with_model("\n".join(msg)).split("\n") However, it didn't help much: the LLM often replied with the same thing the second time. Taking a closer look at the responses revealed something even more frustrating though: sometimes it selected the correct category but then fucked up the spelling. For example, instead of `documentation and planning` it returned the following * document and planning * documentaton and planning * documenta... * documentations and planning Sometimes the response would contain a valid category name but prefixed by some other text *category*: documentation and planning *final decision*: calls-11 I messed aroumd with making it respond with JSON, but nothing _really_ helped. In the end, I created a function called `unfuck_ai_response` and started adding string processing to account for its common mistakes. As a side note, the AI also **staunchly** refused to allow me to use non-words as category names. I defined a category with the name `managering` but it insisted on returning `managerial`. That behaviour makes sense, because _actual words_ are obviously going to have more weight, but it doesn't help with the feeling that AI is a fun sponge. * * * #### Taking Load Off the AI Even with the new function, there were still far too many items that weren't being categorised correctly. Quite a few of the regular items in my todo list can be categorised by looking for specific strings, so I adjusted the script to start handling those simpler items itself: if in_section == "calls": if "1:1" in item: return "call-11" elif "adhoc" in item: return "call-unplanned" else: return "call-meeting" I adjusted the script so that all todo items categorised at `other` would be written to a file before triggering a full run across all of my notes. Even if it didn't make sense to adjust the script for every `other` item, the list allowed me to spot patterns and build the ruleset out. * * * ### Collecting Metrics With the script now capable of emitting semi-reliable metrics, I needed to collect the data somewhere. For this project, I decided to give VictoriaMetrics a try: it accepts a variety of input formats, including Influx line protocol, so making the script generate LP gave me the ability to try VictoriaMetrics whilst retaining the ability to pivot back to InfluxDB if needed. * * * #### Standing VictoriaMetrics up Using the VictoriaMetrics docs as a guide, I was able to come up with a `docker-compose` snippet to stand a container up: victoria_metrics: restart: always image: victoriametrics/victoria-metrics:v1.122.0 container_name: victoria_metrics command: - "-selfScrapeInterval=5s" - "-storageDataPath=victoria-metrics-data" - "-retentionPeriod=100y" - "-disablePerDayIndex" ports: - 8428:8428 volumes: - /home/ben/docker_files/victoria/data:/victoria-metrics-data Writing line protocol in was easy: curl \ -d 'metric,foo=bar field1=3.4' \ http://127.0.0.1:8428/api/v2/write?db=bar So I updated my script: print("Submitting to VictoriaMetrics") res = requests.post( f"{vm_url}/api/v2/write?db=workload_stats", data = '\n'.join(lp_buffer) ) print(res.status_code) print(res.text) Note: VictoriaMetrics doesn't have the concept of separate databases and the name of the DB provided instead gets added as a label. After the next script run, I was able to query data: curl -s 'https://vm.83n.uk/api/v1/series?match[]=todo_list_completed' { "status":"success", "data":[{ "__name__":"todo_list_completed", "db":"workload_stats", "hostname":"flibble" }] } I could also delete that series - something that I used _quite a bit_ whilst continuing to refine the script's output curl -s 'https://vm.83n.uk/api/v1/admin/tsdb/delete_series?match[]=todo_list_completed' * * * #### Visualising I installed the VictoriaMetrics plugin for Grafana and started to build a dashboard Historically, I've written quite a lot of Flux on this blog, but VictoriaMetrics uses MetricsQL. It's a PromQL-alike language, which itself is similar to LogQL, so I didn't experience any real issues writing queries to drive the graphs: todo_list_items_count{db="workload_stats", category!="error", category!="scheduled-for-tomorrow"} The visual query builder is also pretty good. * * * #### Metrics: use of AI The TODO processing script also writes metrics about itself into VictoriaMetrics, so the dashboard includes a section specifically for script run stats. One graph indicates the proportion of TODO items categorised by the script vs categorised by AI: Although I'd been careful to only add general rules (rather than targetting specific items), the ruleset embedded within the script accounted for two thirds of all categorisations. I also wanted to understand how often AI was misbehaving, so I adjusted the script to calculate an AI reliability score: # Calculate a reliability score for the AI # for every retry we had to send the count goes down. if source == "ai": # So the logic here is that we want to generate # a percentage reliability score based on # comparison to the max retry depth. # # We add 1 onto max_retry_depth to account for # the first request (which doesn't get treated # as a retry) # # So, if max_retry_depth is 4, the max becomes 5 # and scores would look like this: # # * we didn't retry at all: 100% # * we retried once: 80% # * we retried 4 times: 20% # * we burned through the retries: 0% # score = (1 - (RETRY_DEPTH / (max_retry_depth + 1))) * 100 For each TODO list run, the script reports mean, min and max reliability scores. It seems that it _generally_ behaves, but when it misbehaves, it does so quite stubbornly: * * * ### Conclusion I now have a script which can _roughly_ categorise my TODO items from each day ready for visualisation in a dashboard to shows where the majority of my efforts are spent. My initial hopes for AI making it quick and easy were quickly, if predictably, dashed: the LLM does a "good enough" job most of the time, but only because substring matching within the script handles ~66% of items. One of the **huge** benefits of having opted for a local model is that I don't have to worry about results suddenly drifting as the result of a provider deploying an update which results in the model responding differently. That consistency has its limit though: the dashboard has looked slightly different every time that I've triggered a full re-run - the LLM is rarely consistent in the categories that it selects. Equally, though, there probably isn't any _reasonable_ way that I could have script logic correctly categorise the 1/3rd that ends up being passed off to the LLM. Although I could either limit the language used or include a category alongside items, the likely result of that added burden is that the list would fail to include so much of my workday. * * * 1. It doesn't _matter_ what their terms say. Data protection by design and by default, if it can be done without sending to OpenAI/Google/Musk then it should be ↩ 2. It's possible that I've skipped a few iterations in this post and was growing fed up at this point ↩ 3. In case you haven't picked up on this: the privacy benefits mean that local AI will always be > remote AI ↩

Bonnie came to live with us at the beginning of last year. She joined our family unspayed and so was still having regular seasons. I don't _think_ that her previous owner had tried to breed her but, as he had a breeder in the family, I'm guessing that it was an option that he wanted to keep. Intact dogs aren't just exposed to the risk of accidental pregnancy, they also have an increased incidence of cancer and Pyometra (on average, 19% of intact females will suffer this before age 10). On top of that, of course, is the disruption to routine that having a season brings: although the dog will still want to play, there's an obvious need to keep them away from males (intact or otherwise: neutered males can still end up fighting over a hormonal bitch). Getting Bonnie spayed was the _responsible_ thing to do. However, Bonnie joined us with some behavioural issues and the early relationship between her and Pepper was sometimes a little shakey. We felt that their developing relationship might be negatively impacted by a surgery recovery period. Over the last year or so, though, Bonnie has made great progress and we decided that the time had come to have her spayed. In order to supervise the dogs while Bonnie recovered, I took some time off work and, frankly, got quite bored and restless, so thought I'd write about her recovery. * * * #### Visiting The Vet When Bonnie first joined us, she suffered quite badly from anxiety, particularly whilst on a lead. Although it's improved considerably, it's very much still a thing. It should come as no surprise then, that she **does not** enjoy visits to the vet. Bonnie's first visit to her new vet was for a vaccination booster and, although she received the jab, it was clearly quite a stressful experience for her (and everyone else!). After that, the vet provided a small supply of anti-anxiety medication (Gabapentin and Trazodone) for me to give to Bonnie ahead of any visit. The next visit to the vet was unplanned: Bonnie had caught her nail and torn it most of the way off. I gave her some tablets a few hours before the appointment and, once we were in, she _even_ let the vet stroke under her chin. The tablets _definitely_ work, but they're not a panacea: they buy a little bit of time for the vet to do what needs to be done, but Bonnie can still eventually get worked up. * * * #### The Morning Of The Procedure In order to minimise stress for Bonnie, the plan was simple: * Dose her with anti-anxiety meds the night before (and morning of) the procedure * Take her into the vets before anyone else might arrive * Do a quick bit of paperwork and then sedate her Just before the sedative was injected, for the vet's safety, I put a plastic muzzle over Bonnie's snout. This was quite new to her, but she let me put it on and take it off without any fuss - it turned out not to be needed, she didn't even growl when the needle went in. Having injected the sedative, the vet and nurse left the room to allow Bonnie to settle. Within about 15 minutes of arriving, Bonnie was curled up unconscious at my feet: The nurse came back in and, after taking a list of additional requests (trim her nails etc), carefully picked Bonnie up and carried her through. I went home, made a cup of tea and then washed her bedding to make sure she'd that have a clean bed to come home to. * * * #### Laparoscopic Procedure We opted for a laparoscopic spay1. A traditional spaying (like the one Pepper had) involves a long incision down the abdomen, giving the vet access to their ovaries and uterus. A laparoscopic procedure, instead, involves a couple of small incisions. The vet pushes a camera through one to help them see (hence "keyhole" surgery) whilst operating through the other. The procedures are also quite different in terms of the impact that they have on the dog's body. Lap spays only remove the ovaries (an _ovariectomy_), whilst traditional spays tend to remove the uterus as well (an _ovariohysterectomy_), inevitably disturbing more of the remaining tissue in the process. For those don't mind a bit of inside imagery, there are videos of Lap Spays on Youtube. Key-hole procedures are more expensive2, but bring a real advantage in terms of recovery time: | Traditional | Laparoscopic ---|---|--- **Recovery Time** | 14 days | 3-4 days The crucial difference is that Bonnie was allowed to run and play again after just a few days (although she'd need to be kept from licking the wound for 10 days). This is a **huge** improvement over the two weeks that Pepper spent on lead and wearing a t-shirt (the vets couldn't get her to stay in a cone, even before we arrived): I dropped Bonnie off at the vets at 08:40. By 11:30 they had phoned me to say that everything had gone well and that she was awake enough to be collected. * * * #### Coming Home I wasn't sure how awake "awake" was actually going to be. Bonnie had been sedated during the nail-tear visit and, that time, "awake" had meant an eyes-wide, tongue hanging, spaced out dog who needed to be carried because she couldn't walk more than a few steps. This time, though, Bonnie trotted into the waiting room wearing a snazzy pink recovery suit: The suit proved to be pretty good: rather than having a buttoned up flap that I'd need to undo whenever she needed the toilet, it was elasticated and so stretched out of the way whenever she squatted. * * * #### Day 0: Dopey Dope Dope Although able to walk on her own, Bonnie was quite clearly still feeling the effects of the methadone that she'd been sedated with. She was slow to respond, easily confused and had pupils like spaceships: Although it was one that I had been prepared for, her lack of re-activeness did pose something of a problem. The last time that Bonnie came home doped up, Pepper was _beside herself_. She couldn't seem to understand why her buddy wasn't responding to her and responded by getting really close and barking anxiously at her 3. This time, although Bonnie was mobile, she still wasn't particularly responsive and Pepper started worrying again. To allow Bonnie to recover in peace, we took Pepper up to littlun's room and left her to ~~be entertained~~ watch littlun play on the Xbox for the evening. Around 1930, we brought Pepper back down and, keeping her on a lead, I got both dogs to settle down next to me and go to sleep. A few hours later, I moved them to their crates and said goodnight. * * * #### Day 1: Hooman, PLAY The next morning, I got up and made their breakfast, squirting a pre-prepared dose of Rheumacam onto Bonnies. When I went through to let them out of their crates, it was clear that Bonnie had slept the remainder of the sedation off - if she hadn't been wearing a bright pink reminder, you could _easily_ have forgotten that she'd had surgery at all. After feeding them, I clipped their leads on and took them for a short walk. During that walk, Bon had a wee and a poo - both are something that you're supposed to get excited about post-sedation4. In preparation for having a post-operative dog in the house, I had put a step-stool in front of the sofa (and another by our bed). Unfortunately, telling a Border Collie not to jump is a bit like telling a toddler not to eat sweet things. Bonnie ignored the step and jumped straight up... _sigh_. The _real_ challenge, though, lay in keeping the two dogs from playing. _This_ was the reason that I needed to take time off work: both dogs _love_ a game of bitey-face and my primary role was to pour a (metaphorical) jug of cold water on any sign of play. Using a precise combination of exasperation, darkened rooms and the words "settle down", we got through the day without any rough play. * * * #### Day 2: Grrrr Day 2's morning walk didn't end as well as the previous one. Normally, once we get back to the house, the dogs sit and receive a small treat (which is used to reinforce the command "come home" in case they ever do get out of the garden). But, after swallowing hers, Bon nipped jealously at Pepper's muzzle. I've written before about Bonnie food guarding, but it's _very_ rare for it to show up when they're both receiving a treat (particularly one that both dogs can finish quickly). There's no way to be sure, but I _think_ the issue was that Bonnie was in some discomfort and felt less tolerant and reasonable than normal: her painkillers were administered once daily, so it's quite likely that the previous day's dose had worn off and the next hadn't kicked in yet. It's equally possible, though, that it was temperature induced grumpiness. The UK was going through (yet) another heatwave, which wasn't exactly ideal timing for a double-coated collie to be wearing an outer body suit. Whatever the cause, with a quick verbal correction, Bonnie went back to being nice and the day continued. However, Bonnie was also fully back to her old self and jumped5 on every possible opportunity to release some of her immense Collie energy. That included trying to chase pigeons in the garden, running up and down the stairs and, of course, finding ball-shaped things to drop in my lap in an attempt to try and tempt me to throw them: * * * #### Day 3: Check the wound... but how? It was time to check the wound. With a less anxious dog, the vet arranges a post-operative appointment to check that the wound is healing and isn't showing any signs of infection. Those appointments can be _extremely_ stressful for an anxious dog, though, so they asked me to email photos instead. Day 3 was _the_ day, but the question of **how** had been running through my mind _for days_. In order to even _see_ the wound, we'd need to do the following: Bonnie seems to trust us, but it's very much on her terms. The chances of success seemed _incredibly_ remote and brought some associated risks: * If the suit came all the way off of her hindquarters, there was no chance we'd be able to get it back on * If she stressed or wriggled too much, she could end up tearing the stitches and re-opening the wound * If she got too stressed, she might bite (crossing a line that she hasn't yet) This was complicated further by the fact that it would need to be _me_ holding her head: much to my family's annoyance, she has the best relationship with me and so was less likely to overreact or _hold a grudge_. Even with the difficulties, though, the alternative was to leave her health to chance, so we **had** to try. * * * ##### Checking The Wound I knelt next to Bonnie, holding her lead short (but not tight). I pressed the flat of my hand against her cheek, sandwiching her head lightly between my palm and my body, effectively holding her in place. The _moment_ that the suit's zip was touched, she growled and tried to flick her head round. Talking soothingly, I tightened my grip a little and nodded to continue. With the zip undone half-way up her back, the back of the suit was pulled gently down her legs. It became clear that there was problem: although a shaved patch of her underside was visible, it wasn't really possible to see her wound and certainly wasn't going to be possible to get a _useful_ photo. I carefully released her head and lent down to have a look myself - there really was just no way that it was going to work: * The suit was elasticated and tight, so we couldn't move it to increase visibility without taking it off her legs (at which point, it'd never go back on) * If we poked a phone into the suit and she wriggled, we'd likely end up rubbing whatever bacteria was on the phone over her wound, potentially introducing an infection that hadn't been present before With my head at an uncomfortable angle, I _thought_ I could occasionally see half a glimpse as she shifted, but she was starting to get quite agitated - we were out of time. I held her head again and we zipped the suit safely back into place before releasing her and rewarding with a treat. * * * ##### Reporting Back I emailed the vet, noted that I hadn't been able to get a picture and provided the evidence that I _had_ been able to collect: * The shaved area was all a normal colour * There was no obvious sign of swelling * There was no malodour * She was mobile and seemed her usual self * There was no evidence of discharge or dampness on the suit itself * Apart from that first day, Pepper had not been sniffing around her Being _a little bit_ of a nerd, I also pointed my thermal imaging camera6 at Bonnie's belly (and, yes, I forgot to set the clock the last time that the batteries died): I'm more than a little out of my lane here but, if there were any infection, I would expect the associated vasodilation to cause a hot/warm spot around the wounds. Although the numerical values can't be taken literally they _were_ about the same as when I pointed it at Pepper's belly (suggesting that Bonnie wasn't feverish). The vet _didn't_ comment on my (potentially) questionable application of science, but agreed that it sounded like things were healing as they should. * * * #### Day 4: Play? Play Day 4 was the point at which Bonnie _could_ be allowed to be more active. However, as we hadn't been able to _physically_ check the wound, I decided that it would be wise to wait a little longer so that we could be reasonably sure that she wouldn't tear it (particularly as there was no way to know whether her earlier pigeon chasing and sofa jumping antics had already done so). Obviously, Bonnie had no way to know that day 4 was _supposed_ to be the day and so also wouldn't _really_ know that the recovery period had been extended. Unfortunately, neither did she know _why_ she wasn't allowed to play or even really _that_ she wasn't supposed to. Inevitably, our lunch time spell out side included more attempts to make me acknowledge the presence of a ball: * * * #### Day 5: Extra Time After five days of me disrupting their play and insisting on calm, the dogs tried a new tactic: one dropped a toy into my lap just before the other launched at my face to kick off a human-dog game of bitey-face. In life, there are moments when it's impossible to be strict and one of those is when your dogs seemingly collaborate to launch a sneak attack on you. A three way game of bitey-face ensued, before one dog decided that I needed the handicap of playing tug-of-war whilst still playing bitey-face with the other. Nearly a week's worth of pent up energy was essentially unleashed on me by both dogs. As the day cooled off, I decided to let Bonnie burn a little more energy off and took her out to the field for a short play with the ball Day 6 had another, slightly longer, trip out to play ball. * * * #### Day 7: Poooooooooooo One the morning of Day 7, I went to get the dogs up, but was hit by a strong smell as soon as I entered the room. As I turned the corner, I saw Bonnie in her pen: had I forgotten to lock the crate last night? I was _sure_ I'd closed it. She was stood, grinning up at me, with the carpet around her covered in shit. Diarrhoea is one of the possible side effects of Rheumocam. Of course, it's _equally_ possible that she found and ate something nasty while in the garden. Whatever the cause was, it made her shit hard enough that, for the first time ever, she'd escaped her crate, breaking the lock in the process: There wasn't any call for immediate concern though: she seemed fine and happy, so I set about cleaning it all up. The prior day's dose had been her last round of painkillers so there were, thankfully, no repeats of this. * * * #### Day 8: Fun With Friends By day 8, Bonnie had been out for a few runs on her own with no sign of discomfort afterwards. But, as she often does when we're out alone, she'd spent a chunk of the time looking around to see if her mates were coming in through any of the field's entrances. So, on day 8, we went back to meeting up with her doggie pals for a run-around. * * * #### Day 10: Black is the new Pink The day had _finally_ come to remove the pink recovery suit. As usual, we put Bonnie on the lead to go for a run, but then I held her head while the suit was unzipped (prompting the expected growling). The suit was carefully pulled down and off her back legs before we released her so that she could step her front legs out herself. Later that night, Bonnie lay on her back on the sofa and we were _finally_ able to get a photo of her (now healed) wound It _really_ is tiny compared to Pepper's scar. * * * #### Day 13 Towards the end of day 13, Bonnie's mood shifted a bit - she became _very_ clingy and cuddly. When Pepper got a bit FOMO, Bonnie stopped "speaking" to her for the rest of the day - when Pepper tried to initiate play, Bonnie turned her back on her and went and curled up in her crate. She wasn't showing any signs of pain or discomfort, so our assumption was that this was the result of hormonal changes. Although Pepper never showed any sign of it, apparently it can take 2-4 weeks for their hormones to settle. * * * #### Day 14 The dogs are back to being friends again, games of bitey face and all. * * * #### Conclusion So, there we have it: Bonnie was able to play again 3-4 days from spaying and was fully recovered within 10. She's no longer at risk from the range of things that can affect intact females (including males breaking in to find her when she's in season) and no longer has to miss out on a month of playing with friends twice a year. Pepper's recovery period had been _much_ longer and included a period where she wasn't even allowed to go on walks. Bonnie's recovery period, whilst exhausting for me7, was **much** better in comparison. If we ever got a third dog (no!), despite the additional cost, I think we'd _definitely_ go for the laparoscopic spay again. We'd also **definitely** go for the surgery suit over a cone-of-shame or (as Pepper had) a loosely tied t-shirt: * Although the suit did make it harder for us to check the wound, a large part of that difficulty came from Bonnie herself - I think we'd have been able to check it with Pep. * Unlike a cone, the suit acted as a barrier to help keep detritus and crap away from the wound. * It also didn't need constant adjustment like Pepper's t-shirt had Apart from day 3 and 10, were were essentially all able to act as if it wasn't even there. We're now a few days beyond day 10 and the only sign that anything even happened is Bonnie's shaved patch. * * * 1. Hence the procedure being known as a "Lap Spay" ↩ 2. Apparently the associated equipment costs between £20-30K and the setup time is quite a bit longer, so it's not like the vet is charging extra just for the sake of it ↩ 3. Awwwwww sweeeeeet ↩ 4. In truth, most dog owners enthuse about any toilet activity that happens outside the house, because that psuedo-excitement helps reinforce the positive behaviour ↩ 5. Sometimes, quite literally ↩ 6. there's a certain irony in having to conduct a _heat_ survey on a dog that's just been spayed. ↩ 7. It would have been worse with Pep, but we only had one dog at the time so there was no need for me to supervise as closely ↩

Yesterday, I read an interesting article about this spring's UK solar output being 42% higher than in the same period last year. The article notes that, although the sun shone a **lot** more, increased sunlight wasn't the only driver of this increase in generation: capacity also expanded. I was curious, though, to see what kind of increase (if any) our solar had achieved over that period. This post looks into our levels of production as well as going down some rabbit holes looking at changes in the UK's solar capacity. * * * ### Performance Of Our Install I track solar generation metrics in InfluxDB, so comparing yields for the two periods was just a case of writing a Flux query to grab the totals: twenty4 = from(bucket: "Systemstats") |> range(start: 2024-01-01T00:00:00Z, stop: 2024-06-01T00:00:00Z) |> filter(fn: (r) => r._measurement == "solar_inverter") |> filter(fn: (r) => r._field == "todayYield") |> aggregateWindow(every: 1d, fn: max) |> sum() |> set(key: "year", value: "2024") twenty5 = from(bucket: "Systemstats") |> range(start: 2025-01-01T00:00:00Z, stop: 2025-06-01T00:00:00Z) |> filter(fn: (r) => r._measurement == "solar_inverter") |> filter(fn: (r) => r._field == "todayYield") |> aggregateWindow(every: 1d, fn: max) |> sum() |> set(key: "year", value: "2025") union(tables: [twenty4, twenty5]) |> keep(columns: ["year", "_value"]) |> group() We **did** see a non-negligible increase, though it was still quite some way away from 42%: Year | kWh | Change ---|---|--- **2024** | 996 | **2025** | 1200 | _+ 20.5%_ Although I'd have _loved_ to to compare the same period in 2023 our panels weren't installed until May that year. As a rough indicator, though, we can compare from the beginning of June to early July: twenty3 = from(bucket: "Systemstats") |> range(start: 2023-06-01T00:00:00Z, stop: 2023-07-12T00:00:00Z) .. etc .. Year | kWh | Change ---|---|--- **2023** | 598 | _-_ **2024** | 550 | _- 8.1%_ **2025** | 605 | _+ 10%_ Six weeks isn't _exactly_ a representative sample, but 24-25 effectively reversed the previous year's loss, which suggests that this spring's year-on-year (YoY) increase _could_ partly be a regression to the mean. Even _if_ that _were_ the case, though, we're still quite a way short of the claimed 42% increase. * * * ### Capacity The original article notes that capacity increases played a role and adding additional panels will lead to an increase in overall output, even if the amount of sunlight remains exactly the same. We can check the extent to which capacity expansion is likely to have contributed, because the Government publishes data on UK solar photovoltaic capacity. The dataset includes a table of capacity and installation sites going back to 2010: The file includes a cover sheet which explains that this data has been pulled from a number of sources: * Ofgem's REGO data set * Microgeneration Certification Scheme * The Central FiTs register Although the cover sheet doesn't _explicitly_ say it, the reliance on these registers implies that the values in the dataset won't account for * Households that haven't applied for a MCS cert (i.e. aren't being paid for exports) * Capacity that has since been disabled * Off-grid capacity But, those factors _probably_ don't change the analysis that we need to do: even if the absolute numbers aren't _quite_ correct, what really matters is the difference between them. The sheet contains the following values for the time periods that we're interested in: | Capacity (MW) | Num Installations ---|---|--- **Jan 23** | 15502.4 | 1326180 **Feb 23** | 15618.5 | 1344094 **Mar 23** | 15712.0 | 1364450 **Apr 23** | 15811.9 | 1380711 **May 23** | 15968.1 | 1399067 **--- break ---** | **--- break ---** | **--- break ---** **Jan 24** | 17003.2 | 1519762 **Feb 24** | 17093.0 | 1533829 **Mar 24** | 17281.8 | 1548582 **Apr 24** | 17417.3 | 1564146 **May 24** | 17493.4 | 1580496 **--- break ---** | **--- break ---** | **--- break ---** **Jan 25** | 18574 | 1714838 **Feb 25** | 18654 | 1735259 **Mar 25** | 18747.9 | 1759978 **Apr 25** | 18829.6 | 1780380 **May 25** | 18919 | 1803478 Because the values are effectively cumulative, May's figures are sufficient for us to calculate the level of YoY change: | Capacity (MW) | YoY ---|---|--- **May 23** | 15968.1 | **May 24** | 17493.4 | _+9.6%_ **May 25** | 18919 | _+8.15%_ In percentage terms, installed capacity grew by less in 24-23 than it had in the prior year. Things get interesting, though, when we look at the year-on-year change in the number of installs: | Installs | YoY ---|---|--- **May 23** | 1399067 | **May 24** | 1580496 | +13% **May 25** | 1803478 | +14.1% Despite the lower _capacity_ growth, the number of installations grew at a _higher_ rate than in 23-24. Is this, perhaps, a sign that there were more small-scale (i.e. domestic) installs? * * * #### Domestic Installs The government's data can help here too, because it includes a second set of totals: The figures for _domestic_ capacity and installs are as follows: | Capacity (MW) | Installs ---|---|--- **May 23** | 4372.5 | 1242413 **May 24** | 4976 | 1392110 **May 25** | 5616.5 | 1552266 Using these figures, the year on year change in UK domestic solar is as follows: | Capacity | Installs ---|---|--- **May 23** | - | - **May 24** | +13.80% | +12.04% **May 25** | +12.85% | +11.50% Interestingly, these numbers **don't** seem to suggest that install count bump was driven by domestic users, because growth in 24-25 was slower than in 23-24. * * * ##### Tangent: Proportion of Domestic Generators As a quick aside, the proportion of overall UK capacity provided by domestic installs is quite surprising: | Domestic | Total | %age ---|---|---|--- **Capacity (MW)** | 5616.5 | 18919 | _29.68%_ **Installs** | 1552266 | 1803478 | _86.07%_ Domestic capacity makes up **nearly 30%** of the UK's installed solar capacity! Domestic forming a high proportion of installations is somewhat less surprising - there are a **lot** more rooftops than there are solar farms - but I'm still quite surprised that it's as high as 86%. * * * #### What drove the increase? The numbers above suggest that increase in the number of installs wasn't driven by domestic. It'd probably be quite easy to shrug, say that it must be driven by commercial capacity and walk away. But, we're down the rabbit hole and the Government's data includes additional information that we can use. So far, we've worked with totals, but the dataset also provides bucketed totals: These group the install count down into capacity bandings, allowing us to see which was most responsible for the YoY increase: | May 23 | May 24 | May 25 ---|---|---|--- **0 to 4kW** | 1236768 | 1360486 | 1522128 **4 to 10kW** | 115489 | 166057 | 219278 **10 to 50kW** | 41645 | 48584 | 56633 **50kW to 5MW** | 4683 | 4860 | 4915 **5 to 25MW** | 428 | 446 | 451 **> 25MW** | 54 | 63 | 73 This translates into the following YoY changes: | May 23 | May 24 delta | May 24 % | May 25 delta | May 25 % ---|---|---|---|---|--- **0 to 4kW** | - | 123718 | +10% | 161642 | +11.88% **4 to 10kW** | - | 50568 | +43.78% | 53221 | +32.09% **10 to 50kW** | - | 6939 | +16.66% | 8049 | +16.56% **50kW to 5MW** | - | 177 | +3.77% | 55 | +1.13% **5 to 25MW** | - | 18 | +4.20% | 5 | +1.12% **> 25MW** | - | 9 | +16.66% | 10 | +15.87% **Total** | | 181429 | | 222982 | These figures point to the answer. There were more micro (`0-4kW`) installs in 24-25, but growth in all other capacity bandings was lower (with growth in the `4-10kW` range quite dramatically so). In percentage terms, the difference is small but the sheer volume of micro installs means that it's still a significant increase in absolute terms. The increase in micro installs was more than sufficient to offset the slowed growth seen in the higher bandings and the combined growth across all bandings was still enough to outpace the previous year. There isn't a corresponding increase in generation capacity because micro installs deliver less bang for their buck: it takes a _lot_ of micro installs to offset even one 50kW install. Although all bandings still show _growth_ , most grew at a lower rate than the year before. The capacity disparity, though, may _already_ be gone as a result of the 373 MW Cleve Hill site coming online. At the smaller end of the scale, there might be a number of possible reasons for the drop in growth: * Increasing market saturation: it might be that most of the households likely to go for a 4-10kW install already have one * Local constraints: Installs over 4kW require additional approval from the DNO, which needs to ensure that the local grid can support the input. The drop in registration rate could be a sign that this approval is getting harder to get * Cost: although the cost of panels has reduced, the cost of living in the UK continues to rise - households might be unwilling to commit to spending the extra for a larger install (or perhaps invest the difference in a battery) * * * ### Where did the 42% come from? The various pages that I could find quoting the 42% figure all referred back to a Carbon Brief analysis: > The figures, revealed in new Carbon Brief analysis, show that the nation’s solar sites have generated a record 7.6 terawatt hours (TWh) of electricity during January to May 2025. > > This is some 42% higher than the 5.4TWh generated in the same period last year Unfortunately, other than opaquely referring to "figures from the National Electricity System Operator", the article doesn't provide much information about the underlying data. However, there is a passage which suggests that _some_ of the difference in my figures might boil down to a misuse of nomenclature: > This article refers to the UK throughout, but the electricity generation data only covers the island of Great Britain. Northern Ireland is part of the all-Ireland electricity system. Carbon Brief's numbers are **not** based on the UK figures that are examined above, but on a _subset_ of them. _Naughty_. A quick YoY analysis shows that Great Britain's growth _was_ slightly higher than when Northern Ireland was included: | Capacity (MW) | YoY | Installs | YoY ---|---|---|---|--- **May 23** | 15591.7 | - | 1351065 | - **May 24** | 17108.8 | _+9.73%_ | 1548780 | _+14.63%_ **May 25** | 18562.2 | _+8.49%_ | 1769494 | _+14.25%_ Although it makes a difference, it still isn't enough. But, _it is_ possible to verify and repeat the 42% figure by looking at energydashboard.co.uk: 2024: 2025: There's a total in the table on the right of the screenshots: | Solar Total (GWh) ---|--- **Q1 2024** | 1986 **Q1 2025** | 2823 This represents a ... _drum roll_... 42% YoY increase. Back at the beginning of this (somewhat excessively long) post, I mentioned regression to the mean. So, before I wrap up, let's see whether this year's jump was anomalous or a sign of things to come. The following figures are for the first 90 days of each year, again taken from energydashboard: Year | GWh | YoY ---|---|--- **2015** | 958 | _-_ **2016** | 1442 | _+5.13%_ **2017** | 1516 | _+6.9%_ **2018** | 1621 | _+19.5%_ **2019** | 1938 | _+19.5%_ **2020** | 1909 | _-2.5%_ **2021** | 1745 | _-9.1%_ **2022** | 2169 | _+24.3%_ **2023** | 1887 | _-13.1%_ **2024** | 1986 | _+5.2%_ **2025** | 2823 | _+42.1%_ So, the first quarter of 2025 saw the biggest YoY increase in UK Solar production for **at least** a decade. With Cleve Hill online, Longfield in the works and numerous others going through planning, those figures will probably see another sizeable jump over the next few years.

I've long had some form of VPN for my devices to use when I'm out and about. Although I used to run OpenVPN, I moved to Tailscale a little while back. Tailscale builds a mesh network using Wireguard protocol and so is able to connect and run quite a bit faster than OpenVPN. Side note: for those wondering, Tailscale is _Canadian_ and can't see the content of connections (although if you're worried about this it's also possible to self-host using Headscale). Although the tailnet has been up for some time, I hadn't got around to setting up split horizon DNS for clients on the tailnet. I was in a bit of a hurry when first setting up and so configured my reverse proxy box to advertise a route to it's own LAN IP. This post talks about configuring my Pi-hole to implement a split horizon: returning the tailnet IP to tailnet clients and the LAN IP to LAN clients. * * * #### Splitting my Split Horizon Many of the DNS names that I wanted to do this for _already_ had a split horizon: Clients on both the LAN and the wider internet connect to the same reverse proxy in my DMZ, but LAN clients connect using the proxy's local IP. The reverse proxy fronts multiple services, most of which have authentication built in. However, it also requires that outside connections pass a separate (and valid) set of authentication credentials before it'll pass their connection on. Having to authenticate twice is a little annoying though, and the split horizon makes it easy to disable the additional authentication when LAN clients connect: satisfy any; allow 192.168.3.0/24; deny all; auth_basic "Authenticate you must"; auth_basic_user_file /etc/nginx/wanaccess.htpasswd; This extra authentication means that I'm not exposing any element of the backing service's authentication stack to the outside world. The underlying idea is that it _shouldn't matter_ that there's an auth bypass zero day in (say) Grafana, because the wider world needs to get past my auth prompt before they can try to detect or exploit it. * * * #### You've Got Access: Why Make The Tailnet Special? Given that there's an ability to access services via the WAN, you might be wondering why it is that I felt that I needed to do something specifically for the tailnet. Unfortunately, the proxy _can't_ enforce additional authentication for some services because those services clients don't support it. Nextcloud is a great example of this: the Nextcloud Desktop sync client authenticates with Nextcloud, but * It uses the `Authorization` header to present it's bearer token, so the reverse proxy will see an unexpected (and, to it, invalid) set of credentials * The client doesn't expose a way to add custom headers to the requests that it makes, so I can't simply send a shared secret and have the proxy check a different header Having the reverse proxy require additional auth breaks off-net Nextcloud clients (and Nextcloud isn't the only service with this issue). * * * ##### Geoblocking Originally, I left the affected services accessible to the world. Unfortunately, I _sometimes_ seem to upset people enough to trigger prolonged attempts at compromising my services. After one such attempt, I decided to reduce attack surface by adding geo-blocking to my reverse proxy, essentially restricting access to areas that I thought we'd be likely to connect from (or _at least_ appear to). This, of course, comes at a cost in flexibility, with access failing if any of the following are true: * We connected from an IP that doesn't have a location in the GeoDB (or is mislocated) * The ISP that we're connecting from does funky routing stuff and/or uses CGNAT * We've travelled somewhere that we wouldn't normally Adding split horizon DNS to the tailnet allows me to avoid these scenarios, because the tailnet subnet can be special cased in _exactly_ the same way that the LAN is. It also increases the likelihood that I can close WAN access off and require that a client be on either the LAN or tailnet. * * * #### The Plan The idea was that a tailnet client would also speak to the Pi-hole, but that names would resolve to a tailnet IP: This is possible because Pi-hole is underpinned by a fork of `dnsmasq` called `pihole-FTL` which has inherited the setting `localise-queries` (in Pi-hole, this is enabled by default). The man page for dnsmasq describes the setting as follows (line breaks mine): > Return answers to DNS queries from /etc/hosts and **--interface-name** and **--dynamic-host** which depend on the interface over which the query was received. > > If a name has more than one address associated with it, and at least one of those addresses is on the same subnet as the interface to which the query was sent, then return only the address(es) on that subnet and return all the available addresses otherwise. > > This allows for a server to have multiple addresses in /etc/hosts corresponding to each of its interfaces, and hosts will get the correct address based on which network they are attached to. > > Currently this facility is limited to IPv4. This means that we can create the following record set in `/etc/pihole/custom.list`: 192.168.3.33 foo.example.com 100.100.3.2 foo.example.com If a query is received over an interface in one of these subnets, only the matching record will be returned (otherwise, both will be returned): Receiving Interface IP | Response ---|--- _192.168.3.13/24_ | 192.168.3.33 _100.100.3.13/24_ | 100.100.3.2 _10.8.0.0/24_ | 192.168.3.33, 100.100.3.2 One small drawback with this is that the records must be in the `hosts` format file - most of my records were in dnsmasq format files, so I had to migrate the ones that I wanted to split. * * * #### Re-Jigging My Docker Container There was, however, a catch. When I first created my pihole container, the docker invocation looked something like this: docker run \ -d \ --name=pihole \ --hostname=pihole \ --restart=unless-stopped \ --e ServerIP=0.0.0.0 \ --e WEBPASSWORD='NotMyRealPass' \ -v $PWD/pihole/conf:/etc/pihole \ -v $PWD/pihole/dnsmasq.d:/etc/dnsmasq.d/ \ -p 53:53 -p 53:53/udp \ -p 8080:80 \ pihole/pihole This meant that the container was using bridged networking, depriving Pi-hole of the means to see which physical interface a query arrived on: it simply saw the other side of a single bridge interface. So, I killed the container and started a new one using host networking: docker run \ -d \ --network=host \ --name=pihole \ --hostname=pihole \ --restart=unless-stopped \ -e ServerIP=0.0.0.0 \ -e WEBPASSWORD='NotMyRealPass' \ -v $PWD/pihole/conf:/etc/pihole \ -v $PWD/pihole/dnsmasq.d:/etc/dnsmasq.d/ \ pihole/pihole However the container failed to start: Pihole's web interface was trying to bind to port 80 which already had something bound to it. As I'd previously mapped 8080 into the container (`-p 8080:80`), I used the environment variable `WEB_PORT` to tell Pi-hole to bind to that port instead: docker run \ -d \ --network=host \ -e WEB_PORT=8080 \ --name=pihole \ --hostname=pihole \ --restart=unless-stopped \ --env=ServerIP=0.0.0.0 \ --env='WEBPASSWORD=NotMyRealPass' \ -v $PWD/pihole/conf:/etc/pihole \ -v $PWD/pihole/dnsmasq.d:/etc/dnsmasq.d/ \ -p 53:53 -p 53:53/udp \ -p 8080:80 \ pihole/pihole * * * ##### DNS Outage Pi-hole came up, but it wasn't responding to queries. Netstat showed `pihole-FTL` listening and bound to all interfaces: $ sudo netstat -lnp | grep :53 tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 2653543/pihole-FTL tcp6 0 0 :::53 :::* LISTEN 2653543/pihole-FTL udp 0 0 0.0.0.0:53 0.0.0.0:* 2653543/pihole-FTL udp6 0 0 :::53 :::* 2653543/pihole-FTL Packet captures showed that queries were coming in, but no responses were being sent. $ sudo tcpdump -i any port 53 21:54:02.345555 enp0s25 In IP 192.168.3.163.32273 > 192.168.3.5.53: 57965+ A? n-deventry.tplinkcloud.com. (44) 21:54:02.512870 enp0s25 In IP 192.168.3.44.63761 > 192.168.3.5.53: 26967+ AAAA? lycraservice-pa.googleapis.com.home. (53) 21:54:02.524346 enp0s25 In IP 192.168.3.44.1270 > 192.168.3.5.53: 2692+ A? lycraservice-pa.googleapis.com.home. (53) 21:54:02.767189 enp0s25 In IP6 2001:820:aa1a:c443:b9c4:44b:df15:bd8e.36925 > 2001:820:aa1a:c443::2.53: 28460+ A? a.nel.cloudflare.com.home. (43) 21:54:02.767189 enp0s25 In IP6 Queries weren't triggering any activity in Pihole's logs either. To restore service to the LAN, I killed the container and brought it back up with bridged networking - DNS sprang straight back to life. It took me a while to figure out what the issue was, but eventually I spotted this setting in Pi-hole's web interface: Pi-hole was configured to only respond to queries received from interface `eth0`. Resolution stopped because the box that I run pihole on doesn't have an `eth0` (it's a `udev`'y style `enp0s25`). I switched this to `Permit all origins` and restarted the container with host networking. This time, queries were answered. * * * #### Configuring Tailscale The box hosting pihole was already part of the tailnet, but I wanted to remove the previous route advertisement. So I ran sudo tailscale down # Previously this was # --advertise-routes=192.168.3.33/32 sudo tailscale set --advertise-routes= sudo tailscale up Then, from another tailnet client (my laptop), I tried resolving a name via both the LAN and tailnet address: $ dig +short foo.example.com @100.99.55.55 100.100.3.2 $ dig +short foo.example.com @192.168.3.13 192.168.3.33 All that was left was to have tailnet clients actually use Pihole. I logged into Tailscale's web interface and added a Split DNS entry: When bringing tailscale up on my Linux laptop, I had to explicitly pass a flag to allow it to use the advertised server sudo tailscale up --accept-dns The android app has a toggle for this, but it was already on. * * * #### Conclusion My devices now have transparent (and slightly more privileged) access to services when I'm out and about. Because Tailscale acts as a mesh network, I don't need to worry about automatically turning the VPN off when I'm at home - devices in the same segment can direct connect to one another rather than making a round-trip via a remote coordinator. As a result of getting this up and running, I've been able to close off WAN access to a number of services (although I still can't can't do that for any service which hosts something I might try to cast, because Chromecasts ignore local DNS... grrr). It all works well enough that I've been able to write, proof-read and publish this post whilst off net. As an added bonus, Tailscale seem to have partnered with Mullvad, so if I'm ever _travelling_ travelling, I can have my devices route all connections via Mullvad and my tailnet.

A couple of years ago, as part of adopting a POSSE (Publish On Own Site, Syndicate Elsewhere) approach, I started to automate syndication of my content into other networks. In the time that's passed since, I've added and removed automation for various networks (at one point, managing to get myself banned from Threads in the process). This post is about adding support for syndicating into a Telegram channel, allowing Telegram users to receive notifications when I publish something new. It covers how to create a bot account and a channel before using Python to submit posts. * * * #### Creating a Telegram Bot Account Telegram bot accounts are provisioned by a bot called the BotFather (`@BotFather`), so I used global search to find it and open a conversation. Once everything was ready, the bot provided me with an authentication token for my bot to use: While I was there, I used some other commands to set the bot's name and profile picture: * * * #### Creating a Channel With the bot created, I needed a channel to post new feed items into. A telegram channel is effectively a one-way group: users can subscribe, but only administrators can post into it. So, to be able to post, the bot needed to be added to the channel as an administrator. In Telegram (for convenience, I used the desktop client), I * Opened the menu * Clicked `Create Channel`. I created a channel for testing, called `bentest`. I then browsed into the channel and clicked the 3-dot menu and selected `Manage Channel`: A little way down the channel management dialog was the option `Administrators`. After clicking that I hit `Add Administrator`, which allowed me to search for my bots username: Clicking the name presented a permissions dialog, submission of which added the bot as a channel administrator. * * * #### Scripting I've written in the past about how my POSSE bot walks through my RSS feed, so I won't focus too much on that here, but as a quick overview: * It fetches my RSS feed using `feedparser` * It iterates through each item in the feed As it works through, the script checks whether it's previously successfully submitted each item, so I added a new conditional block to do the same for Telegram: TELEGRAM_SEEN = check_if_link_seen(TRACKING_MODE, linkhash, storedhash, feed, 'telegram') # Only run if we've not published before *and* Telegram # is configured if not TELEGRAM_SEEN and TELEGRAM_CHANNEL and TELEGRAM_TOKEN: # Try and send if create_Telegram_Post(en)): # Mark as sent write_hash_to_storage( TRACKING_MODE, linkhash, feed, hashtracker, firsthash, 'telegram', en['link'] ) print(f"Posted to Telegram: {en['link']}") POSTED = True Posting into Telegram is handled by the `python-telegram-bot` module: python -m pip install python-telegram-bot The script constructs a simple message containing the post name, link and a teaser comprised of the first 100 words: def create_Telegram_Post(en): ''' Publish into a Telegram channel ''' msg = f"<a href='{entry['link']}'>{entry['title']}</a>\n" msg += ' '.join(entry['description'].split(" ")[0:100]).lstrip() msg += f"... <a href='{entry['link']}'>Read More</a>" An important thing to note here is that, although Telegram allows you to submit HTML, it only supports a very limited subset of tags. If you try and include something that the API doesn't support you'll receive a message like the following: > Can't parse entities: unsupported start tag "br" The script initialises an instance of the bot library and sends the message: bot = telegram.Bot(token=TELEGRAM_TOKEN) status = bot.send_message( chat_id=TELEGRAM_CHANNEL, text=msg, parse_mode="HTML" ) The callchain is wrapped in a `try`/`except` to provide simple exception handling and ensure that the bot can continue working through the feed if one item results in an exception. * * * #### The Channel ID On my first run, I found that the bot couldn't post into the channel `@bentest`: > Failed to post to Telegram: Chat not found After trying a few things and getting nowhere, I called the API to fetch the list of events waiting for the bot: curl -v "https://api.telegram.org/bot${TELEGRAM_TOKEN}/getUpdates"| jq Entries within the output provided the channel id and username (which changed when I customised the `t.me` link... doh): "chat": { "id": -1002619163536, "title": "bentest", "username": "testabcd24", "type": "channel" }, So, rather than `@bentest` I needed to use either the ID or `@testabcd24` -e TELEGRAM_CHANNEL="-1002619163536" Messages then started appearing: * * * #### Switching to Prod As the code was working, I created a new public channel: BenTasker.co.uk Feed. I added the bot to it as an administrator and updated my POSSE cron to use the latest version of my script. When the cron next ran, the bot posted my 10 most recent posts into the channel: It's since automatically published my most recent post in too and (by the time you read this) should hopefully have published _this_ post too: * * * #### Conclusion The `python-telegram-bot` library makes posting into a Telegram channel **incredibly** easy - sending a message can be achieved with a couple of function calls. As a result, my POSSE cron-job can now syndicate into My new Telegram channel, allowing Telegram users to chose to receive notifications whenever I publish something new. It wasn't necessary here, but the library also exposes everything necessary to create an interactive bot so that users can actively interact with it.

I'm going to an in-person event later this year and, in preparation, attendees have been asked to read (and make notes on) a specific book. For me, this presents something of a challenge: although I _used to be_ an avid reader, nowadays I really struggle to be able to sit and read. I _think_ it's a combination of a busy life and having two dogs who'll happily find something (usually ball related) for me to do if I seem even _vaguely_ idle for more than a few seconds. I didn't _particularly_ want to arrive at the event with nothing but excuses, so I decided that the solution was probably to find a way to increase the amount of opportunity that I get to dip into the book. The easiest way to do that, seemed to be to make the book (and any annotations that I might make) available across all of my various devices. That way it would always be to hand if I found myself briefly unoccupied. The book in question _is_ available on Kindle but, aside from Kindle not properly supporting the majority of my devices, I didn't want to build a workflow around Amazon given that they've stopped users from keeping backups of their purchases. I needed something different. Although the solution would ideally not be tied to one provider's ecosystem, finding something open and multi-platform is apparently not that easy. After searching around a bit, I decided that my best hope probably lay with Calibre's content server (it helped that I've previously used Calibre to manage an old Kindle). The catch, though, was that Calibre's quite GUI heavy and I wanted to be able to run this headless on a server rather than on my machine. In this post, I talk about the customisations that I made to achieve my needs as well as a few to make the eventual solution more secure. * * * ### Requirements I had a few requirements that I wanted to meet: * The solution must be FOSS (which Calibre is) * Needs to be separate from my "main" library: on the offchance I end up sending someone a link, I don't need/want them having access to my full library * Should run as a server rather than needing to be on my personal laptop * Should come back up automatically after a restart * **Must** allow annotations and highlighting etc * * * ### Calibre Image A quick hunt around revealed that Linuxserver have a Calibre image, meaning that running Calibre in docker should be quite easy. Using someone else's post as a reference, I span up a container to play with it: docker run \ --name calibre-server \ -p 48080:8080 \ -p 48081:8081 \ -e PASSWORD='password' \ -e TZ='Europe/London' \ -v calibre-test-config:/config \ linuxserver/calibre:latest This worked but it quickly became clear that the image was heavier and more complex than I really wanted or needed. By default, the container runs Kasm (a web native VNC server) allowing access to Calibre's GUI: This is, undoubtedly, a clever and cool use of technology but it's not _particularly_ convenient for use on mobile devices. In the top right, though, there's a `Connect/share` button. This reveals a menu which allowed me to start Calibre's content server on port `8081` (which the container then exposed as `48081`) This was _much_ more convenient: The simple UI allowed me to browse, read and annotate books: Importantly, it worked just the same on Firefox Mobile. * * * #### Shortcomings The functionality that the container offered was _almost_ ideal, but there were a couple of shortcomings. The Content Server needed to be manually enabled whenever the container restarted, which meant that I'd need to continue to expose the web-to-VNC service. My biggest concern, though, came when I `exec`'d into the container to see what processes were running: ps -x PID TTY STAT TIME COMMAND 1 ? Ss 0:00 /package/admin/s6/command/s6-svscan -d4 -- /run/service 17 ? S 0:00 s6-supervise s6-linux-init-shutdownd 20 ? Ss 0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -d3 -c /run/s6/basedir -g 3000 -C -B 36 ? S 0:00 s6-supervise svc-pulseaudio 37 ? S 0:00 s6-supervise s6rc-fdholder 38 ? S 0:00 s6-supervise svc-docker 39 ? S 0:00 s6-supervise svc-nginx 40 ? S 0:00 s6-supervise svc-kasmvnc 41 ? S 0:00 s6-supervise svc-kclient 42 ? S 0:00 s6-supervise s6rc-oneshot-runner 43 ? S 0:00 s6-supervise svc-cron 44 ? S 0:00 s6-supervise svc-de 52 ? Ss 0:00 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/ 214 ? Ss 0:00 bash ./run svc-cron 232 ? Ss 0:00 nginx: master process /usr/sbin/nginx -g daemon off; 242 ? S 0:00 sleep infinity 248 ? Ss 0:00 bash ./run svc-docker 373 ? S 0:00 sleep infinity 374 pts/0 Ss 0:00 bash 584 pts/0 R+ 0:00 ps -x The container was running `docker`, `pulseaudio` and various other services. Some of these were even running with `root` privileges. This simply wouldn't do. * * * ### Customising The Container The docker image had _what_ I needed, the problem was that it wasn't running it in the _way_ that I needed. Calibre's UI isn't actually the only way that you can invoke the content server because there's also the `calibre-server` command. So, I decided to experiment with bypassing the container's entrypoint so that it would only run the bit that I needed. Invoking `calibre-server` worked OK within the _existing_ container, but failed when I tried to use it with a fresh/empty volume because it expects that some setup will have occurred first. Although one option was to run the container "normally" on first run, it would have felt a bit defeatist, so I set about figuring out what was needed and wrote a script that could act as the replacement entrypoint: #!/bin/bash # # Custom entrypoint to configure and launch the # Calibre content-server CALIBRE_LIBRARY_PATH=${CALIBRE_LIBRARY_PATH:-"/config/Calibre_Library/"} CALIBRE_USER=${CALIBRE_USER:-"abc"} # Preconfigure user auth if it doesn't exist if [ ! -f /config/.config/calibre/server-users.sqlite ] then calibre-server --userdb /config/.config/calibre/server-users.sqlite --manage-users -- add $CALIBRE_USER $PASSWORD 2> /dev/null fi # Create a library if one doesn't exist if [ ! -d "$CALIBRE_LIBRARY_PATH" ] then # Create the library dir mkdir -p "$CALIBRE_LIBRARY_PATH" # It won't be considered a library by Calibre yet, we need to add a book # so that the DB gets created cat << " EOM" | sed -e 's/^ //' > /tmp/welcome.md # Welcome Welcome to Calibre-Server, preconfigured by Ben Taskers hacky bash script. You should now be able to upload books to your library for reading and annotation. EOM # Add the book calibredb --library-path "$CALIBRE_LIBRARY_PATH" add /tmp/welcome.md fi # Start the server # # We use basic auth mode here because Calibre will # use digest by default. We're going to want our SSL # reverse proxy to send it's own creds, which is much # easier to configure with basic auth # calibre-server \ --listen-on 0.0.0.0 \ --port 8081 \ --access-log /dev/stdout \ --disable-allow-socket-preallocation \ --enable-auth \ --disable-use-bonjour \ --enable-use-sendfile \ --userdb /config/.config/calibre/server-users.sqlite \ --auth-mode basic \ "$CALIBRE_LIBRARY_PATH" Note: there's a copy of this script on Codeberg. Having written this script, I made adjustments to my docker invocation: * Port `8080` was no longer required as the web2vnc stuff wasn't being run * I mapped in the new entrypoint script * I overrode the container's entrypoint so that it would use the script The run command looked like this: # Start from a clean slate docker volume rm calibre-test-config docker run \ --name calibre-server \ -p 48081:8081 \ -e CALIBRE_USER="abc" \ -e PASSWORD='1Ch4ng3d7h15R3411y!' \ -e TZ='Europe/London' \ -v calibre-test-config:/config \ -v $PWD/entrypoint.sh:/entrypoint.sh \ --entrypoint="/entrypoint.sh" \ linuxserver/calibre:latest Logs from the first run showed the preconfiguration adding my Welcome book: Added book ids: 1 calibre server listening on 0.0.0.0:8081 Subsequent runs correctly detected that the library existed and so didn't attempt to re-insert the book. `calibre-server` was listening and I was able to upload, read and annotate ebooks using my web browser: Just as importantly, the container was now running far fewer processes: docker exec -it 118affff481b ps -x PID TTY STAT TIME COMMAND 1 ? Ss 0:00 /bin/bash /entrypoint.sh 12 ? Sl 0:00 /opt/calibre/bin/calibre-server --listen-on 0.0.0.0 --port 8081 --access-log /dev/stdout --disable-allow-socket-preallocation --e 24 ? S 0:00 /opt/calibre/bin/calibre-parallel --pipe-worker from calibre.utils.safe_atexit import main; main() 33 pts/0 Rs+ 0:00 ps -x Whilst this was a significant improvement, it wasn't perfect: the other stuff was _still_ in the container and so could potentially still present attack surface. Whilst it was true that an adversary would now need to start by exploiting an issue in `calibre-server`, vulnerability chaining _is_ a thing and the existence of other tooling can sometimes help promote a minor flaw into a bad day. Ultimately, this was _still_ a container running with root privileges with tools like `docker` inside it. I decided to look at having the entrypoint script remove unnecessary packages by doing something like: apt remove pulseaudio nginx docker-ce cron However, when I looked at the list of installed packages, I found that the image was **much** fatter than I'd originally realised: dpkg-query --list | wc -l 729 * * * #### Can we use Wolfi? Rather than spending time figuring out which of those packages could safely be removed, I decided that it'd probably make sense to start over from a more secure base: enter Wolfi. Wofli is an (un)distro intended as a secure _and minimal_ base for container images. Wolfi's packages are regularly rebuilt in order to remediate CVEs but, unfortunately, there wasn't a package for Calibre, so I wouldn't get the full benefit of this. However, there not being a package wasn't a major blocker: it just meant that I needed to manually install Calibre into the image. _Ideally_ I'd have liked to build it from source, but Calibre's build process is pretty complex so I settled for pulling binaries (which is also what the linuxserver image does). * * * ##### Calibre Dependencies Calibre depends on `PyQt6` which, in turn, depends on a number of shared libraries, so the list of dependencies was a little longer than I'd like. Unfortunately, I also had to build that list manually because `PyQt6` was installed with `pip` which doesn't pay a _huge_ amount of attention to non Python dependencies. This involved a bit of trial and error because the only clue that something was wrong came when Calibre threw exceptions like this (though only ever one at a time): > ImportError: cannot import name 'QWebEnginePage' from 'qt.webengine' > > ImportError: cannot import name 'QBrush' from 'qt.core' This was odd, because these modules are provided by `PyQt6` which _definitely_ was installed. Running `calibre_postinstall` threw similar exceptions, but it's output also gave a clue about the underlying issue: > Failed to import PyQt module: PyQt6.QtWebEngineCore with error: libxkbfile.so.1: cannot open shared object file: No such file or directory Although `PyQt6` was installed, it couldn't be imported because some of _its_ dependencies weren't. Getting this working was a case of 1. Running `calibre_postinstall` 2. Extracting the missing library name from the output 3. Using `apk search` to identifying the package which provided the library 4. `apk add`ing it (and updating the Dockerfile) 5. `goto 1` until exceptions stop It turned out that I needed 15 additional packages. * * * ##### The `Dockerfile` After working through the dependencies, I arrived at the following Dockerfile: FROM cgr.dev/chainguard/wolfi-base # Install Calibre RUN --mount=type=cache,target=/var/cache/apk \ apk add python3 curl jq py3-pip libxcb libnss \ libpcre2-8-0 libglvnd libfontconfig1 libxkbcommon \ libxcomposite libxdamage libxext libxrandr \ libxtst libdrm alsa-lib mesa-gbm libxkbfile \ && pip install pyqt6 \ && mkdir /opt/calibre /config \ && latest=$(curl -s "https://api.github.com/repos/kovidgoyal/calibre/releases/latest" | jq -r '.assets[] | select(.name | contains("x86_64")) | .browser_download_url' ) \ && curl -L -s -o /tmp/release.txz "$latest" \ && tar xvf /tmp/release.txz -C /opt/calibre \ && rm -f /tmp/release.txz \ && /opt/calibre/calibre_postinstall COPY entrypoint.sh /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] One build later, I had a Wolfi based container running Calibre Content Server. Despite the dependency list being longer than I'd wanted, the container still had far less in it than the linuxserver image had: apk list -I | wc -l 82 I wasn't done yet, though, as there was still room for improvement: * The application was still running as root * There were a few packages that could be removed post-install I tagged the following onto the end of the existing `RUN` statement: && apk del curl jq py3-pip wolfi-keys wolfi-base apk-tools \ && adduser -u 1000 -D calibre \ && chown calibre /config And then added a line to have the container drop permissions: USER calibre My container image was ready, so I built and published it: docker tag test codeberg.org/bentasker/calibre-content-server-docker:8.5.0 docker push codeberg.org/bentasker/calibre-content-server-docker:8.5.0 As a final indicator of the benefits of this, my image was 44% of the size of the linuxserver one: $ docker image ls | grep calibre codeberg.org/bentasker/calibre-content-server-docker 8.5.0 84b9667dc463 7 hours ago 1.29GB linuxserver/calibre 8.5.0 347f2c1b5fe2 39 hours ago 2.91GB Unfortunately, both Calibre and PyQt6 are quite large so there wasn't much scope to reduce the size further. You can see the final Dockerfile on Codeberg. * * * ### Deploying I was ready to actually deploy into production, which meant * Standing a container up on the host box * Acquiring a SSL cert * Configuring my Nginx reverse proxy Note: I could also have dropped it into my Kubernetes cluster, but decided that it was better to keep things simple in case I ended up needing to troubleshoot issues. I don't use named volumes in prod: instead I bind mount from the host filesystem (allowing backups to be performed with `rsync`), so I started by creating the directory structure and setting the owner to the UID used by the container: mkdir -p /home/ben/docker_files/Calibre_Web/data sudo chown 1000 /home/ben/docker_files/Calibre_Web/data Next, I started the container: CALIBRE_PASS="UseSomethingSecure" docker run \ -d \ --restart=always \ --name calibre-server \ -p 48081:8081 \ -e CALIBRE_USER="abc" \ -e PASSWORD="$CALIBRE_PASS" \ -e TZ='Europe/London' \ -v /home/ben/docker_files/Calibre_Web/data:/config \ codeberg.org/bentasker/calibre-content-server-docker:8.5.0 `calibre-server` was up and I was able to hit it directly and browse to my welcome book: * * * #### Fronting with Nginx Note: Because this isn't a fresh deployment of Nginx, some of this may be a little specific to me I created a simple server on port 80: server { listen 80; root /usr/share/nginx/letsencryptbase; index index.php index.html index.htm; server_name calibre.example.com; location / {thoug return 301 https://$host$request_uri; add_header X-Clacks-Overhead "GNU Terry Pratchett"; } location /.well-known/ { try_files $uri 404; } } I won't go into depth here, but next I created DNS records and used `certbot` to acquire a SSL certificate for that subdomain. I was almost ready to configure the HTTPS `server` block. However, first I needed to construct the `authorization` header that the proxy would send to Calibre. I _could_ just have let the proxy pass through the client provided auth header, but I prefer the flexibility of managing auth within Nginx (and I still wanted auth turned on in Calibre so that random devices on the LAN couldn't so easily hit it directly). Basic Authorization is just a base64 encoding of `username:password`, so in a shell I ran echo -n abc:${CALIBRE_PASS} | base64 I took note of the result and started to write the Nginx config: server { listen 443; root /usr/share/nginx/empty; index index.php index.html index.htm; server_name calibre.example.com; ssl on; ssl_certificate /certs/calibre.example.com/fullchain.pem; ssl_certificate_key /certs/live/calibre.example.com/privkey.pem; location / { proxy_pass http://192.168.13.5:48081; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header Host $http_host; # Set this to the output of the shell call proxy_set_header Authorization "Basic ZGlkIHlvdSB0aGluayBJIGZvcmdvdCB0byByZWRhY3QgdGhpcz8="; # Bump the maximum request size up so that # ebooks can be uploaded client_max_body_size 20m; satisfy any; allow 192.168.13.0/24; deny all; auth_basic "Authentication is a must"; auth_basic_user_file /etc/nginx/htpasswd-files/developers; add_header X-Clacks-Overhead "GNU Terry Pratchett"; } } With this live, I was able to use HTTPS to access Calibre. * * * #### Tangent: Obsidian Integration I use Obsidian for a range of things and one of the aspects that I like about it is its extensibility - there's a vast range of community plugins to make it do new and wonderful things. It turned out that there's a Calibre integration plugin for Obsidian, which communicates with the Content Server. So, as well as being able to read books in a web browser, I can also use Obsidian: All of the underlying functionality (annotations etc) works. That's not too surprising as, after all, Obsidian's an electron app and so is really just a glorified web browser. The only minor issue with the plugin is that it doesn't expose a way to provide credentials, so it'll only work from IPs that I've allow-listed (which is fine, because portable devices tend to be on my tailnet). * * * #### Conclusion I now have a web based instance of Calibre which allows me to read and annotate ebooks on any device with a web-browser. Admittedly, highlighting and annotating small passages using a touch screen is a _little_ fiddly but it otherwise seems to work well enough: On Android, using it as a web app seems to work particularly well, with support for swiping to turn the page. Pleasingly, there's also _sort of_ an offline mode (the docs note that it's not as fully featured as it could have been). Hopefully, all of this should enable me to pick the book up and progress a little whenever I've a few minutes spare, something that wouldn't be as possible otherwise. Of course, arguably, the time that I've spent doing and writing about this _could_ instead have been used to, err, _read the book_.

This week, I've been wrong about (at least) two things, believing that: * I was done with the Copytrack stuff: I sent them proof of license and subsequently audited my image use to ensure they'd never have any excuse to darken my virtual doorway again. * I could have a lazy Friday afternoon: I had a day off and intended to doze in front of Four in a bed. Unfortunately, I was wrong on both counts. Not long after I had settled down to watch people complain about the quality of sausages, my phone went "bing", indicating the arrival of an email (soon followed by another). It seems that Copytrack's customer had decided to start filing abuse reports with various service providers, insisting that they either take down my Copytrack blog post or _suspend my service entirely_. I imagine that there's an opportunity here for others to learn from my experiences, so this post takes a look at those complaints. As you might imagine, though, I _am_ going to choose my words a bit more carefully than normal. * * * #### Oh, Do Fuck Off This isn't the first legal threat that I've had to deal with in my time scrawling on the interwebs, so intimidation really wasn't on the cards. I choose my service providers quite carefully, looking to use those that are competent rather than focusing solely on things like price. That's done more with service quality in mind than abuse report handling, but oddly enough, providers that invest in service quality also tend to have good customer facing teams. But still... of all days, this just **had** to happen on what was supposed to be a relatively restful and stress-free day... * * * #### Contents * Oh, Do Fuck Off * Contents * The Complaint * Claim 1: The Content Is Defamatory * Statement Of Fact * Is It Defamatory? * Claim 2: It Violates GDPR * Legal Basis For Processing * Right To Be Forgotten * Rights To Object * Claim 3: It violates the Acceptable Use Policy * Follow Up Claim: Give Me His Info * TL:DR Summary * Received A Complaint? What To Do * Conclusion * * * ### The Complaint I'm not going to publish the originator's details here: inclusion of them would serve no material purpose and I don't _really_ want to give them a second URL to try and complain about. Instead, I'll simply refer to them as the complainant. When an abuse complaint is received, most service providers will send you a copy (for obvious reasons, it's sometimes redacted to strip the originator's details) and ask you to respond within a specific time-frame. Let's walk through the complaint email, which starts by providing an overview: > To the Legal Department, > > I am contacting you in my capacity as the legal representative of REDACTED, a French audiovisual production company headquartered at REDACTED > > This is to formally notify you of defamatory and unlawful content hosted on your infrastructure, accessible at the following URL: https://www.bentasker.co.uk/posts/blog/general/copytrack-sent-me-a-copyright-claim.html > > This article references REDACTED, in the context of a copyright enforcement case managed by COPYTRACK, a company with whom we are legally and professionally affiliated. The email then moves onto giving a more detailed description of the grounds for their complaint (we'll examine each of these in more depth shortly): > 1. **Defamation and Reputational Damage** : The article portrays our company as engaging in “copyright trolling” practices and uses derogatory language that undermines the legitimacy of our copyright enforcement efforts. These statements are entirely unfounded and constitute defamation under the UK Defamation Act 2013, resulting in reputational and financial harm to our business. > > 2. **Breach of UK GDPR / Data Protection Law** : The article discloses identifiable information about our company and myself, in a hostile and non-journalistic context, without consent. This violates the UK GDPR, specifically Article 6 (Lawfulness of Processing) and Article 17 (Right to Erasure). > > 3. **Violation of PlutoDNS Acceptable Use Policy** : Your published terms (https://plutodns.com/terms.html) prohibit the transmission or storage of content that is defamatory, illegal, or infringes the rights of third parties. The content in question clearly falls within these prohibited categories. > > Item three contains a pretty major mistake: PlutoDNS are **not** the provider that the complaint was sent to. In fact, I'd never even heard of them until now and they don't appear to be in any way affiliated with any of my service providers. It's quite an odd mistake to make. The best that I can conclude is that either the complainant was raising complaints about multiple people with multiple providers, or they've been using something like ChatGPT and not properly checking the output. Anyway, the mail then moves on to requesting that the service provider complete specific actions: > In accordance with your obligations as a hosting provider under UK law and the retained EU e-Commerce Directive (2000/31/EC), I formally request that you: > > 1. Remove or disable access to the infringing page mentioned above. > > 2. If moderation is not feasible, suspend or terminate the account responsible for hosting the domain bentasker.co.uk, due to repeated violations of your Acceptable Use Policy. > > There's also a small mistake here, in that none of the providers that complaints were sent to provide me with hosting services. This might sound like semantics, but it matters because the obligations (and exemptions) imposed by the E-Commerce directive vary depending on the nature of the provider: * A hosting provider may themselves become liable if they do not remove/disable once relevant content is reported to them. * A caching provider (e.g. a CDN), though, is exempted until they become aware that the upstream hosting provider has removed/disabled (or they receive a court order etc). * Providers acting as a mere conduit (for example, a connectivity provider) will almost never become liable. Obviously, the complainant has no way to know whether a CDN provider might also be acting as a hosting provider, but that's not the case here. It's inclusion in a complaint mail is also, somewhat unnecessary: providers are very aware of their responsibilities and duties and 2000/31/EC is not in-and-of-itself grounds for a takedown. Mentioning it feels more like a threat that you could try and come after them if they don't accede to your demands. The demand to terminate my account on the basis of "repeated violations of your Acceptable Use Policy" is also mildly interesting, in that the complainant is only actually alleging one occurrence. The complaint email then closes out with a set of links labelled as references and a request that the provider respond within 7 days: > REFERENCES > > Infringing content: https://www.bentasker.co.uk/posts/blog/general/copytrack-sent-me-a-copyright-claim.html > > PlutoDNS Terms of Service: https://plutodns.com/terms.html > > Our company website: REDACTED > > UK ICO: https://ico.org.uk > > CNIL (France): https://www.cnil.fr > > We kindly request written confirmation of the actions taken within 7 working days. Failure to act may result in escalation to the appropriate legal and regulatory authorities in both the UK and the EU. I'm not _quite_ sure what purpose the links to the homepages of the UK and French data protection authorities are actually supposed to serve - again, this feels a bit GPTesque to me. Anyway, moving on, let's take a closer look at each of the claims made. * * * #### Claim 1: The Content Is Defamatory > 1. **Defamation and Reputational Damage** : The article portrays our company as engaging in “copyright trolling” practices and uses derogatory language that undermines the legitimacy of our copyright enforcement efforts. These statements are entirely unfounded and constitute defamation under the UK Defamation Act 2013, resulting in reputational and financial harm to our business. > ##### Statement Of Fact The post being referenced was originally about a different company: Copytrack. In the original version of the post, the complainant's company was mentioned in passing, noting that * Copytrack claimed to be acting on their behalf * The complainant's company was not listed as the author on stock photo sites * I had been unable to find any evidence suggesting a link between the complainants company and the image that Copytrack had contacted me about Ten days later, an update to the post referenced the complainants company some more. This update was made because Copytrack provided documentation indicating their right to act on behalf of the complainant (who in turn had signed a Copytrack provided document stating that they hold the rights to the image). The complainants name appeared because it featured prominently on both of these documents - documents that, in the unlikely event that the dispute proceeded to court, would also enter the public record. Given the concerns raised, on sites across the web, of Copytrack seemingly being (mis)used by copyright trolls, the post goes on to perform some basic due diligence checks on the complainant's company, noting that: * They do appear to be a real (and long standing) business * Their website states that they work with Copytrack * The name on the document does appear to be a real person, at that company The post goes on to note that the company's Trustpilot page carries an interesting statement (translated from French): > VERY IMPORTANT | LEGAL STATEMENT | Protection of Intellectual Property `COMPANY` would like to remind you that any infringement of intellectual property, including the unauthorized use of our visual, audiovisual, or textual content, constitutes an offense punishable by civil and criminal prosecution, in accordance with applicable national and international laws. > > We reserve the right to take any necessary legal action against individuals or legal entities who have published defamatory or false opinions or who have exploited our works without prior authorization. > > Furthermore, any attempt at public defamation or publication of misleading content aimed at harming our reputation may be reported to the relevant platforms as well as to the competent authorities in the author's country of residence. It also notes that the same Trustpilot page carries Copytrack related reviews and quotes a relevant excerpt from one of them. > We have been an Elements member for nearly 5 years and support a lot of small creators, but shady tactics from this company are now making us consider even using stock images going forward. > > They are wasting a lot of our time with aggressive copyright claims for images - and unrealistic threatening demands for hundreds of pounds required in compensation. From there, my post * References a post by another blogger which claims to have made contact with the author of their photo, who denied any knowledge of the complainant or his company * Notes that there are reviews mentioning the complainant on Copytrack's Trustpilot page too, giving an example The final paragraphs of that section note that it's not possible to definitively say that the complainant is acting in bad faith and, in fact, goes so far as to suggest alternative possibilities: > In fairness, there's no way to definitively say whether Copytrack's customer is acting in bad faith - they could as easily be someone who doesn't understand how Copytrack works, or someone who's bought the rights (or believes they have) from some other org. ##### Is It Defamatory? The abuse complaint specifically references the UK Defamation Act 2013. UK defamation law is known to be a bit of a minefield, but the law provides for a number of defences, including Truth and Honest Opinion. **Truth**: if the complainant does not, in fact, own the rights to the image in question, any imputation in my post of them being a copyright troll would, in fact be true and not defamatory. To date, I've not put much effort into tracking down the original author to check with them, but it shouldn't be necessary: Copytrack provided a signed statement from the complainant stating that they had acquired rights to the image via a licensing agreement with the original author. The complainant should therefore be able to clarify the nature (and/or existence) of this agreement (in fact, section 10.2 of Copytrack's enforcement terms require that their customers be able to do so, on demand, something that Copytrack didn't appear to do when I requested proof). **Honest Opinion**: The main thrust of the article is **not** that the complainant is a copyright troll - it's focus is around the idea that Copytrack's mode of operation (and an apparent lack of due diligence) appears to enable (and even encourage) copyright trolling. The complainant's company was **not** randomly selected or presented as an example of this. They appear in the post _because_ they are the originator of the claim that Copytrack sent to me. Where the post does make reference to the rightfulness of this claim, it is quite clearly expressed in an opinionated manner: > Although it was still not clear that the claim is legitimate, I was fed up of having Copytrack in my inbox (with their ever present instructions on how to pay), so emailed over the uncropped screenshot that they'd asked for. It seems unlikely that the content of my post could be construed as being particularly defamatory about the complainant and their business: * That Copytrack said they were acting on behalf of the complainant is factual * That the complainant uses Copytrack is factual and documented, both on their own site **and** in the complaints that they sent to my service providers * Although the post suggests that Copytrack's communication style is reminiscent of copyright trolls, any suggestion that the complainant might be a troll is clearly caveated and never stated as fact * * * #### Claim 2: It Violates GDPR The second claim in the complaint email stated that my post violates GDPR: > 1. **Breach of UK GDPR / Data Protection Law** : The article discloses identifiable information about our company and myself, in a hostile and non-journalistic context, without consent. This violates the UK GDPR, specifically Article 6 (Lawfulness of Processing) and Article 17 (Right to Erasure). > Let's walk through this one bit by bit (The UK's implementation of GDPR is in the Data Protection Act 2018, but for consistency's sake I'll refer to it simply as GDPR): > 1. Breach of UK GDPR / Data Protection Law: The article discloses identifiable information about our company > GDPR defines rules relating to the protection of data about "natural persons". A company is **not** a natural person, and so processing of company data is not subject to GDPR (though, of course, processing of data about individuals within the company may be). There's no need to take my word for it, because the European Commision even has a page on it (for the record, the complainant's company does not appear to be a single person company - their website lists 5 people). > and myself, As noted in the Commission link above, GDPR _does_ apply to personal data relating to people in the course of a professional activity. This is not particularly surprising, given that that would otherwise be a **huge** loophole. So, let's look at what information the post shared about the complainant specifically: * Their name: this featured prominently on documents provided by Copytrack and was also referenced in the article body * A link to the complainants public profile on their company website It's worth noting that there was also some redacted personal data: the Copytrack documents included the complainant's business address (which might, of course also prove to be the complainants home). At no point was this information published unredacted (although, for the record, it appears that the same details _are_ published on the complainant's own site). I didn't seek those address details out: Copytrack sent them to me (and in fact, for other reasons, my 2025-05-10 update actually noted that I believed Copytrack's systems might not be safely handling EU user's data). So, the published personal data is a name: one which (demonstrably) appears in a number of locations around the web (though this, of course, is not a defence under GDPR) and is easily discoverable by searching the complainants company name. Given the wide availability of this information, the risk of harm is likely to be very low. > in a hostile and non-journalistic context, The term "hostile" can be quite loaded - some use it as an extremely strong term, others simply mean a less than friendly environment. It's inclusion here, though, doesn't seem _particularly_ pertinent to the application of GDPR. On the face of it, the reference to a non-journalistic context might also seem irrelevant, but has likely been included because the Data Protection Act 2018 includes exemptions for * The purposes of journalism * Academic purposes * Artistic purposes * Literary purposes It doesn't seem worth going into too much depth assessing whether any of these might apply here, given that there are other aspects of the claim which may obviate the need to do so. Instead, lets move onto the real bones of the GDPR claim: ##### Legal Basis For Processing > without consent. This violates the UK GDPR, specifically Article 6 (Lawfulness of Processing) This part of the complaint echoes a common misconception that consent is the only lawful basis under which GDPR allows personal data to be processed. In reality, though, consent is one of six: > a) the data subject has given consent to the processing of his or her personal data for one or more specific purposes; > > b) processing is necessary for the performance of a contract to which the data subject is party or in order to take steps at the request of the data subject prior to entering into a contract; > > c) processing is necessary for compliance with a legal obligation to which the controller is subject; > > d) processing is necessary in order to protect the vital interests of the data subject or of another natural person; > > e) processing is necessary for the performance of a task carried out in the public interest or in the exercise of official authority vested in the controller; > > f) processing is necessary for the purposes of the legitimate interests pursued by the controller or by a third party, except where such interests are overridden by the interests or fundamental rights and freedoms of the data subject which require protection of personal data, in particular where the data subject is a child. There's very little processing of personal data performed for the purposes of running my blog, but almost all is performed under the basis of Legitimate Interests. My Privacy Policy also clearly explains that I may sometimes share or publish information: It could, perhaps, also be argued that this processing could occur under the basis of being in the public interest: adding to the communal body of work (i.e. comments and posts) warning about concerns with some of the financial demands being made, by a company (Copytrack) seemingly associated with problematic claims and indiscriminate working methods, in the name of the complainant and others. One noteworthy aspect of processing under Legitimate Interests, though, is that it does give the data subject the right to object to that processing. So the complainant _could_ have contacted me and asked that I (re)assess whether my interests in the inclusion of that data were sufficient to override their rights. Within a month of that request, I would need to perform a new Legitimate Interests Assessment (LIA) and then act upon the result. Instead, they opted to file (quite poorly substantiated) complaints with my service providers. This does not _remove_ their right to object, but it does imply that they may be trying to make a point/example (by getting service suspended) rather than actually trying to exercise their GDPR rights. We're still not quite done with the GDPR claim though, because the email continues. ##### Right To Be Forgotten > and Article 17 (Right to Erasure). Article 17 of the GDPR is sometimes known as the "right to be forgotten". It gives data subjects the right to request erasure of data where one of the following applies: > a) The data is no longer required for it's original purpose > > b) the data subject withdraws consent where processing was on the basis of consent **and** no other lawful basis applies > > c) The data subject objects to the processing **and** there are no overriding legitimate interests > > d) The data has been processed unlawfully > > e) The data must be erased to comply with a legal obligation > > f) The data was collected in relation to the offer of information societies in Article 8 The most important thing to understand here, though, is that it places a duty on the **Data Controller** (in this case, me). Outside of a court order, Article 17 is of bugger all relevance to intermediate service providers and is not something which would or should cause someone like a CDN or DNS provider to suspend services. Ignoring that, though, the right to erasure is **not** absolute and Article 17 provides exemptions where processing is necessary: * for exercising the right of freedom and expression * for compliance with a legal obligation * for reasons of public interest in the area of public health * for archiving in the public interest, scientific or historical research (but only if erasure would mean it cannot be achieved) Running a **personal** blog should fall, quite squarely into that first exemption. But, as above, it's all rather moot because Article 17 requires that the complainant make a request to the controller (me) rather than invoking references to it in complaints to random service providers. ##### Rights To Object Instead of attempting to have a third party interrupt my service, the complainant had two valid routes to request that I re-assess the inclusion of their name in my post: * The right to object to processing under Legitimate Interests * The Article 17 right to request erasure Despite specifically referencing GDPR in their complaint email, the complainant has not chosen to exercise either of these rights. Splitting hairs a little, it could also be argued that the phrasing used in the complaint emails is problematic. GDPR provides data subjects with various rights, however the complainant clearly expresses that they are communicating on behalf of their company (which has no GDPR rights): > I am contacting you in my capacity as the legal representative of REDACTED, a French audiovisual production company headquartered at REDACTED None-the-less, having become aware that the complainant has objections, I have proactively completed a Legitimate Interest Assessment to balance their rights against my interests. * * * #### Claim 3: It violates the Acceptable Use Policy Here, I'll be generous and ignore that the complainant is referencing the acceptable use policy of an **entirely unrelated and unassociated company**. > 1. **Violation of PlutoDNS Acceptable Use Policy** : Your published terms (https://plutodns.com/terms.html) prohibit the transmission or storage of content that is defamatory, illegal, or infringes the rights of third parties. The content in question clearly falls within these prohibited categories. > What I have to be less generous about, though, is the apparent mischaracterisation of that company's terms, which don't actually make any reference to defamation or to content which "infringes the rights of third parties". It doesn't seem to be that the author mistakenly inserted the wrong link because searching PlutoDNS's site doesn't reveal any other relevant policies. The most likely explanation seems to be that the complainant is lawyering-by-GPT: asking an LLM to summarise the page and not noticing hallucinated portions of the response. If we look, instead, at my CDN provider's AUP we can see that it covers a range of things: * Material protected by copyright, trademark, trade secret or other IP rights (and without proper authorisation) * Material that is obscene, defamatory, constitutes an illegal threat or violates export control laws * Illegal material (CSAM, malware, scams etc) What this tells us is that, in order to compel a takedown, the complainant would need to adequately demonstrate that the post is defamatory. * * * ### Follow Up Claim: Give Me His Info Unfortunately, this was not some fire-and-forget attempt: when providers processed my dispute and closed the claim, the complainant attempted to argue against it. I'm (obviously) not privy to the full communications, but I understand that the complainant tried to insist that the service providers release details about me, requesting * Records relating to any agreements that we may have (even if indirectly) * Confirmation of the types of service provided (DNS, email, hosting etc). In support of this request, they referenced GDPR Article 15 (Right of access by the data subject). This claim is **absolute bollocks** : In the context of data about **me** , the data subject is **me** and not the complainant. I've noted a few times above that I thought the complainant might be using ChatGPT (or some other LLM) and, for me, this rather cements that. I'm just not sure that I can be convinced that anyone could independently believe that Article 15 of the GDPR provides each of us the right to run around demanding information about _other data subjects_. If true, it really would be quite ironic that someone so apparently invested in copyright enforcement is relying on a system who's creators claim is only possible by ignoring copyright. The information that the complainant wanted would only generally be released to a third party in response to a court order. Even then, it's not quite clear what bearing information about things like who provides my email would have. * * * ### TL:DR Summary I've written quite a lot in this post, so to summarise my response for each of the claims: * **Defamation** : The post is unlikely to be considered defamatory, the sections of the post relating to the complainant are either factual or quite clearly stated as opinion. The post even offers viable alternative explanations. Any mention of copyright trolling is in reference to the way in which Copytrack operates or is offered as commentary on reports of similar issues elsewhere on the web. * **GDPR** : The complainant's name **is** used (and covered by GDPR), however this is performed under a valid lawful basis and (given that the complainant's name is publicly documented, including on their own company website) the risk of harm is assessed as low. The remainder of the complainant's GDPR claims are invalid, as they either misunderstand the principles of GDPR (which applies to the personal data of natural persons, not companies) or misunderstand the applicability of parts of it (such as Articles 15 and 17). In any case, complaints would need to be filed with the data controller (me) or a supervisory authority (the ICO) and not an intermediate service provider. * **Acceptable Use Policy Violations** : Under my various providers AUP's, the complainant would need to demonstrate that the blog post is defamatory or otherwise illegal. They do not appear to be able to do so. * * * ### Received A Complaint? What To Do This seems like a good point to talk about what you should do if you've received notification of an abuse complaint. First, **breathe** : service providers don't generally take action immediately - you'll often have at least a day to respond. Once you've calmed yourself: 1. If your service provider provides the means to do so, acknowledge that you've received the report and are looking into it 2. Sit and re-read both the content and the complaint 3. Try to identify whether there are any possible grounds to the claim: perhaps you wrote something which can be read in a different light? Are you publishing something questionable that someone else said unnecessarily? 4. If you conclude that it's appropriate, make changes to (or remove) the content, but keep a note of exactly what you've changed (I track changes in git, but if you're running Wordpress, something like the Simple History plugin can do this for you). Once you're ready, respond to your service provider, but keep it brief - they don't need to know minutiae, nor do they need to hear how unfair it is. All that they need is the information necessary to be able to quickly assess whether you (or they) need to do more. For all that I've written here, what I actually sent to most service providers was along the lines of: > The grounds for the complaint are invalid. > > The content being complained about is not defamatory (it's factual) and is not in violation of GDPR (which applies to information about individuals, not publicly accessible details about companies). > > Please close the abuse case - FYI the same individual has made complaints to some of my other providers too. Your provider will also have reviewed the content, and if they need more information, they'll generally reply and ask for it. If you believe that this is part of a campaign and that complaints may also be sent to other service providers, it's worth contacting their abuse teams to give them a heads up. They can't simply ignore complaints and will still need to follow their policies, but getting ahead of the curve will help empower them to resolve the complaints quickly. Finally, don't be like me: you _probably_ shouldn't blog about it. It might be tempting to rant and rave (or wose, gloat) but, if further action is subsequently taken, your lawyer may well observe that you've made your own life harder. * * * ### Conclusion Receiving notification of an abuse complaint can be quite stressful: is a provider going to over-react and turn everything off on the say-so of a somewhat dubious claim? The answer to that is almost always "no". If a provider _does_ turn things off without giving you adequate time to respond, you should probably consider that a good sign of a bad service provider. Nuisance reports, by all accounts, are not uncommon and good providers invest in teams able to handle them. Once you've received a report, you need to move relatively quickly but without panicking: assess the details of the complaint against the content being complained of - can it reasonably be construed in the described way? Be aware that, in some cases (like mine), the complainant may also include claims which don't stand up to scrutiny. In a world full of genAI originated inaccuracies, you should also check that everything the complaint says is actually accurate (a **good** provider should also be checking this, but LLMs include such foundational mistakes that they can be easy to miss). By extension, **never ever use ChatGPT (or Copilot, Gemini etc) to respond** , LLM's are incredibly unreliable sources, prone to just making shit up and you **need** to be sure that you'll be able to stand by anything that you've sent in response. Having gathered the facts, send your service provider a brief overview of your findings - if there's no apparent issue, state why. If there was an issue, state how you've addressed it, etc. A good provider will (almost always) want to work with you to resolve things if they feel that you need to do more. Unfortunately, if you're writing critically about things that involve other people, however careful you are, you _will_ sometimes attract complaints and nuisance reports. They're not the end of the world, but you do need to be willing to address them as objectively as possible.