Was looking through old emails for some receipts (the literal kind, not the Twitter kind), and stumbled upon this beauty of an opener:

Date: 2016-12-19.

John Goerzen’s Easily Using SSH with FIDO2/U2F Hardware Security Keys came up yesterday, and I thought it was a good time to fix my mess of private keys. I already own a YubiKey 5C Nano, which sits in my laptop at all times, as well as a 5C NFC, which I figured I could use hopefully with both my phone (NFC) and tablet (USB-C) for SSH when needed.

The ideal was to drop all non-SK keys, and use move to using agent forwarding exclusively when authenticating between hosts — rarely needed, but nice for some remote-to-remote scps or git+ssh pushes. (Agent forwarding is traditionally frowned upon, since someone who has or gains access to your VPS can use your socket to get your agent to auth things, but that issue is greatly reduced when user presence is verified on each use, viz. requiring you to touch your key.)

Turns out all was pretty much that easy! Just two minor hiccups:

• Terminus on iOS supports FIDO2 keys, no payment required (despite what some search results say; looks like it was maybe a paid-only feature during beta but since not). Non-resident Ed25519 keys work very well over NFC on iPhone, but not over USB-C on iPad. The only reference I can find is this from their “ideas” page:

   Unfortunately, iPads and iPhones with USB-C cannot be compatible with OpenSSH-generate FIDO2-based keys. Please generate new FIDO2-based keys in the Termius app. These keys are supported in OpenSSH and all Termius apps.

  Upon testing, Terminus generates a non-resident ECDSA, and that works just great. So, in the end, I have three private keys: an Ed25519 for the 5C Nano, and an Ed25519 and ECDSA for the 5C NFC for use with NFC and USB-C respectively.

• The OpenSSH bundled in macOS (at time of writing, OpenSSH_9.9p2, LibreSSL 3.3.6) doesn’t support the use of these keys. I haven’t checked whether it’s non-resident SKs specifically or what, or whether it’s the version or just a matter of what support is compiled in.

  NixOS/nix-darwin 25.05 carries an OpenSSH_10.0p2, OpenSSL 3.4.1 11 Feb 2025, and it does!

  Using agent forwarding without losing what’s left of one’s humanity implies getting your ssh-agent setup working nicely. How?

I looked into a few different ways, but opted for the simplest: patching OpenSSH (!?). The thought process is as follows:

• /System/Library/LaunchAgents/com.openssh.ssh-agent.plist will put an SSH_AUTH_SOCKET in your environment, which launches the system-provided ssh-agent when first addressed.
• This is a nice, macOS-y way to do things, but we don’t want to go down any rabbit hole that involves disabling SIP, so we can’t just swap the binary path (or binary itself) out.
  • Even if we could, we still need to add launchd support to the ssh-agent in Nix. Thankfully, the needed changes are small, and easy to find; just look for __APPLE_LAUNCHD__ in ssh-agent.c.
• In my experience, disabling the system-provided launch agent doesn’t work reliably.

We apply two patches:

1. Add the __APPLE_LAUNCHD__ bits into the Nix-provided OpenSSH.
2. Change SSH_AUTHSOCKET_ENV_NAME to "VYX_SSH_AUTH_SOCK".

Finally, we install our own launchd user agent (modelled upon the system one, but with our binary), which puts the socket in the VYX_SSH_AUTH_SOCK env var instead. This means we don’t need to worry about the system launch agent; it’ll only get triggered/used when something calls the system ssh binary.

Head teed!

Today it was finally time to write a policy file for one of my Anubis instances. I use Timoni as a fairly thin wrapper over CUE to write templates for my own k8s deployments, and I found it really shone in this particular instance. I’ll just tl;dr and show the code; here’s an excerpt from my blog engine’s bundle.cue, which is the “entrypoint” for compiling its manifests:

anubis: {
  secretName: "anubis-20250816-071240"
  policy: permitPaths: [{
    name:       "permit-atom-xml"
    path_regex: "^/atom\\.xml$"
  }, {
    name:       "permit-feed-xml"
    path_regex: "^/feed\\.xml$"
  }]
}

I’m aiming to expose just a minimum of configurability first. Here’s how the schema side of that is defined in config.cue:

anubis?: {
  // Needs to already exist in the target namespace. Should have key
  // "ED25519_PRIVATE_KEY_HEX".
  secretName: string

  policy?: {
    permitPaths: *[] | [... close({
      name:       string
      path_regex: string
    })]
  }
}

I grabbed the default root bot policy file from https://github.com/TecharoHQ/anubis/blob/main/data/botPolicies.yaml, and converted it to CUE with cue import botPolicies.yaml. Then we put it in the templates package, add a way to inject our config, and use the config to expand upon the defaults:

package templates

#AnubisBotPolicies: {
  #config: #Config

  //# Anubis has the ability to let you import snippets of configuration into the main
  //# configuration file. This allows you to break up your config into smaller parts
  //# that get logically assembled into one big file.

// ...

  }, if #config.kv.anubis.policy.permitPaths != _|_ for setting in #config.kv.anubis.policy.permitPaths {
    name:       setting.name
    path_regex: setting.path_regex
    action:     "ALLOW"
  }, {

// ...

Finally, the bit I really like: creating the ConfigMap (which gets mounted as a volume) with the policy YAML:

#AnubisConfigMap: timoniv1.#ImmutableConfig & {
	Config=#config: #Config
	#Kind:          timoniv1.#ConfigMapKind
	#Meta:          #config.metadata
	#Suffix:        "-anubis-env"

	#Data: {
		"policy.yml": yaml.Marshal(#AnubisBotPolicies & {#config: Config})
	}
}

Note the careful lack of hand-written YAML at any stage! 💛🤍💜🖤

By the wonderful Kinu; base from DeviantArt.

La herramienta del día es nftrace. I couldn’t work out why some pods weren’t able to communicate with each other across the Tailscale mesh. Suspected ACLs, suspected routes weren’t getting installed correctly (p.s. ip route show table 52 (!?)), suspected local firewalls, suspected so much. tcpdump only gets you so far.

Finally, on the target node:

$ doas -s
# nix shell nixpkgs#nftrace nixpkgs#nftables
# nftrace add ip daddr 10.59.1.213
# nftrace monitor

Try the request that isn’t making it through a bunch of times until you can isolate the exact sequence. ^C, nftrace remove, and read carefully:

trace id daac839a inet nftrace-table nftrace-chain packet: iif "tailscale0" ip saddr
    100.67.157.26 ip daddr 10.59.1.213 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 32261
    ip protocol tcp ip length 60 tcp sport 33233 tcp dport 9090 tcp flags == syn tcp
    window 64480
trace id daac839a inet nftrace-table nftrace-chain rule ip daddr 10.59.1.213 meta nftrace
    set 1 (verdict continue)
trace id daac839a inet nftrace-table nftrace-chain policy accept
trace id daac839a ip filter FORWARD packet: iif "tailscale0" oif "cni0" ip saddr 100.67.157.26
    ip daddr 10.59.1.213 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 32261 ip length 60 tcp
    sport 33233 tcp dport 9090 tcp flags == syn tcp window 64480
trace id daac839a ip filter FORWARD rule  counter packets 44827 bytes 28768164 jump
    KUBE-ROUTER-FORWARD (verdict jump KUBE-ROUTER-FORWARD)
trace id daac839a ip filter KUBE-ROUTER-FORWARD rule ip daddr 10.59.1.213  counter packets
    5001 bytes 6279235 jump KUBE-POD-FW-FIAOHC4WHRKERAQ6 (verdict jump
    KUBE-POD-FW-FIAOHC4WHRKERAQ6)
trace id daac839a ip filter KUBE-POD-FW-FIAOHC4WHRKERAQ6 rule  counter packets 5 bytes 300
    jump KUBE-NWPLCY-ZYSQVVSY5LQY7Q46 (verdict jump KUBE-NWPLCY-ZYSQVVSY5LQY7Q46)
trace id daac839a ip filter KUBE-NWPLCY-ZYSQVVSY5LQY7Q46 rule  limit rate 10/minute burst 10
    packets meta mark & 0x00010000 != 0x00010000 counter packets 5 bytes 300 log prefix
    "DROP by policy monitoring/prometheus-k8s" group 100 (verdict continue)
trace id daac839a ip filter KUBE-POD-FW-FIAOHC4WHRKERAQ6 rule  meta mark & 0x00010000 !=
    0x00010000 limit rate 10/minute burst 10 packets counter packets 5 bytes 300 log group
    100 (verdict continue)
trace id daac839a ip filter KUBE-POD-FW-FIAOHC4WHRKERAQ6 rule  meta mark & 0x00010000 !=
    0x00010000 counter packets 5 bytes 300 reject (verdict drop)

What’s that? log prefix "DROP by policy monitoring/prometheus-k8s"?? Guuaaaaauuuuu.