Kayaking through the Erie canal locks is so cool! I used to love visiting lock E-2 which is right by Peebles Island State Park.

I don’t professionally work with databases, though it was interesting to read about how concurrency issues plague that world as well. The trick of always processing rows in the same order is neat!

I don’t make art often, but it’s cool that the tactic I use for getting something on the page goes by the name automatic drawing.

Fun!

I can totally relate. I say that the frustration for me extends to tech in general. I haven’t used a Windows computer in a while, so I can’t easily help someone with issues there. Also, I spin up multiple servers to run this website when people can instead use Bear blog or GitHub pages.

The issue is that cryptography is built upon math problems that are difficult to solve. Quantum computers make solving some of these problems such as integer factorization and discrete logs easier. If someone has a quantum computer that can sufficiently solve those two problems, then they can likely decrypt many ciphertexts that were produced using asymmetric cryptography techniques (think public/private key-pairs). Wikipedia has a great article discussing post-quantum cryptography if you want to read more.

Given all that, if the cost isn’t too high then it’s not a bad idea to look at our current systems and see what we can make quantum-resistant today. Otherwise, we risk being vulnerable to harvest now, decrypt later attacks. This is where an adversary stores encrypted packets until they have a computer powerful enough to break the encryption. You might wonder if encrypted packets from 5 years ago matter. Though if you’re like me, chances are you had to transmit personally identifiable information over the internet for jobs, housing, etc. In the US, it is really difficult to change your social security number.

We’ll explore three different protocols I rely on and how we can make them post-quantum resistant.

SSH

OpenSSH implemented and made default post-quantum key agreement back in April 2022. At the time of writing, the mlkem768x25519-sha256 scheme is used. That’s a mouthful but it essentially describes what the scheme is:

• mlkem: Module lattices key encapsulation mechanism (also known as key-exchange)
• 768: Not sure what this means, sorry.
• x25519: Elliptic curve used in the classical Diffie-Hellman key-exchange.
• sha256: The hash function used to combine the keys generated from ML-KEM and x25519 (see Section 2.4 of previous link).

As you might notice, this is a hybrid quantum/classical algorithm. This is a hedge. If quantum computers arrive which break the classical algorithm, then we’re safe. Similarly if we find out that the post-quantum algorithms are insecure, then we still have the well-developed classical ones. We can only hope that they’re both not found to be insecure.

If you are using a client released after October 2025, then you’ll receive a warning if you’re connecting to a server that doesn’t support post-quantum encryption.

** WARNING: connection is not using a post-quantum key exchange algorithm.
** This session may be vulnerable to "store now, decrypt later" attacks.
** The server may need to be upgraded. See https://openssh.com/pq.html

TLS

Transport Layer Security is a cryptographic protocol that underlies HTTPS and many other applications. You’re likely using it to connect to this website! Similar to OpenSSH, they introduced the hybrid scheme X25519MLKEM768 for post-quantum security. The main difference that I can spot (as a non-cryptographer) is that this scheme does not hash the combined key. If you’re interested, you can read more at this IETF draft.

In terms of major browsers, Firefox has supported this since October 2024 and Chrome since November 2024. To use this scheme, however, both sides need to handle it. At the time of writing, Cloudflare estimates that 70.1% of Internet traffic is post-quantum encrypted. You can even use that link to check whether your own website supports post-quantum key exchange.

If you find that your website is not post-quantum secure, then check out the TLSRef TLS Configurator. This provides the configuration options needed to support modern cryptographic protocols for a variety of popular web servers.

WireGuard

WireGuard is a popular VPN technology that is known for being efficient and simple. By default, it is not post-quantum secure. However, we can use the PresharedKey field to mix a symmetric key whose underlying mathematical problems are not as impacted as asymmetric cryptography.

The simplest solution here then is to run

wg genpsk

and then securely add the key into the PresharedKey field.

[Interface]
Address = ...
...
PrivateKey = ...

[Peer]
PublicKey = ...
...
PresharedKey = ...

Both sides of the connection need to use the same PresharedKey in order for the connection to work.

I’ll reiterate that we need to figure out a way to transfer the key securely for this to work. WireGuard doesn’t have a built-in way to share these keys. If you want to go with this simpler static preshared key route, then I suggest that you use a post-quantum secure channel (like a modern SSH setup) to distribute the keys.

The downside of setting the preshared key on both sides once and then moving on is that the tunnel is then not post-quantum forward secret. This means that if an adversary with access to a sufficiently powerful quantum computer additionally gets access to the preshared key, they can then decrypt all future ciphertexts.

If we want to protect against this, then we’ll need to run a post-quantum key exchange protocol on top of WireGuard. At the time of writing, Rosenpass seems to be the simplest way to set this up. They automatically update the preshared key securely within WireGuard every two minutes.

While that is one solution, the space here seems fragmented. Another approach updates the protocol itself and many VPN providers handle it their own way as well.

Conclusion

Even though we’re not at Y2Q or Q-day, we can still take steps to make sure that we’re transmitting information over the Internet in a quantum-resistant way. In the meantime, I’ll dream about the day when I can run a quantum computer in my pocket.

So this time I went with an object storage setup.

I created a bucket (e.g my-website) and within it I have the following directories

my-website
├── etc
    └── letsencrypt
        └── live
            └── example.com
                ├── cert.pem
                ├── chain.pem
                ├── fullchain.pem
                └── privkey.pem
└── var
    └── www
        ├── website1
        ├── website2
        └── websiten

The core idea is that the webservers will read from this object store to stay up to date with my SSL certificates and my static website files. The rest of the post will go over how I 1) modified my deployment pipeline to push to the object store, 2) push the SSL certificates which are renewed, and 3) pull both the SSL certificates and the website files.

Deploying my website files

Currently, I use Github Actions (labeled as CI/CD in the diagram) to build and deploy my website. Beforehand, I had to create a special SSH key-pair and lock it down in case it leaked. For this new setup, we can instead use application keys to authenticate with our object store.

To lower the threat surface further, we can limit the buckets the application key has access to, whether it has read/write permissions, and what file prefixes the application can access.

For my Github action, I created an application key which has read/write permissions to the prefix var/www/website. We need both permissions if we want to delete files that don’t exist in the build anymore. Here’s the script that I use within the GitHub action to synchronize with the object store after I built the website.

#!/usr/bin/env sh
set -e

# Environmental variables we need to set within the runner
: "${AWS_ACCESS_KEY_ID:?AWS_ACCESS_KEY_ID is not set}"
: "${AWS_SECRET_ACCESS_KEY:?AWS_SECRET_ACCESS_KEY is not set}"
: "${AWS_ENDPOINT_URL:?AWS_ENDPOINT_URL is not set}"
: "${S3_BUCKET:?S3_BUCKET is not set}"
: "${S3_PATH:?S3_PATH is not set}"

# Safety check so we don't wipe our website!
if [ ! -d "public" ] || [ -z "$(ls -A public)" ]; then
  echo "public/ is empty or missing"
  exit 1
fi

aws s3 sync public/ "s3://${S3_BUCKET}/${S3_PATH}/" \
  --delete \
  --exclude "*.bak"

Summarizing how the script works:

• The environmental variables at the beginning are used to authenticate with the object store.
• We sanity check that the public folder exists after running hugo and that it’s non-empty so that we don’t accidentally wipe the object store’s files.
• We use the aws command to perform the sync. Note that this command is baked into the default Ubuntu image used by GitHub’s runners, so no installation step is required.

Pushing SSL Certificates

I use Certbot to request SSL certificates from Let’s Encrypt. After a renewal certificate is issued, the client will run any scripts located within /etc/letsencrypt/renewal-hooks/deploy (documentation). In that case, we want to add a push-certs-to-object-store.sh file which does just that.

#!/bin/bash
set -u

S3_BUCKET="INSERT_BUCKET_NAME"
ENDPOINT="INSERT_ENDPOINT_URL"
AWS_ACCESS_KEY_ID="INSERT_KEY_ID_HERE"
AWS_SECRET_ACCESS_KEY="INSERT_SECRET_KEY_HERE"

podman run --rm \
  -e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
  -e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
  -v "${RENEWED_LINEAGE}:${RENEWED_LINEAGE}:ro,z" \
  -v "/etc/letsencrypt/archive:/etc/letsencrypt/archive:ro,z" \
  docker.io/amazon/aws-cli s3 cp "${RENEWED_LINEAGE}/" \
  "s3://${S3_BUCKET}/${RENEWED_LINEAGE#/}/" \
  --recursive \
  --endpoint-url "${ENDPOINT}"

if [ $? -ne 0 ]; then
  # Insert failure notification technique here
  exit 1
fi

Make sure that this script is executable after saving it. This script is different from the last in that it uses podman to run aws-cli as opposed to executing it directly. This is because I’m using Fedora CoreOS an immutable distribution which highly discourages overlaying packages.

Going over the script:

• ${RENEWED_LINEAGE} is the folder path which contains the renewed certs (e.g /etc/letsencrypt/live/example.com)
• We need to mount the ${RENEWED_LINEAGE} path as well as /etc/letsencrypt/archive since the live folder only contains symbolic links to the files which are actually stored in the archive.
• Since we’re not modifying these files, we can treat them as read-only (the ro flag). Since I have SELinux enabled I threw in the z flag so that Podman can automatically handle the contexts for me.
• We’re using cp instead of sync since the renewal procedure will overwrite all the existing files with new ones. Meaning that we don’t need read permissions for this application key to update the certificates.
• Personally for alerting, I send a curl request to a webhook on failure.

Pulling the files

We can create an application key with read-only permissions in order to pull the SSL certificates and the website files.

Website Files

Here’s the script that I use to synchronize the local copy with that of the object store:

#!/bin/bash
set -e

: "${AWS_ACCESS_KEY_ID:?AWS_ACCESS_KEY_ID is not set}"
: "${AWS_SECRET_ACCESS_KEY:?AWS_SECRET_ACCESS_KEY is not set}"
: "${AWS_ENDPOINT_URL:?AWS_ENDPOINT_URL is not set}"
: "${S3_BUCKET:?S3_BUCKET is not set}"

sync_site() {
    local path="$1"

    mkdir -p "/${path}"

    podman run --rm \
      -e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
      -e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
      -v "/${path}:/${path}:z" \
      docker.io/amazon/aws-cli s3 sync \
      "s3://${S3_BUCKET}/${path}/" "/${path}/" \
      --delete \
      --endpoint-url "${AWS_ENDPOINT_URL}"
}

sync_site "var/www/website1"
sync_site "var/www/website2"
sync_site "var/www/websiten"

I have a corresponding systemd unit file and timer which runs every 15 minutes to check for changes.

SSL Certificates

For my certificates, I only check for new ones daily. The script is very similar…

#!/bin/bash
set -e

: "${AWS_ACCESS_KEY_ID:?AWS_ACCESS_KEY_ID is not set}"
: "${AWS_SECRET_ACCESS_KEY:?AWS_SECRET_ACCESS_KEY is not set}"
: "${AWS_ENDPOINT_URL:?AWS_ENDPOINT_URL is not set}"
: "${S3_BUCKET:?S3_BUCKET is not set}"
: "${S3_PATH:?S3_PATH is not set}"

mkdir -p -m 700 "/${S3_PATH}"

podman run --rm \
  -e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
  -e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
  -v "/${S3_PATH}/":"/${S3_PATH}/":z \
  docker.io/amazon/aws-cli s3 cp \
  "s3://${S3_BUCKET}/${S3_PATH}/" \
  "/${S3_PATH}/" \
  --recursive \
  --endpoint-url "${AWS_ENDPOINT_URL}"

Conclusion

That’s at least how I have it set up at the time of writing. I’ve only been running this setup for two days, so we’ll see how I feel ultimately. Right now, I’m happy that it simplifies my Ansible setup for these servers. Before this, I had to manually setup Syncthing using their webui.

This updated method allows me to copy a few scripts and systemd unit files over and call it a day. I’m also happy that I don’t have to deal with creating a special build user and making sure that’s locked down. Now we’ll see if I can be patient for 15 minutes to see my website changes ;D

This was a fun game, try it out!

Even though brocard is a weird name for them, I enjoyed reading the principles that William uses to triage vulnerabilities.

Using a certificate authority to sign a user’s public SSH key instead of storing them on the machine is pretty neat. I’ve worked on systems before where we had short-lived SSH keys and I imagine that the setup had to be similar to what’s presented here. If I scaled up and down infrastructure in my homelab more, then I can totally see myself implementing this.

Woah! I didn’t know that you can do sparse checkouts with git! That was one of my favorite features from subversion. This makes working with large mono-repos more bearable.

I also own a Honda Fit and am sad that they discontinued them in the US. Not only are smaller vehicles more fuel efficient, they’re also easier to parallel park!

I’m not sure when the government set these CAFE MPG targets, but they seem quite aspirational! Even though I would love a small car that drives 70 MPG, it’s rough that these regulations are incentivizing companies to focus on larger vehicles.

What a crazy and fun project! I can imagine archiving a blog as a journal/periodical where you have volume numbers. For example, “Brandon Rozek’s Ramblings Volume 1”.

As an example, he asked me, in more words, what the expected rank when flipping over the top card of a deck of cards was (A=1, J=11, Q=12, K=13). This is easy to compute directly as 7. Then he asked me the expectation of the sum of the top two cards.

- From “Expectation and Copysets” by Justin Jaffray

What does your intuition say the answer is? Justin continues by stating that computing this expectation is as easy as summing their individual expectations. $$ E[X + Y] = E[X] + E[Y] $$ In other words, expectations are linear. I recommend reading his entire blog post. It’s great and also talks about how this property is used in databases today. After a high-level explanation, he says:

The fact that expectation is linear is easy to show if you just look at the definition, which we will not do here, but I trust you are capable of if you are interested and have not already seen it.

In this episode of Exercise for the Reader (last episode), we’ll look at the definition and show why this property holds. This is true regardless of the underlying probability distribution and whether or not we’re sampling with replacement.

As Justin stated, let’s start with the definition of expectation and then split the sum: $$ \begin{align*} E[X + Y] &= \sum_{x \in X} \sum_{y \in Y} (x + y) \cdot P(X = x, Y=y) \\ &= (\sum_{x \in X} \sum_{y \in Y} x \cdot P(X = x, Y = y)) + (\sum_{x \in X} \sum_{y \in Y} y \cdot P(X = x, Y = y)) \end{align*} $$ Notice that the left-hand-side of the multiplication does not depend on both variables anymore. Also it doesn’t matter whether we do the summation over $X$ first or $Y$. Therefore, we can bring that variable out of the inner sum and simplify this to: $$ E[X + Y] = (\sum_{x \in X} x \sum_{y \in Y} P(X = x, Y = y)) + (\sum_{y \in Y} y \sum_{x \in X} P(X = x, Y = y)) $$ We can then perform marginalization to substitute $\sum_{y \in Y} P(X = x, Y = y)$ with $P(X = x)$ and do the same for the right hand side of the sum. $$ \begin{align*} E[X + Y] &= (\sum_{x \in X}xP(X = x)) + (\sum_{y \in y}yP(Y=y))) \\ &= E[x] + E[Y] \end{align*} $$

Why can we marginalize? For me to show why, we need to peel back the curtain on the notation.

The set $\Omega$ contains the outcomes of all the events that we’re concerned about. So, if we are considering events $X$ and $Y$ with outcomes $x_i$ and $y_i$, respectively. Then, our event space $\Omega$ is equal to $\{ (x_i, y_i) \mid x_i \in X, y_i \in Y\}$.

Therefore when we say $X = x$, what we really mean is the set of outcomes where that is true. In mathematical terms, $\{\omega \in \Omega \mid X(\omega) = x\}$.

Now, let’s show why $\sum_{y \in Y} P(X = x, Y = y) = P(X = x)$. $$ \sum_{y \in Y}P(X = x, Y = y) = \sum_{y \in Y} P(\{\omega \in \Omega \mid X(\omega) = x\} \cap \{\omega \in \Omega \mid Y(\omega) = y\}) $$ One of the three Kolmogorov axioms of probability is countable additivity. This is defined as: $$ \sum_{x \in A}P(X = x) = P(\bigcup_{x \in A}X =x ) $$ Substituting that in and simplifying, we get: $$ \begin{align*} \sum_{y \in Y}P(X = x, Y = y) &= P(\bigcup_{y \in Y}(\{\omega \in \Omega \mid X(\omega) = x\} \cap \{\omega \in \Omega \mid Y(\omega) = y\})) \\ &= P(\{\omega \in \Omega \mid X(\omega) = x\} \cap \bigcup_{y \in Y}\{\omega \in \Omega \mid Y(\omega) = y\}) \end{align*} $$ Notice that the right hand side of the term is just $\Omega$. We can then simplify to, $$ \begin{align*} \sum_{y \in Y}P(X = x, Y = y) &= P(\{\omega \in \Omega \mid X(\omega) = x\} \cap \Omega) \\ &= P(\{\omega \in \Omega \mid X(\omega) = x\}) \\ &= P(X = x) \end{align*} $$ Since countable additivity is an axiom, we’ll stop our derivations there. See you next time.

Great reminder to build what you want to see in the world. Hedgehogs are cool!

Using ADS-B data from planes to capture wind speed data is really cool! I never thought about how we have thousands of little wind sensors flying around in the sky.

Very informative piece on how Iran structures their internet infrastructure. It discusses the incentives developed over many years as well as some of the underlying assumptions that are being challenged.

Nice introduction to BGP and a fun live video of a map showing BGP announcements!

Distributed systems are difficult to get 100% right. This article goes over a recent story where it went wrong.

It’s not too surprising to me that Tesla would use LLMs to present a “human readable summary” to describe a fixed set of inputs. Add that to the lessons learned bucket.

Public service announcement to make sure you’re using post-quantum cryptography. Alex shares Cloudflare’s tool which lets you know if a webserver supports post-quantum TLS key exchange. Also don’t forget to check your SSH server! Luckily, many recent versions of these softwares ship with this enabled out of the box.

I’m thankful to all the open-source maintainers out there that work hard to ensure that the applications we use are polished and secure. It’s always been a lot of work, but I do worry about the amount of “vulnerabilities” filed against these repositories today. Unfortunately (as Hendrik describes), every author believes their issue critical. Even if an act of God would have to occur for their vulnerability to be exploited.

In this post, I’ll share the steps I took and what I learned along the way. Huge credit to Christian who wrote their own succinct version of this post and helped me troubleshoot via email.

Getting an Address

Unlike the clear web, we don’t register a domain with anyone. An address on Tor is a hash of your public key.

This is why onion URLs are long and unreadable. Take a look at the following onion URL which takes you to the Tor homepage.

http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion

Notice that the address starts with http. Unlike the clearweb, all traffic via Tor is encrypted. Our hidden service will use the public key behind the URL during the protocol exchange.

The primary benefit of these addresses are that they are entirely decentralized (we create our own keys). They are also incredibly difficult to spoof. The downside is that they are not human readable. We can get a little bit closer to that though with vanity keys.

To create a vanity key, we can use mkp224o. After following the installation instructions, we can create our own onion URL starting with some prefix by running the following:

./mkp224o prefix

Given the current algorithms and hardware, we can easily generate keys which have a URL prefix length of up to six characters in minutes. Beyond that, the generation quickly becomes infeasible… However, we want this to be the case. If it was feasible to generate a key-pair for an entire onion URL then anyone can spoof your hidden service.

The script by default will create many folders in the current directory which have the onion URLs as their name and the contents of the keys within. Pick your favorite and copy it over to /var/lib/tor/somehiddenservicename.

While you’re at it, make a backup of these keys because if we lose it then we lose control over the domain.

Installing and Configuring Tor

Now that we have our address. Let’s get Tor installed and set up. In this guide, I’ll show how to do so via Podman Quadlets.

We’ll use the Docker container dockur/tor. Since Tor is a cryptographic software, we need to be extremely careful where we download it from. As of this time of writing if we look at the Dockerfile in this repo, we can see that this is a simple wrapper over Alpine’s tor packages. Currently, I feel that it’s safe to trust the Alpine maintainers.

After selecting the Docker container, we need to write the Quadlet definition file. Here’s how I have /etc/containers/systemd/tor.container configured:

[Container]
ContainerName=tor
HostName=tor
Image=docker.io/dockurr/tor
AutoUpdate=registry
Volume=/etc/tor:/etc/tor:ro,Z
Volume=/var/lib/tor:/var/lib/tor:Z,U

[Service]
Restart=always

[Install]
WantedBy=default.target

Before we run it, we’ll need to create our main Tor configuration file. This lives in /etc/tor/torrc.

SocksPort 0
HiddenServiceDir /var/lib/tor/somehiddenservicename
HiddenServicePort 80 IP_OF_NGINX_CONTAINER:80

Replace IP_OF_NGINX_CONTAINER with the internal IP of the Nginx container or of the host machine if it is running on there. Here’s an explanation of each line:

Since we’re proxying traffic to our application, let’s configure our target next.

Configuring Nginx

My website is a static site served by Nginx. These files are linked together via absolute URLs. This adds some complications because the onion hidden service URL is completely different than my clearnet one.

One solution would be to make my URLs relative. However, I want my website to be as portable as possible. For example, you can run my website in a subfolder. Thus, our solution here is to have an entirely separate copy of the website where the only difference is the internal URLs.

I’ve set this version of my website to live in /var/www/brozekhs.

As such, we have to create an Nginx configuration file solely for our hidden service. We’ll make sure to point the root to the folder which contains the hidden service version of my website.

server {
    listen 80;
    server_name ONION_URL;

	root /var/www/brozekhs;
    index index.html;

	# ...

	location / {
		allow IP_OF_TOR_CONTAINER;
        deny all;
        # ...
	}
}

Replace /var/www/brozekhs, ONION_URL and IP_OF_TOR_CONTAINER with their respective values. We need to configure the allow and deny lines because we don’t want someone to confirm that our machine responds to tor addresses over the clearnet.

After this when running my nginx container I came across the following error:

date time [emerg] 1#1: could not build server_names_hash, you should increase server_names_hash_bucket_size: 64

As stated, the solution is to increase the server name hash bucket size. I doubled it from 64 to 128 in the http block of /etc/nginx/nginx.conf:

server_names_hash_bucket_size 128;

Lastly if you have a clearnet site and you want to advertise the onion URL for users visiting the clearnet URL on the Tor browser, then we can add an Onion-Location header to the Nginx config of our clearnet site.

location / {
	# ...
    add_header Onion-Location ONION_URL$request_uri;
}

Replace ONION_URL with your own and make sure that $request_uri stays at the end of the header value. This will help the Tor browser with the redirection.

Start em up!

With that, we can start our two services:

sudo systemctl start nginx
sudo systemctl start tor

When we look at journalctl -U tor, we should see something like the following:

datetime hostname systemd[1]: Started tor.service.
datetime hostname tor[681220]: datetime [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
datetime hostname tor[681220]: datetime [notice] Read configuration file "/etc/tor/torrc".
datetime hostname tor[681220]: datetime [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip.
datetime hostname tor[681220]: datetime [notice] Parsing GEOIP IPv6 file /usr/share/tor/geoip6.
datetime hostname tor[681220]: datetime [notice] Bootstrapped 0% (starting): Starting
datetime hostname tor[681220]: datetime [notice] Starting with guard context "default"
datetime hostname tor[681220]: datetime [notice] Bootstrapped 5% (conn): Connecting to a relay
datetime hostname tor[681220]: datetime [notice] Bootstrapped 10% (conn_done): Connected to a relay
datetime hostname tor[681220]: datetime [notice] Bootstrapped 14% (handshake): Handshaking with a relay
datetime hostname tor[681220]: datetime [notice] Bootstrapped 15% (handshake_done): Handshake with a relay done
datetime hostname tor[681220]: datetime [notice] Bootstrapped 45% (requesting_descriptors): Asking for relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 50% (loading_descriptors): Loading relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 55% (loading_descriptors): Loading relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 60% (loading_descriptors): Loading relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 66% (loading_descriptors): Loading relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits
datetime hostname tor[681220]: datetime [notice] Bootstrapped 90% (ap_handshake_done): Handshake finished with a relay to build circuits
datetime hostname tor[681220]: datetime [notice] Bootstrapped 95% (circuit_create): Establishing a Tor circuit
datetime hostname tor[681220]: datetime [notice] Bootstrapped 100% (done): Done

We won’t be able to access the hidden service until the bootstrapping process reaches 100%. Depending on the state of the network, this might take awhile…

I mention the network state since it’s common for me to see this in the logs:

datetime hostname tor[295456]: datetime [warn] Detected possible compression bomb with input size = 17413 and output size = 515472 (compression factor = 29.60)
datetime hostname tor[295456]: datetime [warn] Possible compression bomb; abandoning stream.
datetime hostname tor[295456]: datetime [warn] Detected possible compression bomb with input size = 23289 and output size = 774624 (compression factor = 33.26)
datetime hostname tor[295456]: datetime [warn] Possible compression bomb; abandoning stream.

This compression or zip bomb appears to be an attempt at a distributed denial of service attack. Luckily, the Tor software is smart enough to guard against this.

Every so often my Tor service goes down and I need to restart the daemon for it to come back up. I haven’t gotten around to writing monitoring scripts for my hidden service so if anyone has any tips feel free to get in touch.

I enjoyed having a live tree in my apartment. I still remember the smell of fresh conifer. After several weeks, the holidays pass and the needles start falling to the ground. I don’t quite remember how we got rid of this tree, but I wouldn’t be surprised if we just threw it in our dumpster.

We later moved to New York. For the next few years, we would travel to see friends and family during the holidays. I used that as an excuse to not have a tree setup at home. Eventually Clare convinces me that we should still be festive at home, and we bought a plastic tree.

Around us in upstate New York, there are several bonfire events in January. There’s one in particular that we’ve attended for several New Year’s Eve. What’s special about this bonfire is that the fuel of the fire is not prepared as chopped wood in advance. Instead, it is sourced from the community.

Let me paint a picture. It’s a cold winter night, and there is a group of around 20-30 people bundled up in full winter gear staying close to the bonfire. As time progresses, the fire gets smaller and smaller, and we get closer to try to feel some of its heat. Then, a person in a pickup truck arrives, gets out of their vehicle, and chucks their Christmas tree from the bed of the truck onto the fire. The fire rages.

Now I find this very cool, but some trees have different fates. Ton writes on his blog that they have the same tree every year. Instead of sawing their tree down, the tree gets delivered to their place with the root ball intact. Once the holidays are over, the growers come back and pick it up.

Through an email exchange, I learn that this is a national service over in the Netherlands. Now that’s pretty cool.

• Text
• Email
• Pen a letter
• Post on a microblog (Mastodon/X/Pixelfed/etc.)
• Write a blog post

But not all of these methods inherently create a conversation or dialogue. When I write a technical blog post, I don’t expect a reply. Similarly, when I toot on Mastodon, I’m fine if no one favorited the post. As such, (micro-)blogging differs greatly from texting and calling someone and is instead much closer to recording a postcast or uploading a video – a one-way transmission of information.

Ploum wrote about how he views the ActivityPub protocol as a conversation, and as such servers should not filter posts based on whether they have a picture. Now I’m not on Pixelfed, so I do not have a stake in this issue. However since I view Pixelfed as a microblogging platform, I tend to see it as more of a one-way transmission of information rather than soliciting a response from my friends. This view would put me in the category of folks that view AcitvityPub as a “content consumption platform.” Though I find that term derogative¹.

Now should Pixelfed filter posts based on whether it contains a picture? I’m not sure. But if I was on Mastodon and I specifically @’d someone on Pixelfed, then I would sure hope that either they received that message or I was shown an error.

So if we’re not having a conversation with these (micro-)blog posts, what are we doing? Some of us are trying to teach, keep a public journal, or share our perspectives. Some of us don’t even want responses². Though I find that many of us do:

Blog posts become invitations that never expire - Dries

When I send a postcard to a loved one, they don’t need to reply. However, I send it because it’s an acknowledgement of our relationship and it’s an invitation to reach out.

Similarly when I write a blog post, by default I’m only transmitting information. However, any of you readers can choose to promote this from a transmission to an exchange. From a communication to a dialogue.

https://meta.wikimedia.org/wiki/Event:Wikipedia_25_Virtual_Celebration

Here's to another 25 years!

The first step before talking technology is to identify exactly what data we want to backup. In my homelab, I rely on Immich for photo storage, Navidrome for music streaming, databases for my website, and other personal documents. This amounts to a little over 250 GB of data that would be very difficult for me to replace if it was lost.

Not all my data lives on that one server though. I also have a storage VPS which runs Nextcloud and Hedgedoc. Both of those services combined have less than 100 GB of data, but just like the homelab, that data is precious. The storage VPS has a total capacity of 1.5 TB.

The 3-2-1 backup strategy coined by Peter Krogh suggests that we should have three copies of our data, stored on two different media, and with one being off-site.

Both of my servers have enough storage capacity to hold a copy of all of my data. For my third copy, I decided to rely on Backblaze b2. This is a S3 object storage service that charges low rates for what I store and the number of transactions I make.

My homelab sits in my house, the storage VPS sits in Montreal, and Backblaze stores my data in Phoenix. Therefore, I have 3 copies of my data and it’s stored in more than 1 location.

In order to address having at least 2 different storage media, I backup my data using two different approaches: Restic and Btrfs Snapshots. I don’t currently have enough data such that hot online storage is cost-prohibitive.

Restic

Restic is a modern and open-source backup software that both deduplicates and encrypts file contents at the blob level. In combination with a metadata local cache, this makes this Restic snappy and efficient to use.

It supports a variety of backup targets:

• Local filesystem
• SFTP
• REST server
• S3
• and several others, including those implemented by Rclone!

While I could use the variety of backup targets to increase the number of approaches my data is backed up, to me, this does not feel sufficiently different enough to warrent the additional complexity. Therefore, I’ll use one backup target for all three locations.

Since Backblaze b2 only supports the S3 protocol, this means then that my choice has been made for me. However even if it wasn’t forced upon me, I would still pick that option. Designed by Amazon for their storage service, the S3 protocol is built with scale in mind and supports concurrent multipart file uploads. Additionally unlike SFTP, S3 separates its accounts from that of the server. There are ways to restrict SSH clients, however, it is not the default behavior. In S3, by default a user cannot do anything.

Thus, after choosing S3, I need to set up a S3 server on both my storage VPS and my homelab. I landed on MinIO for its ease of setup and longevity, however, any of the other solutions would work as well.

Since, we’re storing copies of the data on all the servers, one idea is to set it up in a cluster configuration. However, since I’m not managing the S3 server hosted by Backblaze, I wouldn’t be able to include that in the cluster. Generally, it is not recommended to run a cluster with only two nodes. If we do, it can lead to what’s called a split brain problem where both nodes are unable to communicate with each other and think that they are the leader node.

To avoid this, we can run both MinIO instances inpendently and have Restic separately send the data to all servers.

MinIO Setup

If you already have a bucket setup and configured with the appropriate permissions, then feel free to skip this section.

The following code examples assume that MinIO is installed and the mc client is available with the root credentials to each server (alias) set. These general steps can also be achieved using the management UI. For simplicity, I’ll show how to do this with respect to one server homelab. However, we’ll need to do this for every MinIO instance.

First, create a bucket which will hold our backup data.

# alias_of_server/bucket_name
mc mb homelab/backups

Afterwards, create a backup user. We’ll need to specify the ACCESS_KEY and SECRET_KEY which corresponds to the username and password respectively. Store these as we’ll later need to use them.

export ACCESS_KEY="my_awesome_homelab_user"
export SECRET_KEY="SUPER_SECURE_SECRET_KEY"
mc admin user add homelab $ACCESSKEY $SECRETKEY

Note: For a bit more security, I like to create a different access key and secret key for the other servers.

By default, our user cannot cannot do anything. Let’s make it so that they have access to our backups bucket we created earlier. To do that, we’ll need to create a policy which allows the user to perform operations on that bucket.

We’ll have to create a JSON file and follow the AWS IAM format.

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Action": [
    "s3:ListBucket",
    "s3:PutObject",
    "s3:DeleteObject",
    "s3:GetObject"
   ],
   "Resource": [
    "arn:aws:s3:::backups/*",
    "arn:aws:s3:::backups"
   ]
  }
 ]
}

Assume that we saved the prior JSON at a location specified by the environmental variable $POLICY_PATH, then we can create the policy through the following:

export POLICY_NAME="backup-policy"
mc admin policy create homelab $POLICY_NAME $POLICY_PATH

After creating the policy, we need to assign it to our backup user.

mc admin policy attach homelab $POLICY_NAME --user $ACCESS_KEY

Nginx and Initializing Restic

With our bucket and user configured, now let’s discuss how we’ll perform our backups. Our bucket will have the following directory structure:

backups/
  homelab/
  vps/

Each folder will contain a restic repository. We separate them out instead of having one giant restic repository, so that we don’t have to worry about multiple servers competing for a lock.

Instead of communicating over HTTP, we want a secure way to access our buckets from outside the server. For this, I use nginx to proxy the traffic over to MinIO. I configure Certbot with LetsEncrypt so that the connection can be secureted with HTTPS/TLS. In addition to the instructions listed on the MinIO page, I also restrict the IPs which are allowed to connect. For example, to only allow traffic from within your Wireguard network (ex subnet: 10.10.10.1/24) then you can put the following within the location block of the nginx config.

allow 10.10.10.1/24;
deny all; // Denies all other traffic

After securing the entryway to our S3 server, we’re ready to use Restic. Before jumping in, you might want to create a dedicated account so that we don’t have to use the root user.

Let’s initialize each server’s respective repository. Since our backups will be encrypted, we need to come up with another password and put it within the RESTIC_PASSWORD environmental variable. Save this password in a safe place, since we will not be able to decrypt the backup without it.

export RESTIC_REPOSITORY=s3:https://<domain-of-s3-server>/backups/homelab
export AWS_ACCESS_KEY_ID=$ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY
export RESTIC_PASSWORD="RESTIC_SUPER_SECURE_PASSWORD"

restic init

From here, create a backup script at /usr/local/bin/backup.sh. This is where we specify which folders to backup.

Example:

#!/bin/sh

set -o errexit
set -o nounset

if [ "$EUID" -ne "$(id -u restic)" ]
  then echo "Please run as restic"
  exit
fi

# Usage: rbackup [message] [restic-tag] [backup-dir]
rbackup () {
  echo "$1"
  restic backup \
    --tag "$2" \
    "$3"
}


## BACKUP TO CLOUD
export RESTIC_REPOSITORY=s3:https://<domain-of-cloud-server>/backups/homelab
export AWS_ACCESS_KEY_ID="CLOUD_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="CLOUD_SECRET_ACCESS_KEY"
export RESTIC_PASSWORD="RESTIC_SUPER_SECURE_PASSWORD"

rbackup "Backing up documents" \
  Documents \
  /home/user/Documents

# Backup other great directories...

## BACKUP TO BACKBLAZE
export RESTIC_REPOSITORY=s3:https://<domain-of-backblaze-server>/backups/homelab
export AWS_ACCESS_KEY_ID="BACKBLAZE_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="BACKBLAZE_SECRET_ACCESS_KEY"
export RESTIC_PASSWORD="RESTIC_SUPER_SECURE_PASSWORD"

rbackup "Backing up documents" \
  Documents \
  /home/user/Documents

# Backup other great directories...

Unless you have unlimited storage, you probably want to prune old backups according to some schedule. In the same script I have the following function:

prune () {
  echo "Pruning old snapshots"
  restic unlock
  restic forget \
    --group-by "tags" \
    --keep-daily N_DAYS \
    --keep-weekly N_WEEKS \
    --keep-monthly N_MONTHS \
    --prune
}

Replace N_* to your liking. Restic will then ensure that it keeps enough snapshots such that those time-based rules are satisfied.

With the script written, we can then setup a systemd service and timer so that it runs daily. Write the following to /etc/systemd/system/restic-backup.service.

[Unit]
Description=Executes backup script
Requires=network-online.target
Wants=

[Service]
User=restic
Group=restic
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
Environment="HOME=/home/restic"

[Install]
WantedBy=multi-user.target

Write the timer to /etc/systemd/system/restic-backup.timer

[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target

Btrfs snapshots

Both my servers run Btrfs as the underlying filesystem. One cool feature is that Btrfs can create snapshots. These are immutable point-in-time views of a given subvolume. As such, we’ll need to create a subvolume that will contain the directories we want.

If we’re not starting from scratch, and instead want to create a subvolume from an existing folder, then I recommend performing the following steps from this reddit thread:

mv folder folder_backup
btrfs subvolume create folder
cp --archive --one-file-system --reflink=always folder_backup/. folder

From here, we can create our snapshots. Like with our Restic setup, I like having some daily, weekly, and monthly backups available. I’ll store these snapshots in /snapshots, but you’re free to change the location. Here’s what it looks like on my machine:

/snapshots/
  daily
    20251201
      /home/user/Documents
      ...
      /home/user/Music
    ...
    20251227
  monthly
    ...
  weekly
    ...

Since these are backup snapshots, we want it to be readonly. Here is what it looks like to create a daily snapshot for our documents subvolume:

SNAPSHOT_PATH="/snapshots/daily/$(date +'%Y%m%d')/home/user/Documents"
mkdir -p "$(dirname $SNAPSHOT_PATH)"
btrfs subvolume snapshot -r /home/user/Documents "${SNAPSHOT_PATH}"

We used the date in the folder name so that we can easily detect the oldest snapshots for deletion.

OLDEST_SNAPSHOT=$(ls "${SNAPSHOT_DIR}" | sort | head -n 1)
for SUBVOLUME_PATH in "${SUBVOLUMES[@]}"
do
    SNAPSHOT_PATH="${SNAPSHOT_DIR}/${OLDEST_SNAPSHOT}${SUBVOLUME_PATH}"
    btrfs subvolume delete -c "${SNAPSHOT_PATH}"    
done
rm -rf "${SNAPSHOT_DIR}/${OLDEST_SNAPSHOT}"

I put together a script which takes as input the environment variable SNAPSHOT_DIR and handles creating and pruning snapshots. To get these snapshots at different time intervals, we use different systemd timers and change that input variable. Unlike with our restic setup, this will keep the same $N$ number of snapshots for each of our time intervals. We copy this script to /usr/local/bin/localbtrbak.sh.

#!/bin/bash

set -o nounset

show_usage() {
    echo "Usage: localbtrback"
    exit 1
}

# Check argument count
if [ "$#" -ne 0 ]; then
    show_usage
fi

if [ "$EUID" -ne 0 ]
    then echo "Please run as root"
    exit
fi

if [ -z "$SNAPSHOT_DIR" ]
    then echo "SNAPSHOT_DIR not defined"
    exit
fi

# EDIT TO POINT TO YOUR SUBVOLUMES
SUBVOLUMES=("/home/user/Documents"  "/home/user/Music")

for SUBVOLUME_PATH in "${SUBVOLUMES[@]}"
do
    SNAPSHOT_PATH="${SNAPSHOT_DIR}/$(date +'%Y%m%d')${SUBVOLUME_PATH}"

    # Create folder if not already exists
    mkdir -p "$(dirname $SNAPSHOT_PATH)"

    # Create the readonly snapshot
    btrfs subvolume snapshot -r "${SUBVOLUME_PATH}" "${SNAPSHOT_PATH}"
done

# Calculate the number of snapshots
COUNT=$(ls "${SNAPSHOT_DIR}" | wc -l)

if [ "${COUNT}" -gt 3 ]; then
    OLDEST_SNAPSHOT=$(ls "${SNAPSHOT_DIR}" | sort | head -n 1)
    for SUBVOLUME_PATH in "${SUBVOLUMES[@]}"
    do
        SNAPSHOT_PATH="${SNAPSHOT_DIR}/${OLDEST_SNAPSHOT}${SUBVOLUME_PATH}"
        btrfs subvolume delete -c "${SNAPSHOT_PATH}"    
    done
    rm -rf "${SNAPSHOT_DIR}/${OLDEST_SNAPSHOT}"
fi

For our systemd service in /etc/systemd/system/btrlocalback@.service

[Unit]
Description=Create a local BTRFS snapshot

[Service]
Type=oneshot
Environment=SNAPSHOT_DIR="/snapshots/%i"
ExecStart=/usr/local/bin/localbtrbak.sh

[Install]
WantedBy=multi-user.target

An example daily timer stored in /etc/systemd/system/btrlocalbak@daily.timer

[Unit]
Description=Create a daily local BTRFS snapshot
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target

Conclusion

Here is a visualization of the three servers and where the data gets backed up to:

The bucket image in the diagram denotes the S3 storage, while the cylinder database icon denotes the Btrfs snapshots. Pictorially, this also shows us how the 3-2-1 rule is satisfied.

• Three outgoing arrows on the two servers means that we have three copies of the data
• The two icons on each of the servers show that we’re backing it up in two different ways
• I have more than one off-site backup since all three boxes are in different locations.

If I accidentally delete a file, the Btrfs setup is useful since I can quickly access the old version at /snapshots/daily/<yesterday>/path. However, if my entire server goes down, then I can use one of the restic backups in either server to restore.

The Beginning

Over the summer, I lived and worked in Austin, Texas. For the prior months of the year, I’ve lived a fairly sedimentary lifestyle. I enjoy these summers where I am forced to walk more places and take public transit since I don’t have access to a vehicle.

Two of my coworkers were really into running and one of them even regularly ran a few miles outside in the 95+ degree Fahrenheit weather. This didn’t immediately convince me to start myself, but it definitely planted the seed. Towards the end of the summer, I asked myself “What’s stopping me from running?” and took this as an opportunity to prove something to myself.

What do I need to get started?

At the beginning, I didn’t end up buying anything new. In Austin, I wore some sneakers from Keen footwear which I accidentally soaked a few times in sudden downpours. I also wear a Fitbit watch that has heart rate monitoring built in.

When I returned to New York, I ended up buying a Brooks running shoe. Shoes designed for running are usually lighter and have shock absorption. This makes for a more pleasant workout.

Did you use a training plan?

Initially, I used the couch to 5k running plan developed by the UK’s National Health Service (NHS). After completing that plan, I charted out my own path, with most of my sessions consisting of 30 minutes of running surrounded by 5 minutes of walking.

When I returned to New York, I saw that my city was hosting a turkey trot on Thanksgiving. That was a perfect amount of time to follow the program and see how things turn out!

Training Arc

With a race in mind, I had to make sure to not skimp on my training. Initially, I had an every other day schedule. Though as I adjusted back to my new schedule, this has become a little chaotic.

Running Inside vs Outside

The transition between summer and fall in New York state is lovely. It makes for a great time to run outside. However, as the months rolled by it started getting colder and colder. Now that we’re in the midst of winter, I’ve mostly abandoned running outside.

Some people swear by running outdoors versus using a treadmill. I’m fine with either, but there are definitely trade-offs.

With outdoor running there is much more visual stimuli. Instead of staring at a wall, the scenery changes as you move around. This is a huge pro when it comes to staving off boredom; which I find very important for a successful workout. The uneven ground also prevents repetitive strain injuries.

On the other hand, it’s very easy to keep pace with a treadmill. Another benefit is that you don’t have to plan out a route ahead of time. Though, I find it imperative to have some sort of distraction ready for when I do my treadmill runs. Initially I turned to watching YouTube videos, but I noticed that my posture would then suffer. I find podcasts to be a nice compromise.

Run Slowly

I’m far from an expert in this area, but it’s generally recommended to keep your heart rate low when you’re running. This helps improve your cardiovascular fitness over the long run. An informal test is to see if you can carry a conversation while running.

Formally, it’s recommended to stay in “Zone 2” while running which is about 60-70% of your max heart rate. We can use the Bruce protocol to find out our max heart rate, or for a much simpler approximation, we can use the Tanaka, Monahan, & Seals (2001) formula. $$ hr_{max} = 208 - (0.7 * age) $$ That means for a 30 year old their zone 2 heart rate lies between 112 and 130.

In Zone 2, the body can still supply energy using fat reserves and clear up the lactate before it builds up. It’s easy when running to go past this heart rate. The best way I’ve found to keeping it in zone 2 is to run really slow. Often much slower than you think you should be running. When I was first starting out, I wasn’t going much faster than walking pace.

Running with others

A friend recently got me to attend the local run club in the city. At first, I was nervous about joining since I thought it would only be full of people who ran their whole life. It turns out that it had people at all different points in the running journey. I joined once a week until it started regularly staying below freezing. I hope to attend again next year.

Either way, running can be a great excuse to get together with others!

The Turkey Trot

We then arrive at the big day! I showed up around thirty minutes before the event started. To give an idea of the scale, the Troy turkey trot had over 4000 finishers for the 5k.

At the beginning of the race, everyone lined up by flags marking different paces. I timed myself at a nearby track, and my last 5k was 35 minutes. Feeling optimistic, I lined up at the 11 minute flag.

Bam! The sound of the gun then went off signaling the start of the race – except there was no action. Since I was so far back from the start line, it took over 5 minutes before me and the others around me got to shuffling.

Luckily, when it comes to timing there’s the gun time and the net time. The net time is the time it takes once you cross the “start line” to hit the “finish line”.

With the majority of my running sessions featuring me, myself, and I, running with so many other people gave a huge energy boost. I let the energy get to me, and I didn’t end up running a consistent pace at all. I likely ran a little too fast in the beginning, resulting in short walking breaks towards the end. Also I underestimated the energy it would take to weave around other people.

I hit the finish line at 34 minutes and 14 seconds! It’s pretty exciting to hit a new personal best during a race.

Focusing on the race, I didn’t end up taking too many photos. Instead here’s a blurry snapshot from the finish cam and a selfie of me with my participation medal.

Onto the future!

With that milestone completed, now it’s time to think about what’s next. Do I train for a 10k? A half-marathon? Nothing is set in stone yet, and I’m keeping my eye out for future events.

I mentioned that one of my challenges with running is starving off boredom. While the race does inject a new form of energy, I still currently can’t see running for multiple hours!

Beyond running itself, I find it fun to look at my logs. For example, we can take the times from the Troy turkey trot, and plot my performance on the histogram of all participant net times.

Additionally, from my overall training log, I can see that I run on average 2.5 miles during my 30 minute runs. I’m excited to see how these metrics improve over time.

Let’s play Mario Party tonight!

After many years of friendship, I’ve learned that playing online multiplayer games is almost never as simple as it seems. This night was no different. In this post, I’ll go over what I learned setting up my Nintendo Switch for online play. Luckily, this same concept applies to the PlayStation 5 as well¹.

But first, I’ll take a detour into how peer-to-peer (P2P) gaming typically works. So, feel free to skip down to the solution.

Peer-to-Peer Gaming

Mario Party is a board game where you move characters around in hopes of collecting the most stars. You play as a character from the Mario franchise, and between each turn on the board is a minigame. These minigames provide the illusion that this is a skill-based game. But trust me, you can win by just tapping A.

Sony and Nintendo both aren’t forthcoming with information about how their games and systems work. Therefore, I’ll be making a number of assumptions here.

Nintendo owns Mario Party, and from there we can presume that they use the standard Nintendo libraries when building the game. The homebrew community over the years reverse-engineered many of these libraries and due to this we can look at the protocols they use. The Nintendo networking library, called Pia, uses a service called NEX to match players in games. This match-making protocol includes a network-address-translation (nat) traversal protocol.

For sake of simplicity, I won’t describe the protocol in full-depth. What’s important to know is that when both players message the server, the server knows each player’s IP address and the source port that they used to communicate. The server will then share the other console’s information to facilitate that direct peer-to-peer connection

Unfortunately, I was not able to find either official or unofficial documentation on how P2P gaming works on the PlayStation 5. From browsing around, I have the impression that this is less standardized and developers are either relying on external SDKs or developing their own libraries.

The Problem

Opnsense, in all their great wisdom, wants to protect me from TCP hijacking and spoofing attacks. Therefore, by default during nat my source port is randomized.

Now the game consoles will not tell us directly that this is the issue. Instead it will provide a “score” which is supposed to establish a rough sense of how easy it will be to establish a P2P connection.

Nintendo Switch: Settings -> Internet -> Test Connection

PS5: Settings -> Network -> Test Internet Connection

Before applying any changes, on the switch I had a “D” score on on the PS5 it was “Type 3”. I wish I can tell you what these scores mean, but the documentation is lacking to say the least…

However, I was finally able to determine that nat source port randomization was the problem after stumbling upon this Reddit thread – which now takes us to the solution:

The Solution

We need to tell our router that our gaming devices are special, and therefore does not need the additional security measure of randomizing the source port during nat. That way when the game is establishing a P2P connection, the ports aren’t randomized and the connection can proceed smoothly.

On Opnsense, we can change this setting by navigating to Firewall -> NAT -> Outbound.

Now I do want this security measure to be applied to the rest of my devices. Therefore, we’ll be setting the mode to “Hybrid outbound NAT rule generation” which allows us to specify the custom rules.

From there, we can add a manual rule for each game console with the following:

• Interface: WAN
• TCP/IP Version: IPv4
• Protocol: UDP
• Source Address: Single host or network
  • Insert Switch/PS5 address
  • Replace netmask with 32
• Source Port: any
• Destination Address: any
• Destination Port: Any
• Static Port: Checked

The last option is what tells Opnsense to not randomize the source port.

From there, we can save, apply our settings, and rerun our connection tests. With this change, my Switch reports a NAT type of B and PS5 reports Type 2. Good enough for online gaming :)

What is an immutable Linux distribution? It is a Linux distribution with a read-only core. This prevents accidental modifications which overtime lead to an unstable system.

There are many different options for these immutable distributions, but as you can see from the title of this post, I went with Fedora CoreOS. The reason is simple. All my other servers run Fedora Server, so hopefully the changes I need to make to my playbooks are minimal.

At the time of writing, Fedore CoreOS uses OSTree to perform upgrades over the entire filesystem. These upgrades are atomic which suggests that we are not updating individual packages but the entire Linux distribution as a whole. Apart from the read-only core, there are two writable directories /etc and /var. In fact, your home directory lives in /var/home.

Initial Installation

OVHCloud does not have a way to upload an ISO and boot directly from there. Instead, we’ll have to use their rescue mode feature. Luckily, Timothée wrote up a great guide on how to get started. If this doesn’t exactly match your situation, the official documentation has over 20 different provisioning guides.

When I was following the documentation, one of the parts I got tripped up on was how much configuration to put in my Butane file. As noted above, I already have Ansible setup to copy over various files I need. Then I saw that ignition runs only once during the first boot of the system. Hence, if we wanted to customize our storage partitions or lay out networking, then this is a good place to do this. Otherwise, if we want to setup systemd services and the like, we can always add those via Ansible later.

In other words, if you’re fine with the defaults and there’s a DHCP server running on your network, then the base Butane config outlined in the documentation is sufficient.

From the official documentation (write this to a file such as config.bu):

variant: fcos
version: 1.6.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAA...

Where you replace the ssh-rsa line with your own SSH public key file.

Then, in order to get the ignition file, we can run the following command:

podman run --interactive --rm quay.io/coreos/butane:release --pretty --strict < config.bu > config.ign

Running Software

For the most part, using OSTree to install additional packages is highly discouraged. Instead, it’s suggested to install and run things through containers. The official documentation shows how to set up containers via the Butane configuration above, however, I kept my file as simple as shown above. Instead since /etc/containers/systemd is writable, I wrote Podman Quadlet files directly (Official Quadlet Documentation).

For example, here is my Wireguard Quadlet file

[Unit]
Description=WireGuard VPN Container
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/linuxserver/wireguard:latest
ContainerName=wireguard

# Give the container the ability to add network interfaces
AddCapability=NET_ADMIN

# Have the wireguard network accessible on the host
Network=host

# Mount the WireGuard configuration directory
Volume=/etc/wireguard:/config/wg_confs:Z

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

The following are my Ansible tasks which copy that file over to the machine.

- name: Create /etc/wireguard directory
  become: true
  ansible.builtin.file:
    path: /etc/wireguard
    state: directory
    mode: '0700'
    owner: root
    group: root

- name: Copy Quadlet container file
  become: true
  ansible.builtin.copy:
    src: etc/containers/systemd/wireguard.container
    dest: /etc/containers/systemd/wireguard.container
    mode: '0644'
    owner: root
    group: root
  register: wireguardcontainer

- name: Reload systemd daemon to pick up Quadlet
  become: true
  ansible.builtin.systemd:
    daemon_reload: yes
  when: wireguardcontainer.changed

Now not everything deserves a spot in /etc/containers/systemd. Take fastfetch for example.

             .',;::::;,'.                 core@toolbx
         .';:cccccccccccc:;,.             ------------
      .;cccccccccccccccccccccc;.          OS: Fedora Linux 43 (Toolbx Container Image) x86_64
    .:cccccccccccccccccccccccccc:.        Host: OpenStack Nova (19.3.2)
  .;ccccccccccccc;.:dddl:.;ccccccc;.      Kernel: Linux 6.17.1-300.fc43.x86_64
 .:ccccccccccccc;OWMKOOXMWd;ccccccc:.     Uptime: 22 hours, 16 mins
.:ccccccccccccc;KMMc;cc;xMMc;ccccccc:.    Packages: 366 (rpm)
,cccccccccccccc;MMM.;cc;;WW:;cccccccc,    Shell: bash 5.3.0
:cccccccccccccc;MMM.;cccccccccccccccc:    Terminal: conmon
:ccccccc;oxOOOo;MMM000k.;cccccccccccc:    CPU: 6 x Intel Core (Haswell, no TSX) (6) @ 2.99 GHz
cccccc;0MMKxdd:;MMMkddc.;cccccccccccc;    GPU: Cirrus Logic GD 5446
ccccc;XMO';cccc;MMM.;cccccccccccccccc'    Memory: 920.56 MiB / 11.39 GiB (8%)
ccccc;MMo;ccccc;MMW.;ccccccccccccccc;     Swap: Disabled
ccccc;0MNc.ccc.xMMd;ccccccccccccccc;      Disk (/): 11.53 GiB / 99.44 GiB (12%) - overlay
cccccc;dNMWXXXWM0:;cccccccccccccc:,       Disk (/run/host/boot): 277.41 MiB / 349.87 MiB (79%) - ext4 [Read-only]
cccccccc;.:odl:.;cccccccccccccc:,.        Disk (/run/host/etc): 11.53 GiB / 99.44 GiB (12%) - xfs
ccccccccccccccccccccccccccccc:'.          
:ccccccccccccccccccccccc:;,..             
 ':cccccccccccccccc::;,.

All it does it displays system information. Now this is a very important task when showing off your system on Reddit, but it’s more of a system administration tool than a software service that your server provides. For these types of tools, we can use toolbox.

toolbox create # Run this once
toolbox enter

This will give us a new prompt:

⬢ [core@toolbx ~]$

From here, we can use dnf and treat it similarly as a mutable Fedora server machine.

sudo dnf install fastfetch

Now you’ll notice that by default it mounts your home directory but the /etc and /var directories are those of the container. We can find all the files in the host system by accessing /run/host.

I use this when trying to view my Nginx logs:

goaccess /run/host/var/log/nginx/access.log

System Security

A huge benefit to running all your services via Quadlets is that Podman is able to automatically set the SELinux contexts and configure your firewall rules for you. This means that we don’t have to manually change the SELinux context of every file with chcon. If you’re lazy, this means hopefully we don’t have to disable SELinux!

SELinux: Notice in my Wireguard Quadlet file I had the following

Volume=/etc/wireguard:/config/wg_confs:Z

The part after wg_confs: is an optional comma-separated list of options. Here are some that I found to be particularly relevant:

NFTables: My other Fedora server systems use firewalld as the primary way I interface with the firewall. Instead of using a CLI tool, the idea is that we edit /etc/sysconfig/nftables.conf with the rules we want. I’m still getting used writing my firewall config this way, but I do like how it’s all easily viewable in one place.

Here’s a version of what I have:

#!/usr/sbin/nft -f

# Define the main table
table inet filter {
    
    # Data structure we'll use to keep track of rate-limiting
    set ssh_ratelimit {
        type ipv4_addr
        size 65536
        flags dynamic,timeout
        timeout 30s
    }

    chain input {
        type filter hook input priority filter; policy drop;

        # Allow loopback
        iif lo accept

        # Allow established/related connections
        ct state established,related accept

        # Allow ICMP (ping, etc)
        ip protocol icmp accept
        ip6 nexthdr icmpv6 accept

        # SSH with rate limiting
        tcp dport 22 ct state new limit rate over 12/minute burst 6 packets drop
        tcp dport 22 accept

        # Allow HTTP and HTTPS
        tcp dport { 80, 443 } accept

        # Drop everything else
        drop
    }

    chain forward {
        type filter hook forward priority filter; policy drop;
        
        # Allow established/related connections
        ct state established,related accept
    }

    chain output {
    	# Allow outgoing connections
        type filter hook output priority filter; policy accept;
    }
}

I won’t go into detail how NFTables works here. Notice though how we don’t specify anything about the Podman network. As I said before, Podman will automatically handle that for us.

However, what Podman won’t automatically handle is if we try to change our NFTables configuration without a reboot. This is because reloading NFTables will wipe the existing configuration. As such, we need to manually invoke Podman to regenerate the rules.

Luckily, we can override the NFTables systemd service so that it happens automatically. Create the file /etc/systemd/system/nftables.service.d/override.conf with the following:

[Service]
ExecStartPost=podman network reload --all
ExecReload=podman network reload --all

Conclusion

That wraps up the bits and pieces I had to learn while I was setting my machine up. From there, I’ve been running my CoreOS machine for a month and have not yet had any issues. It’s too early for me to make any grand claims, but it feels incredibly reliable.

By default, updates are automatic. Since these are atomic, it means that the machine reboots regularly to apply these updates. This encourages us to setup everything to survive reboots and not require any manual intervention. Also, boot times are quick with it clocking under 12 seconds for my VPS.

Running everything in containers provides isolation between these services and the host. This is useful of course for security, but also allows everything to update on their own cadence and not conflict.

Overall, using an immutable distribution is different than traditional Linux server management. However hopefully with this setup, it incentivizes us to create less fragile systems. A community of maintainers will keep the stable read-only core in a great state, and if things go wrong, we can copy our container volumes to another machine.

So if you haven’t already, I recommend giving Fedora CoreOS a shot. Next, I have to take a look at how it’s like using an immutable distribution on a desktop.

I came across this recently in Lean when working on Lattice proofs over integers with $\infty$ and $-\infty$ In Lean, we can define this “extended integer” (EInt) by using WithTop and WithBot

import Mathlib.Order.Interval.Basic

-- An Integer with a top (∞) and bottom (-∞) element
def EInt : Type := WithBot (WithTop Int)
deriving LinearOrder

@[simp] def EInt.ninf : EInt := (⊥ : WithBot (WithTop Int))
@[simp] def EInt.inf : EInt := (WithBot.some ⊤ : WithBot (WithTop Int))

notation "-∞" => EInt.ninf
notation "∞" => EInt.inf

-- Helper instances so I can later write numbers and have them casted
instance: IntCast EInt where
  intCast n := WithBot.some (WithTop.some n)

instance: OfNat EInt n where
  ofNat := some (some (Int.ofNat n))

Unfortunately, using WithBot and WithTop is just weird. Look at how we would define functions and write proofs using them.

def EIntFun : EInt → ℤ
  | none => 0
  | some ⊤ => 0
  | some (some _) => 0

lemma EIntFunIsZero (e: EInt) : EIntFun e = 0 := by
  cases e
  case none =>
    rfl
  case some e =>
    cases e
    case top =>
      rfl
    case coe e =>
      rfl

What’s more intuitive is to break it up based on whether it’s $-\infty$, $\infty$ or some integer $z$. Luckily, we’re able to define our own way of splitting up cases in Lean.

def EInt.casesOn.{u} {motive : EInt -> Sort u} (a : EInt)
  (ninf : motive -∞)
  (int : ∀ n : Int, motive ↑n)
  (pinf : motive ∞) :
motive a := by
  cases a
  case none =>
    exact ninf
  case some v =>
    cases v
    case top =>
      exact pinf
    case coe n =>
      exact int n

When we’re doing a proof by cases, we’re trying to prove some motive a. What the above definition says, is that if we’re given some EInt a and three proofs regarding the motive of each of the cases, then performing the cases successfully proves the motive.

Notice how the proof now no longer has any nested cases:

lemma EIntFunIsZero2 (e: EInt) : EIntFun e = 0 := by
  cases e using EInt.casesOn
  case ninf =>
    rfl
  case int z =>
    rfl
  case pinf =>
    rfl

In fact, we can even use this trick to simplify our function

def EIntFun2 (e: EInt) : ℤ := by
  cases e using EInt.casesOn with
  | ninf =>
    exact 0
  | int z =>
    exact 0
  | pinf =>
    exact 0

A more complicated example

Personally I find utility in defining how we want our cases to go prior to the proof itself. For example, let’s break up the domain further by considering the sign of our integers.

def EInt.casesOnSigns.{u} {motive : EInt -> Sort u} (a : EInt)
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) :
motive a := by
  cases a
  case none =>
    exact ninf
  case some v =>
    cases v
    case top =>
      exact pinf
    case coe n =>
      by_cases n < 0
      case pos H =>
        exact nint n H
      case neg H =>
        by_cases n = 0
        case pos H2 =>
          exact zero n H2
        case neg H2 =>
          have H3 : n > 0 := by
            have HH : n ≥ 0 := Int.not_lt.mp H
            have HH2 : n ≠ 0 := H2
            have HH3 : 0 ≠ n := Ne.symm HH2
            exact lt_of_le_of_ne HH HH3
          exact pint n H3

The more complicated we make our cases, the more Lean will struggle to establish definitional equality. I personally find it useful to create helper lemmas for each of the cases to later help establish our motive.

Negative Infinity Case

@[simp]
lemma EInt.casesOnSigns.is_ninf.{u} {motive : EInt -> Sort u}
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) : EInt.casesOnSigns (-∞) ninf nint zero pint pinf = ninf := rfl

The above says that if we perform a cases on $-\infty$ then the result will be equivalent to the ninf case.

Negative Integer Case

lemma EInt.casesOnSigns.is_nint.{u} {motive : EInt -> Sort u}
  (z : Int)
  (Hz : z < 0)
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) :
  EInt.casesOnSigns (↑z) ninf nint zero pint pinf = nint z Hz := by
  unfold casesOnSigns
  show (casesOn (↑z) ninf (fun n => if h : n < 0 then nint n h
                                    else if h : n = 0 then zero n h
                                    else pint n (by omega : n > 0)) pinf) = nint z Hz
  change (if h : z < 0 then nint z h
          else if h : z = 0 then zero z h
          else pint z (by omega : z > 0)) = nint z Hz
  rw [dif_pos Hz]

With our usage of inequalities, we have to help guide Lean through this proof. We first unfold the casesOnSigns definition to get the goal shown in the show tactic. From there, we already know that our EInt is the integer z, so we can simplify the cases to our nested if-then-else statement. After that, since we have the hypothesis that our integer z is negative, we can directly get the nint z h case from the consequent of the outer ite.

Zero Case

@[simp]
lemma EInt.casesOnSigns.is_zero.{u}  {motive : EInt -> Sort u}
  (z : Int)
  (Hn : z = 0)
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) : (EInt.casesOnSigns (z) ninf nint zero pint pinf)  = zero z Hn := by
    subst z
    rfl

Once we substitute zero in, Lean can automatically establish definitional equality.

Positive Case

lemma EInt.casesOnSigns.is_pint.{u}  {motive : EInt -> Sort u}
  (z : Int)
  (Hz : z > 0)
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) : (EInt.casesOnSigns (z) ninf nint zero pint pinf)  = pint z Hz := by
  unfold casesOnSigns
  show (casesOn (↑z) ninf (fun n => if h : n < 0 then nint n h
                                    else if h : n = 0 then zero n h
                                    else pint n (by omega : n > 0)) pinf) = pint z Hz
  change (if h : z < 0 then nint z h
          else if h : z = 0 then zero z h
          else pint z (Hz : z > 0)) = pint z Hz
  have HH: ¬(z < 0) := not_lt_of_gt Hz
  rw [dif_neg HH]
  have HH2: ¬(z = 0) := Int.ne_of_gt Hz
  rw [dif_neg HH2]

Similar to the negative case, but we need to do slightly more work to get to the last alternative within the nested if-then-else statement.

Infinity Case

@[simp]
lemma EInt.casesOnSigns.is_pinf.{u} {motive : EInt -> Sort u}
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) : EInt.casesOnSigns (∞) ninf nint zero pint pinf = pinf := rfl

Like before, now that we have our new definition EInt.casesOnSigns, we can create our function which determines whether an EInt is positive cleanly.

def EInt.isPos (e: EInt) : Bool := by
  cases e using EInt.casesOnSigns with
  | ninf => exact false
  | nint _ => exact false
  | zero => exact false
  | pint _ => exact true
  | pinf => exact true

This is much better than if we worked with the original WithTop and WithBot version! Now let’s prove that our EInt is positive iff it is greater than zero. To do this we’ll need one helper lemma.

lemma EInt.coe_lt_coe {a b : Int}:  ((↑a : EInt) < (↑b: EInt)) ↔ a < b := by
  have H1 : ((↑a : EInt) < (↑b: EInt)) → a < b := by
    intro (H: (↑a : EInt) < (↑b: EInt))
    apply WithTop.coe_lt_coe.mp
    exact WithBot.coe_lt_coe.mp H
  have H2 : a < b → ((↑a : EInt) < (↑b: EInt)) := by
    clear H1
    intro (H : a < b)
    apply WithBot.coe_lt_coe.mpr
    exact WithTop.coe_lt_coe.mpr H
  exact Iff.intro H1 H2

The above lemma states that if the integer $a$ is less than the integer $b$, then the EInt version of $a$ is less than the EInt version of $b$ and vice versa. Now for the main proof

Soundness

If our EInt e is positive, then e is greater than zero.

lemma EInt.isPos_sound (e: EInt) : e.isPos = true → e > 0 := by
  intro H
  cases e using EInt.casesOnSigns
  case ninf =>
    contradiction
  case nint n Hn =>
    unfold EInt.isPos at H
    rw [EInt.casesOnSigns.is_nint n Hn] at H
    contradiction
  case zero n Hz =>
    unfold EInt.isPos at H
    rw [EInt.casesOnSigns.is_zero n Hz] at H
    contradiction
  case pint n Hp =>
    apply EInt.coe_lt_coe.mpr Hp
  case pinf =>
    exact Batteries.compareOfLessAndEq_eq_lt.mp rfl

Completeness

If our EInt e is greater than zero, then e is positive.

lemma EInt.isPos_complete (e: EInt) : e > 0 → e.isPos = true := by
  intro H
  cases e using EInt.casesOnSigns
  case ninf =>
    contradiction
  case nint n Hn =>
    have H2 : n > 0 := EInt.coe_lt_coe.mp H
    have H3 : ¬(n > 0) := not_lt_of_gt Hn
    contradiction
  case zero n Hz =>
    have H2 : n > 0 := EInt.coe_lt_coe.mp H
    have H3 : ¬(n > 0) := Eq.not_gt Hz
    contradiction
  case pint n Hp =>
    unfold EInt.isPos
    rw [EInt.casesOnSigns.is_pint n Hp _ _ _ _ _ ]
  case pinf =>
    unfold EInt.isPos
    rw [EInt.casesOnSigns.is_pinf]

JavaScript arrays are exotic objects according to the ECMAScript specification. Therefore, they may lead to unintuitive behavior if we think of these arrays as C-like.

Let’s play around.

Concept 1: JavaScript arrays are not continguous

First, consider the following array:

let x = [0, 1, 2];

As one might expect, x.length is equal to 3. To tell whether or not an index is in an array, we can use the in operator.

3 in x // Evaluates to false

If we try to access the 3rd index, the result will evaluate to undefined.

x[3] // Evaluates to undefined

Now let’s assign an element to the 4th index. Keep in mind that we’re skipping over the 3rd one.

x[4] = 4;

Now when we check our length property, it’ll say that our array is now of size 5.

x.length // Evaluates to 5

However, the 3rd index still does not exist

3 in x // Evaluates to false

Concept 2: Explicit vs Implicit undefined

Recall that x[3] evaluates to undefined. What happens when we set the value explicitly?

x[3] = undefined;
3 in x // Evaluates to true

So there is a difference on whether we have explicitly set an index to undefined! This distinction is not always used. For example, our trusty for-of loop does not care.

x = [0, 1, 2];
x[4] = 4;
for (a of x) {
    console.log(a)
}

Will print out

0
1
2
undefined
4

Concept 3: Indices are actually strings

Given that we have a length property and that we’ve been indexing with numeric keys, it must mean that arrays have numeric indices. Right?

"0" in ["a", "b"] // Evaluates to true

Okay, it looks like there’s some conversion magic that’s happening behind the scenes here. The ECMAScript specification says that an array index must be strictly less than $2^{32}$. So what happens if it is not?

let x = [0];
x[4294967296] = true
x // Evaluates to [ 0, '4294967296': true ]

It looks like it no longer gets treated as an array item, but instead treats it as an arbitrary key-value pair. Why stop there, this must mean that we can store any sort of arbitrary data in our array.

x.name = "Brandon"
x // Evaluates to [ 0, '4294967296': true, name: 'Brandon' ]

Viewing arrays as objects

Now everything starts to make more sense when we think of these arrays as objects.

let x = [0, 1, 2];
x[4] = 4;

Internally, this corresponds to the object:

{
    "0": 0,
    "1": 1,
    "2": 2,
    "4": 4
}

From this object, we can see that the keys are strings and that the 3rd key is not in the object. Now let’s see what happens when we explicitly set x[5] = undefined.

{
  "0": 0,
  "1": 1,
  "2": 2,
  "4": 4,
  "5": undefined
}

The 5th key is now in our object and it’s set to an undefined value. We also get an undefined value when we try to retrieve a value of a key that is not in our object.

The length of our array is the highest “numeric” key within our object (subject to the size limit). When we iterate over our array using for-of, we’re iterating from "0" to our length.

for (a of x) {
    console.log(a);
}

Is the same as:

console.log(x[0]);
console.log(x[1]);
console.log(x[2]);
console.log(x[3]);
console.log(x[4]);
console.log(x[5]);

That’s an exotic object for you.

https://www.stampsforever.com/vote

The stamps that they make are always cool and it's great to see recognition for some of the earlier designs.

function iterate_helper(collection: set<string>, acc: seq<string>): seq<string>
{
    if collection == {} then acc
    else
        var x :| x in collection;
        var newAcc := acc + [x];
        var newCollection := collection - {x};
        iterate_helper(newCollection, newAcc)
}

The issue is that Dafny will complain with the following error message:

to be compilable, the value of a let-such-that expression must be uniquely determined

Dafny functions must be deterministic. This means that no matter how many times we call a function with some specified input, we will always get the same output. Therefore, as the error message suggests, we need to write a condition that uniquely determines x. One way to achieve this is to specify an order as described in Rustan’s paper:

function iterate_helper(collection: set<string>, acc: seq<string>): seq<string>
{
    if collection == {} then acc
    else
        var x :| x in collection && forall y | y in collection :: x <= y;
        var newAcc := acc + [x];
        var newCollection := collection - {x};
        iterate_helper(newCollection, newAcc)
}

Unfortunately, this code will return two errors:

cannot establish the existence of LHS values that satisfy the such-that predicate

to be compilable, the value of a let-such-that expression must be uniquely determined

However, it’s totally possible to define an ordering over strings. We’ll just need to do some work to convince the verifier of this.

Comparing two strings

Since strings in Dafny are a sequence of characters, it turns out that the <= relation checks whether the left side is a prefix of the right side. Therefore, there is no such thing as a unique minimum element, since there are incomparable elements like “a” and “b”.

As such, our first step is to create a less than or equal to (<=) relation that induces a total order. Consider the following function that determines the order based on the left-most character.

function string_le(s1: string, s2: string): bool
    decreases |s1| + |s2|
{
    if |s1| == 0 && |s2| > 0 then
        true
    else if |s1| > 0 && |s2| == 0 then
        false
    else if |s1| == 0 && |s2| == 0 then
        true
    else
        assert(|s1| > 0);
        assert(|s2| > 0);
        var c1 := s1[0];
        var c2 := s2[0];
        if c1 < c2 then
            true
        else if c1 > c2 then
            false
        else
            string_le(s1[1..], s2[1..])
}

Properties of our comparison function

From here we need to prove that our relation induces a total order. For this, we need to show that it is reflexive, total, anti-symmetric, and transitive.

Reflexivity: All strings are less than or equal to themselves

lemma string_le_reflexive()
    ensures forall s :: string_le(s ,s)
{
    forall s ensures string_le(s, s)
    {
        string_le_reflexive_helper(s);
    }
}

lemma string_le_reflexive_helper(s1: string)
    ensures string_le(s1, s1)
{}

Totality: Given two strings, one is less than or equal to the other.

lemma string_le_totality()
    ensures forall s1, s2 :: string_le(s1, s2) || string_le(s2, s1)
{
    forall s1, s2 ensures string_le(s1, s2) || string_le(s2, s1)
    {
        string_le_totality_helper(s1, s2);
    }
}

lemma string_le_totality_helper(s1: string, s2: string)
    ensures string_le(s1, s2) || string_le(s2, s1)
{}

Antisymmetric: If one string is less than or equal to another string and that other string is also less than or equal to the original string then both strings are equivalent.

lemma string_le_antisymmetric()
    ensures forall s1, s2 :: string_le(s1, s2) && string_le(s2, s1) ==> s1 == s2
{
    forall s1, s2 | string_le(s1, s2) && string_le(s2, s1)
    ensures s1 == s2
    {
        string_le_antisymmetric_helper(s1, s2);
    }
}

lemma string_le_antisymmetric_helper(s1: string, s2: string)
    requires string_le(s1, s2)
    requires string_le(s2, s1)
    ensures s1 == s2
{}

Transitive: If one string is less than or equal to another string, and that other string is less than or equal to some third string, then the first string is less than or equal to that third string.

lemma string_le_transitive()
    ensures forall s1, s2, s3 :: string_le(s1, s2) && string_le(s2, s3) ==> string_le(s1, s3)
{
    forall s1, s2, s3 | string_le(s1, s2) && string_le(s2, s3)
    ensures string_le(s1, s3)
    {
        string_le_transitive_helper(s1, s2, s3);
    }
}

lemma string_le_transitive_helper(s1: string, s2: string, s3: string)
    requires string_le(s1, s2)
    requires string_le(s2, s3)
    ensures string_le(s1, s3)
{}

Then, we conviniently package all the properties together:

lemma string_le_properties()
    ensures forall s :: string_le(s, s)
    ensures forall s1, s2 :: string_le(s1, s2) && string_le(s2, s1) ==> s1 == s2
    ensures forall s1, s2, s3 :: string_le(s1, s2) && string_le(s2, s3) ==> string_le(s1, s3)
    ensures forall s1, s2 :: string_le(s1, s2) || string_le(s2, s1)
{
    string_le_reflexive();
    string_le_antisymmetric();
    string_le_transitive();
    string_le_totality();
}

String sets have a minimum

With the total ordering of strings, we can prove that a smallest element exists within a non-empty set s. First, let us invoke our comparison properties lemma, so the verifier has access to those properties:

string_le_properties();

Since our set is non-empty, we can grab an arbitrary element from s.

var x :| x in s;

For this proof, we’ll approach it inductively. First, let’s consider when s == {x}.

1. By construction, every element in s is equal to x.
2. Then by reflexivity, x is smaller than every element in s.

assert forall y :: y in s ==> y == x;
assert forall y :: y in s ==> string_le(x, y);

Now let’s consider the inductive case. The set in this case has more elements than just x. Let’s consider s' the subset of s without the element x.

var s' := s - {x};
assert s' != {};

Since s' is non-empty, we can by induction say that s' has a smallest element.

string_smallest_exists(s');
var x' :| x' in s' && forall y :: y in s' ==> string_le(x', y);

As s' is the subset of s without x, we can assert that x' and x are not the same:

assert x' != x;

From here, we compare both x and x' (which we’re able to do since <= is total)

Case 1: x <= x': By transitivity, x will be less than all the elements of s'. Since s' is s without x, we can safely say that x is less than every element in s.

assert forall y :: y in s' ==> string_le(x, y);
assert forall y :: y in s ==> string_le(x, y);

Case 2: !(x <= x'). From totality, we have that x' <= x. Since we know from the inductive hypothesis that x' is the minimum of s' and s is s' with the element x, we can conclude that x' is the smallest element of s.

assert !string_le(x, x');
assert string_le(x', x);
assert forall y :: y in s ==> string_le(x', y);

With that, we’ve proven that a smallest string exists! Here’s the lemma in its entirety:

lemma string_smallest_exists(s: set<string>)
    requires s != {}
    decreases s
    ensures exists x :: x in s && forall y :: y in s ==> string_le(x, y)
{
    string_le_properties();

    var x :| x in s;

    // Base Case
    if s == {x} {
        assert forall y :: y in s ==> y == x;
        assert forall y :: y in s ==> string_le(x, y);
	
    // Inductive Case
    } else {
        var s' := s - {x};
        assert s' != {};
        string_smallest_exists(s');
        var x' :| x' in s' && forall y :: y in s' ==> string_le(x', y);
        assert x != x';

        if string_le(x, x') {
            assert forall y :: y in s' ==> string_le(x, y);
            assert forall y :: y in s ==> string_le(x, y);
        } else {
            // x' is smaller than x
            assert !string_le(x, x');
            assert string_le(x', x);
            assert forall y :: y in s ==> string_le(x', y);
        }
    }
}

Select the smallest element from a set

Now that we have determined that a minimum exists in a set, we can use these lemmas to establish that we can uniquely determine the element that we select based on the ordering.

function select_string_from_set(collection: set<string>): string
    requires collection != {}
{
    string_le_properties();
    string_smallest_exists(collection);
    var value :| value in collection && forall y | y in collection :: string_le(value, y);
    value
}

Conclusion

Revisiting our iteration example, we can use our new select_string_from_set function to iterate over a set of strings in a pure deterministic function. More specifically, we’ll visit all the elements in the collection in the order defined by our relation.

function iterate_helper(collection: set<string>, acc: seq<string>): seq<string>
    decreases collection
{
    if collection == {} then acc
    else
        var x := select_string_from_set(collection);
        var newAcc := acc + [x];
        var newCollection := collection - {x};
        iterate_helper(newCollection, newAcc)
}

From here you can generalize beyond a set of strings, as long as you’re able to prove the properties of a total order. The full code for our string sets example is below:

function string_le(s1: string, s2: string): bool
    decreases |s1| + |s2|
{
    if |s1| == 0 && |s2| > 0 then
        true
    else if |s1| > 0 && |s2| == 0 then
        false
    else if |s1| == 0 && |s2| == 0 then
        true
    else
        assert(|s1| > 0);
        assert(|s2| > 0);
        var c1 := s1[0];
        var c2 := s2[0];
        if c1 < c2 then
            true
        else if c1 > c2 then
            false
        else
            string_le(s1[1..], s2[1..])
}

lemma string_le_reflexive()
    ensures forall s :: string_le(s ,s)
{
    forall s ensures string_le(s, s)
    {
        string_le_reflexive_helper(s);
    }
}

lemma string_le_reflexive_helper(s1: string)
    ensures string_le(s1, s1)
{}

lemma string_le_totality()
    ensures forall s1, s2 :: string_le(s1, s2) || string_le(s2, s1)
{
    forall s1, s2 ensures string_le(s1, s2) || string_le(s2, s1)
    {
        string_le_totality_helper(s1, s2);
    }
}

lemma string_le_totality_helper(s1: string, s2: string)
    ensures string_le(s1, s2) || string_le(s2, s1)
{}

lemma string_le_antisymmetric()
    ensures forall s1, s2 :: string_le(s1, s2) && string_le(s2, s1) ==> s1 == s2
{
    forall s1, s2 | string_le(s1, s2) && string_le(s2, s1)
    ensures s1 == s2
    {
        string_le_antisymmetric_helper(s1, s2);
    }
}

lemma string_le_antisymmetric_helper(s1: string, s2: string)
    requires string_le(s1, s2)
    requires string_le(s2, s1)
    ensures s1 == s2
{}


lemma string_le_transitive()
    ensures forall s1, s2, s3 :: string_le(s1, s2) && string_le(s2, s3) ==> string_le(s1, s3)
{
    forall s1, s2, s3 | string_le(s1, s2) && string_le(s2, s3)
    ensures string_le(s1, s3)
    {
        string_le_transitive_helper(s1, s2, s3);
    }
}

lemma string_le_transitive_helper(s1: string, s2: string, s3: string)
    requires string_le(s1, s2)
    requires string_le(s2, s3)
    ensures string_le(s1, s3)
{}

lemma string_le_properties()
    ensures forall s :: string_le(s, s)
    ensures forall s1, s2 :: string_le(s1, s2) && string_le(s2, s1) ==> s1 == s2
    ensures forall s1, s2, s3 :: string_le(s1, s2) && string_le(s2, s3) ==> string_le(s1, s3)
    ensures forall s1, s2 :: string_le(s1, s2) || string_le(s2, s1)
{
    string_le_reflexive();
    string_le_antisymmetric();
    string_le_transitive();
    string_le_totality();
}


lemma string_smallest_exists(s: set<string>)
    requires s != {}
    decreases s
    ensures exists x :: x in s && forall y :: y in s ==> string_le(x, y)
{
    string_le_properties();

    var x :| x in s;

    if s == {x} {
        assert forall y :: y in s ==> y == x;
        assert forall y :: y in s ==> string_le(x, y);
    } else {
        // For sets with more than one element, we use induction-like reasoning
        var s' := s - {x};

        assert s' != {};
        string_smallest_exists(s');

        var x' :| x' in s' && forall y :: y in s' ==> string_le(x', y);
        assert x != x';

        if string_le(x, x') {
            assert forall y :: y in s' ==> string_le(x, y);
            assert forall y :: y in s ==> string_le(x, y);
        } else {
            // x' is smaller than x
            assert !string_le(x, x');
            assert string_le(x', x);
            assert forall y :: y in s ==> string_le(x', y);
        }
    }
}

function select_string_from_set(collection: set<string>): string
    requires collection != {}
{
    string_le_properties();
    string_smallest_exists(collection);
    var value :| value in collection && forall y | y in collection :: string_le(value, y);
    value
}

function iterate_helper(collection: set<string>, acc: seq<string>): seq<string>
    decreases collection
{
    if collection == {} then acc
    else
        var x := select_string_from_set(collection);
        var newAcc := acc + [x];
        var newCollection := collection - {x};
        iterate_helper(newCollection, newAcc)
}

Overall, the techinques I’ve seen website owners use aim to make scraping more difficult. Though it’s a balance. The harder we make it for bots to access a website, the more we turn away regular humans as well. Here’s a short and non-exhaustive list of techinques:

1. User Agent Filtering
2. CAPTCHA solving
3. Rate Limiting
4. Proof of work
5. Identification
6. Paywall

User Agent Filtering

When a person/bot requests a page from a website, the HTTP header of the request has a field called User-Agent. This is to denote the type of client that the requester is using. For example, when I visited a website just now, I sent the user agent Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0.

Filtering based on this string is the easiest technique to employ and also has a low chance of impacting regular humans visiting the website. RFC 9309 Robots Exclusion Protocol, more commonly known as robots.txt, is the most common way of implementing this technique.

How it works is that you create a file named robots.txt at the root directory of your website and write a set of rules that different robots should follow. Here’s an example from Google’s search documentation:

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The * here is the Klenne star which means that it can match any string. Before the bot requests a page, the idea is that they first request this robots.txt file, find the rules that match their user agent, and follow it’s instructions.

As you might imagine, not everyone writes scrapers that follow these rules. This depends on how well-written the bot was and how considerate the developer is. An alternative to this approach is to block the request at the web server. For example, here’s how you would do that using nginx

if ($http_user_agent = "Googlebot"){
    return 403;
}

This returns an empty response with the HTTP code 403 Forbidden.

The downside to this approach is that it’s easy to pretend that you have a different user agent. For example on my machine, the user agent set by curl is curl/8.9.1. However, I can use the same user agent as my browser by adding a flag:

curl --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0" https://brandonrozek.com

CAPTCHA Solving

The Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) is a challenge-response approach to dealing with bots. The idea is that the webserver would present some sort of challenge that is supposedly hard for computers to solve but easy for humans. The human responds to the challenge and then is granted access to the website.

In the paper “Recent advances of Captcha security analysis: a short literature review” by Nghia Trong Dinh and Vinh Truong Hoang, they show that for the majority of CAPTCHA systems, bots are successful at solving them over 50% of the time. Specifically, the best bots are able to solve Google’s image-based CAPTCHAs with 70.78% accuracy.

Unfortunately² the success rate of bots are bound to improve over time. Additionally, CAPTCHA systems are annoying to humans. For example, when I use a VPN, I don’t bother with Google search since I don’t want to select pictures of stairs, fire hydrants, or crosswalks 10 times before being granted a search query.

Rate Limiting

Computers are inherently faster than us. In the paper “How many words do we read per minute? A review and meta-analysis of reading rate” by Marc Brysbaert, he writes that the average human adult reads 238 words per minute of non-fiction silently. Thus, it would take a human on average almost 11 hours to read all my prior blog posts (assuming they don’t get tired or distracted). Meanwhile a bot can scrape this site in under a minute.

From this insight, one technique is to limit the number of requests that an IP address can make at any given time. This is formally known as rate limiting.

It sounds simple in concept but can be difficult to implement without impacting user experience. How many requests is a human reasonably allowed to make in a minute? Human traffic is typically bursty, where a page load can request many different files (CSS, JS, media) in a short period of time. How quick can I expect someone to reasonably click around my website? If this isn’t dialed in properly, then rate limiting can cause frustration with your visitors.

I’m also unsure how successful this is against the LLM web scrapers. Nowadays there are bot farms where they each have their own IP address. It’s difficult to determine whether a request is from a human visitor or part of a larger bot collection network.

Proof of work

We talked about how CAPTCHAs are difficult for computers but easy for humans. Proof of work is difficult for both computers and humans. This helps reduce the number of scrapers by making it costly to request resources from the website. By making the web browser solve some proof of work challenge (usually involving hash functions), the request consumes additional CPU cycles and takes additional time.

Similar to rate limiting, how difficult you make the problem has a direct impact on user experience. The more difficult, the longer it’ll take for the web browser to solve it. This will deter more bots, but after a few seconds will also deter human visitors. According to a study performed by Google and SOASTA Research in 2017, if a user has to wait 3 seconds instead of 1 second, then the probability that they bounce (leave the page) increases by 32%.

Recently, open-source projects Anubis and go-away gained popularity for making it easy to implement this technique. It’s popular for git forges like sourcehut’s as scraping those incurs a lot of CPU cycles in traversing git repositories.

Identification

Another tactic is to ask the requester to provide some information that a human would likely have but a bot less so. Examples include email addresses, phone number, government ID, etc. Of course, a bot can supply false information, but as with the other techinques this adds an additional barrier. Watch out for the fake phone numbers.

Paywall

Lastly, you can require users to pay to see the contents of your website. This is popular with news organizations where they ask you to pay for a subscription in order to see content. This ties in well with the previous tactic, because if the user pays for a subscription, then you likely have a lot of identifying information about that user.

Another interesting idea that I haven’t seen widely implemented is requiring some amount of money per interaction. This can be in the form of the Web Monetization API or via cryptocurrency like Bitcoin on the Lightning network. Stacker news is an example of a Reddit-like platform where users need to pay a small fee in order to upvote a post. The idea is to make it cheap for a human to do on a small scale (like 1 cent per up-vote), but expensive for a bot to do at scale.

Conclusion

We’re in a special time period where everyone is fighting to become the top AI company. Long term, I feel that the scraper activity will die down. Similar to how there weren’t as many web search scrapers out there.

In the meantime, these are multiple techniques to consider if your website is suffering under heavy load. As for myself, I don’t currently implement any of these as my website is mostly static and I haven’t noticed my servers being overloaded.

However if you do, I urge you to exercise some caution. For the most part, we share on the web for information to flow freely, and if we’re not careful we may drive people away.

Using public transportation is a great way to explore the neighborhoods around you. Busses often don’t take highways, and instead will take you through areas that you would’ve otherwise skipped. You can find many great restaurants to eat in the Bay Area right next to the Mountain View Caltrain station. A few years ago when Clare and I visited Portland, Maine, we got to explore the thousand islands by a mail boat.

Even if you have a car, it’s worth taking a look at the transit maps to see if there are any hidden gems.