Though that doesn’t mean that there aren’t people passionate about crabs, creating local-first software, etc. Those folks are still there, even if they’re not the mainstream.

This is why I’m such a big proponent of curating your web experience using RSS feeds. It might take some effort to find people on the small/personal/indie web, but it’s soo worth it.

Home cooking to me is a necessity (lest I go broke) as well as a craft. Recently, I’ve been cooking Paella weekly slightly tweaking each time to get to a recipe that I enjoy.

In an age of abundance, restraint becomes the only scarce thing left, which means saying ’no’ is more valuable than ever.

I fully agree! We all excel at different tasks, and with time and focus we can cultivate them. Just because we can do something, doesn’t mean we should. There’s always a trade-off.

The piece was written in the context of building a product in a business, but honestly many of these points apply to our lives in general.

What’s our story? What do we care about? Saying yes to everything muddies the waters. At the same time, however, don’t say no to everything. The most rewarding oppurtunities are not always the most obvious.

It annoys me how much of a grip Facebook still has on being able to digitally participate in your local community.

Same with Instagram. Though I get it – it’s so easy to pull up your phone and post on these platforms.

Recently, I found The 518 which is a newsletter which covers events happening in the capital region of New York. Luckily they surface all the events posted on Instagram for me. Though I do wonder what my life would be like if I acquiesced and signed up for Meta’s platforms.

In this post, I’ll share the steps I took and what I learned along the way. Huge credit to Christian who wrote their own succinct version of this post and helped me troubleshoot via email.

Getting an Address

Unlike the clear web, we don’t register a domain with anyone. An address on Tor is a hash of your public key.

This is why onion URLs are long and unreadable. Take a look at the following onion URL which takes you to the Tor homepage.

http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion

Notice that the address starts with http. Unlike the clearweb, all traffic via Tor is encrypted. Our hidden service will use the public key behind the URL during the protocol exchange.

The primary benefit of these addresses are that they are entirely decentralized (we create our own keys). They are also incredibly difficult to spoof. The downside is that they are not human readable. We can get a little bit closer to that though with vanity keys.

To create a vanity key, we can use mkp224o. After following the installation instructions, we can create our own onion URL starting with some prefix by running the following:

./mkp224o prefix

Given the current algorithms and hardware, we can easily generate keys which have a URL prefix length of up to six characters in minutes. Beyond that, the generation quickly becomes infeasible… However, we want this to be the case. If it was feasible to generate a key-pair for an entire onion URL then anyone can spoof your hidden service.

The script by default will create many folders in the current directory which have the onion URLs as their name and the contents of the keys within. Pick your favorite and copy it over to /var/lib/tor/somehiddenservicename.

While you’re at it, make a backup of these keys because if we lose it then we lose control over the domain.

Installing and Configuring Tor

Now that we have our address. Let’s get Tor installed and set up. In this guide, I’ll show how to do so via Podman Quadlets.

We’ll use the Docker container dockur/tor. Since Tor is a cryptographic software, we need to be extremely careful where we download it from. As of this time of writing if we look at the Dockerfile in this repo, we can see that this is a simple wrapper over Alpine’s tor packages. Currently, I feel that it’s safe to trust the Alpine maintainers.

After selecting the Docker container, we need to write the Quadlet definition file. Here’s how I have /etc/containers/systemd/tor.container configured:

[Container]
ContainerName=tor
HostName=tor
Image=docker.io/dockurr/tor
AutoUpdate=registry
Volume=/etc/tor:/etc/tor:ro,Z
Volume=/var/lib/tor:/var/lib/tor:Z,U

[Service]
Restart=always

[Install]
WantedBy=default.target

Before we run it, we’ll need to create our main Tor configuration file. This lives in /etc/tor/torrc.

SocksPort 0
HiddenServiceDir /var/lib/tor/somehiddenservicename
HiddenServicePort 80 IP_OF_NGINX_CONTAINER:80

Replace IP_OF_NGINX_CONTAINER with the internal IP of the Nginx container or of the host machine if it is running on there. Here’s an explanation of each line:

Since we’re proxying traffic to our application, let’s configure our target next.

Configuring Nginx

My website is a static site served by Nginx. These files are linked together via absolute URLs. This adds some complications because the onion hidden service URL is completely different than my clearnet one.

One solution would be to make my URLs relative. However, I want my website to be as portable as possible. For example, you can run my website in a subfolder. Thus, our solution here is to have an entirely separate copy of the website where the only difference is the internal URLs.

I’ve set this version of my website to live in /var/www/brozekhs.

As such, we have to create an Nginx configuration file solely for our hidden service. We’ll make sure to point the root to the folder which contains the hidden service version of my website.

server {
    listen 80;
    server_name ONION_URL;

	root /var/www/brozekhs;
    index index.html;

	# ...

	location / {
		allow IP_OF_TOR_CONTAINER;
        deny all;
        # ...
	}
}

Replace /var/www/brozekhs, ONION_URL and IP_OF_TOR_CONTAINER with their respective values. We need to configure the allow and deny lines because we don’t want someone to confirm that our machine responds to tor addresses over the clearnet.

After this when running my nginx container I came across the following error:

date time [emerg] 1#1: could not build server_names_hash, you should increase server_names_hash_bucket_size: 64

As stated, the solution is to increase the server name hash bucket size. I doubled it from 64 to 128 in the http block of /etc/nginx/nginx.conf:

server_names_hash_bucket_size 128;

Lastly if you have a clearnet site and you want to advertise the onion URL for users visiting the clearnet URL on the Tor browser, then we can add an Onion-Location header to the Nginx config of our clearnet site.

location / {
	# ...
    add_header Onion-Location ONION_URL$request_uri;
}

Replace ONION_URL with your own and make sure that $request_uri stays at the end of the header value. This will help the Tor browser with the redirection.

Start em up!

With that, we can start our two services:

sudo systemctl start nginx
sudo systemctl start tor

When we look at journalctl -U tor, we should see something like the following:

datetime hostname systemd[1]: Started tor.service.
datetime hostname tor[681220]: datetime [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/
datetime hostname tor[681220]: datetime [notice] Read configuration file "/etc/tor/torrc".
datetime hostname tor[681220]: datetime [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip.
datetime hostname tor[681220]: datetime [notice] Parsing GEOIP IPv6 file /usr/share/tor/geoip6.
datetime hostname tor[681220]: datetime [notice] Bootstrapped 0% (starting): Starting
datetime hostname tor[681220]: datetime [notice] Starting with guard context "default"
datetime hostname tor[681220]: datetime [notice] Bootstrapped 5% (conn): Connecting to a relay
datetime hostname tor[681220]: datetime [notice] Bootstrapped 10% (conn_done): Connected to a relay
datetime hostname tor[681220]: datetime [notice] Bootstrapped 14% (handshake): Handshaking with a relay
datetime hostname tor[681220]: datetime [notice] Bootstrapped 15% (handshake_done): Handshake with a relay done
datetime hostname tor[681220]: datetime [notice] Bootstrapped 45% (requesting_descriptors): Asking for relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 50% (loading_descriptors): Loading relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 55% (loading_descriptors): Loading relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 60% (loading_descriptors): Loading relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 66% (loading_descriptors): Loading relay descriptors
datetime hostname tor[681220]: datetime [notice] Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits
datetime hostname tor[681220]: datetime [notice] Bootstrapped 90% (ap_handshake_done): Handshake finished with a relay to build circuits
datetime hostname tor[681220]: datetime [notice] Bootstrapped 95% (circuit_create): Establishing a Tor circuit
datetime hostname tor[681220]: datetime [notice] Bootstrapped 100% (done): Done

We won’t be able to access the hidden service until the bootstrapping process reaches 100%. Depending on the state of the network, this might take awhile…

I mention the network state since it’s common for me to see this in the logs:

datetime hostname tor[295456]: datetime [warn] Detected possible compression bomb with input size = 17413 and output size = 515472 (compression factor = 29.60)
datetime hostname tor[295456]: datetime [warn] Possible compression bomb; abandoning stream.
datetime hostname tor[295456]: datetime [warn] Detected possible compression bomb with input size = 23289 and output size = 774624 (compression factor = 33.26)
datetime hostname tor[295456]: datetime [warn] Possible compression bomb; abandoning stream.

This compression or zip bomb appears to be an attempt at a distributed denial of service attack. Luckily, the Tor software is smart enough to guard against this.

Every so often my Tor service goes down and I need to restart the daemon for it to come back up. I haven’t gotten around to writing monitoring scripts for my hidden service so if anyone has any tips feel free to get in touch.

In this article, Kunal writes about how Wikipedia is able to flourish in part due to a lack of censorship by the US. This, however, should not be taken for granted. Free speech (like many rights) have to be continously fought for.

If as a society we had better incentives, then we would have a lot more resources to get me to my office. Imagine if there was a van that can pick me up from my apartment and take me directly to my office.

• Text
• Email
• Pen a letter
• Post on a microblog (Mastodon/X/Pixelfed/etc.)
• Write a blog post

But not all of these methods inherently create a conversation or dialogue. When I write a technical blog post, I don’t expect a reply. Similarly, when I toot on Mastodon, I’m fine if no one favorited the post. As such, (micro-)blogging differs greatly from texting and calling someone and is instead much closer to recording a postcast or uploading a video – a one-way transmission of information.

Ploum wrote about how he views the ActivityPub protocol as a conversation, and as such servers should not filter posts based on whether they have a picture. Now I’m not on Pixelfed, so I do not have a stake in this issue. However since I view Pixelfed as a microblogging platform, I tend to see it as more of a one-way transmission of information rather than soliciting a response from my friends. This view would put me in the category of folks that view AcitvityPub as a “content consumption platform.” Though I find that term derogative¹.

Now should Pixelfed filter posts based on whether it contains a picture? I’m not sure. But if I was on Mastodon and I specifically @’d someone on Pixelfed, then I would sure hope that either they received that message or I was shown an error.

So if we’re not having a conversation with these (micro-)blog posts, what are we doing? Some of us are trying to teach, keep a public journal, or share our perspectives. Some of us don’t even want responses². Though I find that many of us do:

Blog posts become invitations that never expire - Dries

When I send a postcard to a loved one, they don’t need to reply. However, I send it because it’s an acknowledgement of our relationship and it’s an invitation to reach out.

Similarly when I write a blog post, by default I’m only transmitting information. However, any of you readers can choose to promote this from a transmission to an exchange. From a communication to a dialogue.

https://meta.wikimedia.org/wiki/Event:Wikipedia_25_Virtual_Celebration

Here's to another 25 years!

Part of it, I think is that watching sports is already a hobby. So why not make a game that engages people in their hobby more.

Another reason I think is that first-person perspective games are rare in general. Even in the shooter genre, we see third-person perspectives.

Funny enough, vehicle simulation games such as Truck Simulator and Flight Simulator are closer to the actual experience. I remember when I first played the former I was frustrated that I couldn’t park an 18-wheeler easily ;D

What this means for us is that we need to be skeptical and take responses with a grain of salt. Though that isn’t a new issue. What’s more interesting to me is how we treat these LLMs similar to how we treat news outlets: trusting that coverage in an unfamiliar area is correct even if we don’t trust it’s reporting in an area that we’re familiar with.

The first step before talking technology is to identify exactly what data we want to backup. In my homelab, I rely on Immich for photo storage, Navidrome for music streaming, databases for my website, and other personal documents. This amounts to a little over 250 GB of data that would be very difficult for me to replace if it was lost.

Not all my data lives on that one server though. I also have a storage VPS which runs Nextcloud and Hedgedoc. Both of those services combined have less than 100 GB of data, but just like the homelab, that data is precious. The storage VPS has a total capacity of 1.5 TB.

The 3-2-1 backup strategy coined by Peter Krogh suggests that we should have three copies of our data, stored on two different media, and with one being off-site.

Both of my servers have enough storage capacity to hold a copy of all of my data. For my third copy, I decided to rely on Backblaze b2. This is a S3 object storage service that charges low rates for what I store and the number of transactions I make.

My homelab sits in my house, the storage VPS sits in Montreal, and Backblaze stores my data in Phoenix. Therefore, I have 3 copies of my data and it’s stored in more than 1 location.

In order to address having at least 2 different storage media, I backup my data using two different approaches: Restic and Btrfs Snapshots. I don’t currently have enough data such that hot online storage is cost-prohibitive.

Restic

Restic is a modern and open-source backup software that both deduplicates and encrypts file contents at the blob level. In combination with a metadata local cache, this makes this Restic snappy and efficient to use.

It supports a variety of backup targets:

• Local filesystem
• SFTP
• REST server
• S3
• and several others, including those implemented by Rclone!

While I could use the variety of backup targets to increase the number of approaches my data is backed up, to me, this does not feel sufficiently different enough to warrent the additional complexity. Therefore, I’ll use one backup target for all three locations.

Since Backblaze b2 only supports the S3 protocol, this means then that my choice has been made for me. However even if it wasn’t forced upon me, I would still pick that option. Designed by Amazon for their storage service, the S3 protocol is built with scale in mind and supports concurrent multipart file uploads. Additionally unlike SFTP, S3 separates its accounts from that of the server. There are ways to restrict SSH clients, however, it is not the default behavior. In S3, by default a user cannot do anything.

Thus, after choosing S3, I need to set up a S3 server on both my storage VPS and my homelab. I landed on MinIO for its ease of setup and longevity, however, any of the other solutions would work as well.

Since, we’re storing copies of the data on all the servers, one idea is to set it up in a cluster configuration. However, since I’m not managing the S3 server hosted by Backblaze, I wouldn’t be able to include that in the cluster. Generally, it is not recommended to run a cluster with only two nodes. If we do, it can lead to what’s called a split brain problem where both nodes are unable to communicate with each other and think that they are the leader node.

To avoid this, we can run both MinIO instances inpendently and have Restic separately send the data to all servers.

MinIO Setup

If you already have a bucket setup and configured with the appropriate permissions, then feel free to skip this section.

The following code examples assume that MinIO is installed and the mc client is available with the root credentials to each server (alias) set. These general steps can also be achieved using the management UI. For simplicity, I’ll show how to do this with respect to one server homelab. However, we’ll need to do this for every MinIO instance.

First, create a bucket which will hold our backup data.

# alias_of_server/bucket_name
mc mb homelab/backups

Afterwards, create a backup user. We’ll need to specify the ACCESS_KEY and SECRET_KEY which corresponds to the username and password respectively. Store these as we’ll later need to use them.

export ACCESS_KEY="my_awesome_homelab_user"
export SECRET_KEY="SUPER_SECURE_SECRET_KEY"
mc admin user add homelab $ACCESSKEY $SECRETKEY

Note: For a bit more security, I like to create a different access key and secret key for the other servers.

By default, our user cannot cannot do anything. Let’s make it so that they have access to our backups bucket we created earlier. To do that, we’ll need to create a policy which allows the user to perform operations on that bucket.

We’ll have to create a JSON file and follow the AWS IAM format.

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Action": [
    "s3:ListBucket",
    "s3:PutObject",
    "s3:DeleteObject",
    "s3:GetObject"
   ],
   "Resource": [
    "arn:aws:s3:::backups/*",
    "arn:aws:s3:::backups"
   ]
  }
 ]
}

Assume that we saved the prior JSON at a location specified by the environmental variable $POLICY_PATH, then we can create the policy through the following:

export POLICY_NAME="backup-policy"
mc admin policy create homelab $POLICY_NAME $POLICY_PATH

After creating the policy, we need to assign it to our backup user.

mc admin policy attach homelab $POLICY_NAME --user $ACCESS_KEY

Nginx and Initializing Restic

With our bucket and user configured, now let’s discuss how we’ll perform our backups. Our bucket will have the following directory structure:

backups/
  homelab/
  vps/

Each folder will contain a restic repository. We separate them out instead of having one giant restic repository, so that we don’t have to worry about multiple servers competing for a lock.

Instead of communicating over HTTP, we want a secure way to access our buckets from outside the server. For this, I use nginx to proxy the traffic over to MinIO. I configure Certbot with LetsEncrypt so that the connection can be secureted with HTTPS/TLS. In addition to the instructions listed on the MinIO page, I also restrict the IPs which are allowed to connect. For example, to only allow traffic from within your Wireguard network (ex subnet: 10.10.10.1/24) then you can put the following within the location block of the nginx config.

allow 10.10.10.1/24;
deny all; // Denies all other traffic

After securing the entryway to our S3 server, we’re ready to use Restic. Before jumping in, you might want to create a dedicated account so that we don’t have to use the root user.

Let’s initialize each server’s respective repository. Since our backups will be encrypted, we need to come up with another password and put it within the RESTIC_PASSWORD environmental variable. Save this password in a safe place, since we will not be able to decrypt the backup without it.

export RESTIC_REPOSITORY=s3:https://<domain-of-s3-server>/backups/homelab
export AWS_ACCESS_KEY_ID=$ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY
export RESTIC_PASSWORD="RESTIC_SUPER_SECURE_PASSWORD"

restic init

From here, create a backup script at /usr/local/bin/backup.sh. This is where we specify which folders to backup.

Example:

#!/bin/sh

set -o errexit
set -o nounset

if [ "$EUID" -ne "$(id -u restic)" ]
  then echo "Please run as restic"
  exit
fi

# Usage: rbackup [message] [restic-tag] [backup-dir]
rbackup () {
  echo "$1"
  restic backup \
    --tag "$2" \
    "$3"
}


## BACKUP TO CLOUD
export RESTIC_REPOSITORY=s3:https://<domain-of-cloud-server>/backups/homelab
export AWS_ACCESS_KEY_ID="CLOUD_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="CLOUD_SECRET_ACCESS_KEY"
export RESTIC_PASSWORD="RESTIC_SUPER_SECURE_PASSWORD"

rbackup "Backing up documents" \
  Documents \
  /home/user/Documents

# Backup other great directories...

## BACKUP TO BACKBLAZE
export RESTIC_REPOSITORY=s3:https://<domain-of-backblaze-server>/backups/homelab
export AWS_ACCESS_KEY_ID="BACKBLAZE_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="BACKBLAZE_SECRET_ACCESS_KEY"
export RESTIC_PASSWORD="RESTIC_SUPER_SECURE_PASSWORD"

rbackup "Backing up documents" \
  Documents \
  /home/user/Documents

# Backup other great directories...

Unless you have unlimited storage, you probably want to prune old backups according to some schedule. In the same script I have the following function:

prune () {
  echo "Pruning old snapshots"
  restic unlock
  restic forget \
    --group-by "tags" \
    --keep-daily N_DAYS \
    --keep-weekly N_WEEKS \
    --keep-monthly N_MONTHS \
    --prune
}

Replace N_* to your liking. Restic will then ensure that it keeps enough snapshots such that those time-based rules are satisfied.

With the script written, we can then setup a systemd service and timer so that it runs daily. Write the following to /etc/systemd/system/restic-backup.service.

[Unit]
Description=Executes backup script
Requires=network-online.target
Wants=

[Service]
User=restic
Group=restic
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
Environment="HOME=/home/restic"

[Install]
WantedBy=multi-user.target

Write the timer to /etc/systemd/system/restic-backup.timer

[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target

Btrfs snapshots

Both my servers run Btrfs as the underlying filesystem. One cool feature is that Btrfs can create snapshots. These are immutable point-in-time views of a given subvolume. As such, we’ll need to create a subvolume that will contain the directories we want.

If we’re not starting from scratch, and instead want to create a subvolume from an existing folder, then I recommend performing the following steps from this reddit thread:

mv folder folder_backup
btrfs subvolume create folder
cp --archive --one-file-system --reflink=always folder_backup/. folder

From here, we can create our snapshots. Like with our Restic setup, I like having some daily, weekly, and monthly backups available. I’ll store these snapshots in /snapshots, but you’re free to change the location. Here’s what it looks like on my machine:

/snapshots/
  daily
    20251201
      /home/user/Documents
      ...
      /home/user/Music
    ...
    20251227
  monthly
    ...
  weekly
    ...

Since these are backup snapshots, we want it to be readonly. Here is what it looks like to create a daily snapshot for our documents subvolume:

SNAPSHOT_PATH="/snapshots/daily/$(date +'%Y%m%d')/home/user/Documents"
mkdir -p "$(dirname $SNAPSHOT_PATH)"
btrfs subvolume snapshot -r /home/user/Documents "${SNAPSHOT_PATH}"

We used the date in the folder name so that we can easily detect the oldest snapshots for deletion.

OLDEST_SNAPSHOT=$(ls "${SNAPSHOT_DIR}" | sort | head -n 1)
for SUBVOLUME_PATH in "${SUBVOLUMES[@]}"
do
    SNAPSHOT_PATH="${SNAPSHOT_DIR}/${OLDEST_SNAPSHOT}${SUBVOLUME_PATH}"
    btrfs subvolume delete -c "${SNAPSHOT_PATH}"    
done
rm -rf "${SNAPSHOT_DIR}/${OLDEST_SNAPSHOT}"

I put together a script which takes as input the environment variable SNAPSHOT_DIR and handles creating and pruning snapshots. To get these snapshots at different time intervals, we use different systemd timers and change that input variable. Unlike with our restic setup, this will keep the same $N$ number of snapshots for each of our time intervals. We copy this script to /usr/local/bin/localbtrbak.sh.

#!/bin/bash

set -o nounset

show_usage() {
    echo "Usage: localbtrback"
    exit 1
}

# Check argument count
if [ "$#" -ne 0 ]; then
    show_usage
fi

if [ "$EUID" -ne 0 ]
    then echo "Please run as root"
    exit
fi

if [ -z "$SNAPSHOT_DIR" ]
    then echo "SNAPSHOT_DIR not defined"
    exit
fi

# EDIT TO POINT TO YOUR SUBVOLUMES
SUBVOLUMES=("/home/user/Documents"  "/home/user/Music")

for SUBVOLUME_PATH in "${SUBVOLUMES[@]}"
do
    SNAPSHOT_PATH="${SNAPSHOT_DIR}/$(date +'%Y%m%d')${SUBVOLUME_PATH}"

    # Create folder if not already exists
    mkdir -p "$(dirname $SNAPSHOT_PATH)"

    # Create the readonly snapshot
    btrfs subvolume snapshot -r "${SUBVOLUME_PATH}" "${SNAPSHOT_PATH}"
done

# Calculate the number of snapshots
COUNT=$(ls "${SNAPSHOT_DIR}" | wc -l)

if [ "${COUNT}" -gt 3 ]; then
    OLDEST_SNAPSHOT=$(ls "${SNAPSHOT_DIR}" | sort | head -n 1)
    for SUBVOLUME_PATH in "${SUBVOLUMES[@]}"
    do
        SNAPSHOT_PATH="${SNAPSHOT_DIR}/${OLDEST_SNAPSHOT}${SUBVOLUME_PATH}"
        btrfs subvolume delete -c "${SNAPSHOT_PATH}"    
    done
    rm -rf "${SNAPSHOT_DIR}/${OLDEST_SNAPSHOT}"
fi

For our systemd service in /etc/systemd/system/btrlocalback@.service

[Unit]
Description=Create a local BTRFS snapshot

[Service]
Type=oneshot
Environment=SNAPSHOT_DIR="/snapshots/%i"
ExecStart=/usr/local/bin/localbtrbak.sh

[Install]
WantedBy=multi-user.target

An example daily timer stored in /etc/systemd/system/btrlocalbak@daily.timer

[Unit]
Description=Create a daily local BTRFS snapshot
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target

Conclusion

Here is a visualization of the three servers and where the data gets backed up to:

The bucket image in the diagram denotes the S3 storage, while the cylinder database icon denotes the Btrfs snapshots. Pictorially, this also shows us how the 3-2-1 rule is satisfied.

• Three outgoing arrows on the two servers means that we have three copies of the data
• The two icons on each of the servers show that we’re backing it up in two different ways
• I have more than one off-site backup since all three boxes are in different locations.

If I accidentally delete a file, the Btrfs setup is useful since I can quickly access the old version at /snapshots/daily/<yesterday>/path. However, if my entire server goes down, then I can use one of the restic backups in either server to restore.

The Beginning

Over the summer, I lived and worked in Austin, Texas. For the prior months of the year, I’ve lived a fairly sedimentary lifestyle. I enjoy these summers where I am forced to walk more places and take public transit since I don’t have access to a vehicle.

Two of my coworkers were really into running and one of them even regularly ran a few miles outside in the 95+ degree Fahrenheit weather. This didn’t immediately convince me to start myself, but it definitely planted the seed. Towards the end of the summer, I asked myself “What’s stopping me from running?” and took this as an opportunity to prove something to myself.

What do I need to get started?

At the beginning, I didn’t end up buying anything new. In Austin, I wore some sneakers from Keen footwear which I accidentally soaked a few times in sudden downpours. I also wear a Fitbit watch that has heart rate monitoring built in.

When I returned to New York, I ended up buying a Brooks running shoe. Shoes designed for running are usually lighter and have shock absorption. This makes for a more pleasant workout.

Did you use a training plan?

Initially, I used the couch to 5k running plan developed by the UK’s National Health Service (NHS). After completing that plan, I charted out my own path, with most of my sessions consisting of 30 minutes of running surrounded by 5 minutes of walking.

When I returned to New York, I saw that my city was hosting a turkey trot on Thanksgiving. That was a perfect amount of time to follow the program and see how things turn out!

Training Arc

With a race in mind, I had to make sure to not skimp on my training. Initially, I had an every other day schedule. Though as I adjusted back to my new schedule, this has become a little chaotic.

Running Inside vs Outside

The transition between summer and fall in New York state is lovely. It makes for a great time to run outside. However, as the months rolled by it started getting colder and colder. Now that we’re in the midst of winter, I’ve mostly abandoned running outside.

Some people swear by running outdoors versus using a treadmill. I’m fine with either, but there are definitely trade-offs.

With outdoor running there is much more visual stimuli. Instead of staring at a wall, the scenery changes as you move around. This is a huge pro when it comes to staving off boredom; which I find very important for a successful workout. The uneven ground also prevents repetitive strain injuries.

On the other hand, it’s very easy to keep pace with a treadmill. Another benefit is that you don’t have to plan out a route ahead of time. Though, I find it imperative to have some sort of distraction ready for when I do my treadmill runs. Initially I turned to watching YouTube videos, but I noticed that my posture would then suffer. I find podcasts to be a nice compromise.

Run Slowly

I’m far from an expert in this area, but it’s generally recommended to keep your heart rate low when you’re running. This helps improve your cardiovascular fitness over the long run. An informal test is to see if you can carry a conversation while running.

Formally, it’s recommended to stay in “Zone 2” while running which is about 60-70% of your max heart rate. We can use the Bruce protocol to find out our max heart rate, or for a much simpler approximation, we can use the Tanaka, Monahan, & Seals (2001) formula. $$ hr_{max} = 208 - (0.7 * age) $$ That means for a 30 year old their zone 2 heart rate lies between 112 and 130.

In Zone 2, the body can still supply energy using fat reserves and clear up the lactate before it builds up. It’s easy when running to go past this heart rate. The best way I’ve found to keeping it in zone 2 is to run really slow. Often much slower than you think you should be running. When I was first starting out, I wasn’t going much faster than walking pace.

Running with others

A friend recently got me to attend the local run club in the city. At first, I was nervous about joining since I thought it would only be full of people who ran their whole life. It turns out that it had people at all different points in the running journey. I joined once a week until it started regularly staying below freezing. I hope to attend again next year.

Either way, running can be a great excuse to get together with others!

The Turkey Trot

We then arrive at the big day! I showed up around thirty minutes before the event started. To give an idea of the scale, the Troy turkey trot had over 4000 finishers for the 5k.

At the beginning of the race, everyone lined up by flags marking different paces. I timed myself at a nearby track, and my last 5k was 35 minutes. Feeling optimistic, I lined up at the 11 minute flag.

Bam! The sound of the gun then went off signaling the start of the race – except there was no action. Since I was so far back from the start line, it took over 5 minutes before me and the others around me got to shuffling.

Luckily, when it comes to timing there’s the gun time and the net time. The net time is the time it takes once you cross the “start line” to hit the “finish line”.

With the majority of my running sessions featuring me, myself, and I, running with so many other people gave a huge energy boost. I let the energy get to me, and I didn’t end up running a consistent pace at all. I likely ran a little too fast in the beginning, resulting in short walking breaks towards the end. Also I underestimated the energy it would take to weave around other people.

I hit the finish line at 34 minutes and 14 seconds! It’s pretty exciting to hit a new personal best during a race.

Focusing on the race, I didn’t end up taking too many photos. Instead here’s a blurry snapshot from the finish cam and a selfie of me with my participation medal.

Onto the future!

With that milestone completed, now it’s time to think about what’s next. Do I train for a 10k? A half-marathon? Nothing is set in stone yet, and I’m keeping my eye out for future events.

I mentioned that one of my challenges with running is starving off boredom. While the race does inject a new form of energy, I still currently can’t see running for multiple hours!

Beyond running itself, I find it fun to look at my logs. For example, we can take the times from the Troy turkey trot, and plot my performance on the histogram of all participant net times.

Additionally, from my overall training log, I can see that I run on average 2.5 miles during my 30 minute runs. I’m excited to see how these metrics improve over time.

Let’s play Mario Party tonight!

After many years of friendship, I’ve learned that playing online multiplayer games is almost never as simple as it seems. This night was no different. In this post, I’ll go over what I learned setting up my Nintendo Switch for online play. Luckily, this same concept applies to the PlayStation 5 as well¹.

But first, I’ll take a detour into how peer-to-peer (P2P) gaming typically works. So, feel free to skip down to the solution.

Peer-to-Peer Gaming

Mario Party is a board game where you move characters around in hopes of collecting the most stars. You play as a character from the Mario franchise, and between each turn on the board is a minigame. These minigames provide the illusion that this is a skill-based game. But trust me, you can win by just tapping A.

Sony and Nintendo both aren’t forthcoming with information about how their games and systems work. Therefore, I’ll be making a number of assumptions here.

Nintendo owns Mario Party, and from there we can presume that they use the standard Nintendo libraries when building the game. The homebrew community over the years reverse-engineered many of these libraries and due to this we can look at the protocols they use. The Nintendo networking library, called Pia, uses a service called NEX to match players in games. This match-making protocol includes a network-address-translation (nat) traversal protocol.

For sake of simplicity, I won’t describe the protocol in full-depth. What’s important to know is that when both players message the server, the server knows each player’s IP address and the source port that they used to communicate. The server will then share the other console’s information to facilitate that direct peer-to-peer connection

Unfortunately, I was not able to find either official or unofficial documentation on how P2P gaming works on the PlayStation 5. From browsing around, I have the impression that this is less standardized and developers are either relying on external SDKs or developing their own libraries.

The Problem

Opnsense, in all their great wisdom, wants to protect me from TCP hijacking and spoofing attacks. Therefore, by default during nat my source port is randomized.

Now the game consoles will not tell us directly that this is the issue. Instead it will provide a “score” which is supposed to establish a rough sense of how easy it will be to establish a P2P connection.

Nintendo Switch: Settings -> Internet -> Test Connection

PS5: Settings -> Network -> Test Internet Connection

Before applying any changes, on the switch I had a “D” score on on the PS5 it was “Type 3”. I wish I can tell you what these scores mean, but the documentation is lacking to say the least…

However, I was finally able to determine that nat source port randomization was the problem after stumbling upon this Reddit thread – which now takes us to the solution:

The Solution

We need to tell our router that our gaming devices are special, and therefore does not need the additional security measure of randomizing the source port during nat. That way when the game is establishing a P2P connection, the ports aren’t randomized and the connection can proceed smoothly.

On Opnsense, we can change this setting by navigating to Firewall -> NAT -> Outbound.

Now I do want this security measure to be applied to the rest of my devices. Therefore, we’ll be setting the mode to “Hybrid outbound NAT rule generation” which allows us to specify the custom rules.

From there, we can add a manual rule for each game console with the following:

• Interface: WAN
• TCP/IP Version: IPv4
• Protocol: UDP
• Source Address: Single host or network
  • Insert Switch/PS5 address
  • Replace netmask with 32
• Source Port: any
• Destination Address: any
• Destination Port: Any
• Static Port: Checked

The last option is what tells Opnsense to not randomize the source port.

From there, we can save, apply our settings, and rerun our connection tests. With this change, my Switch reports a NAT type of B and PS5 reports Type 2. Good enough for online gaming :)

This post goes over why one would never want to first encrypt and then compress, and how the other way around can be dangerous when an adversary can control the payload.

One summer, I had a roommate who was studying for a job in finance. They told me that estimation is a key skill. In other words, trying to get into the “ballpark” of the solution. Maybe they’ll excel in these contests.

Though sadly, sometimes we can’t be open about our preferences. For our sanity’s sake, I hope we’re not in the Abilene Paradox too often.

I hereby raise awareness of Nighthawk’s Solstice, which celebrates the day of the earliest sunset of the winter.

Reading this post got me excited. I don’t have to wait until the winter solstice to get more sun in the evening? I agree that the Nighthawk’s solstice should be discussed way more.

What is an immutable Linux distribution? It is a Linux distribution with a read-only core. This prevents accidental modifications which overtime lead to an unstable system.

There are many different options for these immutable distributions, but as you can see from the title of this post, I went with Fedora CoreOS. The reason is simple. All my other servers run Fedora Server, so hopefully the changes I need to make to my playbooks are minimal.

At the time of writing, Fedore CoreOS uses OSTree to perform upgrades over the entire filesystem. These upgrades are atomic which suggests that we are not updating individual packages but the entire Linux distribution as a whole. Apart from the read-only core, there are two writable directories /etc and /var. In fact, your home directory lives in /var/home.

Initial Installation

OVHCloud does not have a way to upload an ISO and boot directly from there. Instead, we’ll have to use their rescue mode feature. Luckily, Timothée wrote up a great guide on how to get started. If this doesn’t exactly match your situation, the official documentation has over 20 different provisioning guides.

When I was following the documentation, one of the parts I got tripped up on was how much configuration to put in my Butane file. As noted above, I already have Ansible setup to copy over various files I need. Then I saw that ignition runs only once during the first boot of the system. Hence, if we wanted to customize our storage partitions or lay out networking, then this is a good place to do this. Otherwise, if we want to setup systemd services and the like, we can always add those via Ansible later.

In other words, if you’re fine with the defaults and there’s a DHCP server running on your network, then the base Butane config outlined in the documentation is sufficient.

From the official documentation:

variant: fcos
version: 1.6.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAA...

Where you replace the ssh-rsa line with your own SSH public key file.

Running Software

For the most part, using OSTree to install additional packages is highly discouraged. Instead, it’s suggested to install and run things through containers. The official documentation shows how to set up containers via the Butane configuration above, however, I kept my file as simple as shown above. Instead since /etc/containers/systemd is writable, I wrote Podman Quadlet files directly (Official Quadlet Documentation).

For example, here is my Wireguard Quadlet file

[Unit]
Description=WireGuard VPN Container
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/linuxserver/wireguard:latest
ContainerName=wireguard

# Give the container the ability to add network interfaces
AddCapability=NET_ADMIN

# Have the wireguard network accessible on the host
Network=host

# Mount the WireGuard configuration directory
Volume=/etc/wireguard:/config/wg_confs:Z

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

The following are my Ansible tasks which copy that file over to the machine.

- name: Create /etc/wireguard directory
  become: true
  ansible.builtin.file:
    path: /etc/wireguard
    state: directory
    mode: '0700'
    owner: root
    group: root

- name: Copy Quadlet container file
  become: true
  ansible.builtin.copy:
    src: etc/containers/systemd/wireguard.container
    dest: /etc/containers/systemd/wireguard.container
    mode: '0644'
    owner: root
    group: root
  register: wireguardcontainer

- name: Reload systemd daemon to pick up Quadlet
  become: true
  ansible.builtin.systemd:
    daemon_reload: yes
  when: wireguardcontainer.changed

Now not everything deserves a spot in /etc/containers/systemd. Take fastfetch for example.

             .',;::::;,'.                 core@toolbx
         .';:cccccccccccc:;,.             ------------
      .;cccccccccccccccccccccc;.          OS: Fedora Linux 43 (Toolbx Container Image) x86_64
    .:cccccccccccccccccccccccccc:.        Host: OpenStack Nova (19.3.2)
  .;ccccccccccccc;.:dddl:.;ccccccc;.      Kernel: Linux 6.17.1-300.fc43.x86_64
 .:ccccccccccccc;OWMKOOXMWd;ccccccc:.     Uptime: 22 hours, 16 mins
.:ccccccccccccc;KMMc;cc;xMMc;ccccccc:.    Packages: 366 (rpm)
,cccccccccccccc;MMM.;cc;;WW:;cccccccc,    Shell: bash 5.3.0
:cccccccccccccc;MMM.;cccccccccccccccc:    Terminal: conmon
:ccccccc;oxOOOo;MMM000k.;cccccccccccc:    CPU: 6 x Intel Core (Haswell, no TSX) (6) @ 2.99 GHz
cccccc;0MMKxdd:;MMMkddc.;cccccccccccc;    GPU: Cirrus Logic GD 5446
ccccc;XMO';cccc;MMM.;cccccccccccccccc'    Memory: 920.56 MiB / 11.39 GiB (8%)
ccccc;MMo;ccccc;MMW.;ccccccccccccccc;     Swap: Disabled
ccccc;0MNc.ccc.xMMd;ccccccccccccccc;      Disk (/): 11.53 GiB / 99.44 GiB (12%) - overlay
cccccc;dNMWXXXWM0:;cccccccccccccc:,       Disk (/run/host/boot): 277.41 MiB / 349.87 MiB (79%) - ext4 [Read-only]
cccccccc;.:odl:.;cccccccccccccc:,.        Disk (/run/host/etc): 11.53 GiB / 99.44 GiB (12%) - xfs
ccccccccccccccccccccccccccccc:'.          
:ccccccccccccccccccccccc:;,..             
 ':cccccccccccccccc::;,.

All it does it displays system information. Now this is a very important task when showing off your system on Reddit, but it’s more of a system administration tool than a software service that your server provides. For these types of tools, we can use toolbox.

toolbox create # Run this once
toolbox enter

This will give us a new prompt:

⬢ [core@toolbx ~]$

From here, we can use dnf and treat it similarly as a mutable Fedora server machine.

sudo dnf install fastfetch

Now you’ll notice that by default it mounts your home directory but the /etc and /var directories are those of the container. We can find all the files in the host system by accessing /run/host.

I use this when trying to view my Nginx logs:

goaccess /run/host/var/log/nginx/access.log

System Security

A huge benefit to running all your services via Quadlets is that Podman is able to automatically set the SELinux contexts and configure your firewall rules for you. This means that we don’t have to manually change the SELinux context of every file with chcon. If you’re lazy, this means hopefully we don’t have to disable SELinux!

SELinux: Notice in my Wireguard Quadlet file I had the following

Volume=/etc/wireguard:/config/wg_confs:Z

The part after wg_confs: is an optional comma-separated list of options. Here are some that I found to be particularly relevant:

NFTables: My other Fedora server systems use firewalld as the primary way I interface with the firewall. Instead of using a CLI tool, the idea is that we edit /etc/sysconfig/nftables.conf with the rules we want. I’m still getting used writing my firewall config this way, but I do like how it’s all easily viewable in one place.

Here’s a version of what I have:

#!/usr/sbin/nft -f

# Define the main table
table inet filter {
    
    # Data structure we'll use to keep track of rate-limiting
    set ssh_ratelimit {
        type ipv4_addr
        size 65536
        flags dynamic,timeout
        timeout 30s
    }

    chain input {
        type filter hook input priority filter; policy drop;

        # Allow loopback
        iif lo accept

        # Allow established/related connections
        ct state established,related accept

        # Allow ICMP (ping, etc)
        ip protocol icmp accept
        ip6 nexthdr icmpv6 accept

        # SSH with rate limiting
        tcp dport 22 ct state new limit rate over 12/minute burst 6 packets drop
        tcp dport 22 accept

        # Allow HTTP and HTTPS
        tcp dport { 80, 443 } accept

        # Drop everything else
        drop
    }

    chain forward {
        type filter hook forward priority filter; policy drop;
        
        # Allow established/related connections
        ct state established,related accept
    }

    chain output {
    	# Allow outgoing connections
        type filter hook output priority filter; policy accept;
    }
}

I won’t go into detail how NFTables works here. Notice though how we don’t specify anything about the Podman network. As I said before, Podman will automatically handle that for us.

However, what Podman won’t automatically handle is if we try to change our NFTables configuration without a reboot. This is because reloading NFTables will wipe the existing configuration. As such, we need to manually invoke Podman to regenerate the rules.

Luckily, we can override the NFTables systemd service so that it happens automatically. Create the file /etc/systemd/system/nftables.service.d/override.conf with the following:

[Service]
ExecStartPost=podman network reload --all
ExecReload=podman network reload --all

Conclusion

That wraps up the bits and pieces I had to learn while I was setting my machine up. From there, I’ve been running my CoreOS machine for a month and have not yet had any issues. It’s too early for me to make any grand claims, but it feels incredibly reliable.

By default, updates are automatic. Since these are atomic, it means that the machine reboots regularly to apply these updates. This encourages us to setup everything to survive reboots and not require any manual intervention. Also, boot times are quick with it clocking under 12 seconds for my VPS.

Running everything in containers provides isolation between these services and the host. This is useful of course for security, but also allows everything to update on their own cadence and not conflict.

Overall, using an immutable distribution is different than traditional Linux server management. However hopefully with this setup, it incentivizes us to create less fragile systems. A community of maintainers will keep the stable read-only core in a great state, and if things go wrong, we can copy our container volumes to another machine.

So if you haven’t already, I recommend giving Fedora CoreOS a shot. Next, I have to take a look at how it’s like using an immutable distribution on a desktop.

I came across this recently in Lean when working on Lattice proofs over integers with $\infty$ and $-\infty$ In Lean, we can define this “extended integer” (EInt) by using WithTop and WithBot

import Mathlib.Order.Interval.Basic

-- An Integer with a top (∞) and bottom (-∞) element
def EInt : Type := WithBot (WithTop Int)
deriving LinearOrder

@[simp] def EInt.ninf : EInt := (⊥ : WithBot (WithTop Int))
@[simp] def EInt.inf : EInt := (WithBot.some ⊤ : WithBot (WithTop Int))

notation "-∞" => EInt.ninf
notation "∞" => EInt.inf

-- Helper instances so I can later write numbers and have them casted
instance: IntCast EInt where
  intCast n := WithBot.some (WithTop.some n)

instance: OfNat EInt n where
  ofNat := some (some (Int.ofNat n))

Unfortunately, using WithBot and WithTop is just weird. Look at how we would define functions and write proofs using them.

def EIntFun : EInt → ℤ
  | none => 0
  | some ⊤ => 0
  | some (some _) => 0

lemma EIntFunIsZero (e: EInt) : EIntFun e = 0 := by
  cases e
  case none =>
    rfl
  case some e =>
    cases e
    case top =>
      rfl
    case coe e =>
      rfl

What’s more intuitive is to break it up based on whether it’s $-\infty$, $\infty$ or some integer $z$. Luckily, we’re able to define our own way of splitting up cases in Lean.

def EInt.casesOn.{u} {motive : EInt -> Sort u} (a : EInt)
  (ninf : motive -∞)
  (int : ∀ n : Int, motive ↑n)
  (pinf : motive ∞) :
motive a := by
  cases a
  case none =>
    exact ninf
  case some v =>
    cases v
    case top =>
      exact pinf
    case coe n =>
      exact int n

When we’re doing a proof by cases, we’re trying to prove some motive a. What the above definition says, is that if we’re given some EInt a and three proofs regarding the motive of each of the cases, then performing the cases successfully proves the motive.

Notice how the proof now no longer has any nested cases:

lemma EIntFunIsZero2 (e: EInt) : EIntFun e = 0 := by
  cases e using EInt.casesOn
  case ninf =>
    rfl
  case int z =>
    rfl
  case pinf =>
    rfl

In fact, we can even use this trick to simplify our function

def EIntFun2 (e: EInt) : ℤ := by
  cases e using EInt.casesOn with
  | ninf =>
    exact 0
  | int z =>
    exact 0
  | pinf =>
    exact 0

A more complicated example

Personally I find utility in defining how we want our cases to go prior to the proof itself. For example, let’s break up the domain further by considering the sign of our integers.

def EInt.casesOnSigns.{u} {motive : EInt -> Sort u} (a : EInt)
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) :
motive a := by
  cases a
  case none =>
    exact ninf
  case some v =>
    cases v
    case top =>
      exact pinf
    case coe n =>
      by_cases n < 0
      case pos H =>
        exact nint n H
      case neg H =>
        by_cases n = 0
        case pos H2 =>
          exact zero n H2
        case neg H2 =>
          have H3 : n > 0 := by
            have HH : n ≥ 0 := Int.not_lt.mp H
            have HH2 : n ≠ 0 := H2
            have HH3 : 0 ≠ n := Ne.symm HH2
            exact lt_of_le_of_ne HH HH3
          exact pint n H3

The more complicated we make our cases, the more Lean will struggle to establish definitional equality. I personally find it useful to create helper lemmas for each of the cases to later help establish our motive.

Negative Infinity Case

@[simp]
lemma EInt.casesOnSigns.is_ninf.{u} {motive : EInt -> Sort u}
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) : EInt.casesOnSigns (-∞) ninf nint zero pint pinf = ninf := rfl

The above says that if we perform a cases on $-\infty$ then the result will be equivalent to the ninf case.

Negative Integer Case

lemma EInt.casesOnSigns.is_nint.{u} {motive : EInt -> Sort u}
  (z : Int)
  (Hz : z < 0)
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) :
  EInt.casesOnSigns (↑z) ninf nint zero pint pinf = nint z Hz := by
  unfold casesOnSigns
  show (casesOn (↑z) ninf (fun n => if h : n < 0 then nint n h
                                    else if h : n = 0 then zero n h
                                    else pint n (by omega : n > 0)) pinf) = nint z Hz
  change (if h : z < 0 then nint z h
          else if h : z = 0 then zero z h
          else pint z (by omega : z > 0)) = nint z Hz
  rw [dif_pos Hz]

With our usage of inequalities, we have to help guide Lean through this proof. We first unfold the casesOnSigns definition to get the goal shown in the show tactic. From there, we already know that our EInt is the integer z, so we can simplify the cases to our nested if-then-else statement. After that, since we have the hypothesis that our integer z is negative, we can directly get the nint z h case from the consequent of the outer ite.

Zero Case

@[simp]
lemma EInt.casesOnSigns.is_zero.{u}  {motive : EInt -> Sort u}
  (z : Int)
  (Hn : z = 0)
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) : (EInt.casesOnSigns (z) ninf nint zero pint pinf)  = zero z Hn := by
    subst z
    rfl

Once we substitute zero in, Lean can automatically establish definitional equality.

Positive Case

lemma EInt.casesOnSigns.is_pint.{u}  {motive : EInt -> Sort u}
  (z : Int)
  (Hz : z > 0)
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) : (EInt.casesOnSigns (z) ninf nint zero pint pinf)  = pint z Hz := by
  unfold casesOnSigns
  show (casesOn (↑z) ninf (fun n => if h : n < 0 then nint n h
                                    else if h : n = 0 then zero n h
                                    else pint n (by omega : n > 0)) pinf) = pint z Hz
  change (if h : z < 0 then nint z h
          else if h : z = 0 then zero z h
          else pint z (Hz : z > 0)) = pint z Hz
  have HH: ¬(z < 0) := not_lt_of_gt Hz
  rw [dif_neg HH]
  have HH2: ¬(z = 0) := Int.ne_of_gt Hz
  rw [dif_neg HH2]

Similar to the negative case, but we need to do slightly more work to get to the last alternative within the nested if-then-else statement.

Infinity Case

@[simp]
lemma EInt.casesOnSigns.is_pinf.{u} {motive : EInt -> Sort u}
  (ninf : motive -∞)
  (nint : ∀ n : Int, n < 0 → motive ↑n)
  (zero: ∀ n : Int, n = 0 → motive ↑n)
  (pint : ∀ n : Int, n > 0 → motive ↑n)
  (pinf : motive ∞) : EInt.casesOnSigns (∞) ninf nint zero pint pinf = pinf := rfl

Like before, now that we have our new definition EInt.casesOnSigns, we can create our function which determines whether an EInt is positive cleanly.

def EInt.isPos (e: EInt) : Bool := by
  cases e using EInt.casesOnSigns with
  | ninf => exact false
  | nint _ => exact false
  | zero => exact false
  | pint _ => exact true
  | pinf => exact true

This is much better than if we worked with the original WithTop and WithBot version! Now let’s prove that our EInt is positive iff it is greater than zero. To do this we’ll need one helper lemma.

lemma EInt.coe_lt_coe {a b : Int}:  ((↑a : EInt) < (↑b: EInt)) ↔ a < b := by
  have H1 : ((↑a : EInt) < (↑b: EInt)) → a < b := by
    intro (H: (↑a : EInt) < (↑b: EInt))
    apply WithTop.coe_lt_coe.mp
    exact WithBot.coe_lt_coe.mp H
  have H2 : a < b → ((↑a : EInt) < (↑b: EInt)) := by
    clear H1
    intro (H : a < b)
    apply WithBot.coe_lt_coe.mpr
    exact WithTop.coe_lt_coe.mpr H
  exact Iff.intro H1 H2

The above lemma states that if the integer $a$ is less than the integer $b$, then the EInt version of $a$ is less than the EInt version of $b$ and vice versa. Now for the main proof

Soundness

If our EInt e is positive, then e is greater than zero.

lemma EInt.isPos_sound (e: EInt) : e.isPos = true → e > 0 := by
  intro H
  cases e using EInt.casesOnSigns
  case ninf =>
    contradiction
  case nint n Hn =>
    unfold EInt.isPos at H
    rw [EInt.casesOnSigns.is_nint n Hn] at H
    contradiction
  case zero n Hz =>
    unfold EInt.isPos at H
    rw [EInt.casesOnSigns.is_zero n Hz] at H
    contradiction
  case pint n Hp =>
    apply EInt.coe_lt_coe.mpr Hp
  case pinf =>
    exact Batteries.compareOfLessAndEq_eq_lt.mp rfl

Completeness

If our EInt e is greater than zero, then e is positive.

lemma EInt.isPos_complete (e: EInt) : e > 0 → e.isPos = true := by
  intro H
  cases e using EInt.casesOnSigns
  case ninf =>
    contradiction
  case nint n Hn =>
    have H2 : n > 0 := EInt.coe_lt_coe.mp H
    have H3 : ¬(n > 0) := not_lt_of_gt Hn
    contradiction
  case zero n Hz =>
    have H2 : n > 0 := EInt.coe_lt_coe.mp H
    have H3 : ¬(n > 0) := Eq.not_gt Hz
    contradiction
  case pint n Hp =>
    unfold EInt.isPos
    rw [EInt.casesOnSigns.is_pint n Hp _ _ _ _ _ ]
  case pinf =>
    unfold EInt.isPos
    rw [EInt.casesOnSigns.is_pinf]

JavaScript arrays are exotic objects according to the ECMAScript specification. Therefore, they may lead to unintuitive behavior if we think of these arrays as C-like.

Let’s play around.

Concept 1: JavaScript arrays are not continguous

First, consider the following array:

let x = [0, 1, 2];

As one might expect, x.length is equal to 3. To tell whether or not an index is in an array, we can use the in operator.

3 in x // Evaluates to false

If we try to access the 3rd index, the result will evaluate to undefined.

x[3] // Evaluates to undefined

Now let’s assign an element to the 4th index. Keep in mind that we’re skipping over the 3rd one.

x[4] = 4;

Now when we check our length property, it’ll say that our array is now of size 5.

x.length // Evaluates to 5

However, the 3rd index still does not exist

3 in x // Evaluates to false

Concept 2: Explicit vs Implicit undefined

Recall that x[3] evaluates to undefined. What happens when we set the value explicitly?

x[3] = undefined;
3 in x // Evaluates to true

So there is a difference on whether we have explicitly set an index to undefined! This distinction is not always used. For example, our trusty for-of loop does not care.

x = [0, 1, 2];
x[4] = 4;
for (a of x) {
    console.log(a)
}

Will print out

0
1
2
undefined
4

Concept 3: Indices are actually strings

Given that we have a length property and that we’ve been indexing with numeric keys, it must mean that arrays have numeric indices. Right?

"0" in ["a", "b"] // Evaluates to true

Okay, it looks like there’s some conversion magic that’s happening behind the scenes here. The ECMAScript specification says that an array index must be strictly less than $2^{32}$. So what happens if it is not?

let x = [0];
x[4294967296] = true
x // Evaluates to [ 0, '4294967296': true ]

It looks like it no longer gets treated as an array item, but instead treats it as an arbitrary key-value pair. Why stop there, this must mean that we can store any sort of arbitrary data in our array.

x.name = "Brandon"
x // Evaluates to [ 0, '4294967296': true, name: 'Brandon' ]

Viewing arrays as objects

Now everything starts to make more sense when we think of these arrays as objects.

let x = [0, 1, 2];
x[4] = 4;

Internally, this corresponds to the object:

{
    "0": 0,
    "1": 1,
    "2": 2,
    "4": 4
}

From this object, we can see that the keys are strings and that the 3rd key is not in the object. Now let’s see what happens when we explicitly set x[5] = undefined.

{
  "0": 0,
  "1": 1,
  "2": 2,
  "4": 4,
  "5": undefined
}

The 5th key is now in our object and it’s set to an undefined value. We also get an undefined value when we try to retrieve a value of a key that is not in our object.

The length of our array is the highest “numeric” key within our object (subject to the size limit). When we iterate over our array using for-of, we’re iterating from "0" to our length.

for (a of x) {
    console.log(a);
}

Is the same as:

console.log(x[0]);
console.log(x[1]);
console.log(x[2]);
console.log(x[3]);
console.log(x[4]);
console.log(x[5]);

That’s an exotic object for you.

https://www.stampsforever.com/vote

The stamps that they make are always cool and it's great to see recognition for some of the earlier designs.

function iterate_helper(collection: set<string>, acc: seq<string>): seq<string>
{
    if collection == {} then acc
    else
        var x :| x in collection;
        var newAcc := acc + [x];
        var newCollection := collection - {x};
        iterate_helper(newCollection, newAcc)
}

The issue is that Dafny will complain with the following error message:

to be compilable, the value of a let-such-that expression must be uniquely determined

Dafny functions must be deterministic. This means that no matter how many times we call a function with some specified input, we will always get the same output. Therefore, as the error message suggests, we need to write a condition that uniquely determines x. One way to achieve this is to specify an order as described in Rustan’s paper:

function iterate_helper(collection: set<string>, acc: seq<string>): seq<string>
{
    if collection == {} then acc
    else
        var x :| x in collection && forall y | y in collection :: x <= y;
        var newAcc := acc + [x];
        var newCollection := collection - {x};
        iterate_helper(newCollection, newAcc)
}

Unfortunately, this code will return two errors:

cannot establish the existence of LHS values that satisfy the such-that predicate

to be compilable, the value of a let-such-that expression must be uniquely determined

However, it’s totally possible to define an ordering over strings. We’ll just need to do some work to convince the verifier of this.

Comparing two strings

Since strings in Dafny are a sequence of characters, it turns out that the <= relation checks whether the left side is a prefix of the right side. Therefore, there is no such thing as a unique minimum element, since there are incomparable elements like “a” and “b”.

As such, our first step is to create a less than or equal to (<=) relation that induces a total order. Consider the following function that determines the order based on the left-most character.

function string_le(s1: string, s2: string): bool
    decreases |s1| + |s2|
{
    if |s1| == 0 && |s2| > 0 then
        true
    else if |s1| > 0 && |s2| == 0 then
        false
    else if |s1| == 0 && |s2| == 0 then
        true
    else
        assert(|s1| > 0);
        assert(|s2| > 0);
        var c1 := s1[0];
        var c2 := s2[0];
        if c1 < c2 then
            true
        else if c1 > c2 then
            false
        else
            string_le(s1[1..], s2[1..])
}

Properties of our comparison function

From here we need to prove that our relation induces a total order. For this, we need to show that it is reflexive, total, anti-symmetric, and transitive.

Reflexivity: All strings are less than or equal to themselves

lemma string_le_reflexive()
    ensures forall s :: string_le(s ,s)
{
    forall s ensures string_le(s, s)
    {
        string_le_reflexive_helper(s);
    }
}

lemma string_le_reflexive_helper(s1: string)
    ensures string_le(s1, s1)
{}

Totality: Given two strings, one is less than or equal to the other.

lemma string_le_totality()
    ensures forall s1, s2 :: string_le(s1, s2) || string_le(s2, s1)
{
    forall s1, s2 ensures string_le(s1, s2) || string_le(s2, s1)
    {
        string_le_totality_helper(s1, s2);
    }
}

lemma string_le_totality_helper(s1: string, s2: string)
    ensures string_le(s1, s2) || string_le(s2, s1)
{}

Antisymmetric: If one string is less than or equal to another string and that other string is also less than or equal to the original string then both strings are equivalent.

lemma string_le_antisymmetric()
    ensures forall s1, s2 :: string_le(s1, s2) && string_le(s2, s1) ==> s1 == s2
{
    forall s1, s2 | string_le(s1, s2) && string_le(s2, s1)
    ensures s1 == s2
    {
        string_le_antisymmetric_helper(s1, s2);
    }
}

lemma string_le_antisymmetric_helper(s1: string, s2: string)
    requires string_le(s1, s2)
    requires string_le(s2, s1)
    ensures s1 == s2
{}

Transitive: If one string is less than or equal to another string, and that other string is less than or equal to some third string, then the first string is less than or equal to that third string.

lemma string_le_transitive()
    ensures forall s1, s2, s3 :: string_le(s1, s2) && string_le(s2, s3) ==> string_le(s1, s3)
{
    forall s1, s2, s3 | string_le(s1, s2) && string_le(s2, s3)
    ensures string_le(s1, s3)
    {
        string_le_transitive_helper(s1, s2, s3);
    }
}

lemma string_le_transitive_helper(s1: string, s2: string, s3: string)
    requires string_le(s1, s2)
    requires string_le(s2, s3)
    ensures string_le(s1, s3)
{}

Then, we conviniently package all the properties together:

lemma string_le_properties()
    ensures forall s :: string_le(s, s)
    ensures forall s1, s2 :: string_le(s1, s2) && string_le(s2, s1) ==> s1 == s2
    ensures forall s1, s2, s3 :: string_le(s1, s2) && string_le(s2, s3) ==> string_le(s1, s3)
    ensures forall s1, s2 :: string_le(s1, s2) || string_le(s2, s1)
{
    string_le_reflexive();
    string_le_antisymmetric();
    string_le_transitive();
    string_le_totality();
}

String sets have a minimum

With the total ordering of strings, we can prove that a smallest element exists within a non-empty set s. First, let us invoke our comparison properties lemma, so the verifier has access to those properties:

string_le_properties();

Since our set is non-empty, we can grab an arbitrary element from s.

var x :| x in s;

For this proof, we’ll approach it inductively. First, let’s consider when s == {x}.

1. By construction, every element in s is equal to x.
2. Then by reflexivity, x is smaller than every element in s.

assert forall y :: y in s ==> y == x;
assert forall y :: y in s ==> string_le(x, y);

Now let’s consider the inductive case. The set in this case has more elements than just x. Let’s consider s' the subset of s without the element x.

var s' := s - {x};
assert s' != {};

Since s' is non-empty, we can by induction say that s' has a smallest element.

string_smallest_exists(s');
var x' :| x' in s' && forall y :: y in s' ==> string_le(x', y);

As s' is the subset of s without x, we can assert that x' and x are not the same:

assert x' != x;

From here, we compare both x and x' (which we’re able to do since <= is total)

Case 1: x <= x': By transitivity, x will be less than all the elements of s'. Since s' is s without x, we can safely say that x is less than every element in s.

assert forall y :: y in s' ==> string_le(x, y);
assert forall y :: y in s ==> string_le(x, y);

Case 2: !(x <= x'). From totality, we have that x' <= x. Since we know from the inductive hypothesis that x' is the minimum of s' and s is s' with the element x, we can conclude that x' is the smallest element of s.

assert !string_le(x, x');
assert string_le(x', x);
assert forall y :: y in s ==> string_le(x', y);

With that, we’ve proven that a smallest string exists! Here’s the lemma in its entirety:

lemma string_smallest_exists(s: set<string>)
    requires s != {}
    decreases s
    ensures exists x :: x in s && forall y :: y in s ==> string_le(x, y)
{
    string_le_properties();

    var x :| x in s;

    // Base Case
    if s == {x} {
        assert forall y :: y in s ==> y == x;
        assert forall y :: y in s ==> string_le(x, y);
	
    // Inductive Case
    } else {
        var s' := s - {x};
        assert s' != {};
        string_smallest_exists(s');
        var x' :| x' in s' && forall y :: y in s' ==> string_le(x', y);
        assert x != x';

        if string_le(x, x') {
            assert forall y :: y in s' ==> string_le(x, y);
            assert forall y :: y in s ==> string_le(x, y);
        } else {
            // x' is smaller than x
            assert !string_le(x, x');
            assert string_le(x', x);
            assert forall y :: y in s ==> string_le(x', y);
        }
    }
}

Select the smallest element from a set

Now that we have determined that a minimum exists in a set, we can use these lemmas to establish that we can uniquely determine the element that we select based on the ordering.

function select_string_from_set(collection: set<string>): string
    requires collection != {}
{
    string_le_properties();
    string_smallest_exists(collection);
    var value :| value in collection && forall y | y in collection :: string_le(value, y);
    value
}

Conclusion

Revisiting our iteration example, we can use our new select_string_from_set function to iterate over a set of strings in a pure deterministic function. More specifically, we’ll visit all the elements in the collection in the order defined by our relation.

function iterate_helper(collection: set<string>, acc: seq<string>): seq<string>
    decreases collection
{
    if collection == {} then acc
    else
        var x := select_string_from_set(collection);
        var newAcc := acc + [x];
        var newCollection := collection - {x};
        iterate_helper(newCollection, newAcc)
}

From here you can generalize beyond a set of strings, as long as you’re able to prove the properties of a total order. The full code for our string sets example is below:

function string_le(s1: string, s2: string): bool
    decreases |s1| + |s2|
{
    if |s1| == 0 && |s2| > 0 then
        true
    else if |s1| > 0 && |s2| == 0 then
        false
    else if |s1| == 0 && |s2| == 0 then
        true
    else
        assert(|s1| > 0);
        assert(|s2| > 0);
        var c1 := s1[0];
        var c2 := s2[0];
        if c1 < c2 then
            true
        else if c1 > c2 then
            false
        else
            string_le(s1[1..], s2[1..])
}

lemma string_le_reflexive()
    ensures forall s :: string_le(s ,s)
{
    forall s ensures string_le(s, s)
    {
        string_le_reflexive_helper(s);
    }
}

lemma string_le_reflexive_helper(s1: string)
    ensures string_le(s1, s1)
{}

lemma string_le_totality()
    ensures forall s1, s2 :: string_le(s1, s2) || string_le(s2, s1)
{
    forall s1, s2 ensures string_le(s1, s2) || string_le(s2, s1)
    {
        string_le_totality_helper(s1, s2);
    }
}

lemma string_le_totality_helper(s1: string, s2: string)
    ensures string_le(s1, s2) || string_le(s2, s1)
{}

lemma string_le_antisymmetric()
    ensures forall s1, s2 :: string_le(s1, s2) && string_le(s2, s1) ==> s1 == s2
{
    forall s1, s2 | string_le(s1, s2) && string_le(s2, s1)
    ensures s1 == s2
    {
        string_le_antisymmetric_helper(s1, s2);
    }
}

lemma string_le_antisymmetric_helper(s1: string, s2: string)
    requires string_le(s1, s2)
    requires string_le(s2, s1)
    ensures s1 == s2
{}


lemma string_le_transitive()
    ensures forall s1, s2, s3 :: string_le(s1, s2) && string_le(s2, s3) ==> string_le(s1, s3)
{
    forall s1, s2, s3 | string_le(s1, s2) && string_le(s2, s3)
    ensures string_le(s1, s3)
    {
        string_le_transitive_helper(s1, s2, s3);
    }
}

lemma string_le_transitive_helper(s1: string, s2: string, s3: string)
    requires string_le(s1, s2)
    requires string_le(s2, s3)
    ensures string_le(s1, s3)
{}

lemma string_le_properties()
    ensures forall s :: string_le(s, s)
    ensures forall s1, s2 :: string_le(s1, s2) && string_le(s2, s1) ==> s1 == s2
    ensures forall s1, s2, s3 :: string_le(s1, s2) && string_le(s2, s3) ==> string_le(s1, s3)
    ensures forall s1, s2 :: string_le(s1, s2) || string_le(s2, s1)
{
    string_le_reflexive();
    string_le_antisymmetric();
    string_le_transitive();
    string_le_totality();
}


lemma string_smallest_exists(s: set<string>)
    requires s != {}
    decreases s
    ensures exists x :: x in s && forall y :: y in s ==> string_le(x, y)
{
    string_le_properties();

    var x :| x in s;

    if s == {x} {
        assert forall y :: y in s ==> y == x;
        assert forall y :: y in s ==> string_le(x, y);
    } else {
        // For sets with more than one element, we use induction-like reasoning
        var s' := s - {x};

        assert s' != {};
        string_smallest_exists(s');

        var x' :| x' in s' && forall y :: y in s' ==> string_le(x', y);
        assert x != x';

        if string_le(x, x') {
            assert forall y :: y in s' ==> string_le(x, y);
            assert forall y :: y in s ==> string_le(x, y);
        } else {
            // x' is smaller than x
            assert !string_le(x, x');
            assert string_le(x', x);
            assert forall y :: y in s ==> string_le(x', y);
        }
    }
}

function select_string_from_set(collection: set<string>): string
    requires collection != {}
{
    string_le_properties();
    string_smallest_exists(collection);
    var value :| value in collection && forall y | y in collection :: string_le(value, y);
    value
}

function iterate_helper(collection: set<string>, acc: seq<string>): seq<string>
    decreases collection
{
    if collection == {} then acc
    else
        var x := select_string_from_set(collection);
        var newAcc := acc + [x];
        var newCollection := collection - {x};
        iterate_helper(newCollection, newAcc)
}

Overall, the techinques I’ve seen website owners use aim to make scraping more difficult. Though it’s a balance. The harder we make it for bots to access a website, the more we turn away regular humans as well. Here’s a short and non-exhaustive list of techinques:

1. User Agent Filtering
2. CAPTCHA solving
3. Rate Limiting
4. Proof of work
5. Identification
6. Paywall

User Agent Filtering

When a person/bot requests a page from a website, the HTTP header of the request has a field called User-Agent. This is to denote the type of client that the requester is using. For example, when I visited a website just now, I sent the user agent Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0.

Filtering based on this string is the easiest technique to employ and also has a low chance of impacting regular humans visiting the website. RFC 9309 Robots Exclusion Protocol, more commonly known as robots.txt, is the most common way of implementing this technique.

How it works is that you create a file named robots.txt at the root directory of your website and write a set of rules that different robots should follow. Here’s an example from Google’s search documentation:

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The * here is the Klenne star which means that it can match any string. Before the bot requests a page, the idea is that they first request this robots.txt file, find the rules that match their user agent, and follow it’s instructions.

As you might imagine, not everyone writes scrapers that follow these rules. This depends on how well-written the bot was and how considerate the developer is. An alternative to this approach is to block the request at the web server. For example, here’s how you would do that using nginx

if ($http_user_agent = "Googlebot"){
    return 403;
}

This returns an empty response with the HTTP code 403 Forbidden.

The downside to this approach is that it’s easy to pretend that you have a different user agent. For example on my machine, the user agent set by curl is curl/8.9.1. However, I can use the same user agent as my browser by adding a flag:

curl --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:139.0) Gecko/20100101 Firefox/139.0" https://brandonrozek.com

CAPTCHA Solving

The Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) is a challenge-response approach to dealing with bots. The idea is that the webserver would present some sort of challenge that is supposedly hard for computers to solve but easy for humans. The human responds to the challenge and then is granted access to the website.

In the paper “Recent advances of Captcha security analysis: a short literature review” by Nghia Trong Dinh and Vinh Truong Hoang, they show that for the majority of CAPTCHA systems, bots are successful at solving them over 50% of the time. Specifically, the best bots are able to solve Google’s image-based CAPTCHAs with 70.78% accuracy.

Unfortunately² the success rate of bots are bound to improve over time. Additionally, CAPTCHA systems are annoying to humans. For example, when I use a VPN, I don’t bother with Google search since I don’t want to select pictures of stairs, fire hydrants, or crosswalks 10 times before being granted a search query.

Rate Limiting

Computers are inherently faster than us. In the paper “How many words do we read per minute? A review and meta-analysis of reading rate” by Marc Brysbaert, he writes that the average human adult reads 238 words per minute of non-fiction silently. Thus, it would take a human on average almost 11 hours to read all my prior blog posts (assuming they don’t get tired or distracted). Meanwhile a bot can scrape this site in under a minute.

From this insight, one technique is to limit the number of requests that an IP address can make at any given time. This is formally known as rate limiting.

It sounds simple in concept but can be difficult to implement without impacting user experience. How many requests is a human reasonably allowed to make in a minute? Human traffic is typically bursty, where a page load can request many different files (CSS, JS, media) in a short period of time. How quick can I expect someone to reasonably click around my website? If this isn’t dialed in properly, then rate limiting can cause frustration with your visitors.

I’m also unsure how successful this is against the LLM web scrapers. Nowadays there are bot farms where they each have their own IP address. It’s difficult to determine whether a request is from a human visitor or part of a larger bot collection network.

Proof of work

We talked about how CAPTCHAs are difficult for computers but easy for humans. Proof of work is difficult for both computers and humans. This helps reduce the number of scrapers by making it costly to request resources from the website. By making the web browser solve some proof of work challenge (usually involving hash functions), the request consumes additional CPU cycles and takes additional time.

Similar to rate limiting, how difficult you make the problem has a direct impact on user experience. The more difficult, the longer it’ll take for the web browser to solve it. This will deter more bots, but after a few seconds will also deter human visitors. According to a study performed by Google and SOASTA Research in 2017, if a user has to wait 3 seconds instead of 1 second, then the probability that they bounce (leave the page) increases by 32%.

Recently, open-source projects Anubis and go-away gained popularity for making it easy to implement this technique. It’s popular for git forges like sourcehut’s as scraping those incurs a lot of CPU cycles in traversing git repositories.

Identification

Another tactic is to ask the requester to provide some information that a human would likely have but a bot less so. Examples include email addresses, phone number, government ID, etc. Of course, a bot can supply false information, but as with the other techinques this adds an additional barrier. Watch out for the fake phone numbers.

Paywall

Lastly, you can require users to pay to see the contents of your website. This is popular with news organizations where they ask you to pay for a subscription in order to see content. This ties in well with the previous tactic, because if the user pays for a subscription, then you likely have a lot of identifying information about that user.

Another interesting idea that I haven’t seen widely implemented is requiring some amount of money per interaction. This can be in the form of the Web Monetization API or via cryptocurrency like Bitcoin on the Lightning network. Stacker news is an example of a Reddit-like platform where users need to pay a small fee in order to upvote a post. The idea is to make it cheap for a human to do on a small scale (like 1 cent per up-vote), but expensive for a bot to do at scale.

Conclusion

We’re in a special time period where everyone is fighting to become the top AI company. Long term, I feel that the scraper activity will die down. Similar to how there weren’t as many web search scrapers out there.

In the meantime, these are multiple techniques to consider if your website is suffering under heavy load. As for myself, I don’t currently implement any of these as my website is mostly static and I haven’t noticed my servers being overloaded.

However if you do, I urge you to exercise some caution. For the most part, we share on the web for information to flow freely, and if we’re not careful we may drive people away.

Using public transportation is a great way to explore the neighborhoods around you. Busses often don’t take highways, and instead will take you through areas that you would’ve otherwise skipped. You can find many great restaurants to eat in the Bay Area right next to the Mountain View Caltrain station. A few years ago when Clare and I visited Portland, Maine, we got to explore the thousand islands by a mail boat.

Even if you have a car, it’s worth taking a look at the transit maps to see if there are any hidden gems.