For the whole animated version, check out my tilde.club page.

The picture above is a PNG screenshot, but for the card itself I wanted to use SVGs as the primary graphics format to create the heart.

If you were to ask me why, I would claim it’s because SVGs are flexible when it comes to size. Though if I was to be honest, it’s beause SVGs are a plaintext format that I can edit.

Looking around for a tool to create this, I came across a SVG Pixel Art project from Jero Soler.

The editor provides a very familiar experience to Gimp and Photoshop. It includes advance features like layering which I didn’t feel the need to use for this quick drawing.

The available colors leave some to be desired. After downloading the SVG, I was able to edit the fill attribute with the hex values of the tacky colors I wanted to use instead.

I recommend trying the website out yourself next time you want to make some pixel art. For me, it’s less daunting and more familiar to use than Inkscape, the more prevalent svg editing tool.

try:
    for _ in range(5):
        sleep(1)
except KeyboardInterrupt:
    # Awesome task 1
    # Awesome task 2...
    pass

Especially if you end up capturing the same exceptions and handling it the same way.

try:
    for _ in range(5):
        sleep(1)
except KeyboardInterrupt:
    # Awesome task 1
    # Awesome task 2...
    pass

try:
    for _ in range(2):
        sleep(1)
except KeyboardInterrupt:
    # Awesome task 1
    # Awesome task 2...
    pass

One way to make our code more DRY (don’t-repeat-yourself) is to make use of Python’s context managers.

@contextmanager
def handle_sigint():
    try:
        yield
    except KeyboardInterrupt:
        # Awesome task 1
        # Awesome task 2...
        pass

Using the context manager, everything within the indented block gets executed within the try block.

with handle_sigint():
    for _ in range(5):
        sleep(1)

with handle_sigint():
    for _ in range(2):
        sleep(1)

In fact, we can write this in a generic way to give us an alternative syntax for handling exceptions.

@contextmanager
def handle_exception(f, *exceptions):
    try:
        yield
    except exceptions as e:
        f(e)

For example, let’s tell the user that we’re explicitly ignoring their exception

def ignore(e):
    print("Ignoring", e.__class__.__name__)

with handle_exception(ignore, NotImplementedError, KeyboardInterrupt):
    for _ in range(5):
        sleep(1)

Instead of presenting the states in a list, I thought it would be very cool to visually show it on a map. Ideally the map would be encoded in a SVG so that:

• The image will scale regardless of the device size
• I can manually edit the source to show the visited regions

Now imagine my surprise after searching on Kagi to come across a SVG in the public domain on Wikipedia. Even better, the following is shown near the top of the file

/* Individual states can be colored as follows: */
.ks,.mt,.pa {fill:#0000FF}
.ca,.de {fill:#FF0000}
/* In this example, Kansas, Montana and Pennsylvania are colored blue,
and California and Delaware are colored red.
Place this code in the empty space below. */

Such an elegant way of coloring in the states! Now it’s up to me to come up with some categories. As of the time of writing, I decided on:

• Green: Places I’ve lived
• Blue: Places I’ve visited
• Darker Gray: Places I’ve driven through

Here’s how I colored by map:

For the last couple weeks, I’ve been brainstorming new pages for my website and came up with https://brandonrozek.com/visited.

At the time of writing, it only has the colored in states and a list of cities I’ve been to. Though I’m welcome to any suggestions you might have. Feel free to get in touch!

Three games that are featured in the message-of-the-day in Tilde.club are:

• Botany: Plant raising game where players can help water each other’s plants
• Asciifarm: Multiplayer farming/fighting game
• Dungeon Crawl Stone Soup: Rouge-like dungeon adventure

These games are designed to be multiplayer, but it doesn’t need to in order to be a social game. A large example of this recently, is the game Wordle.

Wordle is designed to be played by oneself. A challenge is issued once a day, and the player gets a score based on their performance. The players can then compare the scores with each other and strive to beat each other on future days.

WordGuess is a Wordle inspired game I made designed to run on a pubnix. It relies on the Linux permission system to authenticate players and keeps a leaderboard showing players scores for each of the days.

Below is how it looks like for a player to SSH into the pubnix and play the game:

For those on the Tilde.club server, it’s already setup and ready to play. Run the following command to start:

python /home/brozek/WordGuess/client.py

After playing the game you’re welcome to see the scores of others that day:

python /home/brozek/WordGuess/leaderboard.py

If you’re not on Tilde.club. You’re welcome to set it up yourself on any Linux server or pubnix. The instructions are listed on the Github.

One of my favorite pastimes of late is watching my university’s hockey team. RPI Men’s hockey compete in the ECAC hockey league which is a division 1 league under the NCAA. In other words, hockey is a pretty big deal at the university,

Hockey is a full-contact sport with players going back and forth on the ice at high speeds. You can tell it’s a strenuous sport because the players only stay on the ice for a shift of on average 45 seconds.

Part of the fun is the camaraderie among the attendees. We have a full set of vocal cheers that we scream when penalties are announced, goals are obtained, and other in-game events. We call them vocal cheers, however, the majority of them consists of yelling that the other team sucks.

I’ve attended at least 5 games at this point. I wouldn’t say that I’m an expert at the rules of hockey, but I’ve picked up on the most common calls.

For resets:

• Icing: When a player shoots a puck before the center line and it passes the goal line on the other side. This resets the game with the puck going to the defending zone of the team that committed the infraction.
• Puck out of bounds: When a player unintentionally shoots a puck out of the rink either hitting the audience area or the netting. This resets the game to the closest face off spot from where the puck was hit.

It’s also not hockey without talking about penalties.

• Boarding: Violently pushing a player onto the walls of the rink
• Cross-checking: Hitting an opponent with a stick when both hands are on the stick and no part of the stick is on the ice.
• Hooking: Using the stick to “hook” or slow down an opponent.
• Slashing: Swinging the stick at an opponent.
• Tripping: Causing an opponent to fall.
• Interference: Impeding a player that does not have a puck.

Most of these penalties (unless injury occurs) result in a 2 minute timeout for the player that committed the infraction. The player does not get replaced, resulting in the team that committed the infraction having one less active player. This time period is marked as a power play for the other team.

There are 3 periods in ice hockey with each period lasting 20 minutes. If there are any special performances at RPI, it typically happens after the first period. For example, we’ve had several ice skating performances. After the second period, RPI holds the “chuck the puck” competition and the 50/50 raffle. Zambonis come out after these events to resurface the ice.

Lastly, we cannot forget about the pep band and our good ol’ mascot Puckman. They don’t come out every game but it’s great when they do.

We’ll consider the following message for the rest of the post:

from dataclasses import dataclass

@dataclass
class QueryUserMessage:
    auth_key: str
    username: str

Marshalling

Let’s say we have a message we want to send:

message = QueryUserMessage("lkajdfsas", "brozek")

We first need to get its dictionary representation. Luckily the standard library has us there:

from dataclasses import asdict

message_dict = asdict(message)

Then we can use the json module to give us a string representation

import json

message_str = json.dumps(message_dict)

Finally, we can encode it into bytes and send it away:

# Default encoding is "utf-8"
message_bytes = message_str.encode()
# Assuming connetion is defined...
connection.sendall(message_bytes)

To make this easier for myself, I create a custom json encoder and a function that uses the connection to send off the message

class DataclassEncoder(json.JSONEncoder):
    def default(self, o):
        return asdict(o)
def send_message(connection, message_dataclass):
    contents = json.dumps(message_dataclass, cls=DataclassEncoder).encode()
    connection.sendall(contents)

Un-marshalling

On the other end, let us receive the bytes and decode it into a string:

MESSAGE_BUFFER_LEN = 1024
message_bytes = connection.recv(MESSAGE_BUFFER_LEN)
message_str = message_bytes.decode()

We can use the json module to turn it into a Python dictionary

message_dict = json.loads(message_str)

In this post, we can make use of the fact that we only have one message class. In other cases, you would either want to rely on some protocol or pass in the message type ahead of time. Therefore, we can pass the fields of the dictionary straight to the constructor.

message = QueryUserMessage(**message_dict)

Conclusion

In production use cases, we’ll need to introduce a gambit of error-handling to capture failures in json de-serialization and class instantiation. I hope, however, that this serves as a good starting point.

Some things to consider:

1. If you have multiple types of messages, maybe including a field in the dictionary that is a string which represents the message type. Both applications can then maintain a map between these strings and the class constructors.
2. If it’s possible to have messages larger than the buffer length, then consider either setting it higher or sending the size of the message beforehand.
3. Using a standard HTTP library ;)

For example, consider one user has the following in ~/.cadastre/home.txt:

0 0
####################
#  ________        #
# /~\__\___\       #
# | |  |   | ,,,,  #
#           ,,,,,  #
#    ~~~~~  ,,,,,  #
#    ~~~~~     _   #
#    ~~~~~    (*)  #
# ~deepend     |   #
#                  #
####################

While another user has:

0 1
~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~
~~        ~~~~~~~~~~~~~
~    ~~~      ~~~~~~~~~~
~~~~~~~~~      ~~~    ~
~~~~~~~~~~~         ~~~
~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~

Then the stitched together town will be:

#################### ~~~~~~~~~~~~~~~~~~~~~~~~
#  ________        # ~~~~~~~~~~~~~~~~~~~~~~~
# /~\__\___\       # ~~~~~~~~~~~~~~~~~~~~~~~
# | |  |   | ,,,,  # ~~~~~~~~~~~~~~~~~~~~~~~
#           ,,,,,  # ~~        ~~~~~~~~~~~~~
#    ~~~~~  ,,,,,  # ~    ~~~      ~~~~~~~~~~
#    ~~~~~     _   # ~~~~~~~~~      ~~~    ~
#    ~~~~~    (*)  # ~~~~~~~~~~~         ~~~
# ~deepend     |   # ~~~~~~~~~~~~~~~~~~~~~~~
#                  # ~~~~~~~~~~~~~~~~~~~~~~~
#################### ~~~~~~~~~~~~~~~~~~~~~~~

Ascii art is something that always interested me, but I never took the time to make. Cadastre limits us to 24x12 characters which serves as a good starting canvas for myself.

You might be able to tell, but I’m a big fan of mountains.

+=======================+
.    /\                 .
.   /  \  /\            .
.  /    \/  \/\         .
. /      \ _ \ \        .
.     ____//            .
. ___/____/      @@@    .
. ___/         @@@@@@@  .
.             @@@@@@@@@ .
.              @@@@@@@  .
. ~brozek        | |    .
.........................

Check out my plot among the village over at tilde.club. For a more lively instance check out tilde.town.

Thanks to ~troido for creating this cool piece of software!

from dataclasses import dataclass

@dataclass
class Person:
    name: str
    birth_year: int

The above code is equivalent to:

class A:
    def __init__(name: str, birth_year: int):
        self.name = name
        self.birth_year = birth_year
        self.__post__init__()

Notice the call to __post__init__ at the end. We can override that method to do whatever we’d like. I have found two great use cases for this.

Use Case 1: Derived Fields

Straight from the Python documentation, this use case is for when we want to use some variables to create a new variable.

For example, to compute a new field age from a person’s birth_year:

class Person:
    name: str
    birth_year: int
    age: int = field(init=False)
    
    def __post_init__(self):
        # Assuming the current year is 2024 and their birthday already passed
        self.age = 2024 - self.birth_year

Use Case 2: Validation

Another use case is to make sure that the user instantiates the fields of a data class in a way we expect.

class Person:
    name: str
    birth_year: int
    
    def __post__init__(self):
        assert self.birth_year > 0
        assert isinstance(self.name, str)

Nothing is stopping us from combining both of these use cases within the __post_init__ method!

.........                @@@@@    @@@@@      ...........           
.........               @     @  @     @     ...........           
 .........                 @@@   @     @      ..........           
  ........               @@      @     @       .........           
   ........             @@@@@@@   @@@@@  th     .......            
     ......           -----------------------    ......            
       .....            C  E  N  T  U  R  Y        ....            
         ....         -----------------------       ...            
           ...        @@@@@ @@@@@ @   @ @@@@@        ..            
            ==          @   @      @ @    @          ==            
          __||__        @   @@@@    @     @        __||__          
         |      |       @   @      @ @    @       |      |         
_________|______|_____  @   @@@@@ @   @   @  _____|______|_________

It was a Star Wars inspired animation, brought to you by Telnet!

telnet towel.blinkenlights.nl 23

It’s worth watching the entire thing.

Fast forward a few years, and I’ve come across other great telnet nuggets

Remember that telnet communication is all over plaintext, so don’t accidentally leak any secrets. Have fun discovering the quirky old world of Telnet!

Step 1: Get your token

Go to your preferences. Log in if not already.

Then click on the Watchlist tab. Scroll down to Tokens. Click the Manage Tokens button.

Hold on to the token for the next step.

Step 2: Import the RSS feed to your feed reader.

Direct URL: https://en.wikipedia.org/w/api.php?action=feedwatchlist&allrev&wlowner=USERNAME&wltoken=TOKEN

Replace USERNAME and TOKEN with their respective information.

That’s it! Each new item in the RSS feed will look like the following (feed reader dependent):

Reinforcement learning
December 31, 2023 at 11:28 PM by OAbot

[[Wikipedia:OABOT|Open access bot]]: arxiv updated in citation with #oabot. (OAbot) 

At a glance, you can see:

Line 1: Title of Wikipedia Page Edited

Line 2: Time and Author of Edit (may be username or IP)

Rest: Description of edit.

From here you can check the diffs to make sure that the edit stands.

Resources

https://m.mediawiki.org/wiki/Help:Watchlist https://en.m.wikipedia.org/wiki/Wikipedia%3aSyndication

First if you don’t already have it, install entr.

sudo dnf install entr

This terminal application works by monitoring files passed within standard in. Therefore, to run a $command if any file under some $directory changes run the following:

find $directory -type f | entr $command

An example can be running a build script if anything under the source folder changes:

find src -type f | entr ./build.sh

Now this doesn’t capture files that get added to the directory. To do this, we need to get entr to monitor the parent folder for any changes. We can do this with the -d flag.

From the man page:

-d Track the directories of regular files provided as input and exit if a new file is added.

Since entr exits when a file gets added with this flag, the common solution on the internet is to wrap it in a while loop.

$ while sleep 0.1 ; do find $directory -type f | entr -d $command ; done

This solution works great as a one-liner, but it doesn’t let me CTRL-C out when I’m finished. Therefore, I wrote a shell script that incorporates this solution while also adding a trap.

#!/bin/bash

set -o nounset
set -o pipefail

show_usage() {
    echo "Usage: entr-dir [dir] [command]"
    exit 1
}

# Check argument count
if [ "$#" -lt 2 ]; then
    show_usage
fi

# Make sure that the command entr exists
if ! command -v entr > /dev/null ; then
    echo "entr not found. Exiting..."
    exit 1
fi

DIR="$1"; shift

if [[ ! -d "$DIR" ]]; then
    echo "First argument must be a directory. Exiting..."
    exit 1
fi

# Allow for CTRL-C to exit script
trap "exit 0;" SIGINT

while sleep 0.1; do
    find "$DIR" -type f | entr -d "$@"
done

I plopped this in ~/.local/bin/entr-dir, gave it the +x permission, and now I can easily monitor and build projects when files get changed/added/deleted.

entr-dir src ./build.sh

SIGINT

If we have a while loop within a bash script

while sleep 0.1; do
  do_something
done

We probably want to be able to CTRL-C to exit the whole script.

finish () {
  exit 0
}
trap finish SIGINT

while sleep 0.1; do
  do_something
done

ERR

If we encounter an error in a bash script, one thing we might want to do is dump the environment information and other things that can be useful for debugging.

set -o errexit
set -o nounset
set -o pipefail

print_debug () {
  declare exit_code=$?
  env >&2
  other_debug_info >&2
  exit "$exit_code"
}

trap print_debug ERR

command_that_may_fail

EXIT

Like the last one, this function will occur if the script fails. However, this also occurs if the script is successful. It’s similar to the finally clause within a try-catch programming paradigm.

This is useful for cleaning up build files, de-provisioning services, etc.

cleanup () {
  rm a.tmp b.tmp
  deprovision
}

trap cleanup EXIT

long_compile_may_fail

Other Resources

http://redsymbol.net/articles/bash-exit-traps/

https://jakski.github.io/pages/error-handling-in-bash.html

Welcome to nginx!

After installing nginx fresh on a server, this classic quote is part of a welcome page that is automatically setup within a default configuration file. On Redhat-based systems this lives under /etc/nginx/conf.d and for Ubuntu-based ones under /etc/nginx/sites-available/.

This page mostly exists to make sure that everything is squared away after installation. However, how can we get rid of the page once we have all the site configurations loaded?

We’ll show how to get rid of the default welcome page by changing the default.conf file.

When you type in https://brandonrozek.com in your web browser, your system will first (1) look up the IP (at the time of writing it’s 173.255.230.230) and then (2) send a HTTP request with the header including Host('brandonrozek.com'). It’s the Host header that lets nginx know which content to serve.

So if nginx comes across a Host header that doesn’t have a specified configuration for, then it follows some default behavior. This is described under the configuration file with the host default_server.

One good default behavior is to not bother responding to the request. For this purpose, the best HTTP response code is 444 No Response.

The nginx configuration on the HTTP side looks like the following:

server {
    listen 80 default_server;
    listen [::]:80;

    server_name _;
    return 444;
}

Under standard HTTPS, a SSL handshake occurs. Though this seems pointless if all we’re doing is returning nothing. Therefore, we can reject the handshake as well.

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    ssl_reject_handshake on;

    server_name _;
    return 444;
}

All together, a modified default.conf looks like:

server {
    listen 80 default_server;
    listen [::]:80;

    server_name _;
    return 444;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    ssl_reject_handshake on;

    server_name _;
    return 444;
}

I run btrfs as my primary file system on that server. The file system supports multiple raid levels: RAID 0, RAID 1, RAID 10, RAID 5, and RAID 6. Wikipedia has two great articles describing the standard RAID levels and nested RAID levels.

Since at the time I only had 3 drives, it limited me to RAID 0, 1, and 5. There are still some complications with RAID 5 in btrfs, so it leaves me with two options:

• RAID 0: No data redundancy. Data is split evenly across all disks
• RAID 1: Mirroring. Data is present across all disks.

Now the data isn’t vital to me. I have backups elsewhere. However, I would prefer not to deal with the process of restoring terrabytes of data from a backup when a hard drive fails. So at the time, I used RAID 1.

For Christmas, I received another 3.5 inch hard drive for my server. This now unlocks the two other RAID levels 6 and 10. RAID 6 suffers from similar issues as RAID 5. What about RAID 10?

RAID 10 is a nested RAID level because it has both a RAID 1 and RAID 0 setup with it. Wikipedia has a great diagram which I’ll show here describing how it’s laid out.

For four hard drives, we split them into pairs of two. Each pair is configured as RAID 1. Meaning that for a pair (A, B), all the data in A is also in B. Then all of the pairs (A, B), (C, D) get combined into a RAID 0 configuration. This means that data from one pair (A, B) will not appear in another pair (C, D).

RAID 10, just like RAID 1 protects against a single drive failure. For RAID 1, this is because the data is stored on all the other hard drives. For RAID 10, this is because the data is on the hard drives corresponding pair.

RAID 10, however, may or may not be safe if two hard drives fail. If you’re lucky, only a single hard drive from each of the pairs fail. However, if both A and B fail and they’re in a (A, B) pair, the data is lost.

The benefit? You get to utilize more of the storage space compared to a RAID 1 configuration. In RAID 1, you can only use up to the smallest disk size you have. This is because the data needs to exist on all the hard drives. For RAID 10, it’s dependent on how the drives are paired up. If all the hard drives are the same size, however, then you can use double the space than a corresponding RAID 1 configuration.

Hard drives don’t fail every day, in fact it’s common for hard drives to last beyond 3-5 years. Therefore it makes sense to me to sacrifice the ability for all-but-one hard drives to fail for more storage capacity. Especially since I regularly obtain hard drives many months apart, to avoid the bathtub curve.

The steps for adding the hard drive and converting the raid level weren’t too involved.

To add a drive:

sudo btrfs device add /dev/sdX MOUNTPOINT

Replacing sdX with the appropriate hard drive reference.

To convert to RAID 10:

sudo btrfs balance start -dconvert=raid10 -mconvert=raid10 MOUNTPOINT

Depending on the amount of data you have, it can take many hours (sometimes days) to balance a btrfs file system. Therefore, it might make sense to run the above command in a tmux session.

Having keys automatically expire is annoying. Though, it does give me a chance to reflect if these keys are still useful to me. Currently I use GPG keys for:

• Code signing
• Receiving encrypted messages
• Decentralized Identity

So to me, having a GPG key is still worth it. Now to go about renewing my keys. This post will show how I go about the renewing process itself and what services I update. Mostly for me in the future.

Renewing my GPG key

First, find your key

gpg --list-keys

/home/brandon/.gnupg/pubring.kbx
------------------------------
pub   ed25519 2022-12-14 [SC] [expires: 2023-12-14]
      5F37830BFA46FF7881F47AC78DF79C3DC5FC658A
uid           [ultimate] Brandon Rozek <brozek@brandonrozek.com>
uid           [ultimate] Brandon Rozek <hello@brandonrozek.com>
sub   cv25519 2022-12-14 [E] [expires: 2023-12-14]
sub   dsa2048 2022-12-17 [S] [expires: 2023-12-14]

The fingerprint is the line below pub and for me starts with 5F37. Let’s store that in a variable for easy reference later.

export FPR=5F37830BFA46FF7881F47AC78DF79C3DC5FC658A

If we want to extend the expiration date to a year from today, we can use the following command:

gpg --quick-set-expire $FPR 1y

Alternatively, you can specify an exact date with the ISO format YYYY-MM-DD or keep it relative with respect to days d, weeks w, and months m.

When we check the key again, we should see an updated expiration date

gpg --list-keys

/home/brandon/.gnupg/pubring.kbx
------------------------------
pub   ed25519 2022-12-14 [SC] [expires: 2024-12-28]
      5F37830BFA46FF7881F47AC78DF79C3DC5FC658A
uid           [ultimate] Brandon Rozek <brozek@brandonrozek.com>
uid           [ultimate] Brandon Rozek <hello@brandonrozek.com>
sub   cv25519 2022-12-14 [E] [expires: 2023-12-14]
sub   dsa2048 2022-12-17 [S] [expires: 2023-12-14]

Notice that the two subkeys still have the old expiration date. We’ll need to update that as well. We’ll need to get their fingergrints with the following command

gpg --list-keys --verbose --with-subkey-fingerprints

gpg: enabled compatibility flags:
gpg: using pgp trust model
/home/rozek/.gnupg/pubring.kbx
------------------------------
pub   ed25519 2022-12-14 [SC] [expires: 2024-12-27]
      5F37830BFA46FF7881F47AC78DF79C3DC5FC658A
uid           [ultimate] Brandon Rozek <brozek@brandonrozek.com>
uid           [ultimate] Brandon Rozek <hello@brandonrozek.com>
sub   cv25519 2022-12-14 [E] [expires: 2023-12-14]
      D502A12A65F9997DAE4609C97DAEAD7BFFA8F9D3
sub   dsa2048 2022-12-17 [S] [expires: 2023-12-14]
      89859D1EDF70D6DC2F6BFFF226E457DA82C9F480

Store the fingerprints again for easy reference:

export SFPR1=D502A12A65F9997DAE4609C97DAEAD7BFFA8F9D3
export SFPR2=89859D1EDF70D6DC2F6BFFF226E457DA82C9F480

Extend the expiration of the subkeys:

gpg --quick-set-expire $FPR 1y $SFPR1
gpg --quick-set-expire $FPR 1y $SFPR2

Updating Services

I currently allow for two ways to query my keys: OpenGPG keyserver and WKD. To update my keys on my own WKD keyserver, I followed the steps in my tutorial on WKD.

For OpenGPG, I followed the instructions on their usage page:

gpg --export your_address@example.net | curl -T - https://keys.openpgp.org 

References

https://www.gnupg.org/documentation/manuals/gnupg24/gpg.1.html

https://whynothugo.nl/journal/2023/07/13/extending-an-expired-gpg-key/

https://brandonrozek.com/blog/decentralized-pgp-keys-wkd/

https://keys.openpgp.org/about/usage

However, since I’ve recently learned how to setup LXC containers, my host OS is Fedora, and the steamcmd provides instructions for Debian, Ubuntu, and Arch. I’ve decided to set up a system container using LXC.

Also who knows how secure these game servers are. It’s likely a good idea to virtualize them when possible…

This guide assumes you have a Ubuntu Jammy (22.04) amd64 system container configured and that you’re within the root shell of that container. For help with that, check out my post.

Installing Starbound

Since we start off with a root account, let’s create another user which we’ll call steam.

adduser steam

We’ll give sudo privileges to our steam user. You can always remove this privilege after setting it up.

usermod -aG sudo steam

Now let’s go into our steam user

su steam

Before using apt, it’s always a good idea to make sure our system is up to date.

sudo apt update && sudo apt upgrade

Steam requires 32-bit libraries to work properly, so we’ll need to setup the multiverse repository and add i386 support.

sudo apt install software-properties-common
sudo add-apt-repository multiverse
sudo dpkg --add-architecture i386

Finally, we can update our local cache and install steamcmd.

sudo apt update
sudo apt install steamcmd

Now let’s open our steam shell.

steamcmd

We should now see a steam> at the beginning of our prompt. We’ll tell steam to install Starbound under the Starbound folder of our $HOME directory.

force_install_dir /home/steam/starbound

I’m unsure if we need to login or do this anonymously. When I set this up, I’ve logged into my steam account. Write to me if you were able to get this to work anonymously.

login <username> <password>

After logging in (it might ask for a 2FA code), we can install Starbound using the following command.

app_update 211820

We can then leave the steam shell.

quit

At this point, we should check to see if the server binary runs as expected.

/home/steam/starbound/linux/starbound_server

If this works, great! I didn’t face any issues at this step, but feel free to write in if you did.

Configuring Networking

We now have a Starbound server running from within our LXC container. Since we want players outside of the host system to access the Starbound server, we’ll need to setup some firewall rules and TCP forwarding.

First we’ll need to allow traffic to go into our container network.

sudo ufw allow in on lxdbr0
sudo ufw route allow in on lxdbr0

On the host, we need to open up the Starbound port

sudo ufw allow 21025

Then we need to route TCP traffic back and forth. For this I used nginx.

sudo dnf install nginx

We need to get the container IP.

lxc-ls --fancy

Add the following to the end of /etc/nginx/nginx.conf

stream {
    upstream starbound {
        server CONTAINERIP:21025;
    }

    server {
        listen 21025;
        proxy_pass starbound;
    }
}

Start and enable the nginx service

sudo systemctl enable --now nginx

If you face an error here, it might be because of selinux. Either disable that or configure it to allow for binding of 21025.

At this point, you can try to connect to the Starbound server using the game! I didn’t face any unexpected issues at this part, but feel free to write in if you did.

Locking down Starbound (Optional)

This part is optional, but if you don’t want any arbitrary person to connect to your Starbound server, you’ll need to lock it down.

Inside your container exit the /home/steam/starbound/storage/starbound_server.config

Disallow anonymous connections

"allowAnonymousConnections": false

Add users with usernames and passwords

"serverUsers" : {
  "user1" : {
    "admin" : false,
    "password" : "somethinghere"
  },
  "user2" : {
    "admin" : false,
    "password" : "somethinghere2"
  }
}

Keep in mind that we didn’t set up TLS so I wouldn’t put a password here that gets used anywhere else.

Automatically Starting up the Starbound server

First make sure the container autostarts on boot.

Then within the container, we can set up a systemd service to start the starbound server.

Edit /etc/systemd/system/starbound.service

[Unit]
Description=StarboundServer
After=network.target

[Service]
WorkingDirectory=/home/steam/starbound/linux
User=steam
Group=steam
Type=simple
ExecStart=/home/steam/starbound/linux/starbound_server
RestartSec=15
Restart=always
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

Then start and enable the service

sudo systemctl enable --now starbound

First, these system containers allows me to follow instruction documentation for projects that do not treat docker/podman as a first class distribution method. Maybe the project relies on being able to access systemd, or perhaps I don’t want the additional burden of ensuring the upgrade path of a custom docker-based method.

Second, it allows me to provide a lightweight virtualized environment for my friends and family. With this approach, we don’t have to set caps on the amount of CPU and memory usage from the start (as opposed to a VM), and instead impose them later if others are violating fair use.

In this guide, we’ll focus on setting up unprivileged containers. These are containers that are started up and maintained by a user other than root.

This technology is moving fast. It’s likely that this guide will be out of date by the time you’re reading this. I do hope, however, that this can serve a as a good jumping off point for other queries.

Setup

First, let’s install lxc

sudo dnf install lxc lxc-templates

To setup networking for our containers, we’ll also need to install dnsmasq.

sudo dnf install dnsmasq

We now need to tell LXC that our user is allowed to create a certain number of network devices on our lxcbr0 bridge that LXC configures for us.

echo $(id -un) veth lxcbr0 50 | sudo tee -a /etc/lxc/lxc-usernet

I can’t imagine myself creating more than 50 system containers, but do adjust that number as you see fit.

The last command we’ll need to run as root on the host system is to enable and start the lxc-net service.

sudo systemctl enable --now lxc-net

For our user on the host, we need to setup the LXC configuration file. Here, we’re going to map the user UID/GIDs into our container.

This set of commands were taken from LXC getting started.

mkdir -p ~/.config/lxc
cp /etc/lxc/default.conf ~/.config/lxc/default.conf
MS_UID="$(grep "$(id -un)" /etc/subuid  | cut -d : -f 2)"
ME_UID="$(grep "$(id -un)" /etc/subuid  | cut -d : -f 3)"
MS_GID="$(grep "$(id -un)" /etc/subgid  | cut -d : -f 2)"
ME_GID="$(grep "$(id -un)" /etc/subgid  | cut -d : -f 3)"
echo "lxc.idmap = u 0 $MS_UID $ME_UID" >> ~/.config/lxc/default.conf
echo "lxc.idmap = g 0 $MS_GID $ME_GID" >> ~/.config/lxc/default.conf

Now we can download and create our container.

systemd-run --unit=my-unit --user --scope -p "Delegate=yes" -- lxc-create -t download -n test

At the time of writing, we unfortunately have to wrap the lxc-create command within a systemd-run because of a semi-recent change in how cgroups work.

After running that command, you should see a large table of distributions and questions about which one to choose. Of course this is personal preference, but I selected ubuntu jammy amd64.

Once downloaded, we can then start our container.

systemd-run --unit=my-unit --user --scope -p "Delegate=yes" -- lxc-start test

If we see an error message here, then we can add a log file to check for details.

systemd-run --unit=my-unit --user --scope -p "Delegate=yes" -- lxc-start test --logfile=~/lxc.log

The error I commonly saw when setting this up was:

lxc-start test 20231130034412.168 ERROR    start - start.c:print_top_failing_dir:99 - Permission denied - Could not access /home/brandon. Please grant it x access, or add an ACL for the container root

This means we need to give $MS_UID access to open the /home/brandon folder. Though what it’s really trying to do is access /home/brandon/.local/share/lxc for it’s rootfs and config.

setfacl -m u:$MS_UID:x /home/rozek

Then try running the lxc-start command from before again. Sometimes when setting this up it worked from here, other times it wanted me to add the +x permission to other folders.

If you run it and don’t see a bunch of errors, then it’s hopefully a success! Check that it’s running:

[brandon@host ~]$ lxc-ls --fancy
NAME      STATE   AUTOSTART GROUPS IPV4       IPV6 UNPRIVILEGED 
test      RUNNING 0         -      10.0.3.43  -    true   

We can then attach to our container to get a shell within it.

lxc-attach test

If the above command doesn’t work, you might need to wrap it in a systemd-run

systemd-run --unit=my-unit2 --user --scope -p "Delegate=yes" -- lxc-attach test

Auto-starting LXC containers at boot

When we start a container with the systemd-run command, it’s tied to that particular terminal session. If we want to start these containers when the machine boots up, we can rely on systemd.

First let’s create a service file under ~/.config/systemd/user/lxc@.service

[Unit]
Description=Start LXC container
After=lxc-net.service
Wants=lxc-net.service

[Service]
Type=oneshot
Delegate=yes
RemainAfterExit=yes
ExecStart=/usr/bin/lxc-start %i
ExecStop=/usr/bin/lxc-stop %i
TimeoutStartSec=0

[Install]
WantedBy=default.target

Then we can start our test container

systemctl --user start lxc@test

Enable it on bootup

systemctl --user enable lxc@test

Since this is a user service, we need to make sure linger is on for it to respect the enable on boot setting.

sudo loginctl enable-linger rozek

Other Resources

https://linuxcontainers.org/lxc/getting-started/

https://www.redhat.com/sysadmin/exploring-containers-lxc

https://wiki.archlinux.org/title/Linux_Containers

These web badges are often small images that shows a product or message. Historically, it dates back to when Netscape in 1996 made the Netscape Now! badge.

Since then, the Geocities picked it up and standardized many of the web badges to have a pixel size of 88x31. These badges are sometimes animated as GIFs, though they usually don’t hold dynamic information.

In recent times, we’ve seen badges hold dynamic information such as code coverage on GitHub. Also, Wikipedia has Userboxes that people can add to their User Pages to show dynamic information about themselves. Ruben’s User Page showcases many of these.

Though to get started with my own website, I decided to look primarily at the traditional 88x31 style web badges. Looking around the web, there are some webpages that have large listings of web badges:

https://web.badges.world/

https://cyber.dabamos.de/

https://capstasher.neocities.org/

From there, we can (1) pick the ones we like (2) download them, and (3) upload them to our webserver. We can then create the img tag in our footer to showcase them.

<img width="88px" height="31px" src="/badges/fedora.gif" />

I also recently bought the Google Pixel buds (August 2023) and agree with you on how magical they are. Beforehand, I primarily used the Sony WH-100XM4 headphones for a couple years. While I still believe that those are superior in terms of audio quality, the Google Pixel buds have it beat in terms of convenience. They allow me to have one earbud in to not only enjoy whichever podcast I’m listening to at the moment, but also be able to quickly hear my fiance when she wants to strike a conversation with me.

While the Google Pixel buds are my new default. I do believe there are situations when other solutions are appropriate.

• For video calls, I often use a cheap pair of 2.5mm wired earphones. This is mainly to not include latency introduced by bluetooth.
• When I’m in an extended noisy non-interactive situation like a airline flight. I find that the Sony WH-100XM4 headphones cancels out more noise which is needed for an enjoyable experience.

This isn’t to say that GitHub is evil and that we should move to Bitbucket, Gitlab, Source Hut, etc. But this is more of a commentary on what happens when we depend on one service to host our code. Git is designed to be decentralized, we should make use of that fact!

Also, alleged illegal activity is not the only reason repositories are taken down from the Internet. Sometimes, the developer themselves decide to take it down.

As individuals, we can maintain mirrors of repositories we care about. That way if it ever gets removed from a particular service, we’re not out of luck.

The simplest way to go about this is to git clone the repositories you care about, and regularly pull through a cron job or systemd timer. Through systemd maybe it’ll look like this:

/etc/systemd/system/refresh-hugo.service

[Unit]
Description=Pull latest changes from Hugo

[Service]
Type=oneshot
WorkingDirectory=/home/brandonrozek/repo/hugo
ExecStart=/usr/bin/git pull

[Install]
WantedBy=multi-user.target

/etc/systemd/system/refresh-hugo.timer

[Unit]
Description=Pull latest changes from Hugo every week
Requires=refresh-hugo.service

[Timer]
Unit=refresh-hugo.service
OnCalendar=weekly
Persistent=true

[Install]
WantedBy=timers.target

Alternatively, you can use a self-hosted git server instance such as Forgejo to set up pull mirrors through the migration tool. As of the time of writing, this is what I currently do with repositories at https://git.brandonrozek.com/github

I do recommend only mirroring/pulling infrequently such as weekly. We don’t want to induce unnecessary load on these centralized services.

• If the client has multiple VPNs connected, only one DNS server can be set.
• There may be more latency using a DNS server within a VPN than using a default one provided by the ISP.
• You may not have permission to host the DNS server within the VPN network.

To address these set of issues, we’ll go over how to change /etc/hosts on the local client machine depending on which network it’s connected to.

In this setup, we’ll have a default /etc/hosts file. I’ll show how to then swap it with one for a particular connection. To do this, we need a way for a script to run when NetworkManager connects or disconnects from a network.

Luckily, NetworkManager-dispatcher handles this for us. To get a complete understanding on how to write scripts for this, reference

man 8 networkmanager-dispatcher

In essence, scripts within /etc/NetworkManger/dispatcher.d/ get executed in alphabetical order with two arguments set and a lot of environmental variables.

What we’ll care about in our scripts are:

• $1 the first argument passed to the script is the interface
• $2 the second argument refers to the event being triggered.
  • Possible options include: pre-up, up, pre-down, down, vpn-up, vpn-pre-up, vpn-pre-down, vpn-down, hostname, dhcp4-change, dhcp6-change, connectivity-change, reapply.
• $CONNECTION_UUID refers to a particular connection profile in NetworkManager. This is so we know which /etc/hosts file to swap with which connection.

Doing some quick in dirty tests, I found the following events were triggered when connecting to a particular network:

dhcp4-change -> up -> connectivity-change

And, the following events were triggered when disconnecting from a particular network:

connectivity-change -> down

My first instinct was to use the connectivity-change event, however, the CONNECTION_UUID variable is not set for those. Instead we’ll use the up/down events.

For our example, here’s what our default /etc/hosts/ file will look like:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.10.10.3  home-server.brandonrozek.com
10.10.10.4  home-desktop.brandonrozek.com

When we’re connected to my home network, we’ll swap my /etc/hosts/ to look like:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.0.30  home-server.brandonrozek.com
192.168.0.40  home-desktop.brandonrozek.com

The following script we’ll store at /etc/NetworkManager/dispatcher.d/swap_home.sh which will swap the /etc/hosts file with the one stored at /etc/NetworkManager/hosts.home when I connect to my home network.

#!/usr/bin/env bash

interface=$1
event=$2

if [[ $interface != "wlp0s20f3" ]] || [[ $event != "up" ]] then
        exit 0
fi

# This environmental variable is set on UP/DOWN events
if [[ $CONNECTION_UUID != "901a1b68-e622-4be6-a61f-a8dc999212b3" ]] then
        exit 0
fi

cp /etc/NetworkManager/hosts.home /etc/hosts

In this script, you might have to replace wlp0s20f3 to reflect the interface that you’re using for connecting to the network. Additionally, you’ll have to replace the CONNECTION_UUID with the UUID of the connection you’re trying to swap under. You can use nmcli c to show the UUIDs for each of your network connections.

Similarly, when we disconnect from the network, we’ll need to set it back to our default /etc/hosts file.

#!/usr/bin/env bash

interface=$1
event=$2

if [[ $interface != "wlp0s20f3" ]] || [[ $event != "down" ]] then
        exit 0
fi

cp /etc/NetworkManager/hosts.default /etc/hosts

I’ll show in this post, that we can have a nice simple¹ key-value store using sqlite3. This gives us the benefit that we don’t need to use system resources to run a daemon the entire time and only spin up a process when we need it.

For our key-value store, we’re going to use a table with two columns:

• A key, which we’ll call name. This will be a unique TEXT type that has to be set.
• The value, which we’ll call value (Creative, I know.) For our purposes, this will also be a TEXT type.

The SQL to create this table is

CREATE TABLE config(
    name TEXT NOT NULL UNIQUE,
    value TEXT
);

Let’s say we want to get the value of the key author. This is a SELECT statement away:

SELECT value FROM config where name='author';

Now let’s say that we want to insert a new key into the table.

INSERT INTO config(name, value) VALUES ('a', '1');

What about updating?

UPDATE config SET value='2' WHERE name='a';

The tricky part is if we want to insert if the key does not exist, and update if it does. To handle this we’ll need to resolve the conflict.

INSERT INTO config(name, value) VALUES ('a', '3') ON CONFLICT(name) DO UPDATE SET value=excluded.value;

Lastly if you want to export the entire key-value store as a CSV:

sqlite3 -header -csv data.db "SELECT * FROM config;"

This is nice and all, but it’s inconvinient to type out all these SQL commands. Therefore, I wrote two little bash scripts.

sqlite3_getkv

#!/bin/sh

set -o errexit
set -o nounset
set -o pipefail

show_usage() {
    echo "Usage: sqlite3_getkv [db_file] [key]"
    exit 1
}

# Check argument count
if [ "$#" -ne 2 ]; then
    show_usage
fi

# Initalize database file is not already
sqlite3 "$1" "CREATE TABLE IF NOT EXISTS config(name TEXT NOT NULL UNIQUE, value TEXT);"

# Get value from key
sqlite3 "$1" "SELECT value FROM CONFIG where name='$2';"

ssqlite3_setkv

#!/bin/sh

set -o errexit
set -o nounset
set -o pipefail

show_usage() {
    echo "Usage: sqlite3_setkv [db_file] [key] [value]"
    exit 1
}

# Check argument count
if [ "$#" -ne 3 ]; then
    show_usage
fi

# Initalize database file is not already
sqlite3 "$1" "CREATE TABLE IF NOT EXISTS config(name TEXT NOT NULL UNIQUE, value TEXT);"

# Set key-value pair
sqlite3 "$1" "INSERT INTO config(name, value) VALUES ('$2', '$3') ON CONFLICT(name) DO UPDATE SET value=excluded.value;"

Example Usage:

$ ./sqlite3_setkv.sh test.db a 4
$ ./sqlite3_setkv.sh test.db c 5
$ ./sqlite3_getkv.sh test.db a
4
$ ./sqlite3_setkv.sh test.db a 5
$ ./sqlite3_getkv.sh test.db a
5

Step 1: Set a transcode location in Plex.

Log into your admin account. Then go to Settings -> Transcoder.

From there select the “Show Advanced” button and scroll down to the “Set Transcoder temporary directory”.

I’ve set it to /transcode and then hit the Save button.

Step 2: Setup tmpfs

We can use tmpfs to setup a filesystem that will be stored in RAM at the transcode directory. If you have Plex installed not in a container, then you can follow the Arch Wiki Instructions for setting it up.

I use docker-compose for my setup, so it involved adding some limes to my yaml.

  plex:
    image: lscr.io/linuxserver/plex:1.32.5
    # ...
    tmpfs:
      - /transcode
    # ...

I don’t promise massive performance increase by setting up Plex this way, but I do think it makes sense:

• Improved write speeds for the transcoded files
• The transcoded files aren’t persistent

If you’re of a different opinion, let me know.

I keep the majority of my git repositories in the folder ~/repo/ on all my devices. Inspired by a recent blog post by Clayton Errington, I wanted a way to quickly check within a folder which repositories need updating. Their blog post has a script written in PowerShell. I decided to write my own bash implementation, and also ignore the bit about modified files since I mostly care about the state of my commits with respect to the origin remote.

Before writing a recursive implementation, let’s first discuss how to check the ahead/behind status for a single repository.

First things first, you need to make sure that we have all the references from the remote.

git remote update

To print out how many commits the local main branch is ahead of the one located on the origin remote we can use:

git rev-list --count origin/main..main

Similarly for checking how many commits the local main branch is behind we can use:

git rev-list --count main..origin/main

Instead of looking at the main branch, maybe we can to check whichever branch we’re currently at.

branch=$(git rev-parse --abbrev-ref HEAD)

We can wrap all of this into a nice bash function. We’ll additionally check if there is a .git in the current folder as none of the git commands will work without it.

check_remote() {
  if [ -d .git ]; then
    git remote update > /dev/null 2> /dev/null
    branch=$(git rev-parse --abbrev-ref HEAD)
    ahead=$(git rev-list --count origin/$branch..$branch)
    behind=$(git rev-list --count $branch..origin/$branch)
    echo "$ahead commits ahead, $behind commits behind"
  fi
}

I currently have 15 repositories in my ~/repo folder. Now I can cd into each of them and run this bash function. Or, I can have bash do it for me :)

Let’s write a function called process that does just that. We’ll pass in a folder as an argument stored in $1

process() {
  if [ -d "$1/.git" ]; then
    pushd "$PWD" > /dev/null
    cd "$1"
    echo -n "$1 "
    check_remote
    popd > /dev/null
  fi
}

The pushd command will keep track of the folder that we’re currently in. Then we cd into the directory that has .git folder. Print the name of the folder so we can associate the ahead/behind counts, and then run the check_remote function. Lastly we popd back to the folder we started from.

All that’s left is to get the list of folders to process:

find . -type d -print0

Feed it into a while read loop passing in each folder into the process function.

find . -type d -print0 | while read -d $'\0' folder
do
  process $folder
done

All together the script will look like:

#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

show_usage() {
  echo "Usage: git-remote-status [-R]"
  exit 1
}

check_remote() {
  if [ -d .git ]; then
    git remote update > /dev/null 2> /dev/null
    branch=$(git rev-parse --abbrev-ref HEAD)
    ahead=$(git rev-list --count origin/$branch..$branch)
    behind=$(git rev-list --count $branch..origin/$branch)
    echo "$ahead commits ahead, $behind commits behind"
  fi
}

if [ "$#" -eq 0 ]; then
  check_remote
  exit 0
fi

if [ "$1" != "-R" ]; then 
  show_usage
  exit 1
fi


process() {
  if [ -d "$1/.git" ]; then
    pushd "$PWD" > /dev/null
    cd "$1"
    echo -n "$1 "
    check_remote
    popd > /dev/null
  fi
}
export -f process

find . -type d -print0 | while read -d $'\0' folder
do
  process $folder
done

This gives us two options. If we pass in no flags, then it’ll print out the ahead/behind status for the current folder. If we pass in -R, then we recursively check all the subfolders as well.

Example Output of git-remote-status -R:

./project1 0 commits ahead, 3 commits behind
./project2 1 commits ahead, 0 commits behind
./project3 1 commits ahead, 2 commits behind
./project4 0 commits ahead, 0 commits behind
./project5 0 commits ahead, 0 commits behind

Installing MQTT Broker

On the server you want to install MQTT on:

sudo dnf install mosquitto

We won’t go over how to setup TLS or authenticated users. Instead for a quick test, we’ll configure it to allow for anonymous connections over any interface.

Add the following lines to /etc/mosquitto/mosquitto.conf

listener 1883
allow_anonymous true

Then enable the systemd service to start mosquitto

sudo systemctl enable --now mosquitto

Make sure it’s running:

sudo systemctl status mosquitto

Testing the Broker

Most tutorials have you test the publish/subscribe on the local machine. Though given we’re working with smart devices, we need to make sure it works on another machine first.

Approach 1: Install the mosquitto tools on that machine. Then you’ll need to open two terminals.

• On terminal 1, subscribe to all messages mosquitto_sub -h [MQTT_BROKER_ADDR] -t "#"
• On terminal 2, publish a message: mosquitto_pub -h [MQTT_BROKER_ADDR] -t "test" -m "Hello, World"

On the first terminal, you should see the string Hello, World.

Approach 2: Use curl.

• On terminal 1, subscribe to all messages with curl mqtt://[MQTT_BROKER_ADDR]/%23 --OUTPUT - --trace -
• On terminal 2, publish a message curl -d 'Hello, World' mqtt://[MQTT_BROKER_ADDR]/test/

Similarly, you should see the string Hello, World along with a bunch of debugging information.

Google calls this Emoji Kitchen. Xavier Salazar created a nice website to visualize all the combinations. These emojis are not part of any text encoding standard, but instead are image files.

Now it’s too hard for me to keep track of all these emojis. However, if there’s one series that particularly resonates with me. I would say it’s the turtle. Just look how awesome he is.

Luckily, Hurricane Electric provides a free IPv6 tunnel broker service. The Arch Linux Wiki has a great guide on how to set it up. This post will start off with setting up the service, and we’ll discuss a couple bits of consideration we need to take to get IPv6 traffic to a webserver running in a docker/podman container.

Hurricane Electric is a Tier 2 Internet service provider. This means that they peer with other Internet service providers to route network traffic with. Since at least 2010, Hurricane Electric has offered a free IPv6 tunnel. This gives me confidence that they should at least continue to offer this service in the near future. In order to route IPv6 traffic to your IPv4 server, they make use of the the 6in4 tunneling protocol. This encapsulates the IPv6 packet with a IPv4 header.

To get setup, create on account on https://tunnelbroker.net. After verifying your email¹, login and click on “Create Regular Tunnel”. From there it will ask you for the IPv4 address of your server. Insert that and then click the tunnel server that is the closest geographically to where the server is located.

Once the tunnel is created, you’ll need to take note of four addresses within the “IPv6 Tunnel Endpoints” section:

• Server IPv4 Address
• Server IPv6 Address
• Client IPv4 Address
• Client IPv6 Address

In terms of configuring the networking on the server, I found the systemd script on the Arch Wiki the most straightforward setup. Within /etc/systemd/system/he-ipv6.service

[Unit]
Description=he.net IPv6 tunnel
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/ip tunnel add he-ipv6 mode sit remote server_IPv4_address local client_IPv4_address ttl 255
ExecStart=/usr/bin/ip link set he-ipv6 up mtu 1480
ExecStart=/usr/bin/ip addr add client_IPv6_address dev he-ipv6
ExecStart=/usr/bin/ip -6 route add ::/0 dev he-ipv6
ExecStop=/usr/bin/ip -6 route del ::/0 dev he-ipv6
ExecStop=/usr/bin/ip link set he-ipv6 down
ExecStop=/usr/bin/ip tunnel del he-ipv6

[Install]
WantedBy=multi-user.target

Then enable/start the service with:

sudo systemctl enable --now he-ipv6.service

With this, your server should now be IPv6 routable. You can test this by trying to ping your server from another IPv6 enabled device.

ping client_ipv6_address

If that works, next you’ll have to make sure that docker/podman and nginx are configured to accept TCPv6 traffic. Finally, since the tunnel is on a different network interface, you may need to allow routing at the firewall level.

For example, to allow from traffic between the podman and tunnel interfaces:

sudo ufw route allow in on he-ipv6 out on podman1
sudo ufw route allow in on podman1 out on he-ipv6

LaTex beamer has the \invisible command which holds the space for the text but doesn’t display it. We can create a new command with an if statement stating whether or not we want to use \invisible.

For this, I called the variable showanswer, but you can call it whatever you like. Put the following code in the preamble of your beamer document.

\newif\ifshowanswer
%\showanswertrue % Uncomment when you want to show the answer
\newcommand{\HideAns}[1]{\unless\ifshowanswer \invisible{#1} \else #1 \fi}

To write if not in LaTex, we need to use \unless\if.

Now let’s see this in action. Here is a slide I gave in a recent lecture about Unification:

\begin{frame}{Examples}
	\begin{enumerate}
		\item $f(x, y) \eq f(g(x), b)$
		\begin{itemize}
			\item Answer: \HideAns{None - Occurs Check}
		\end{itemize}
		
		\item $f(x, a) \eq f(a, b)$
		\begin{itemize}
			\item Answer: \HideAns{None - Function Clash}
		\end{itemize}
		
		\item $g(x) \eq g(y)$
		\begin{itemize}
			\item Answer: \HideAns{$x \mapsto y$}
		\end{itemize}
	\end{enumerate}
\end{frame}

The left shows the hidden version, and the right shows the version I can distribute afterwards.

Minimal Working Example:

\documentclass{beamer}
\usetheme{Copenhagen}
\usepackage[utf8]{inputenc}

\newcommand{\eq}{\stackrel{?}{=}}

\newif\ifshowanswer
% \showanswertrue % Uncomment when you want to show the answer
\newcommand{\HideAns}[1]{\unless\ifshowanswer \invisible{#1} \else #1 \fi}

\begin{document}
\begin{frame}{Examples}
	\begin{enumerate}
		\item $f(x, y) \eq f(g(x), b)$
		\begin{itemize}
			\item Answer: \HideAns{None - Occurs Check}
		\end{itemize}
		\item $f(x, a) \eq f(a, b)$
		\begin{itemize}
			\item Answer: \HideAns{None - Function Clash}
		\end{itemize}
		\item $g(x) \eq g(y)$
		\begin{itemize}
			\item Answer: \HideAns{$x \mapsto y$}
		\end{itemize}
	\end{enumerate}
\end{frame}
\end{document}

What if this installation fails and I need to recreate the server?

Luckily that didn’t come to pass, but the thought was enough to scare me into learning Ansible.

Honestly, I’ve been skeptical of it for a while. I thought that having bash automation scripts was good enough and that Ansible will come and go.

Except, that it’s been around for the last 11 years…

Consider me converted! My favorite module is lineinfile. It’s all too common in Linux for setup instructions to contain “edit x line in y file.”

Now, I have Playbooks written for configuring my server, but what about the data?

I was chatting with my good friend Stefano Coronado and he suggested using block volumes. Most VPS providers offer this feature and its persistent data storage that can be attached to one or more servers within the same data center.

This of course does not substitute having good backups, but having separation between configuration and data is a great idea.

Lets say that I have block storage mounted on /mnt/storage.

It’s easy to point directly to it within docker-compose:

database:
  image: docker.io/mongo:5.0
  volumes:
    - /mnt/storage/database/data/db:/data/db

If you’re using an application outside of docker and is not easily configurable, we can make use of symlinks.

For example, if an application is expecting configuration to be at /etc/special then:

sudo ln -s /mnt/storage/special /etc/special

sets up a symlink at /etc/special to point at /mnt/storage/special.

Learn more on how to specifically setup and mount a block volume depending on your hosting provider:

• Linode
• DigitalOcean
• Vultr
• AWS

Each device is given a unique identifier, which it can then query a discovery server for its IP address. The Syncthing foundation runs a public discovery server to facilitate this.

Once the IPs are known, if a direct connection can’t be established, then Syncthing will make use of a relay. A relay only forwards encrypted packets like a router and is unable to inspect its contents.

The community runs a pool of public relay servers that are configured by default as well.

For privacy reasons, one might not want to make use of these public discovery and relay servers. Also, it may be more performant to run your own relay servers. We’ll show how you can configure Syncthing to work without relying on these community services.

We’ll go over:

• How to setup a private relay server
• How to configure the clients to disable discovery and use the private relay server.

Setting up the private relay server

First we’ll need to have strelaysrv installed. On Fedora, this is under the syncthing-tools package.

sudo dnf install syncthing-tools

Then we need to make sure that port 22067 is open to TCP traffic on our firewall. Optionally, strelaysrv can expose a /status HTTP endpoint on port 22070, but I didn’t enable it myself.

sudo ufw allow 22067/tcp

If applicable, make sure that these are enabled in your cloud firewall.

When strelaysrv runs for the first time, it’s going to create a set of public/private keys. We’ll store this in /etc/strelaysrv.

sudo mkdir /etc/strelaysrv

To run as a daemon on startup, we’ll write a systemd service to /etc/systemd/system/strelaysrv.service.

[Unit]
Description=Syncthing Relay Server
After=network.target
Documentation=man:strelaysrv(1)

[Service]
ExecStart=/usr/bin/strelaysrv -keys=/etc/strelaysrv -status-srv="" -token=INSERT_TOKEN_HERE

# Hardening
User=brandonrozek
Group=brandonrozek
ProtectSystem=strict
ReadWritePaths=/etc/strelaysrv
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true

[Install]
WantedBy=multi-user.target

Let’s discuss the flags given in ExecStart:

• -keys indicate the location that the public/private keys will be written to or read from
• -status-srv="" disables the status endpoint on port 22070. You can remove that flag if you want that feature.
• -token="INSERT_TOKEN_HERE" first prevents the server from joining the public pool and requires clients to specify the token in order to use the relay.
  • NOTE: Replace the token with a random output, such as uuidgen.

The hardening section is better left to a future blog post. In the meantime, read the Freedesktop documentation for more information.

Now we can enable the service on boot and start the Syncthing relay service.

sudo systemctl enable --now strelaysrv.service

You’ll need to note the relay:// URL shown at sudo systemctl status strelaysrv for the next section.

Client Configuration

On the client side, we have to prevent the Syncthing service from listening to traffic from public relays and not rely on the public servers for discovery. These instructions are for the GUI interface on localhost:8384.

Lets say we have a relay at relay:examplerelay.com?id=SOME-ID&token=SOME-TOKEN

Change Settings->Connections->Sync Protocol Listen from default to:

relay:examplerelay.com:22067?id=SOME-ID&token=SOME-TOKEN

If we want to allow for direct connections, then we’ll need to have port 22000/UDP open on the client side.

sudo ufw allow 22000/udp

Then edit Settings->Connections->Sync Protocol Listen to:

quic://0.0.0.0:22000, relay://examplerelay.com:22067/?id=SOME-ID&token=SOME-TOKEN

Uncheck ‘Global Discovery’ within that menu. Optionally, uncheck “Local Discovery” as well.

Local discovery only sends packets within the local network so it’s safe to leave it enabled if desired. If you’re worried about information leakage in a public WiFi network, then leave it disabled.

Since we disabled the discovery server, how do we find a connection to the different devices? Here, we can make use of the relay, and/or hard code static ip addresses that some devices may have.

For example, say that I have a home-server on the following IP addresses:

• 192.168.1.58 within my home network
• 10.10.2.7 within my Wireguard network.

Then under home-server->Settings->Advanced->Addresses, I can set it to:

quic://192.168.1.58, quic:10.10.2.7, relay://examplerelay.com:22067

This will try to establish a direct connection using the local and wireguard IP addresses and fallback to the relay if not possible.

Conclusion

With these configuration changes, Syncthing no longer relies on public discovery and relay servers and instead makes use of your own relay server.

This helps improve privacy and perhaps even efficiency as well! If you run into any troubles during this configuration let me know and I can expand upon this post.

Beginning from April through Octoboer, Smorgasburg Williamsburg offers a variety of different food stalls starting in the morning. We decided to start the day there with a late breakfast.

We had some steamed bao buns (bun buns), drank juice out of a hallowed out pineapple, and ate taiyaki. There wasn’t a lot of shaded seating available, so we ended up eating on the grass near a tree.

Afterwards, we walked around Williamsburg and visited a few shops.

We then took the train to “Down Under the Manhattan Bridge Overpass” (DUMBO). There we stumbled upon an art fair, and took some pictures on the way to the Brooklyn bridge.

Photo of Manhattan Bridge:

Photo of Brooklyn Bridge:

We walked all the way to the beginning of the prominade on Tillary street to get onto the Brooklyn bridge. From where we started this resulted in a lot of unnecessary walking. If I was to do this again, I would’ve climbed the staircase located in the underpass of Washington street and Prospect street.

After crossing the bridge and reaching our 20k steps for the day, we found the nearest bench in city hall park and looked up places to eat. While doing so, we saw a biker run over the wings of a pigeon. At least for the pigeon, a little girl fed it Pringles afterwards.

We ended up eating at a HotPot place in the Manhattan Chinatown nearby. It’s harder to justify shared buffet style restaurants like unlimited hot pot or sushi when it’s just Clare and I. So how can we let go of that chance? The food turned out great and we’re looking out for the next excuse to eat it again.

First, we can set some limits to container in general within docker-compose.

Example:

mongo:
  image: docker.io/mongo:5.0
  mem_limit: 256m
  cpus: 0.25

This says that the mongo container can only use a maximum of 256MB of memory and can only use up to 25% of one CPU.

To see a detailed list of possible options, check out the docker documentation.

Keep in mind that your memory and CPUs are not virtualized, therefore the container can see all the resources available, it just cannot request them.

This may lead to some problems depending on the codebase. For example, in MongoDB the WiredTiger internal cache is by default set to 50% of the total amount of RAM minus 1GB.

Luckily we can change this default, by adding a flag within the docker-compose file.

mongo:
  image: docker.io/mongo:5.0
  mem_limit: 256m
  cpus: 0.25
  command: --wiredTigerCacheSizeGB 0.25

The wiredTigerCacheSizeGB takes values between 0.25 GB and 10000 GB.

First, we need to identify the repo id that we wish to remove.

dnf repolist

Example output:

repo id                          repo name
fedora                           Fedora 38 - x86_64
fedora-cisco-openh264            Fedora 38 openh264 (From Cisco) - x86_64
fedora-modular                   Fedora Modular 38 - x86_64
updates                          Fedora 38 - x86_64 - Updates
updates-modular                  Fedora Modular 38 - x86_64 - Updates
ancient-repo                     Fedora 1

We can remove all the packages from the repo id ancient-repo with the following command:

dnf repository-packages ancient-repo remove

Use this opportunity to double check that what it’s going to remove is okay for your system. Some applications may not have newer versions in the other repositories, so you might have to hunt those down separately.

Also keep in mind, that this command will uninstall any unused dependencies from other repositories as well.

After removing the packages within the deprecated repository, we can remove it from the repo list.

rm /etc/yum.repos.d/ancient-repo.repo

Replacing ancient-repo with your repo id.

To install:

npm install -g typescript

Typecheck a file:

tsc --checkJs --allowJs example.js --noEmit

Since tsc is primarily a compile, it by default produces code output. We’re only concerned with seeing any type errors, so we throw away the compiled code with the flag --noEmit.¹

Now it’s likely that we’re using features of the standard library or the DOM in our Javascript code. We need to include a couple flags for Typescript to understand them.

You can see tsc --helpfor the long list of options. Here’s an example for targetting ES2022 with the DOM library.

tsc --checkJs --allowJs --target es2022 --lib es2022,dom example.js --noEmit

For example, given the state: (color shirt1 red) (color shirt2 blue), we know that (color shirt1 blue) is false solely because it wasn’t listed in the state.

This means that for an individual predicate $P$, it can either represent true or false. To represent uncertainty, we need to extend to the use of two predicates instead of one.

This approach is popular within conformant planning citing back to Palacios and Geffner in 2006. Consider our predicate $P$ for example. Instead of that predicate existing imagine we have the predicates. $K P$ and $K \neg P$. Note that $K$ is not a higher-order predicate or operator but is instead part of the predicate name. Then our truth value for $P$ is as follows:

We’re still in the closed world assumption setting. The intuition behind this compilation is that the agent “knows what it knows and knows what it doesn’t know.” This approach doesn’t include any epistemic axions, but may be good enough to get started modeling within some of these systems.

This is one way of encoding uncertainty in some predicate of $P$, but not the only way. An alternative is what I call the discovery encoding, mainly since I haven’t seen it in use in literature. If you have a citation with this encoding being used, please share with me.

This operates very similarly to optionals within functional programming. The idea is that one shouldn’t even consider the valuation of $P\_Value$ until $P\_Discovered$ is true. Since once $P\_Discovered$ is true, then $P\_Value = P$.

References

Palacios, Héctor, and Hector Geffner. “Compiling uncertainty away: Solving conformant planning problems using a classical planner (sometimes).” AAAI. 2006.

However, if the internal network does have access to the Internet, we can use a different tool. Letsencrypt can issue a valid SSL certificate for your network without being able to directly access your network in question! It is able to do this though the DNS-01 challenge.

The way this work is that Letsencrypt asks your client to create a DNS TXT entry containing some special token. The client then either manually, manually with hooks, or via a plugin edits the zone file on the DNS nameserver to add that entry. The Letsencrypt server then only needs to access that DNS nameserver in order to verify that you own the domain; Issuing your certificate upon success.

This is an example on how to get a certificate for an example internal domain insiderexample.org using the Linode DNS provider.

certbot certonly \
  --dns-linode \
  --dns-linode-credentials ~/.secrets/certbot/linode.ini \
  -d insiderexample.org

This easily works for one domain, but what if we have many sub-domains we want added to the certificate? With the HTTP-01 challenge, we need to test each sub-domain individually. Alternatively, we can use the DNS-01 challenge to get issued a wildcard certificate. With one wildcard certificate (e.g *.exampledomain.com) we can secure a.exampledomain.com, b.exampledomain.com and many more!

Letsecnrypt verifies that the user is allowed to claim all these subdomains, by seeing if the user has access to the DNS zone file for that domain. The idea is that if the user is able to change the DNS anyways, then the user could’ve gone through the process of installing a webserver at that IP. With access to the DNS zone file, the user would have to create a TXT record.

Now this process could be done manually on Certbot by using the --manual flag. However, in the spirit of automation, there are many plugins existent to help streamline the process. This list of DNS providers shows which provider supports this feature and in what clients.

If your provider isn’t supported on this list, not all hope is lost. Under the manual plugin one can use the hooks --manual-auth-hook and --manual-cleanup-hook to execute external scripts to access the DNS provider’s API.

This wiki page for Certbot shows the list of supported providers and how to structure the command line arguments. For example, I use Nginx as my webserver and Linode as my DNS nameserver.

certbot certonly \
  --dns-linode \
  --dns-linode-credentials ~/.secrets/certbot/linode.ini \
  -d *.exampledomain.com

However without JavaScript, transitioning between pages can flash a white screen in between. This can be jarring for websites using a non-white background. Single page applications attempt to resolve this by requesting webpages via JavaScript and dynamically reshuffling the page.

Pjax is a technique designed to marry these two ideas. The name Pjax stands for pushState + Ajax. The idea is that when a person clicks a link, instead of the browser loading an entirely new page, the local JavaScript would issue an Ajax request. Once the request is fulfilled it would replace hte current screen contents with the request and update the browser’s history via a pushState. Browsers that don’t support this feature will result in a normal full page load when the link is clicked.

Over the years several JavaScript libraries have popped up to fill this need. There was the classic pjax library which handled this. Then intercooler.js came around to allow you to specify Ajax requests via HTML attributes. Finally, htmx extends these HTML attributes beyond Ajax and into CSS transitions and WebSockets.

This post will show how you can use htmx to easily create page transitions in a progressively enhanced way. The quickest way to achieve this is with an attribute called hx-boost. This can be used by <a> tags in HTML for links and <form> tag for form submissions. By default, these two HTML elements will issue a full page load to a new URL once they are clicked. However with hx-boost, it will replace the full page load with a ajax request and replace the innerHTML of the HTML node’s contents with the response.

Another nice feature of htmx is that we don’t need to add hx-boost for every HTML node. Instead, this attribute can be inherited. This allows us to put hx-boost="true" on the body of the document and every form and link will be boosted. If we want to simulate a page transition, instead of replacing the inner HTML of the link or form, we can replace the entire inner HTML of the body of the document using hx-indicator.

<body hx-boost="true" hx-indicator="body">
   
</body>

Though we have to do a little more work if we want a smooth page transition. HTMX uses classes to denote the lifecycle of the request. When an element sends off a request is, the htmx-request class is added to whichever node is spcified in hx-indicator (itself by default). We can use this to have a simple fade out transition in the beginning of the request.

.htmx-request {
  opacity: 0;
  transition: opacity 200ms ease-in;
}

Once the AJAX request is satisfied, the htmx-request class is replaced with htmx-swapping to indicate that the content is about to be replaced.

.htmx-swapping {
  opacity: 0;
}

Unless a swap delay is specified, the content is then quickly replaced which then puts the node in a settled state. This is when we should fade in to see the new content

.htmx-settling {
  opacity: 1;
  transition: opacity 200ms ease-in;
}

Using the htmx library, we can have nice page transitions with one line of HTML code, an optional 11 lines of CSS, and zero lines of custom JavaScrpt code. The small transitions are especially nice in low-bandwidth scenarios when a response to an ajax request isn’t immediately received. It gives some visual indication that clicking the link still did something.

The rumor has been circulating for a few months now, even when Threads was referred to its codename Project 92. As such, many people have already voiced their opinion of this move. A group of moderators even banded together to make a website declaring their opposition of Meta joining the Fediverse.

The general argument is that Facebook is bad and has a bad track record. Thus, many people from the Mastodon community want to disassociate from Meta. The tool for disassociation on the Fediverse is called de-federation.

The idea behind the Fediverse is that servers can federate with each other. This means that servers can share data (whether its statuses or posts) with other servers as they talk a common language. Admins of each of these servers can select which platforms they want to block. Refusing to display content from blocked servers is called de-federation.

This technique is commonly used to block content from servers that often promote illegal or offensive content. Though it can also be used in cases where servers do not align with the ethics of a community. Since Facebook “does not match the moral compass” of several Mastodon communities, they seek to de-federate proactively. Before any questionable content appears.

The hope of these admins, is that if enough servers de-federate, it effectively nulls the benefits of adding ActivityPub support. Blocking the millions of users that signed up for Threads from interacting with other Mastodon users.

Of course, there will be some server admins that prioritize free speech and alternative values, but we’ll get into that later…

Instead of going further into the ethics of the situation. I’m curious on the potential ramifications of Threads both sucessfully and unsucessfully federating with the majority of the Fediverse.

Successful Federation

Lets say somehow Threads manages to federate with enough servers, such that the majority of current Mastodon users can interact with Threads users.

In this world, Threads will be by far the largest user instance. This would not be too unexpected, since they have the engineers on staff to build infrastructure that can support millions of people. As a consequence, the majority of content would then be generated via Threads.

This content will likely not follow the cultural norms of the existing Mastodon community. The majority of these cultural norms are built with an accessibility first mindset. Examples of this include: adding captions to photos, title-casing hashtags, adding content warnings to inappropriate or sensitive topics.

In a world where Meta is the largest Fediverse player, they will likely try to push the ActivityPub standards to whatever their priority of the time is. There’s a few ways this can play out, they can try to be the majority representation of the board updating the ActivityPub standard. They could also just implement features themselves and bully others to adopt and support the extensions to the standard they craft.

What features do I imagine Meta will want to add? I would imagine any that pertain to advertising and monetization, but we can leave that to your imagination.

Failed Federation

In this world, enough large instances de-federate from Threads, that most current users of Mastodon wouldn’t see content from them.

The ideal in this case is that some sort of status quo is maintained and users of these differing communities do not need to interact with each other.

Though I’m not sure if that’s the case. There might be some internal resistance from Mastodon users that want to use the Fediverse to keep in touch with their friends and family. Sadly the majority of non-techie friends and family will likely be using Threads or another platform from a major player. This added pressure can either cause the de-federated servers to federate again. Alternatively, it can cause some of these users to migrate over to free speech-instances or create another account on these or on Threads itself.

Unfortunately for many free-speech instances, they often harbor offensive speech. The question then is: does Threads federate with these servers and then have to take many active measures to make sure the most extreme content doesn’t surface to its users?

Or does it just block the free-speech instances entirely? If Threads de-federates from the free speech instances, then there wouldn’t be enough other instances to federate with making this a failed ActivityPub attempt.

There could be an attempt to fork the ActivityPub standard, but without some buy-in from the existing community, this is likely to fail.

Conclusion

Historically, this reminds me of when Google Talk/Hangouts supported XMPP. At the time I used Pidgin with the OTR plugin to communicate with friends and family. Sadly they henceforth abandoned XMPP support and I don’t use much of that or Hangouts anymore.

Will we see a similar approach from Meta to extend, embrace, and extinguish the ActivityPub protocol?

Don’t forget to bring a pair of binoculars!

Now I can buy the first pair of binoculars I find in a supermarket. Though, I always enjoy over-analyzing purchasing decisions and this isn’t an exception.

What considerations do you need to make when buying binoculars?

Most pairs of binoculars report a NxM measurement.

• N here denotes the magnification factor
• M denotes the diameter of the lens in millimeters

This isn’t a case where you buy the highest value in each section and call it a day. There are trade-offs:

• Magnification Factor
  • Higher means you can see farther away.
  • Lower means you have a greater field of view.
• Diameter Length
  • Higher means that more light gets captured which makes the overall image brighter.
  • Lower means that the binocular weighs less which can be important for long hikes.

For a typical bird watching session, you normally don’t look at birds that are too far away. In this case, you want to prioritize a greater field of view which can make the viewing feel more stable. The commonly recommended magnification factor is 8.

Binocular weight is generally only a massive concern for people hiking long distances. If the bird watching session is a quick one, then you should prioritize having a brighter picture than a lighter binocular. The commonly recommended diameter length is 42mm.

Therefore, the commonly recommended specification for bird-watching binoculars is 8x42.

Once you’re undergoing the bird watching session, how do you know what bird you’re looking at? In the ideal case, you’re already an expert and can identify it once you see it. Second best case, you have a guide that can tell you what it is. For the majority of us, however, we will have neither. In this case, we can rely on technology.

Merlin is a mobile app that can help identify birds.

Developed by the Cornel lab of Orthnology, Merlin can provide a database of bird photos and sounds, as well as tools to help identify them.

If you remember before a session, you can download localized bird data. This way during the session you can look at photos and sounds of birds that are typically found nearby your location.

Not only do they provide a database, but the app also allows for identification by both visuals and sound. When you ask the app the identify a bird visually, it’ll ask for your location, size of bird, colors found on the bird, and its behavior.

Birds often have fairly distinct calls. Merlin can also help identify based on these sounds.

So how did my bird watching tour go? Our group spotted the following birds in New York:

• Turkey Vulture
• Cooper’s Hawk
• Red-Shouldered Hawk
• Red-tailed Hawk
• Red-eyed Vireo
• Blue Jay
• Tree Swallow
• Barn Swallow
• Gray Catbird
• Wood thrush
• Cedar Waxwing
• American Goldfinch
• Song sparrow
• Red-winged blackbird
• Common yellowthroat
• Yellow Warbler

It’s easy to remember to take photos when a special event is going on. Perhaps you’re visiting a location for the first time, or you are hanging out with people. What’s harder to remember is taking photos in locations you constantly go to or around the house.

What is interesting about ordinary times like these? They are a dime a dozen. If I photograph my simple moments everyday, I’ll clutter my photo library.

That may be true, but I argue that they at least deserve a spot in your photo library.

I find that my life constantly changes. It may be because I’m still young and moving around places. But I believe this is still the case even if you settle down in a location.

For example, I used to go to Spillin’ the Beans in Troy, NY for breakfast every week. Now they closed down. Since I have taken a couple photos of that location, I can look back and remember my times there.

People come and go in our lives. Friends may move to different locations or kids move out of the house. What we might at one point taken for granted and call ordinary, may be gone the next moment.

We ourselves undergo change over time. Both through age and sometimes in the way we style ourselves.

Therefore, next time you go out and visit a coffee shop or park that you frequent. Snap a photo. Future you will thank you later for the memory.

How do we know when to buy and sell positions then? One primary goal is to avoiding buying high and selling low. There’s a simple procedure we can follow that guarantees the opposite. That procedure is called rebalancing.

The process of rebalancing assumes that you have some fixed goal distribution of your assets. For example, let’s say you want your portfolio to consist of 80% stocks and 20% bonds. Over time as valuations fluctuate, your portfolio will fall out of balance. For example, it can be 87% stocks and 13% bonds.

There could be a couple reasons this happened. The valuation of the stock positions could’ve risen and/or the valuation of the bond positions could’ve fallen. To get the portfolio back to the 80/20 split, we can sell off some of the stock positions and/or buy some more of the bond positions. Given the circumstances that we’re rebalancing in, it means we’re likely selling the stock positions at its high and/or buying bond positions at its low.

So when should one rebalance? There’s not a hard-fast rule and it’s dependent on taxation and trading fees, but generally people either follow a schedule (1-4 times a year) or rebalance when their distribution strays by more than say 5%.

Diversification is the only free lunch in investing

To diversify with individual securities, one would need to buy equities and bonds from many organizations of differing sectors and capital size. For most people this can turn out to be prohibitively expensive.

Mutual funds and ETFs address this problem by pooling together money from multiple parties and purchasing securities based off some stated objective. Depending on the holdings, the fund may be considered well diversified. Some funds don’t require minimum investments which makes it easier for people to get started.

Not all funds are created equal however. Before investing in any fund, you should be informed and take a look at their prospectus to get an idea of their objectives and current assets before investing.

One way to characterize a fund is if it has an active or passive investment style. The goal of an active fund is to beat overall market trends. This means that the fund manager is often buying and selling securities to take advantage of price fluctuations. Though since it’s neigh impossible to predict market conditions, it is common for active funds to not beat the overall market.

Passive funds take a more buy-and-hold type of mentality. The core assumption is that market valuations increase over time. Therefore if a fund holds a given security over a long period of time, then is likely to result in some positive return.

Though what specific securities do passive funds buy and hold? One common category of passive funds is indexing. This is where the fund holds securities in the same weighted proportion as reported by some index. For example the most commonly known index, the S&P 500, consists of 500 of the largest stocks in the United States weighted by their market value.

For large-cap indexes (those that contains large companies), there is little turnover in the companies within the index. This makes it perfect for passively managed funds. The S&P 500 has a historical turnover rate of 4.4%.

Recall the saying about diversification being a free lunch. What would be considered one of the most diversified funds? If we’re looking at solely the US, we can consider the Dow Jones US Total Stock Market Index. This contains a market cap weighting of 4193 companies at this time of writing. That’s 8 times larger than the S&P 500.

Therefore, given the assumption that equities on average increase in value over time. A well diversified passive fund can be a great investment. These funds generally have lower expense ratios (fees) as not as much human attention is needed to maintain the portfolio.

A risk that’s unique to some index funds is tracking error. A fund may not have enough money to purchase all securities at the reported proportions. Therefore, some will perform statistical techniques to approximate the index. Also since index funds are bound by the index they’re following, the performance will be correlated and the fund may not take advantage of price fluctuations or attempt to lower drawdowns.

Disclaimer: I am not an investment analyst and this does not constitute investment advice. This information is what I’ve learned recently and I encourage you to read up on these concepts and look at the prospectus before investing.

This is often used to simplify circuit design. Instead of having a custom chip for every other operator, it’s generally more cost effective to only have two distinct chips but use more of them. But what if we want to find other combinations?

In formal terms, we’re looking for other functional complete sets.

In this post, we’ll provide an overview for finding these sets. Finding some examples of functional complete sets along the way.

The intuition here is to know that our set $\mathcal{S}$ is functionally complete if we can replicate the other operators using the ones only within $\mathcal{S}$. Let’s refresh ourselves in what the complete set of operators are: $$ \{\mathtt{AND}, \mathtt{OR}, \mathtt{NOT}, \mathtt{IF}, \mathtt{IFF}\} $$ The first step would normally be to remove one operator from the above set and see if you can derive it using the other operators. For brevity of this post though, we’ll skip sets of length four and look directly at sets of length three.

Example 1: Show that $\mathcal{S} = \{\mathtt{AND}, \mathtt{OR}, \mathtt{NOT}\}$ is functionally complete.

In this example, we’re missing both $\mathtt{IF}$ and $\mathtt{IFF}$. Using the operators of $\mathcal{S}$ we can define the missing operators to be (in prefix notation): $$ \begin{align*} (\mathtt{IF}~A~B) &\equiv (\mathtt{OR}~(\mathtt{NOT}~A)~B) \\ (\mathtt{IFF}~A~B) &\equiv (\mathtt{AND}~(\mathtt{OR}~A~B) (\mathtt{OR}~(\mathtt{NOT}~A)~ (\mathtt{NOT}~B))) \end{align*} $$ For now, it’ll be beyond the scope on how to find the correct combination of operators within $\mathcal{S}$.

Since we’re able to represent the missing operators in $\mathcal{S}$ the set is functionally complete.

Example 2: Show that $\mathcal{S} = \{\mathtt{NOT}, \mathtt{AND}, \mathtt{IF}\}$ is functionally complete.

We have to show that we can represent the missing operators $\mathtt{OR}$ and $\mathtt{IFF}$ using only elements of $\mathcal{S}$. $$ \begin{align*} (\mathtt{IFF}~A~B) &\equiv (\mathtt{AND}~ (\mathtt{IF}~ A~ B)~ (\mathtt{IF}~ B~ A)) \\ (\mathtt{OR}~ A~ B) &\equiv (\mathtt{NOT}~ (\mathtt{AND}~ (\mathtt{NOT}~ A)~ (\mathtt{NOT}~ B))) \end{align*} $$ This shows that the set $\mathcal{S}$ is functionally complete.

At this point we know that $\{\mathtt{AND}, \mathtt{OR}, \mathtt{NOT}\}$ and $\{\mathtt{NOT}, \mathtt{AND}, \mathtt{IF}\}$ are functionally complete. Now comes the exciting part, finding functionally complete sets with only two operators.

• What knowledge can we derive from $\{\mathtt{AND}, \mathtt{OR}, \mathtt{NOT}\}$?

Let’s consider all possible sets of length two: $$ \{\mathtt{AND}, \mathtt{NOT}\} \\ \{\mathtt{AND}, \mathtt{OR}\} \\ \{\mathtt{OR}, \mathtt{NOT}\} $$ Example 3: Is ${\mathtt{AND}, \mathtt{NOT}}$ functionally complete?

We need to show that we can derive $\mathtt{OR}$. $$ (\mathtt{OR}~ A~ B) \equiv (\texttt{NOT}~ (\texttt{AND}~ (\texttt{NOT}~ A)~ (\texttt{NOT}~ B))) $$ This means using only $\mathtt{AND}$ and $\mathtt{NOT}$ we can represent $\{\mathtt{AND}, \mathtt{OR}, \mathtt{NOT}\}$ which is functionally complete. Therefore, $\mathcal{S} = \{ \mathtt{AND}, \mathtt{NOT}\}$ is functionally complete.

Example 4: Is $\{\mathtt{AND}, \mathtt{OR}\}$ functionally complete?

No, because we cannot represent $\mathtt{NOT}$ using both $\mathtt{AND}$ and/or $\mathtt{OR}$. In a future iteration of this point I might describe why this is the case.

Example 5: Is $\mathcal{S} = \{\mathtt{OR}, \mathtt{NOT}\}$ functionally complete?

We need to show that we can derive $\mathtt{AND}$. $$ (\mathtt{AND}~ A~ B) \equiv (\texttt{NOT}~ (\texttt{OR}~ (\texttt{NOT}~ A)~ (\texttt{NOT} ~B)) $$ Therefore, $\mathcal{S}$ is functionally complete.

Similarly, we can look at $\{\mathtt{NOT}, \mathtt{AND}, \mathtt{IF}\}$ and consider pairs of operators that might still be functionally complete.

Example 6: Is $\{\mathtt{AND}, \mathtt{IF}\}$ functionally complete?

No, because you cannot represent $\mathtt{NOT}$.

Example 7: Is $\{\mathtt{NOT}, \mathtt{IF}\}$ functionally complete?

Here we’re missing $\mathtt{AND}$, so if we can derive it using the two operators, then the set is functionally complete. $$ (\mathtt{AND}~ A~ B) \equiv (\mathtt{NOT}~ (\mathtt{IF}~ A~ (\mathtt{NOT}~ B))) $$

There are no single operators that are functionally complete in this example. Though the above gives somewhat of a procedure for finding all the sets of functionally complete operators. Here’s a summary:

1. Start with your base set of operators $\mathcal{S} = {x_1, x_2, \dots, x_n}$
2. Remove $x \in \mathcal{S}$ such that $\mathcal{S}^\prime = \mathcal{S} - {x}$
3. Derive $x$ using the other operators in $\mathcal{S}^\prime$.
4. If possible, add $\mathcal{S}^\prime$ to $\mathcal{S}_{n - 1}$ the set of functionally complete operators of length $n - 1$
5. Repeat steps (2) and (3) on each $\mathcal{S}^\prime \in \mathcal{S}_{n - 1}$ to create $\mathcal{S}_{n - 2}$
6. Keep going until no new functionally complete sets are found.

I know that this is a high level article and that there’s some details that are left out, but please feel free to reach out if you have any questions.

1. Form a new government agency charged with licensing large AI models, and allow it to revoke it for companies whose don’t comply.
2. Create a set of safety standards for AI models.
3. Require independent audits, by independent experts.

Okay I can get behind the idea behind “safety standards” but what about this new government agency? Why would Sam Altman the CEO of OpenAI want the creation of a new government agency?

Some (including me) speculate that he’s trying to create a regulatory moat for OpenAI.

We can see where this goes. There are thousands of AI startups trying to sell models of various kinds, the idea is that they all need to register and be regularly monitored by a government agency?

This would probably lead to long wait times for these AI licenses with a few companies holding the majority of the licenses. Startups funded through Y-combinator rarely go beyond two years.

It’s a great way of trying to stamp out competition really. Currently it’s highly expensive to train one of these large language models (LLMs) but would that always be the case? In the inevitable future where other companies have the compute to generate one of these models, OpenAI needs to find a way to keep its market share.

Now you might think I’m being to pessimistic. What if Sam Altman is trying to think about the best for society? If that’s the case, then I don’t think he would threaten to not provide AI services in the EU due to their stringent regulations.

He’s walked back the comment since then, but let’s take a brief look at what the EU regulations look like.

• Most current version I’m aware of: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai]
• Summary in Presentation Form: https://www.ceps.eu/wp-content/uploads/2021/04/AI-Presentation-CEPS-Webinar-L.-Sioli-23.4.21.pdf

Generative models are considered at the time of writing to be a “high-risk” AI model. The additional regulations the EU imposes on these companies are:

• adequate risk assessment and mitigation systems;
• high quality of the datasets feeding the system to minimise risks and discriminatory outcomes;
• logging of activity to ensure traceability of results;
• detailed documentation providing all information necessary on the system and its purpose for authorities to assess its compliance;
• clear and adequate information to the user;
• appropriate human oversight measures to minimise risk;
• high level of robustness, security and accuracy.

I feel like most of those bullet points fall under Altman’s “safety standards for AI models”. Though I think what he really has an issue with is the data privacy requirements set forth by the EU. In fact this is an often criticized part of these models trained on Internet data:

• Personally identifiable information
• Copyright information
• Conspiracy/Misinformation
• etc.

My personal opinion is that these models should not be trained on the above data, but this isn’t a new discussion. More recently, the issue of copyright has been brought up with generative code products such as GitHub Copilot. I am interested to see how it makes its way through the US court systems here.

Another question I think we need to answer as a society is whether to hold companies accountable for defamation. There are already instances where ChatGPT incorrectly accused a law professor of sexual harrasment. With the push to replace search with these automatically generated responses, we can see how misinformation may be propagated and then further reinforced by these models.

I don’t want to say that it’s a bad idea for congress to regulate AI. I’m mostly suggesting that perhaps we shouldn’t call upon people who hold direct financial interest to come up with the regulations that will apply to themselves. That might lead these actors to bring attention to certain issue while avoiding others…

1. /wp-login.php

This is the login page of a Wordpress website. Since this blog engine powers 43% of the Internet, it’s not surprising that this is a common target. Sadly for the bots, I don’t run this website using Wordpress.

2. xmlrpc.php

This one I haven’t heard of before until looking it up. I know XML is a data format, and RPC means remote procedure call, but what is the attacker trying to exploit? Again it’s Wordpress. It seems that this is some API gateway that Wordpress provides to connect with mobile devices, provide pingbacks, and others.

3. /api/v1/instance

Through the power of search this seems to be a API call to Mastodon! This specifically grabs generic information about the instance such as the number of users, number of statuses, restrictions, etc. I’ve considered at some point running a Mastodon instance, but maybe it’s better to leave it to the pros :)

4. /.env

It seems that the Javascript community likes using a .env file to keep environmental variables that hold the secrets of your application. Yikes! Make sure that you’re blocking this if you have it!

5. /inbox

Given #3, I feel like this is ActivityPub related. Though looking at how the actors are usually structured it’s generally /username/inbox. Maybe it’s related to email servers instead? I’m unsure.

6. /status.php

I’m not sure what this is. Doesn’t seem to be Wordpress related. Maybe the attacker is hoping to get the output of phpinfo() in those “getting started with PHP” tutorials?

James Oswald suggests that perhaps it’s the firewall status page in pfsense.

7. /.git/config

I can see a situation where someone has a git repository of their website on the server itself and they push to it. Personally, I rsync the generated HTML files. Generally the config will contain the URLs of remote repositories and other settings. Not entirely sure what’s sensitive, but maybe someone can let me know.

There you have it! The top automated attacks made to my website. If you have any additional information on any of these URL patterns please get in touch. I am curious what these bots are trying to do with the response of each of these queries.

While this would likely get you more views, the idea is that the personal website then loses some personal touch. Some argue that personal websites should not be driven by any agenda other than yourself. No data, no click-bait titles, etc.

Of course this goes into why you created your website to begin with? For myself, this website’s primary purpose is for me. I often forget how to perform many tasks and use this website as a reference guide.

Though one of my core principles is in sharing knowledge. An important secondary goal of mine is that others find my website useful in some way. The best way to achieve this is to carefully craft each and every one of my blog posts.

Personally, I find this difficult. Striving for perfection in a post often leads to its incompletion. In fact, it’s not uncommon for me to sit on a topic for multiple months (sorry OpenStreetMap) only to then force myself to write something on the topic to remove it from my backlog.

Now I could revisit older posts and spruce them up, but over the years I’ve written more than 300 blog posts covering a variety of technical content. It would be a slog to comb over each individual one.

This is where analytics come into play for me. This allows me to identify popular blog posts and spruce up those instead. It does mean that unpopular blog posts are unlikely to get corrected¹. However since time is limited, it’s nice to know that I’m improving a post that many look at and potentially rely on.

Also throughout the years, I’ve edited the engine that creates these posts. This means that there are some broken links in the Internet that gets surfaced every now and again in my error logs :)

I understand that to some looking at an analytics dashboard can cause fixation. That doesn’t mean, however, that we should dismiss analytics entirely. It’s a tool, and if there’s some benefit and it’s right for you, then we should make use of it.

I wrote a script that can selectively upload blog posts generated from a Hugo website onto Medium. This post will go into further detail each of the following steps that make it happen:

• Creating labels on the YAML frontmatter
• Outputting a JSON feed to make it easier to create scripts
• Writing a python script to ingest the JSON feed and upload new specified posts onto Medium
• Updating the YAML frontmatter with information from Medium

Step 1 Define Labels: This advice not only goes for Medium but any other site you want to syndicate to. Within each of the content files within Hugo, there’s generally a frontmatter section that holds the metadata such as title, date, and tags. Within the YAML frontmatter, I created two new fields:

• medium_enabled: Whether or not I want this post to be uploaded to Medium
• medium_post_id: The post identifier used within Medium, empty if non-existent.

We don’t need to include these two new fields in every new post we create. Instead in Step 2, we’ll consider some sane defaults. Though it could be useful to include some default values within the archetype file so that you don’t forget they exist.

For example:themes/themeName/archetypes/default.md

---
title: "{{ replace .TranslationBaseName "-" " " | title }}"
date: {{ dateFormat "2006-01-02T15:04:05Z07:00" .Date }}
draft: true
tags: []
medium_enabled: false
---

Step 2 Setup JSON Feed: Following the Hugo template system, I created a item.json.json, list.json.json, and single.json.json. For this post, the first two are important.

list.json.json

{
    "version": "https://jsonfeed.org/version/1.1",
    "title": "{{ if eq  .Title  .Site.Title }}{{ .Site.Title }}{{ else }}{{ with .Title }}{{.}} | {{ end }}{{ .Site.Title }}{{ end }}",
    "home_page_url": "{{ .Site.BaseURL }}",
    "feed_url": "{{ .Permalink }}",
    "description": "{{ .Description }}",
    "icon": "{{ .Site.BaseURL }}img/{{ .Site.Params.avatar }}",
    "language": "en-US",
    "authors": [
        {{with $.Site.Author.name }}
            { "name": "{{ . }}" }
        {{ end }}
    ],
    "items": [
        {{ range $index, $page := .Pages }}{{ if ne $index 0 }},{{ end }}
            {{ .Render "item"}}
        {{ end }}
    ]
}

My list.json.json follows the JSON Feed specification. The idea here is that this file will contain a list of blog posts within the items attribute. For each item, we have the next template file that defines how it’s structured.

item.json.json

{
    "id": "{{ .Permalink }}",
    "url": "{{ .Permalink }}",
    "title": {{ .Title | jsonify }},
    "authors": [
        {{with $.Site.Author.name }}
            { "name": "{{ . }}" }
        {{ end }}
    ],
    "content_html": {{ .Content | jsonify }},
    "date_published": "{{ .Date.Format "2006.01.02" }}",
    "tags": {{ .Params.tags | jsonify }},
    "_syndication": {
        "medium": {
            "enabled": {{ .Params.medium_enabled | default "false" }},
            "post_id": {{ .Params.medium_post_id | jsonify }}
        }
    }
}

Notice the _syndication attribute. Attributes beginning with an underscore are ignored by JSON feed readers. Within this attribute, we can store our enabled and post_id attributes. If the post does not specify the relevant attributes in the frontmatter YAML, we’ll assume false for enabled and null to for post_id.

Finally for this step we need to tell Hugo to generate this JSON feed. Within your website’s config.toml:

# ...

[outputFormats.json]
  name = "json"
  mediaType = "application/feed+json"
  baseName = "index"
  isPlainText = false
  rel = "alternate"
  isHTML = false
  noUgly = true
  permalinkable = false

[outputs]
    section = ["json", "html", "rss"]
	# ...

Step 3 Ingest Feed and Syndicate via External Script: Now that we can store the metadata on whether or not we want to syndicate a particular post and its relevant medium id if it has already been, we can create an external script that ingests this data and uploads if necessary.

We’ll use Python for this task. The core loop of the file will look like the following:

# Generate the necessary feed files
subprocess.run(['hugo'], check=True)

# Grab blog's feed
data = ""
with open("public/blog/index.json", "r", encoding="UTF-8") as feed_file:
    data = feed_file.read()
feed_data = json.loads(data)

# Go through each post and check syndication status
for post in feed_data['items']:
    medium_enabled = post['_syndication']['medium']['enabled']
    medium_post_id = post['_syndication']['medium']['post_id']
    if medium_enabled and medium_post_id is None:
        # Syndicate.....
        medium_result = syndicate_post(post)
        update_front_matter(post, medium_result)
        time.sleep(60) # Throttle to not hit rate limit

The last couple steps were necessary for Hugo to produce the JSON feed needed which contains the appropriate syndication fields. Now the magic is within the syndicate_post method.

For the magic we need to make use of the Medium API. You’ll need to know both your access token and your authorId. Your author id can be displayed publically, for example mine is 18bad95d2020608a45ef502ef0db83d2cad2e28886d8d3eeef71a6bd089fc2a4e. However, your access token must be kept private. Otherwise, others can impersonate you. For my script, I put my access token within a secret file that I don’t commit to the repository.

# Grab secret
medium_secret = ""
with open("secrets/medium.secret", "r") as secret_file:
    medium_secret = secret_file.read().strip()

The secret is used within the request headers

request_headers = {
        "Host": "api.medium.com",
        "Authorization": f"Bearer {medium_secret}",
        "Content-Type": "application/json",
        "Accept": "application/json",
        "Accept-Charset": "utf-8",
}

Following the API documentation we then create a JSON object that represents our new blog post with its corresponding metadata.

# Structure API Request
blog_post = dict(
    title=post['title'],
    contentFormat="html",
    content=post['content_html'],
    tags=post['tags'],
    canonicalUrl=post['url'],
    publishStatus="public",
    license="all-rights-reserved",
    notifyFollowers=False
)

You’re welcome to change the publish status, license, and follower notification as you see fit. Afterwards, following the API documentation we send a POST request to the Medium API

# Send Request
conn = HTTPSConnection("api.medium.com")
conn.request(
    "POST",
    f"/v1/users/{AUTHOR_ID}/posts",
    json.dumps(blog_post).encode("utf-8"),
    request_headers
)

We’ll need to hold onto the response from the website as that’ll contain our post id.

# Check and parse response
response = conn.getresponse()
if response.status != 201:
    raise Exception(f"Medium API rejected the request with response code {response.status}")
medium_response = json.loads(response.read().decode('utf-8'))
conn.close()

Step 4 Update Frontmatter: Now that we have the post id within the Medium response, we can look towards the original markdown file that contains our content and modify the frontmatter so that the medium_post_id field is populated.

First we need to get the appopriate markdown file

# Figure out the file path of the post
ORIG_URL = urlparse(post['id'])
file_path = "content" + ORIG_URL.path[:-1] + ".md"

Using the Frontmatter Python library, we can read the existing YAML frontmatter and edit the post id with what we received from Medium

# Read existing frontmatter and edit the post id
item = {}
with open(file_path, "r", encoding="UTF-8") as content_file:
    item = frontmatter.load(content_file)
    item['medium_post_id'] = medium_data['data']['id']

The library also helps us edit the file with updated fronmatter

# Write out new frontmatter
with open(file_path, "w", encoding="UTF-8") as content_file:
    content_file.write(frontmatter.dumps(item))

Conclusion: This constitutes my somewhat generic framework for how you can selectively syndicate content within a Hugo website. To summarize, we do this by keeping track of the syndication through frontmatter additions and a JSON feed. We then created an external script that ingests this feed and syndicates if necessary, finally updating the original frontmatter with the server response.

Downsides to this approach is that the external script occurs outside of the hugo call. This adds an extra layer of complication when managing your website. However, keeping it all within Hugo is not only difficult but inflexible. The way I envision this is that Hugo is a database of sorts that keeps track of the syndication status and external scripts operates over this database.

How do you go about syndicating content to other websites? Is your approach different from mine? Please get in touch and share :)

Relevant files from my website repository:

• https://github.com/Brandon-Rozek/website/blob/52b47f1311e7bf98b97c69bd0e838d8e3cdf8392/scripts/syndicate_medium.py
• https://github.com/Brandon-Rozek/website/blob/52b47f1311e7bf98b97c69bd0e838d8e3cdf8392/config.toml
• https://github.com/Brandon-Rozek/website-theme/blob/fb42d6b950b1d7752e199dc16fd0d39292a16c93/layouts/_default/item.json.json
• https://github.com/Brandon-Rozek/website-theme/blob/fb42d6b950b1d7752e199dc16fd0d39292a16c93/layouts/_default/list.json.json

Example

We start by defining a Person class that holds a name

class Person:
    def __init__(self, name):
        self.name = name

In order to put it within a set, we need to define a hash method.

class Person:
    def __init__(self, name):
        self.name = name
    def __hash__(self):
        return hash(("Person", self.name))
    
people = {
    Person("Brandon"),
    Person("Clare"),
    Person("James"),
    Person("John")
} 

The property we’ll focus on in this post is set membership. In other words, does the set contain a specified object?

print(Person("Brandon") in people) # False

This might seem strange since we can clearly see a Person("Brandon") in the set! The problem is that we didn’t override the __eq__ method within the class. Currently it only considers two Person objects the same if they share the same reference.

To test Person equivalence in a more intuitive way, we’ll say that two Person objects are equivalent if they share the same name.

class Person:
    def __init__(self, name):
        self.name = name
    def __hash__(self):
        return hash(("Person", self.name))
   	def __eq__(self, other):
        return isinstance(other, Person) and self.name == other.name

print(Person("Brandon") in people) # True

Now let’s mutate the names so they all start with Awesome

for p in people:
    p.name = "Awesome " + p.name

Let’s test for membership again

print(Person("Awesome Brandon") in people) # False
print(Person("Brandon") in people) # False

What happened? Why is both Awesome Brandon and Brandon not in the set? This has to do with the way that the sets keep track of the objects within them.

When an object is inserted, it’s placed into bins according to their hash. Therefore, when checking for set membership, we look to see if there are items within that hash bin.

Originally when we added Person("Brandon") the following hash is computed:

print(hash(("Person", "Brandon"))) # 8879556234447129293

The object Person("Brandon") is then placed in bin 8879556234447129293.

When we mutate the object, we don’t remove the item from the set and add it back in with a recomputed hash. Instead the mutated object stays within the same hash bin.

So what happens then when we ask if Person("Awesome Brandon") is within the set? First, it checks to see if there are any non-empty bins with the same hash.

print(hash(("Person", "Awesome Brandon"))) # 8677222781184413080

Our set contains no items within the 8677222781184413080 hash bin. Therefore, the set membership will immediately fail disregarding completely the __eq__ method we define.

Aside: Another implicit contract we have is that for a given hash method, two equivalent objects always produce the same hash. What happens when that’s not the case?

import random

class Person:
    def __init__(self, name):
        self.name = name
    def __hash__(self):
        # Add randomness!
        return hash(("Person", self.name, random.randint(0, 5)))
    def __eq__(self, other):
        return isinstance(other, Person) and self.name == other.name
    def __str__(self):
        return f"Person(name={self.name})"


people = {
    Person("Brandon"),
    Person("Clare"),
    Person("James"),
    Person("John")
}

print(Person("Brandon") in people) # False

Again, the set membership returns false disregarding our __eq__ method since the __hash__ outputs don’t match.

Workaround: If we can’t mutate the items in a set directly, what should we do instead? The solution here is to remove the old items and insert newly created items with the modified data.

people = {
    Person("Brandon"),
    Person("Clare"),
    Person("James"),
    Person("John")
}

to_remove = set()
to_add = set()
for p in people:
    to_remove.add(p)
    to_add.add(Person("Awesome " + p.name))

people = (people - to_remove).union(to_add)

print(Person("Awesome Brandon") in people) # True

Almost nothing is truly free. Outside of technology, it costs money to keep clean and maintain parks. Similarly, a server needs to be available to respond to a search query. In fact these are often multiple servers, each that run on electricity and operate within an air conditioned data center.

There is a common saying among privacy activists:

If you are not paying for the product, then you are the product.

This has some truth to it. A company can only operate at a loss for so long therefore the easiest way to monetize is through advertisements. For an advertisement to be effective, the company has to collect information about your behavior patterns.

Expectation of free can drive out competition. Who can compete with a company that is offering their services for free? Knowing this, some large companies often operate at a loss for a certain time period. Why? In order to drive out competition. Google at one point offered unlimited photo storage as long as you saved the photo as “High Quality”. This has since changed in June 2021. This marketing tactic is known as the “bait and switch”. It relies on the fact that it’s often effort for people to change services and that they would instead eat the cost once it’s introduced later on.

I argue that the lack of competition hinders innovation. Currently Google reigns supreme over search. There’s some competition from Bing and other smaller search engines, though the market share isn’t enough for Google to try to improve the product.¹

In a world where large tech companies offer expensive services for free. It is hard for a small company to bootstrap and provide a competitive take in the area.²

What about open source? It turns out that there’s often a way to financially support these projects. The ways are varied but they include paying for support, hosting, or even solely donations. I argue that if no funding model exists, then it’s likely for the project to be unsustainable. At the end of the day humans are behind it all and we all require to have our needs met.

Though another interesting component of open source is that it is often driven by the needs of people. Often times when a developer contributes a feature it’s because they want to use the feature themselves. In that way they provide value to the project. I encourage people to contribute to the projects they depend on in some way (if possible), whether financially, through documentation, or code.