juliu.is

Tidying up your $HOME with Nix

Fri, 17 Feb 2023 12:00:00 GMT

I have four machines I often use:

a 2022 M1 Macbook, my work laptop
a 2017 Intel Macbook, my personal laptop
a 2011 iMac, my… music player
a Raspberry Pi 3, my ad blocker

I’d like to have the same development environment on all of these machines, mostly for my bash, git, vim, and tmux configs. I’d also like to be able to use all the nice little utilities I’m used to, such as:

fzf to fuzzy search everything
exa as a replacement for ls
ripgrep to search for stuff
zoxide as a smarter cd
delta for viewing diffs

As many of us do, I used to have a repo with a scrappy script that would symlink all my configuration files and install a list of packages via homebrew or apt.

Except that it would often break. For example, I would install a vim plugin that depends on Ruby: at install time, the plugin would be dynamically linked against the existing version of Ruby. Whenever I updated Ruby, the vim plugin would then start crashing. The fix? Reinstall the vim plugin so it would be dynamically linked against the correct version of Ruby. I had to do this every time I updated Ruby. Ugh!

Oh, remember my old iMac from 2011? Apparently, it’s so old that homebrew stopped supporting it. Many times you won’t be able to find bottled versions of programs and you’ll have to compile them from source, which takes ages. And sometimes the compilation crashes with random errors. It seems I’m out of luck and I should give up, even though it’s a perfectly functioning machine. Should I try to install Linux on it?

It would be great if there was a way to provision a development environment that worked on all of my machines, even the ones which are a bit obsolete.

Oh, but there is

I’ve already written about Nix, so feel free to go and read that one first. Don’t feel like it? Fine.

Let’s install it:

$ sh <(curl -L https://nixos.org/nix/install)

This should work now:

$ nix-env --version
nix-env (Nix) 2.13.2

I’m going to make you enable a feature called Nix flakes, which we’ll use to lock the versions of our dependencies.

$ mkdir -p ~/.config/nix

$ cat <<EOF >> ~/.config/nix/nix.conf
experimental-features = nix-command flakes
EOF

In order to get Nix to pick this up we need to restart the Nix daemon:

$ sudo launchctl kickstart -k system/org.nixos.nix-daemon

Great! Now we can create the folder to store all our configs:

$ mkdir -p ~/.config/nixpkgs && cd ~/.config/nixpkgs

We are going to create a file called flake.nix, where we’ll define some inputs and some outputs. The inputs are going to be:

nixpkgs, the repository of all nix packages
home-manager, the tool to manage your env

What is going to be the output? Our dev environment :)

{
  description = "Home Manager configuration";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { nixpkgs, home-manager, ... }: let
    arch = "x86_64-darwin"; # or aarch64-darwin  in {
    defaultPackage.${arch} =
      home-manager.defaultPackage.${arch};

    homeConfigurations.YOUR_USER = # REPLACE ME      home-manager.lib.homeManagerConfiguration {
        pkgs = nixpkgs.legacyPackages.${arch};
        modules = [ ./home.nix ];
      };
    };
}

Just be sure to choose the right architecture for your system, x86_64-darwin for Intel Macs and aarch64-darwin for ARM Macs. Later I’m going to explain how to support multiple architectures. Oh, and also make sure to put your username instead of YOUR_USER 🙏

You can see that we include a file called home.nix, which doesn’t exist yet. Let’s create it:

{ pkgs, ... }: {
  home.username = "YOUR_USER"; # REPLACE ME  home.homeDirectory = "/Users/YOUR_USER"; # REPLACE ME  home.stateVersion = "22.11";
  programs.home-manager.enable = true;
}

Before we can try it out, flakes require all config files to be in a Git repository:

git init && git add .

Perfect, now we’re ready to go:

nix run . switch

This command will install nixpkgs-unstable and home-manager, lock their versions in a flake.lock file, and enable the first generation of our dev environment. When it completes, we can check that it has been installed correctly:

$ home-manager --version
22.11

Hurray! 🎆

Adding packages

Let’s modify our home.nix so that it looks like this:

{ pkgs, ... }: {
  home.username = "arkham";
  home.homeDirectory = "/Users/arkham";
  home.stateVersion = "22.11";
  programs.home-manager.enable = true;

  home.packages = [    pkgs.sl  ];}

If we run home-manager switch, we’ll be greeted with an output like this one:

$ home-manager switch
Starting Home Manager activation
Activating checkFilesChanged
Activating checkLaunchAgents
Activating checkLinkTargets
Activating writeBoundary
Activating copyFonts
Activating installPackages
replacing old 'home-manager-path'
installing 'home-manager-path'
building '/nix/store/vj49hfv01fdgwd3igl9pi1kz4rlmj6x5-user-environment.drv'...
Activating linkGeneration
Cleaning up orphan links from /Users/arkham
Creating profile generation 2
Creating home file links in /Users/arkham
Activating onFilesChange
Activating setupLaunchAgents

Now we can run:

$ sl

And we’ll see something like this chugging along:

CHOO CHOO! 🚂 Notice that we downloaded a precompiled version of sl, which is very similar to the bottled programs that homebrew provides. No compilation needed! You can search here for any package you’d like to install.

If you’re like me and love trains, you might already be satisfied. If you want to see how home-manager manages both programs and configuration files, keep on reading.

Let’s go back inside ~/.config/nixpkgs and copy our git config:

$ cd ~/.config/nixpkgs

$ cp -L ~/.gitconfig gitconfig

And change home.nix to look like this:

{ pkgs, ... }: {
  home.username = "arkham";
  home.homeDirectory = "/Users/arkham";
  home.stateVersion = "22.11";
  programs.home-manager.enable = true;

  home.packages = [
    pkgs.sl
  ];

  programs.git = {    enable = true;    includes = [{ path = "~/.config/nixpkgs/gitconfig"; }];  };}

You’ll see that we specified that we want programs.git to be enabled. In this way home-manager can install the program and manage its configuration at the same time. As a rule of thumb, if it’s supported by home-manager I’ll install the program using programs.<program>. If not, I’ll add it to home.packages.

If we run home-manager switch -b bak, we’ll notice some things:

the old config has been backed up as ~/.gitconfig.bak
but now ~/.gitconfig doesn’t exist anymore
yet somehow, still, our git settings are working

This is because home-manager stores the git configuration inside ~/.config/git/config. If we look at that path we’ll be surprised:

$ ls -al ~/.config/git/config
lrwxr-xr-x 1 arkham staff 81 Feb 12 00:01 /Users/arkham/.config
/git/config -> /nix/store/mcnp2v50ndgxvkis1labl6784r8qwbj8-home
-manager-files/.config/git/config

Our config file is read-only and is a link to an obscure path in the /nix/store. Home-manager does this to ensure that you don’t accidentally edit your generated config files, since your changes would be overridden next time you run home-manager switch.

Ready to try something else? Let’s remove the homebrew copy of git:

$ brew uninstall git

You’ll notice that git is still working:

$ which git
/Users/arkham/.nix-profile/bin/git

$ ls -al $(which git)
lrwxr-xr-x 69 root  1 Jan  1970 /Users/arkham/.nix-profile/bin/
git -> /nix/store/ampvjly5y69drxcxyipjfzk2xghwn94i-home-manager
-path/bin/git

Oh god, these symlinks are EVERYWHERE.

Bring on the heat

Now we’re going to do the unthinkable. We’re going to move our precious shell configuration inside home-manager. Yes.

First let’s make sure we are in the right directory:

$ cd ~/.config/nixpkgs

I’m going to show you how to do it for Bash and Zsh. You can choose one:

$ cp -L ~/.bashrc bashrc
$ cp -L ~/.bash_profile bash_profile

$ cp -L ~/.zshrc zshrc

Now we can change our home.nix:

{ pkgs, ... }: {
  home.username = "arkham";
  home.homeDirectory = "/Users/arkham";
  home.stateVersion = "22.11";
  programs.home-manager.enable = true;

  home.packages = [
    pkgs.sl
    pkgs.bashInteractive # don't ask me why  ];

  programs.git = {
    enable = true;
    includes = [{ path = "~/.config/nixpkgs/gitconfig"; }];
  };

  programs.bash = {    enable = true;    profileExtra = builtins.readFile ./bash_profile;    initExtra = builtins.readFile ./bashrc;  };  programs.zsh = {    enable = true;    initExtra = builtins.readFile ./zshrc;  };}

Ready? Go! home-manager switch -b bak 😱

If you open a new shell, you’ll see (surprise!) that everything is still working, and all your customizations are still there. The only difference is that those configuration files are now symlinks to the Nix store.

Note that by default your terminal will still run the shell provided by the OS. In my terminal, I customized the starting shell to be the one provided by Nix:

But what is the advantage of letting home-manager take care of those files? Aren’t those glorified symlinks after all? Isn’t it all pointless?

Let me show you why this is VERY NICE.

Let’s change our home.nix again:

{ pkgs, ... }: {
  home.username = "arkham";
  home.homeDirectory = "/Users/arkham";
  home.stateVersion = "22.11";
  programs.home-manager.enable = true;

  home.packages = [
    pkgs.sl
    pkgs.bashInteractive
  ];

  programs.git = {
    enable = true;
    includes = [{ path = "~/.config/nixpkgs/gitconfig"; }];
  };

  programs.bash = {
    enable = true;
    profileExtra = builtins.readFile ./bash_profile;
    initExtra = builtins.readFile ./bashrc;
  };

  programs.zsh = {
    enable = true;
    initExtra = builtins.readFile ./zshrc;
  };

  programs.fzf = {    enable = true;  };  programs.zoxide = {    enable = true;  };  programs.direnv = {    enable = true;    nix-direnv.enable = true;  };}

fzf, zoxide, and direnv are programs that need to be hooked in your shell. So normally you’d have to install the program and then edit your shell config to load them up. Instead, home-manager will do that for you now. Neat, right?

Have you noticed that last line that says nix-direnv.enable = true? That single line will cache your direnv activated nix shells and make them super fast. That’s the advantage of using home-manager, you get all this stuff for free, with only one line of code. Do you want a nicer diffing experience in git? Enable this option in your programs.git block and you’re done. Do you want fzf and tmux to work nicer together? Enable this other option and you’re set.

This page is very useful to look at which programs are included in home-manager and which options are supported. You can view the same information locally by running:

$ man home-configuration.nix

You can also check my Nix configuration on Github for inspiration.

I like this. What do I do next?

You can start by moving more and more packages from homebrew to home-manager. Take it easy, as they can coexist peacefully. For example, you could check out:

how to port your git configuration
how to port your vim configuration
how to port your bash configuration

How do I update all my packages?

Go inside ~/.config/nixpkgs and run

$ nix flake update

$ home-manager switch

How do I manage multiple machines with different architectures?

Check out this section of my flake.nix. You can create multiple blocks where homeConfigurations is followed by YOUR_USER@YOUR_HOSTNAME. Remember you can always find your user with whoami and your hostname with hostname. As you can see, I’ve extracted the shared configuration inside a withArch helper function.

How do I manage multiple machines with different setups?

The easiest way is to have multiple home.nix files, one for each machine:

{
  description = "Home Manager configuration";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { nixpkgs, home-manager, ... }: {
    defaultPackage = {
      "x86_64-darwin": home-manager.defaultPackage.x86_64-darwin;
      "aarch64-darwin": home-manager.defaultPackage.aarch64-darwin;
    };

    homeConfigurations."alice@wonderland" =      home-manager.lib.homeManagerConfiguration {
        pkgs = nixpkgs.legacyPackages.x86_64-darwin;
        modules = [ ./home-alice.nix ];
      };
    };

    homeConfigurations."bob@bikinibottom" =      home-manager.lib.homeManagerConfiguration {
        pkgs = nixpkgs.legacyPackages.aarch64-darwin;
        modules = [ ./home-bob.nix ];
      };
    };
}

Then in each file you can share modules, helper functions, or whatever your heart desires.

I don’t want to use unstable software. How do I use the stable version?

In your flake.nix do this in your inputs block:

{
  description = "Home Manager configuration";

  inputs = {    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-22.11-darwin";    home-manager = {      url = "github:nix-community/home-manager/release-22.11";      inputs.nixpkgs.follows = "nixpkgs";    };  };
  outputs = { nixpkgs, home-manager, ... }: let
    arch = "x86_64-darwin"; # or aarch64-darwin
  in {
    defaultPackage.${arch} =
      home-manager.defaultPackage.${arch};

    homeConfigurations.YOUR_USER = # REPLACE ME
      home-manager.lib.homeManagerConfiguration {
        pkgs = nixpkgs.legacyPackages.${arch};
        modules = [ ./home.nix ];
      };
    };
}

I’ve changed my mind. How do I get both stable and unstable?

In your flake.nix do this:

{
  description = "Home Manager configuration";

  inputs = {    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-22.11-darwin";    home-manager = {      url = "github:nix-community/home-manager/release-22.11";      inputs.nixpkgs.follows = "nixpkgs";    };    nixpkgs-unstable = {      url = "github:nixos/nixpkgs/nixpkgs-unstable";    };  };
  outputs = { nixpkgs, nixpkgs-unstable, home-manager, ... }:    let
      arch = "x86_64-darwin"; # or aarch64-darwin
    in {
      defaultPackage.${arch} =
        home-manager.defaultPackage.${arch};

      homeConfigurations.YOUR_USER = # REPLACE ME
        home-manager.lib.homeManagerConfiguration {
          pkgs = nixpkgs.legacyPackages.${arch};
          modules = [ ./home.nix ];
          extraSpecialArgs = {            pkgs-unstable = nixpkgs-unstable.legacyPackages.${arch};          };        };
    };
}

Then in your home.nix you can do:

{ pkgs, pkgs-unstable, ... }: {  home.username = "arkham";
  home.homeDirectory = "/Users/arkham";
  home.stateVersion = "22.11";
  programs.home-manager.enable = true;

  home.packages = [
    pkgs.sl
    pkgs-unstable.cowsay  ];
}

Now you can run:

$ cowthink "Nix is pretty neat."
 _____________________
( Nix is pretty neat. )
 ---------------------
        o   ^__^
         o  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

I need a config file which is unsupported. How do I symlink it into my home?

Move the file inside ~/.config/nixpkgs:

$ mv .inputrc ~/.config/nixpkgs/inputrc

Then add a section like this to your home.nix:

home.file.".inputrc".source = ./inputrc;

If the config is very short you can also pass it inline:

home.file.".gemrc".text = "gem: --no-ri --no-rdoc";

I’ve made a terrible mistake and screwed up my configs. Halp!

No big deal. Run:

$ home-manager generations
2023-02-14 20:59 : id 11 -> /nix/store/d8xwz7hik6i2iji4wl9py7p6rxn2vn56-home-manager-generation
2023-02-13 22:07 : id 10 -> /nix/store/sxgdyj9lbx496rczgn7rig2vkjly6nwj-home-manager-generation
2023-02-13 01:13 : id 9 -> /nix/store/wdllhq7pf0sghn3kb0w1hjqc9962i02j-home-manager-generation
2023-02-12 00:01 : id 8 -> /nix/store/fm9bg8phff5zm0hp3mp5safk9v6dgdrc-home-manager-generation
2023-02-11 19:28 : id 7 -> /nix/store/wdllhq7pf0sghn3kb0w1hjqc9962i02j-home-manager-generation
2023-02-11 18:59 : id 6 -> /nix/store/i6979nf0135r6qid4dpq2m3vmy177k1w-home-manager-generation
2023-02-11 18:01 : id 5 -> /nix/store/6iybn1scyl9yh0xb2scp4hfza65xnnfa-home-manager-generation

Each symlink points to an immutable copy of a previous home configuration. Look at the timestamp, grab the Nix store path, append /activate, and run it:

$ /nix/store/fm9bg8phff5zm0hp3mp5safk9v6dgdrc-home-manager-
generation/activate
Starting Home Manager activation
Activating checkFilesChanged
Activating checkLaunchAgents
Activating checkLinkTargets
Activating writeBoundary
Activating copyFonts
Activating installPackages
replacing old 'home-manager-path'
installing 'home-manager-path'
Activating linkGeneration
Cleaning up orphan links from /Users/arkham
Creating profile generation 12
Creating home file links in /Users/arkham
Activating onFilesChange
Activating setupLaunchAgents

🎉 🎉 🎉

Nix is taking a lot of space on my machine.

Unfortunately that is a side effect of the Nix philosophy of tracking every dependencies’ dependencies. You can remove programs that you installed in the past by running:

$ nix-collect-garbage -d

This is bad. How do I uninstall it?

Uninstall home-manager:

$ home-manager uninstall

Restore your backups in your home directory:

$ cd ~

$ mv .gitconfig.bak .gitconfig

$ mv .bashrc.bak .bashrc
$ mv .bash_profile.bak .bash_profile

$ mv .zshrc.bak .zshrc

Then follow this guide to uninstall Nix.

Is homebrew completely useless now?

No, I still use it to install casks. ⚰️

Still reading?

Fun fact: this blog uses Nix flakes.

Many thanks to the wonderful human beings that have read earlier versions of this post: Pedro Piñera Buendía, Isaac Roldán, Fabrizio Monti, Andrew MacMurray, Austin Erlandson, and Georges Boris.

Peeling zeroes in Idris

Thu, 28 Oct 2021 12:00:00 GMT

🎶 We’re no strangers to lists, you know the rules and so do I 🎶

Anyway, here’s one:

theBeatles : List String
theBeatles = ["John", "Paul", "George", "Ringo"]

If we want to get an element from a list at a specific index, we can reach for a function like index:

index : Int -> List a -> Maybe a

Which we would use like this:

> index 0 theBeatles
Just "John"

> index 6 theBeatles
Nothing

The reason this function returns a Maybe is that we could pass any number to it. It seems thus inescapable that we need to handle the case when the given number is “out of bounds”. It seems impossible to conjure a function which at the same time is safe and behaves like this:

index : Int -> List a -> a

This seems peculiar because unless something quite monstrous happens we know that there will be no further changes to the Beatles lineup. So the problem is that we know this but the type system doesn’t, and it has to treat that list as it would treat any other list.

Is there no way out?

Enter Idris

In Idris there is another data structure that can describe lists called Vect:

theBeatles : Vect 4 String
theBeatles = ["John", "Paul", "George", "Ringo"]

What’s going on here?

The type also contains information about the length of the list. Let’s see what happens if we forget one of the Fab Four:

theBeatles : Vect 4 String
theBeatles = ["John", "Paul", "George"]

Idris complains:

When unifying Vect 0 String and Vect 1 String.
Mismatch between: 0 and 1.

Test:6:39--6:40
 2 |
 3 | import Data.Vect as Vect
 4 |
 5 | theBeatles : Vect 4 String
 6 | theBeatles = ["John", "Paul", "George"]
                                           ^

Don’t worry too much about the specifics of the error message, just look at that little caret that Idris puts at the end. It’s almost as the compiler is telling us “Hey, something’s missing there”.

With this data structure we can write something like this:

> Vect.index 0 theBeatles
"John"

Note that we get the raw value, no Maybe nonsense here. What about when we pass a number that’s out of bounds?

> Vect.index 6 theBeatles
Error: Can't find an implementation for
       IsJust (integerToFin 6 4).

(Interactive):1:12--1:14
 1 | Vect.index 6 theBeatles

Woah, now the compiler understands. Cool. But what’s the type signature of this index function?

> :t Vect.index
Vect.index : Fin len -> Vect len elem -> elem

Uh, what that’s Fin type? It describes a natural number strictly less than some bound. The name comes from “finite sets”, i.e. sets that have a finite number of elements. Its definition is:

data Fin : Nat -> Type where
  FZ : Fin (S k)
  FS : Fin k -> Fin (S k)

Ok.

We might need to take a little step back to unpack it.

Natura, artis magistra

Natural numbers are integers greater or equal than zero. A neat trick to represent them is to create a recursive type:

data Nat
  = Z
  | S Nat

Where Z represents zero, while S Nat represents the successor of a certain natural number.

Using this type we can construct the following values:

Z           == 0
S Z         == 1
S (S Z)     == 2
S (S (S Z)) == 3

You can see that Idris does a little bit of magic under the hood so that we can write 1 instead of S Z. Neat.

A somewhat interesting fact is that we can also get the previous natural number by removing a layer of S:

prev : Nat -> Maybe Nat
prev Z = Nothing
prev (S x) = Just x

> prev 0
Nothing

> prev 1
Just 0

> prev 2
Just 1

Now that we know how to express natural numbers in Idris, we can take a look at our finite subsets of natural numbers.

The Meat and Potatoes

Let’s look at that definition again:

data Fin : Nat -> Type where
  FZ : Fin (S k)
  FS : Fin k -> Fin (S k)

This looks a tad different from the previous data Nat = Z | S Nat definition, but it’s quite close:

We are defining a function called Fin that takes a natural number and returns a type.
One possible constructor is FZ, which stands for “finite zero”.
The other constructor is FS, which takes an existing Fin and returns its successor.

From this explanation it’s not clear how we are restricting these natural numbers, so let’s play around with some finite numbers. Since a literal 0 can be an Int, a Nat, and many other things, we have to use the special the function to tell Idris which specific type we want:

> the (Fin 1) 0
FZ

So everything is good when we ask for a 0 in Fin 1, the set of natural numbers strictly less than 1. How about this?

> the (Fin 0) 0

It blows up!

Error: Can't find an implementation
for IsJust (integerToFin 0 0).

(Interactive):1:13--1:14
 1 | the (Fin 0) 0

Why is this happening? By definition the minimum value we can create is FZ. The type of this value is Fin (S k), and by definition S k must be greater than zero. So this means that the smallest set we can create is Fin 1, which is the set that only contains zero.

Let’s look at Fin 2, the set that contains 0 and 1.

> the (Fin 2) 0
FZ

> the (Fin 2) 1
FS FZ

As expected, the zeroes in Fin 1 and Fin 2 are different:

> the (Fin 1) 0 == the (Fin 2) 0
Error: When unifying Fin 1 and Fin 2.
Mismatch between: 0 and 1.

(Interactive):1:1--1:14
 1 | the (Fin 1) 0 == the (Fin 2) 0
     ^^^^^^^^^^^^^

Ok, time for a little surprise. Look at these values again:

> the (Fin 2) 0
FZ

> the (Fin 2) 1
FS FZ

Those FZ values are not the same! The first one is of type Fin 2, while the second one is of type Fin 1. This makes sense when we look at the type of the FS function:

FS : Fin k -> Fin (S k)

Or making it more concrete:

FS : Fin 1 -> Fin 2

we get a 0 of type Fin 1
we apply FS
we get a 1 of type Fin 2

How would we get a 2 of type Fin 2? We would need to apply the FS function twice, going from Fin 0 to Fin 1 to Fin 2, right?

The problem is, we simply cannot do that.

The smallest valid finite set is Fin 1, which means we just cannot apply that function twice to create that value. You can only apply it once, giving us 1, which is the largest element in Fin 2.

If we look at Fin 3, things start making more sense:

> the (Fin 3) 0
FZ

> the (Fin 3) 1
FS FZ

> the (Fin 3) 2
FS (FS FZ)

> the (Fin 3) 3
Error: Can't find an implementation
for IsJust (integerToFin 3 3).

(Interactive):1:13--1:14
 1 | the (Fin 3) 3

It’s like we’re starting with this beefy zero, and we need to peel a layer to be able to create the next number. Then we keep peeling and peeling, up until we just can’t peel anything anymore. Fin 1 is the end of all hopes, the last laugh, the final curtain.

And Now for Something Completely Different

If you’ve followed so far, you’re in for a treat 🍭

We’ve talked a lot about Vect and Fin, but you don’t need any of that to reap the benefits we’ve talked about! Let’s go back to lists:

theBeatles : List String
theBeatles = ["John", "Paul", "George", "Ringo"]

Look at this:

> List.index 1 theBeatles
"Paul"

What??? How is it possible it just returns a value, no Maybe in sight?

> List.index 5 theBeatles
Error: Can't find an implementation for InBounds 5
  ["John", "Paul", "George", "Ringo"].

(Interactive):1:1--1:24
 1 | List.index 5 theBeatles
     ^^^^^^^^^^^^^^^^^^^^^^^

Just how. Let’s dig in:

> :t List.index
List.index :
  (n : Nat) ->
  (xs : List a) ->
  {auto 0 _ : InBounds n xs} ->
  a

If we look at the third argument we notice some curly braces around it. I won’t go too much in detail, but that’s an implicit argument that doesn’t need to be passed in when calling the function.

The interesting bit for us is that InBounds type:

data InBounds : Nat -> List a -> Type where
  InFirst : InBounds 0 (x :: xs)
  InLater : InBounds k xs -> InBounds (S k) (x :: xs)

That looks complicated. But how about we compare it to this?

data Fin : Nat -> Type where
  FZ : Fin (S k)
  FS : Fin k -> Fin (S k)

There is a base case. We use x :: xs to signal that the list must not be empty, so that we can never get an InFirst value of type InBounds 0 [].
There is a function that we can call recursively to peel a layer. In this case, not only do we go from k to S k, but we also specify that the list gets an extra element in front.

If you squint a bit, you’ll see it’s the exact type trick we’ve seen before. If we tried to call List.index 1 ["John"] we’d have to find a way to call the InLater function to peel off a layer. Or more concretely:

InLater : InBounds k xs -> InBounds (S k) (x :: xs)
-- becomes
InLater : InBounds 0 [] -> InBounds 1 ["John"]

The problem is constructing that InBounds 0 []. By definition InFirst has to be InBounds 0 (x :: xs), so we simply cannot create such value. Thus the type system cannot find a valid implementation of InBounds 1 ["John"], which is exactly the error message Idris prints out. QED

I hope you enjoyed this adventure in the land of zeroes and dependent types.

Fin 1.

Ditch your version manager

Fri, 17 Sep 2021 12:00:00 GMT

You probably use a variety of tools to manage your Ruby, Node, Python, Elixir versions, such as rvm, rbenv, nvm, or asdf. They all work reasonably well, right?

Well, not quite.

The fundamental issue is that many programs depend on other programs being present in the same system. For many years, if you wanted to install nokogiri, a popular XML parser written in Ruby, you’d have to remember to install libxml beforehand. Furthermore, some of these programs depend on OS-specific configurations, such as libraries shipped with Mac OS or Linux.

It’s not uncommon to check out a project and having to install a plethora of libraries and tools, and having to hope that the versions that you installed are compatible with the ones used by the original authors. Then you’d have to figure out a way to install an older version of a specific package.

I can hear what you’re wondering now: what about Docker? The idea of having a container image that contains all your dependencies sounds very promising indeed. The issue with docker images is that all those dependencies are still intertwined, only the whole dependency tree has become invisible: for example, if the version of Ubuntu you are using has reached its End-Of-Life, you will have to manually untangle all the dependencies and update everything in one go. If you’re working on a large project with many dependencies, please do accept my condolences.

My ideal dependency manager would allow me to specify each and every dependency that is required to work on my projects. It should be easily reproducible, declarative and easy to upgrade.

Does it sound too good to be true?

Welcome to the jungle

Let me introduce you to Nix.

Nix is a tool that takes a unique approach to package management and system configuration. Learn how to make reproducible, declarative and reliable systems.

Let’s try it out.

$ sh <(curl -L https://nixos.org/nix/install)

$ nix-env --version
nix-env (Nix) 2.3.10

Before we can start on our first project we will need to install direnv, a tool which will let us build a custom environment in a specific folder. You’ll need to:

nix-env -iA nixpkgs.direnv
hook direnv into your shell
restart your shell

Now we’re ready and raring to go! 🏄‍♀️

Let’s create a new folder:

$ mkdir nix_hello_world && cd nix_hello_world

$ hello
-bash: hello: command not found

As you can see, no hello command is available yet.

We can write our first configuration in a file called default.nix. We’ll be using the Nix Expression Language, a pure, lazy, functional programming language. Exciting times!

let
  pkgs = import <nixpkgs> { };
in
pkgs.mkShell {
  buildInputs = [ pkgs.hello ];
}

And we can tell direnv to load this file and update our environment.

$ echo "use nix" > .envrc
direnv: error /Users/arkham/Desktop/nix_hello_world/.envrc
is blocked. Run `direnv allow` to approve its content

$ direnv allow
<loads of nix output>

$ hello
Hello, world!

🎉

What did we do?

we imported the whole set of packages provided by Nix and called it pkgs
we used the mkShell built-in function to create a new environment where we can use Nix programs
we specify we want the GNU hello program

What if we wanted ruby? Couldn’t be simpler:

let
  pkgs = import <nixpkgs> { };
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
    pkgs.ruby
  ];
}

By saving this file and returning to your shell (you might need to press Enter to let direnv reload), Nix will pick up the change and install the new dependency.

$ ruby --version
ruby 2.7.4p191 (2021-07-07) [x86_64-darwin17]

$ which ruby
/nix/store/r6siyyqmnidkn2y8y5hl7payykb51kb0-ruby-2.7.4/bin/ruby

What’s with that super long path? Don’t worry, it’s just how Nix stores its data (waves hands.. iMmUTaBIlitY).

What if we needed node?

let
  pkgs = import <nixpkgs> { };
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
    pkgs.ruby
    pkgs.nodejs
  ];
}

$ node --version
v14.17.2

$ which node
/nix/store/zj1sqykb3s5a0fakm94qwz7fg6g2zxpm-nodejs-14.17.2/bin/node

You see the pattern here.

How about a different version of Ruby or Node? Let’s say that our project depends on Ruby 2.6 and Node 10. We can go and search for those specific versions, then change our default.nix accordingly:

let
  pkgs = import <nixpkgs> { };
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
    pkgs.ruby_2_6
    pkgs.nodejs-10_x
  ];
}

$ ruby --version
ruby 2.6.8p205 (2021-07-07) [x86_64-darwin17]

$ node --version
v10.24.1

🎉

Some bad news

Unfortunately, if you tried to follow this article step by step, you’ll have noticed that the versions of ruby and node you installed are probably slightly different from the ones above.

Wasn’t Nix supposed to give us reproducible builds?

Yes, but…

All the software that we installed depends on the specific version of the nixpkgs channel that we installed on our system. We can check which version that is by running:

$ nix repl
Welcome to Nix version 2.3.10. Type :? for help.

nix-repl> (import <nixpkgs> {}).lib.version
"21.05pre292875.60563458051"

We can then grab the commit part (the third segment) and view it on GitHub.

If only we were pointing to the same commit, then we would have exactly the same dependencies. But it would be foolish to require every project to depend on the same version of nixpkgs. We need a way to specify this on a per-project basis, right? We can use a tool called niv to do exactly that.

niv

niv lets us pin our dependencies to a specific version, thus ensuring a full reproducibility of our setup.

Let’s install it:

$ nix-env -iA nixpkgs.niv

Now we can initialize a niv project:

$ niv init --no-nixpkgs
Initializing
  Creating nix/sources.nix
  Creating nix/sources.json
  Not importing 'nixpkgs'.
Done: Initializing

This command generates a couple files:

$ tree
.
└── nix
    ├── sources.json
    └── sources.nix

1 directory, 3 files

The sources.json will contain all the dependencies that we’re going to use. Don’t worry about the sources.nix file, it’s mostly glue to allow us to load a JSON file in Nix.

For the sake of reproducibility, change your sources.json to look like this:

{
    "nixpkgs": {
        "branch": "release-21.05",
        "description": "Nix Packages collection",
        "homepage": "",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "5f244caea76105b63d826911b2a1563d33ff1cdc",
        "sha256": "1xlgynfw9svy7nvh9nkxsxdzncv9hg99gbvbwv3gmrhmzc3sar75",
        "type": "tarball",
        "url": "https://github.com/NixOS/nixpkgs/archive/5f244caea76105b63d826911b2a1563d33ff1cdc.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    }
}

Now we can create our top-level Nix configuration:

let
  sources = import ./nix/sources.nix { };
  pkgs = import sources.nixpkgs { };
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
    pkgs.ruby_2_6
    pkgs.nodejs-10_x
  ];
}

Going back to the shell, you will see that the version of ruby has changed:

$ ruby --version
ruby 2.6.7p197 (2021-04-05) [x86_64-darwin17]

Same for node:

$ which node
/nix/store/d08y5bbjq5pric7rwifim6bf4zvwmasc-nodejs-10.24.1/bin/node

This time you’re going to see exactly the same versions!

Another good news is that configuration will work forever. That’s right, the moment you commit this configuration to your project you will ensure that it will never stop working.

Need to install a new tool? Pop it in the Nix config. Need to patch a program and make sure every one is using it? Create an overlay, pop it in the Nix file and grab some pop corn. Need to update your packages? Run niv update and all the new dependencies will be pulled in.

Nix is a version manager for everything. You’ll never have to tell the new colleague “oh right, you need to install this version of LLVM to get this thing to work”. They can pull the repo, direnv allow and BAM, fully-working, reproducible project setup.

What are you waiting for?!

Elm at NoRedInk

Wed, 28 Apr 2021 12:00:00 GMT

At NoRedInk we have one of the largest Elm apps in the world. It serves millions of teachers and students, and our frontend code is almost exclusively written in Elm. In this post, we will explore the structure of our codebase and the patterns that we use to stay sane. One of the most common questions I get about Elm is:

“Does it scale? And if so, how?”

I don’t think that there can ever be a comprehensive answer to that question, but I hope this post can be a useful reference to resources and techniques we’ve used to solve real-world problems, so you can dig deeper at your own leisure.

A wild monorepo appears
Rails conventions
Our Elm programs
How we write Elm files
Nesting the Elm Architecture
When to nest TEA modules
Interacting with JavaScript
Testing
Tooling

A wild monorepo appears

We have a glorious monorepo that contains all the services that power our infrastructure. Our biggest service (which we affectionately call the monolith) is written in Rails and contains most of our Elm code:

$ cloc --include-ext=elm monolith/ui/src
-----------------------------------------------
Language    files    blank    comment      code
-----------------------------------------------
Elm          1506    49759      16535    211835
-----------------------------------------------

Nothing much to add here. Moving on :)

$ cloc --include-ext=elm monolith/ui/tests
-----------------------------------------------
Language    files    blank    comment      code
-----------------------------------------------
Elm           569    10275       1309    200586
-----------------------------------------------

As you can see, we write loads of tests. As much as the Elm compiler gives you that wonderful confidence while refactoring, we still want to make sure that our code is working as intended. We will talk more in-depth about testing later.

$ cloc --include-ext=elm monolith/ui/generated
-----------------------------------------------
Language    files    blank    comment      code
-----------------------------------------------
Elm           129     2428       1728     26399
-----------------------------------------------

We end up auto-generating a lot of Elm code: we do this for multiple purposes, such as automatically generating types from graphql, or ensuring that JSON payloads sent from Rails controllers match the definitions inside our Elm decoders.

$ cloc --include-ext=elm content-creation
-----------------------------------------------
Language    files    blank    comment      code
-----------------------------------------------
Elm            80     3245       1182     13941
-----------------------------------------------

At last, here’s a snapshot of Elm code in one of our Haskell services. I won’t spend much time describing how that works in this post, let me know if you are interested and I’ll write a follow-up. For now, let’s take a deeper look at how we integrate with Rails.

Rails conventions

Following the Rails architecture, each REST resource is managed by its dedicated controller. Here is what happens when a teacher goes to manage their classes:

they visit the /teach/classes URL in their browser
that route is managed by the Teach::ClassesController controller
that controller will fetch stuff from the database, clean it up and load a corresponding view, app/views/teach/classes/index.html.haml

That view looks like this:

- content_for :title, "Manage Classes | NoRedInk"

- content_for(:javascript) do
  = javascript_include_tag "shake/Page/Teach/Classes/index.js"

= elm_mount(@elm_flags, 'teach-classes-elm')

What is that elm_mount helper? We define it to be like this:

def elm_mount(flags, prefix = 'elm')
  tag.div(id: prefix + '-flags',
          class: "elm-flags",
          data: {flags: flags.to_json}) +
    tag.div(id: prefix + '-host')
end

In short, we need some conventions for the names of two DOM nodes:

one that contains the flags that we pass from Rails to Elm: we use this to pass data to the Elm application that needs to be available on boot
one that will contain the actual Elm app when it’s mounted

The javascript file that we include is the result of the compilation of the Elm app. We use a build system called Shake to generate all our assets. The actual entry point looks like this:

import { Elm } from "./Main.elm"

import * as NriProgram from "Nri/Program.js"
import setupReadAloud from "ReadAloud/setup.js"

NriProgram.domready(function () {
  const { subscribe, send } = NriProgram.mountPorts(
    Elm.Page.Teach.Classes.Main,
    "ui/src/Page/Teach/Classes/index.js",
    "teach-classes-elm"
  )

  setupReadAloud({ subscribe, send })
})

The NriProgram that you see here is a small wrapper around common operations that we need to perform, such as:

passing an environment object to our apps so that we can have different settings in test, development and production
setting up some basic analytics and reporting
grabbing the flags and the div where the Elm app will be mounted
giving us the ability to easily set up ports using subscribe and send

Another interesting bit is the convention that we use to name our Elm modules. Since the URL is going to be /teach/classes, the associated Elm module is going to be Page.Teach.Classes.Main. Here are some other examples:

Page.Admin.RelevantTerms.Main corresponds to /admin/relevant_terms
Page.Learn.ChooseSourceMaterials.Main corresponds to /learn/choose_source_materials
Page.Preferences.Main corresponds to /preferences
Page.Teacher.Courses.Assignments.Main corresponds to /teach/courses/:id/assignments

In total, we have over a hundred Elm apps that serve different Rails controllers. They consist of a mixture of normal Elm applications and single-page apps. By adapting the Rails motto of “convention over configuration” we’re pretty confident we can scale this approach indefinitely.

Our Elm programs

At this point, you probably won’t be surprised to learn that we use a custom wrapper as the entrypoint of our Elm programs. Here is how it looks like:

main : Nri.Program.Program Model Msg
main =
    Nri.Program.program
        { moduleName = "Page.Interests.Main"
        , flagsDecoder = \_ -> decode
        , perform = perform
        , init = init
        , view = view
        , update = update
        , subscriptions = \_ -> Sub.none
        }

Let’s compare it to Browser.element from elm/browser:

main : Browser.Program flags model msg
main =
    Browser.element
        { init = init
        , view = view
        , update = update
        , subscriptions = \_ -> Sub.none
        }

As you can see, the types are slightly different and we include a couple of different fields in our version.

What are the advantages of using a custom Nri.Program?

reducing boilerplate: for example, we can automatically handle decoding failures so that we don’t have to worry about them on each individual page.
handling generic behaviours: for example, we can automatically detect the user’s input method to make our product more accessible.

The differences don’t stop there. Let’s look at the type of the view function:

view : Env -> Model -> Html Msg

Here we take an additional Env argument. This record contains information about the current release, the logged-in user and the environment where the code is running. For example, if we detect that we are running in test mode, there is no need to run fancy UI animations, thus we can disable them. This in turn makes our tests faster and less flaky. Win-win!

If you look at the record that we pass to Nri.Program.program you will notice an additional perform field. Let’s look at the type signature:

perform : Env -> Effect -> Cmd Msg

This is easier to understand if we look at the type signature of update:

update : Env -> Msg -> Model -> ( Model, Effect )

The curious change is that a standard update function in Elm returns a (Model, Cmd Msg) tuple, while ours returns (Model, Effect). Why is that? Cmd Msg is an opaque type that is meant to be passed as-is to the Elm runtime. But in order to test that the correct side effect has been emitted we need to be able to inspect it: so we define our own type and we call it Effect.

Then we can use the fantastic elm-program-test library (you can find it here) to test our Elm programs. Meanwhile, in our normal application, we can use the perform function to convert the Effect representation of a side effect to its Cmd Msg counterpart.

Similarly, we have another custom wrapper for single-page applications called Nri.Spa.Program which mostly wraps Browser.application.

The general lesson here is that just because Elm provides an update function with the shape msg -> model -> (model, Cmd msg) your version doesn’t need to. You might need more arguments, or different types, or more return values. As long as you convert the result of your function to match the API of Browser.element you can customize your API as much as you want.

How we write Elm files

We don’t believe in the mantra:

Prefer small files

Here is a little snapshot of some large files in our codebase:

2251 src/Page/Learn/GuidedDrafts/Main.elm
1773 src/Page/Curriculum/BrowseAndAssignLayout.elm
1725 src/Page/Teach/Classes/Page/Classes/Explore.elm
1691 src/Page/Teach/Assignment/Form/GuidedDraft/Customize.elm
1524 src/Page/Teach/Courses/PeerReviews/Progress.elm
1280 src/Page/Admin/Alignments/Main.elm
1188 src/Page/Learn/Home/Main.elm
1184 src/Page/Teach/Assignment/Form/Model.elm
1136 src/Page/Preview/PeerReview/Main.elm

As Evan Czaplicki explains in this talk, larger files in Elm are not a problem. We tend to delineate the different sections related to the model, update and view functions with comments. Then we extract functions around their data structures, rather than their perceived role.

Another point I want to emphasize is that in Elm you don’t have to get everything right from the start. Choose the simplest architecture that works. If you need to change it later, the compiler will assist you every step of the way. Specifically for this sort of mechanical changes, the chance of introducing bugs is very close to zero. If it compiles, it works.

We regularly merge refactors in the thousands of lines that don’t introduce a single regression to the product. There is really nothing to be afraid of! 🍧

Nesting the Elm Architecture

When we need to combine multiple Elm applications, we nest them under one another. So for example if we have a modal in a page with its own state and messages, we will do this:

type alias Model =
    { modal : Modal.Model
    }

type Msg = ModalMsg Modal.Msg

update : Env -> Msg -> Model -> ( Model, Effect )
update env msg model =
    case msg of
        ModalMsg modalMsg ->
            let
                ( newModal, modalEffect ) =
                    Modal.update env modalMsg model.modal
            in
            ( { model | modal = newModal}
            , ModalEffect modalEffect
            )

view : Env -> Model -> Html Msg
view env model =
    div []
        [ Modal.view env model.modal |> Html.map ModalMsg
        , span [] [ text "Hello there" ]
        ]

Let’s break this down.

We have a top-level Msg type that describes the behaviour of the top-level application. Its only variant is a ModalMsg Modal.Msg which wraps the message type of the modal.

When we receive a ModalMsg we pattern match to extract the inner message and we run it against the modal state stored in the model through Modal.update. Then we replace the modal with the updated version and do something with the side effect.

Something else to note is that we need to use Html.map to wrap the Modal.view function. In this way, we ensure that all the messages that are emitted are wrapped with the ModalMsg message constructor.

This is how we build big apps in Elm. That’s the secret sauce. By applying this concept repeatedly you can scale your Elm apps as much as you want. Here is the actual Msg type of our Teach.Classes.Main module:

type Msg
    = ChangePage Route
    -- Nested TEA
    | TimedAlerts Alert.Msg
    | ClassesMsg Classes.Msg
    | ModalMsg Modal.Msg
    | EditClassMsg EditClass.Msg
    -- HTTP responses
    | ReceiveClasses
        Class.RedirectToClass
        (WebData
            { activeClasses : List Class.Class
            , archivedClassCount : Int
            }
        )
    | ReceiveArchivedClasses (WebData (List Archived.Class))
    -- DOM focus
    | Focus String
    | Focused (Result Dom.Error ())
    -- User actions
    | OpenModal Modal.Kind String
    | CloseModal
    | SetModalKind Modal.Kind
    | ModalError Http.Error
    | AddStudent Modal.CreateStudentAccountsManuallyModel
    | Archived Int
    | Unarchived Int
    | ToggleTooltip String Bool

When to nest TEA modules

If you looked at the section above and you thought “welp, that looks like a lot of work”, I hear you. As a matter of fact, you don’t need to do the whole nesting dance every time we want to reuse a module. In fact, most of the UI components that we use don’t have their own state, so they don’t need even need an update function. Let’s look at how we use a button:

import Nri.Ui.Button.V10 as Button

view : Env -> Model -> Html Msg
view env model =
    div []
        [ Button.link "Edit"
            [ Button.css [ Css.marginBottom (Css.px 10) ]
            , Button.onClick EditAssignment
            , Button.icon UiIcon.calendar
            , Button.small
            ]
        ]

It seems obvious, but if there is no state, then there is no need to do extra work.

If you are curious about how we structure our UI components, you can take a gander at noredink-ui (check out the preview here). We’re always including simple examples for each UI component so that they act as the best documentation possible (component / example).

If you feel that it is too cumbersome to nest TEA applications, it may be because it just is. If you choose it as your default approach to scale Elm code, you will find yourself constantly wrapping and unwrapping messages for little or no benefit. I would recommend reaching for it as a last resort, like using a rocket launcher to plant daffodils in your garden: it works, but it’s a bit heavy-handed.

Instead you might find some abstractions which don’t require another model and another set of update messages. You can do lots with a view function that takes a config record and an extensible record:

viewSidebar :
    { onClick : msg, onClose : msg }
    -> { a | sidebarItems : List SidebarItem }
    -> Html msg

As mentioned above, choose the simplest solution that works. Later you can refactor and make it feel good. You can watch this talk by Richard Feldman if you want to learn more.

A good point to reiterate is that, when writing Elm, you don’t have to get everything right from the start. The compiler gives you an unprecedented level of confidence when refactoring code. So go out there and make mistakes, then fix them, then keep going.

Interacting with JavaScript

We mainly use ports to interact with JavaScript code. We have two simple rules for ports:

All ports must return values as Json.Value. If a Int -> Cmd msg port receives a float instead, it will cause a crash. This is one of the simplest ways to introduce runtime errors in your application. By treating all values as JSON blobs, we must decode them and deal with the eventual decoding failure. Repeat with me, ports need JSON values.
All port functions must be documented in the Elm module. We don’t want to go hunting in JavaScriptLand how a port is being used. A couple of lines explaining what the port triggers can go a long way.

In cases where ports are not enough, we use custom elements. You might want to do this for many reasons, such as reusing existing React components inside your Elm application. In our case, we have integrated rich text editing using a custom element that wraps quilljs. If you’re interested in learning more about integrating custom elements in Elm, I recommend watching this great talk by Luke Westby.

Testing

We write tests for our Elm code at four different layers:

Unit tests: these are pure tests around data structures. Create a piece of data, run a function on it and assert some results.
View tests: here we construct a model, pass it to the view function and write assertions against the result using elm-explorations/test.
Integration tests: here we load up an auto-generated JSON file (using rails_edge_test, find it here), pass it to the elm-program-test program, interact with the elements on the page, and assert side effects.
Acceptance tests: we write these in Capybara as happy-path tests. They are extremely useful to test JavaScript interop.

I can recommend this excellent talk by Tessa Kelly if you want to learn how to write testable Elm.

Tooling

Here’s a selection of tools that we use:

elm-format, the Holy Grail of code formatters. I love repeating “garbage in, code out”. In this case, it happens to be beautifully formatted, while respecting your preference in terms of whitespace, and applying tiny cleanups of your code on the fly. What’s not to like?
elm-css, so that our CSS is typed. We use it so much that the view function in our Nri.Program returns a Html.Styled.Html msg by default.
accessible-html, or how to ensure that accessibility is a first-class citizen in your app. I think that this library is also a great case study on how to create a light wrapper around another library.
elm-review, a must-have addition to any Elm project. It will detect unused variables, unused variants, unused modules, and much much more. Use it!
elm-json-decode-pipeline, another approach at writing JSON decoders. I love how easy it is to read and modify decoders written in this style.

That’s all for today, and if this sort of work interests you do check out this page. Thanks for reading 👋

My eternal gratitude to the wonderful folks that have read through the drafts of this post: @juanedi, @brianhicks, @rtfeldman, and @michaelglass.

There and back again

Fri, 26 Mar 2021 12:00:00 GMT

When writing Elm, it’s very common to use custom types to structure your data. Let’s say we want to describe the flashy suits of the Power Rangers, as featured in the 1975 classic Himitsu Sentai Gorenger:

type Suit
    = Green
    | Yellow
    | Red
    | Blue

This looks great, so we decide to go to the next level. We’re going to build a fully-fledged Elm application to choose our favourite Power Ranger suit color. We start with a simple model:

type alias Model =
    { chosenSuit : Maybe Suit }

We then define an initial state of the model, where no suit has been chosen:

initialModel : Model
initialModel =
    { chosenSuit = Nothing }

Now we can write a view function to display something:

import Html exposing (Html, div, text)
import Html.Attributes as Attrs

view : Model -> Html Msg
view model =
    div []
        [ Html.h2 [] [ text "Choose your Power Ranger Suit" ]
        , Html.select []
            [ Html.option
                [ Attrs.value "" ]
                [ text "Choose" ]
            , Html.option
                [ Attrs.value "Green" ]
                [ text "Green" ]
            , Html.option
                [ Attrs.value "Yellow" ]
                [ text "Yellow" ]
            , Html.option
                [ Attrs.value "Red" ]
                [ text "Red" ]
            , Html.option
                [ Attrs.value "Blue" ]
                [ text "Blue" ]
            ]
        ]

This is unfortunate. We have to use strings and pass them to Attrs.value and to Html.text, which means we can’t use the beautiful custom type we wrote above. To be able to do that, we have to define a toString function:

toString : Suit -> String
toString suit =
    case suit of
        Green ->
            "Green"

        Yellow ->
            "Yellow"

        Red ->
            "Red"

        Blue ->
            "Blue"

And use it to dry up the view:

selectOptions : List (Html Msg)
selectOptions =
    List.map
        (\color ->
            Html.option
                [ Attrs.value (toString color) ]
                [ text (toString color) ]
        )
        [ Green, Yellow, Red, Blue ]

view : Model -> Html Msg
view model =
    div []
        [ Html.h2 [] [ text "Choose your Power Ranger Suit" ]
        , Html.select []
            (Html.option [ Attrs.value "" ] [ text "Choose" ]
                :: selectOptions
            )
        ]

You can play around with this Ellie app to see how that looks like.

If you’re a bit more experienced with Elm, you will notice that you can choose an option in the select, but that won’t update the Model. Let’s prove that assumption by changing the view to:

view : Model -> Html Msg
view model =
    div []
        [ Html.h2 [] [ text "Choose your Power Ranger Suit" ]
        , Html.select []
            (Html.option [ Attrs.value "" ] [ text "Choose" ]
                :: selectOptions
            )
        , Html.p []
            [ case model.chosenSuit of
                Nothing ->
                    text "No suit was chosen."

                Just suit ->
                    text ("Your choice: " ++ toString suit)
            ]
        ]

As you can see, no matter how you change the view, the value won’t be updated. This happens because we haven’t defined a onInput event on the select :(

import Html.Events as Events

view : Model -> Html Msg
view model =
    div []
        [ Html.h2 [] [ text "Choose your Power Ranger Suit" ]
        , Html.select [ Events.onInput ChooseSuit ]
            (Html.option [ Attrs.value "" ] [ text "Choose" ]
                :: selectOptions
            )
        , Html.p []
            [ case model.chosenSuit of
                Nothing ->
                    text "No suit was chosen."

                Just suit ->
                    text ("Your choice: " ++ toString suit)
            ]
        ]

type Msg
    = ChooseSuit String

update : Msg -> Model -> Model
update msg model =
    case msg of
        ChooseSuit value ->
            { model | chosenSuit = fromString value }

What’s that fromString function? Here it is:

fromString : String -> Maybe Suit
fromString str =
    case str of
        "Green" ->
            Just Green

        "Yellow" ->
            Just Yellow

        "Red" ->
            Just Red

        "Blue" ->
            Just Blue

        _ ->
            Nothing

In this Ellie app, we can see that everything is working nicely :)

An interesting pair

We notice a terrible bug in our application: we forgot about the Pink Power Ranger! Let’s fix that by adding a new variant:

type Suit
    = Green
    | Yellow
    | Red
    | Blue
    | Pink

If we save the file, we’ll notice the Elm compiler being helpful as ever:

This `case` does not have branches for all possibilities:

19|>    case suit of
20|>        Green ->
21|>            "Green"
22|>
23|>        Yellow ->
24|>            "Yellow"
25|>
26|>        Red ->
27|>            "Red"
28|>
29|>        Blue ->
30|>            "Blue"

Missing possibilities include:

    Pink

Cool, we need to add the missing case in toString:

toString : Suit -> String
toString suit =
    case suit of
        Green ->
            "Green"

        Yellow ->
            "Yellow"

        Red ->
            "Red"

        Blue ->
            "Blue"

        Pink ->
            "Pink"

If we save now the app compiles. But if we try using the app, we can see that there is no Pink option :( It turns out we forgot to add the option inside the selectOptions function. Maybe we can also extract the list of suits into another function:

allSuits : List Suit
allSuits =
    [ Green, Yellow, Red, Blue, Pink ]

selectOptions : List (Html Msg)
selectOptions =
    List.map
        (\color ->
            Html.option
                [ Attrs.value (toString color) ]
                [ text (toString color) ]
        )
        allSuits

Try the app now. This is what happens when you choose ‘Pink’ in the dropdown:

This makes me sad, since I love to repeat:

If it compiles, it works

And in the last minute we already bumped into two cases where the code compiles, but doesn’t work. As you probably guessed, the bug is hidden in our fromString function:

fromString : String -> Maybe Suit
fromString str =
    case str of
        "Green" ->
            Just Green

        "Yellow" ->
            Just Yellow

        "Red" ->
            Just Red

        "Blue" ->
            Just Blue

        "Pink" ->
            Just Pink

        _ ->
            Nothing

This is the key difference between the toString function and the fromString function: toString takes a Suit, so the compiler can warn us if we forget to consider one of the possible alternatives. In the case of fromString, the valid inputs are all possible strings, so the compiler can’t do much to warn us.

But as humans, we can tell that these two functions are really similar. Is there nothing we can do to express it in code? Let’s go for a little trip.

A trip in Haskell Land

Let’s define the same type in Haskell:

data Suit
  = Green
  | Yellow
  | Red
  | Blue
  | Pink

In order to implement the toString function we can derive the Show typeclass, which will know how to convert our custom type to a string:

data Suit
  = Green
  | Yellow
  | Red
  | Blue
  | Pink
  deriving (Show)

toString :: Suit -> String
toString = show

And we take it for a spin:

Prelude> toString Green
"Green"

Prelude> toString Blue
"Blue"

Prelude> toString Pink
"Pink"

Now let’s figure out a way to define the fromString function. We can start by defining an allSuits function. We will do so by deriving other two typeclasses, named Enum and Bounded. The first one tells the compiler that our type is an enumeration of sequentially ordered types, while the second specifies that our type has a minimum value and a maximum value.

data Suit
  = Green
  | Yellow
  | Red
  | Blue
  | Pink
  deriving (Show, Enum, Bounded)

allSuits :: [Suit]
allSuits =
  [minBound .. maxBound]

Which results in:

Prelude> allSuits
[Green, Yellow, Red, Blue, Pink]

Cool! Now we can write that fromString function:

import qualified Data.Map as Map

suitMap :: Map.Map String Suit
suitMap =
  Map.fromList $
    map
      (\suit -> (toString suit, suit))
      allSuits

fromString :: String -> Maybe Suit
fromString value =
  Map.lookup value suitMap

Here it is working:

Prelude> fromString "Red"
Just Red

Prelude> fromString "Pink"
Just Pink

Prelude> fromString "Geronimo"
Nothing

As you can see, in Haskell Land we were able to define the relationship between toString and fromString. If we add a new variant to the Suit type, it will be picked up and everything will work as intended. This technique is described in more detail in the wonderful Haskell Mini-Patterns Handbook.

Coming back home

Is there a way to achieve something similar in Elm?

Yes, there is! 🍨

Truth be told, we can’t do exactly what Haskell does, but we can emulate Haskell’s behaviour by using a wonderful library called elm-review.

elm-review analyzes your Elm code and reports patterns that violate certain rules. We can use it to enforce additional guarantees on our projects, such as removing unused functions, unused values, unused modules, and unused dependencies. You can read more about it here and here.

In our case, we can write a rule that detects functions named like allSuits and tell it to warn us when we forget to include a type constructor. I’ve shamefully stolen a gist from Jeroen Engels and pushed it as an elm-review package here.

If you’d like to follow along you can pull this repo and then run:

$ npx elm-review --template \
  Arkham/elm-review-no-missing-type-constructor/example
✔ Fetching template information
-- ELM-REVIEW ERROR ------------------- src/Main.elm:46:1

NoMissingTypeConstructor: `allSuits` does not contain all
the type constructors for `Suit`

45| allSuits : List Suit
46| allSuits =
    ^^^^^^^^
47|     [ Green, Yellow, Red, Blue ]

We expect `allSuits` to contain all the type constructors
for `Suit`.

In this case, you are missing the following constructors:
, Pink

I found 1 error in 1 file.

🎉 🎉 🎉

Now we can implement a solution which is very similar to the Haskell one!

import Dict exposing (Dict)

suitDict : Dict String Suit
suitDict =
    Dict.fromList <|
        List.map
            (\suit -> ( toString suit, suit ))
            allSuits

fromString : String -> Maybe Suit
fromString value =
    Dict.get value suitDict

Now when we add a new variant of Suit, the Elm compiler will nudge us to update toString, while elm-review will nudge us to add the missing variant to allSuits, which in turn is going to make sure that toString and fromString will never go out of sync. Here’s a link to the final version of the code for you to peruse.

Thanks for reading! 👋

A simple tmux config

Mon, 22 Mar 2021 12:00:00 GMT

I friggin’ love tmux. It makes my day-to-day terminal experience so. much. better. Being efficient in tmux means you don’t need tabs any longer. It means you can update your terminal emulator while keeping your work intact. It means you can sling panes and windows across your terminal as if you were the Last Airbender. It’s that good.

I thought I’d share a couple of tricks I’ve picked up over the years. I’d heartily recommend reading and understanding each line of your configuration: in this way, when things go wrong (and they will), you’ll be more likely to have a hunch of the cause of the problem.

Here’s a short demo:

Mac OS install

$ brew install tmux

tat

First of all I want you to add this function to your ~/.bashrc or ~/.zshrc or whichever shell you’re using:

# tat: tmux attach
function tat {
  name=$(basename `pwd` | sed -e 's/\.//g')

  if tmux ls 2>&1 | grep "$name"; then
    tmux attach -t "$name"
  elif [ -f .envrc ]; then
    direnv exec / tmux new-session -s "$name"
  else
    tmux new-session -s "$name"
  fi
}

This function:

gets the name of the current directory and removes periods, which tmux doesn’t like.
if any session with the same name is open, it re-attaches to it.
otherwise, it checks if an .envrc file is present and starts a new tmux session using direnv exec.
otherwise, starts a new tmux session with that name.

This is what I do every time I start working on any project: open the terminal, cd into the folder, then tat.

Now let’s take a look at the tmux configuration.

General configuration

Your config should live in ~/.tmux.conf.

Mouse

set -g mouse on

Welcome to the twenty-first century.

Command history

set -g history-limit 102400

Those who cannot remember the past are condemned to repeat it.

Window management

set -g base-index 1
setw -g pane-base-index 1

By default, tmux will number panes and windows starting from 0. I find indexing things from 1 to be more ergonomic, especially when you want to quickly switch between windows.

set -g renumber-windows on

By default, after you delete a window tmux will remember the previous index of the other windows. Why? I dunno, but this gets rid of it.

Keys modes

# Use emacs keybindings in the status line
set-option -g status-keys emacs

# Use vim keybindings in copy mode
setw -g mode-keys vi

# Fix ESC delay in vim
set -g escape-time 10

Even if you’re a hardcore vim user, you’ll want to use emacs keybinding in the status line.

Key Bindings

Prefix

unbind C-b
set -g prefix C-a

The prefix is the entry point to each command you’re going to use. I chose Ctrl-a because it’s easier to press with my left hand.

bind C-a send-prefix

Since we set Ctrl-a to be our prefix, it will be intercepted by tmux and won’t be forwarded to any program. With this setting, we can press it twice to send it through anyways.

Copy mode

unbind-key -T copy-mode-vi v

bind-key -T copy-mode-vi v \
  send-keys -X begin-selection

bind-key -T copy-mode-vi 'C-v' \
  send-keys -X rectangle-toggle

bind-key -T copy-mode-vi y \
  send-keys -X copy-pipe-and-cancel "pbcopy"

bind-key -T copy-mode-vi MouseDragEnd1Pane \
  send-keys -X copy-pipe-and-cancel "pbcopy"

In tmux, you can enter navigation mode by pressing Ctrl-a [. Once you’re in that mode, you can navigate the buffer as if you were inside a VIM buffer. Now you can:

search for things with /; find next matches with n and previous matches with N.
start a visual selection with v; you can switch to rectangle mode with Ctrl-v.
copy the visual selection into your OS clipboard with y; this will automatically cancel the selection. The same works by just highlighting text with your mouse.

Quick reload

bind r source-file ~/.tmux.conf \; display "Config reloaded!"

A blessing when you’re testing out changes in your config.

More windows management

bind c new-window -c '#{pane_current_path}'

Press Ctrl-a c to open a new window with the same path.

bind '\' split-window -h -c '#{pane_current_path}'
bind - split-window -v -c '#{pane_current_path}'

Use Ctrl-a \ to open a vertical split (it’s the same key with the | symbol). Use Ctrl-a - to open a horizontal split.

bind b break-pane -d

Press Ctrl-a b to break the current pane into a new window.

Another shortcut I use all the time is Ctrl-a z to expand the current pane full-screen. Press it again to return to your previous pane configuration.

Vim integration

First of all, install vim-tmux-navigator and add these lines to your ~/.vimrc:

nnoremap <C-J> <C-W>j
nnoremap <C-K> <C-W>k
nnoremap <C-H> <C-W>h
nnoremap <C-L> <C-W>l

These will allow us to switch between Vim splits by just pressing Ctrl-j instead of Ctrl-w j. Then add this to your ~/.tmux.conf:

is_vim="ps -o state= -o comm= -t '#{pane_tty}' \
    | grep -iqE '^[^TXZ ]+ +(\\S+\\/)?g?(view|n?vim?x?)(diff)?$'"
bind-key -n 'C-h' if-shell "$is_vim" 'send-keys C-h'  'select-pane -L'
bind-key -n 'C-j' if-shell "$is_vim" 'send-keys C-j'  'select-pane -D'
bind-key -n 'C-k' if-shell "$is_vim" 'send-keys C-k'  'select-pane -U'
bind-key -n 'C-l' if-shell "$is_vim" 'send-keys C-l'  'select-pane -R'
tmux_version='$(tmux -V | sed -En "s/^tmux ([0-9]+(.[0-9]+)?).*/\1/p")'
if-shell -b '[ "$(echo "$tmux_version < 3.0" | bc)" = 1 ]' \
    "bind-key -n 'C-\\' if-shell \"$is_vim\" 'send-keys C-\\'  'select-pane -l'"
if-shell -b '[ "$(echo "$tmux_version >= 3.0" | bc)" = 1 ]' \
    "bind-key -n 'C-\\' if-shell \"$is_vim\" 'send-keys C-\\\\'  'select-pane -l'"

bind-key -T copy-mode-vi 'C-h' select-pane -L
bind-key -T copy-mode-vi 'C-j' select-pane -D
bind-key -T copy-mode-vi 'C-k' select-pane -U
bind-key -T copy-mode-vi 'C-l' select-pane -R
bind-key -T copy-mode-vi 'C-\' select-pane -l

Now you will be able to seamlessly navigate between Vim splits and tmux panes using Ctrl-h, Ctrl-j, Ctrl-k, and Ctrl-l. You can switch between the last two panes by repeatedly pressing Ctrl-\.

Cool status bar

set-option -g status-justify left
set-option -g status-left '#[bg=colour72] #[bg=colour237] #[bg=colour236] #[bg=colour235]#[fg=colour185] #S #[bg=colour236] '
set-option -g status-left-length 16
set-option -g status-bg colour237
set-option -g status-right '#[bg=colour236] #[bg=colour235]#[fg=colour185] %a %R #[bg=colour236]#[fg=colour3] #[bg=colour237] #[bg=colour72] #[]'
set-option -g status-interval 60

set-option -g pane-active-border-style fg=colour246
set-option -g pane-border-style fg=colour238

set-window-option -g window-status-format '#[bg=colour238]#[fg=colour107] #I #[bg=colour239]#[fg=colour110] #[bg=colour240]#W#[bg=colour239]#[fg=colour195]#F#[bg=colour238] '
set-window-option -g window-status-current-format '#[bg=colour236]#[fg=colour215] #I #[bg=colour235]#[fg=colour167] #[bg=colour234]#W#[bg=colour235]#[fg=colour195]#F#[bg=colour236] '

Here’s a preview:

Fin

If you’d like, you can get a copy of the whole config here. If you don’t feel like maintaining this configuration yourself you can try out tmux-sensible.

Feel free to reach out on Twitter for help. That’s all, thanks for reading! 🌊

z-index by examples

Mon, 01 Feb 2021 12:00:00 GMT

I’ve been writing software “for a living” for more than ten years. I’m afraid I have to admit that, for more than ten years, I’ve been using z-index without really understanding it.

z-index is a CSS property that is used to position elements above or below one another. It’s your weapon of choice if you have two overlapping elements and you want to decide which one goes on top. It’s quite useful for modals, tooltips, banners, etc. You can set it to an integer and a larger value will generally result in the element being rendered on top of the rest.

That’s pretty much all I knew about it.

Usually, this is great in software: “Learn enough to be dangerous” they say. But invariably I would find myself fighting with z-index and spending hours trying to understand what is wrong with Life, the Universe, and Everything.

I’m pretty sure it’s not just me. It’s common to bump into CSS like this:

z-index: 99999

Which is pretty much the equivalent of the developer saying:

this is too wild, byeee

Well, let’s tackle this beast with some examples.

The setup

I’ve created a little app to make things easy. It looks like this:

The boxes on the right are positioned statically. This is the default when you don’t specify a position property, therefore a statically positioned box is also known as a non-positioned box. I’ve added some negative margins so they overlap with one another.

On the left-hand side, you can type some styles which will be applied to the boxes. Notice that you can also share the setup by copying the URL. Let’s get started now!

Ordering matters

Here’s the HTML that powers the page:

<div class="purple"></div>
<div class="blue">
  <div class="green"></div>
  <div class="yellow"></div>
</div>
<div class="red"></div>

As you can see, ordering in the HTML matters. The red box is positioned above the rest because it appears later in the code.

z-index doesn’t work by itself

Look at this:

That’s rule number one:

z-index works only on positioned elements.

So z-index only works on elements that have a position property set to:

relative
absolute
fixed
sticky

There we go:

Would you be able to guess what happens when we remove the z-index property from this example? Go on, I’ll wait.

This is what we see:

It turns out that positioned boxes appear on top of non-positioned boxes. If you’re inclined, the spec goes into much more detail.

But if all the boxes are positioned, we revert to following the order in the HTML source. Look at this:

And at that point, specifying a z-index does work!

Using the powers of z-index we were able to reverse the default stacking order of the boxes. Go us.

A new mystery

Let’s look at this example:

Out of the three positioned boxes, the yellow one has z-index: 1, and therefore appears on top. Good, the world is making sense.

But what if we give a z-index to the blue box, the parent of the yellow box? Our example would look like this:

What’s happening here? Our yellow box is now rendered below the red one?! All this because we added a z-index: 0 to the blue one?! The world is making no sense again. 😿

Let’s take a bit of a detour…

Auto is not zero

If we don’t set the z-index property of an element, its default value is going to be auto. Such elements will appear in front of elements with negative z-index values and below elements with positive z-index values. This should convince you that I’m not lying:

So in a way that element behaves as if we set z-index: 0. Indeed, if we add that CSS rule, we see no notable changes in the output:

But things changed a lot! Giving a value to z-index to a relatively positioned box creates a new stacking context.

A stacking what? MDN says:

The stacking context is a three-dimensional conceptualization of HTML elements along an imaginary z-axis relative to the user, who is assumed to be facing the viewport or the webpage. HTML elements occupy this space in priority order based on element attributes.

I thought this is what we were trying to do all along. Let’s keep reading.

The interesting part comes later:

Within a stacking context, child elements are stacked according to the same rules previously explained. Importantly, the z-index values of its child stacking contexts only have meaning in this parent.

That’s the key point. A stacking context will force the z-index of its child stacking contexts to only have a local meaning. Let’s look at our mysterious example again:

Setting z-index: 0 on the blue box changed the meaning of the z-index declaration on the yellow box. Before, the yellow box was out there playing with the big boys. Now, it can only affect stacking within the blue box.

This is a great cause of frustration. No matter how high you set a z-index property, you will never be able to ‘escape’ the parent stacking context. And every time you are struggling with making z-index work you can bet that it’s because something created a stacking context that you can’t escape from.

With this newfound understanding, spend some time explaining why this example makes perfect sense:

Also remember that before we said that positioned elements appear on top of non-positioned elements? Well, now we can generalize by saying that elements that form a stacking context appear on top of non-positioned elements.

The usual suspects

The MDN article lists all cases when a new stacking context is formed. I’m going to go through a list of usual suspects.

Opacity less than 1

Transform and filter effects

Flex child with z-index

Grid child with z-index

Some of these examples might look surprising. In general, the underlying reason why these configurations create a new stacking context is that they render to an offscreen context. But in practice, you don’t need to remember all of them: when you bump into a situation where z-index isn’t working as intended, you can quickly check if there’s a runaway stacking context that’s keeping you locked up.

I recommend going through each one of the examples, playing around with them, and explaining in your head why they make perfect sense. It might be helpful to install a browser extension to help check your intuition and solidify your understanding (Chrome, Firefox).

That’s all I have for you today. As always, thanks for reading!

Permutate parsers, don't validate

Wed, 13 Jan 2021 12:00:00 GMT

“Parse, don’t validate” has been one of my favourite programming articles for some time. The main gist of the article is that, when writing in a type-driven fashion, your snappy slogan should be:

Parse, don’t validate.

The core difference between parsing and validating can be explained by looking at two very similar functions:

parseInt :: String -> Maybe Int
parseInt str = Text.Read.readMaybe str

validateInt :: String -> Bool
validateInt str = Text.Read.readMaybe str /= Nothing

As you can see, they look very similar. The main difference is that parseInt returns a useful value, the Int that we wanted to parse, while validateInt takes that useful value and throws it away. This is also mentioned in the wonderful Haskell Mini-Patterns Handbook as the “Evidence” pattern.

The key issue here is that by calling a function that returns Bool you lose the information about earlier performed validation. Instead, you can keep this information by explicitly pattern-matching on the validation or result.

In this post, I would like to go through a practical example that shows the power of bringing this concept to its limits. Which brings us to…

Advent of Code 2020, Day 4

We are tasked with parsing a batch of passports composed of these fields:

- byr (Birth Year)
- iyr (Issue Year)
- eyr (Expiration Year)
- hgt (Height)
- hcl (Hair Color)
- ecl (Eye Color)
- pid (Passport ID)
- cid (Country ID)

All the fields are required except for the cid field, which is optional. Note that the fields can be written in any order, this will be important later. Our batch is composed of multiple passports separated by empty lines (the input.txt):

ecl:gry pid:860033327 eyr:2020 hcl:#fffffd
byr:1937 iyr:2017 cid:147 hgt:183cm

iyr:2013 ecl:amb cid:350 eyr:2023 pid:028048884
hcl:#cfa07d byr:1929

hcl:#ae17e1 iyr:2013
eyr:2024
ecl:brn pid:760753108 byr:1931
hgt:179cm

hcl:#cfa07d eyr:2025 pid:166559648
iyr:2011 ecl:brn hgt:59in

The first passport is valid - all required fields are present.
The second passport is invalid - it is missing hgt.
The third passport is interesting: the only missing field is the optional cid, which makes it valid.
The fourth passport is missing two fields, cid and byr. Missing cid is fine, but missing byr is not, so this passport is invalid.

The challenge is to count how many passports are valid in the given batch.

Let’s write some code to open the file and parse each group of passport fields:

module Main where

import qualified Data.List.Split as S

main :: IO ()
main = do
  contents <- readFile "input.txt"
  let entries = map parseEntry (S.splitOn "\n\n" contents)
  print entries

data PassportEntry = PassportEntry
  deriving (Show)

parseEntry :: String -> PassportEntry
parseEntry text = undefined

Nothing too fancy here, we’re using Data.List.Split from the split package to do the heavy lifting. And the implementation of parseEntry has been conveniently stubbed so that the code compiles.

Now how should our PassportEntry data structure look like? I’d love to eventually represent passports as:

data Passport = Passport
  { birthYear :: Int,
    issueYear :: Int,
    expirationYear :: Int,
    height :: String,
    hairColor :: String,
    eyeColor :: String,
    passportId :: String,
    countryId :: Maybe Int
  }

If we imagine parsing each field sequentially, we can see that we won’t be able to construct this data structure in a single operation. We’ll have to accumulate the data up until we’re ready to create a proper Passport.

One way to store the fields is to insert them into a hash. First of all, we’re going to use a custom data type to represent the keys of the hash. Why is that? We really don’t want to be making typos later when comparing raw strings like "ecl" and "elc". We’ll use a HashMap from the Data.HashMap.Strict module:

import qualified Data.HashMap.Strict as HM

data PassportField
  = BirthYear
  | IssueYear
  | ExpirationYear
  | Height
  | HairColor
  | EyeColor
  | PassportId
  | CountryId
  deriving (Eq, Show)

type PassportEntry = HM.HashMap PassportField String

Of course things can’t be that easy. We also need to make our type implement the Hashable typeclass:

{-# LANGUAGE DeriveGeneric #-}

import qualified Data.HashMap.Strict as HM
import Data.Hashable
import GHC.Generics (Generic)

data PassportField
  = BirthYear
  | IssueYear
  | ExpirationYear
  | Height
  | HairColor
  | EyeColor
  | PassportId
  | CountryId
  deriving (Eq, Show, Generic)

instance Hashable PassportField

type PassportEntry = HM.HashMap PassportField String

Don’t worry about what we’ve added. Just take them as God-given truths. 👼

Okay, now we can implement our parseEntry function:

import Data.Maybe (mapMaybe)
import qualified Data.Char as Char

parseEntry :: String -> PassportEntry
parseEntry line =
  HM.fromList $
    mapMaybe parseTag $
      S.splitWhen Char.isSpace line

parseTag :: String -> Maybe (PassportField, String)
parseTag value =
  case S.splitOn ":" value of
    ["byr", byr] ->
      Just (BirthYear, byr)
    ["iyr", iyr] ->
      Just (IssueYear, iyr)
    ["eyr", eyr] ->
      Just (ExpirationYear, eyr)
    ["hgt", height] ->
      Just (Height, height)
    ["hcl", color] ->
      Just (HairColor, color)
    ["ecl", color] ->
      Just (EyeColor, color)
    ["pid", pid] ->
      Just (PassportId, pid)
    ["cid", cid] ->
      Just (CountryId, cid)
    _ ->
      Nothing

We try to parse each field (such as byr:2002) into a PassportField type, then end up building a hash using HM.fromList. We can take this for a spin:

Prelude> :l Main.hs

*Main> main
[ fromList
    [ (CountryId, "147"),
      (BirthYear, "1937"),
      (IssueYear, "2017"),
      (HairColor, "#fffffd"),
      (ExpirationYear, "2020"),
      (EyeColor, "gry"),
      (Height, "183cm"),
      (PassportId, "860033327")
    ],
  fromList
    [ (CountryId, "350"),
      (BirthYear, "1929"),
      (IssueYear, "2013"),
      (HairColor, "#cfa07d"),
      (ExpirationYear, "2023"),
      (EyeColor, "amb"),
      (PassportId, "028048884")
    ],
  fromList
    [ (BirthYear, "1931"),
      (IssueYear, "2013"),
      (HairColor, "#ae17e1"),
      (ExpirationYear, "2024"),
      (EyeColor, "brn"),
      (Height, "179cm"),
      (PassportId, "760753108")
    ],
  fromList
    [ (IssueYear, "2011"),
      (HairColor, "#cfa07d"),
      (ExpirationYear, "2025"),
      (EyeColor, "brn"),
      (Height, "59in"),
      (PassportId, "166559648")
    ]
]

Nice and tidy! 🦾

Now our goal is to verify which one of these groups is valid. First of all, we should define a list of required fields:

requiredFields :: [PassportField]
requiredFields =
  [ BirthYear,
    IssueYear,
    ExpirationYear,
    Height,
    HairColor,
    EyeColor,
    PassportId
  ]

We can then define a validation function:

isEntryValid :: PassportEntry -> Bool
isEntryValid entry =
  all (`HM.member` entry) requiredFields

And change our main function to use that:

main :: IO ()
main = do
  contents <- readFile "input.txt"
  let entries = map parseEntry (S.splitOn "\n\n" contents)
  print $ length $ filter isEntryValid entries

Running this yields 2, which is the correct answer! Here is all the code we have written so far, if you’re feeling like you need a refresher.

Advent of Code 2020, Day 4, Part II

In the second part of the challenge, these new rules are added:

- byr (Birth Year) - four digits; between 1920 and 2002.
- iyr (Issue Year) - four digits; between 2010 and 2020.
- eyr (Expiration Year) - four digits; between 2020 and 2030.
- hgt (Height) - a number followed by either cm or in:
  - If cm, the number must be between 150 and 193.
  - If in, the number must be between 59 and 76.
- hcl (Hair Color) - a '#' followed by six chars 0-9 or a-f.
- ecl (Eye Color) - one of: amb blu brn gry grn hzl oth.
- pid (Passport ID) - a nine-digit number.
- cid (Country ID) - ignored, missing or not.

Here is a new batch for us to peruse:

pid:087499704 hgt:74in ecl:grn iyr:2012 eyr:2030 byr:1980
hcl:#623a2f

eyr:1972 cid:100
hcl:#18171d ecl:amb hgt:170 pid:186cm iyr:2018 byr:1926

the first passport is valid
the second passport is invalid (look at the eyr field)

These new requirements are a bit annoying. Our simple approach of checking if all required fields are present won’t work any longer. We can instead implement a isFieldValid function to check if all fields are valid.

isFieldValid :: (PassportField, String) -> Bool
isFieldValid (field, value) =
  case field of
    BirthYear ->
      let v = toInt value
       in length value == 4 && v >= 1920 && v <= 2002
    IssueYear ->
      let v = toInt value
       in length value == 4 && v >= 2010 && v <= 2020
    ExpirationYear ->
      let v = toInt value
       in length value == 4 && v >= 2020 && v <= 2030
    Height ->
      case span Char.isDigit value of
        (num, "cm") ->
          let n = toInt num
           in n >= 150 && n <= 193
        (num, "in") ->
          let n = toInt num
           in n >= 59 && n <= 76
        _ ->
          False
    HairColor ->
      case (length value, value) of
        (7, '#' : rest) ->
          all (`elem` allowedHexChars) rest
        _ ->
          False
    EyeColor ->
      value `elem` validEyeColors
    PassportId ->
      length value == 9 && all Char.isDigit value
    CountryId ->
      all Char.isDigit value

toInt :: String -> Int
toInt = read

validEyeColors :: [String]
validEyeColors =
  ["amb", "blu", "brn", "gry", "grn", "hzl", "oth"]

allowedHexChars :: [Char]
allowedHexChars =
  ['0' .. '9'] <> ['a' .. 'f']

Then we can change our isEntryValid to use this function:

isEntryValid :: PassportEntry -> Bool
isEntryValid entry =
  requiredFieldsPresent && allFieldsValid
  where
    requiredFieldsPresent =
      all (`HM.member` entry) requiredFields

    allFieldsValid =
      all isFieldValid (HM.toList entry)

Running this program on our second data sample yields 1, and it will be good enough to solve the Advent of Code challenge and get us those sweet sweet stars.

🎉 🎉 🎉

A moment of reflection

If we look back at the current state of our code, we can see that we are doing a lot of validations.

We do a lot of work to verify if something is valid, then throw it all out of the window to return a meagre Bool. German folks from the sixteenth century would have told us snarkily:

das Kind mit dem Bade ausschütten

Yes, literally throwing the baby out with the bathwater.

This would be even more true if we were given a new challenge:

Now find the unique set of eye colors in all valid passports

With our current code, we know which passport is valid, but we have no way of extracting the eye color of a valid passport. This is why earlier we were mentioning this sort of Passport representation:

data Passport = Passport
  { birthYear :: Int,
    issueYear :: Int,
    expirationYear :: Int,
    height :: String,
    hairColor :: String,
    eyeColor :: String,
    passportId :: String,
    countryId :: Maybe Int
  }

If we had a function like parsePassport that went from String to Maybe Passport we could then write some code like this:

Set.fromList $
  map eyeColor $
  mapMaybe parsePassport (S.splitOn "\n\n" contents)

But let’s not get too ahead of ourselves. Let’s try to refactor our current code to do something similar. First we can try to write a function like this one:

entryToPassport :: PassportEntry -> Maybe Passport

This function takes the intermediate representation of a collection of passport fields and returns a ‘validated’ passport. We can also reuse our isFieldValid function by using this trick:

parseField ::
  (PassportField, String) -> Maybe (PassportField, String)
parseField tuple =
  if isFieldValid tuple
    then Just tuple
    else Nothing

We are still reusing the validating logic, but we end up returning something useful instead. Remember, we are slowly migrating our code from validating data to parsing data.

Using our new helper we can finally implement the entryToPassport function. We’ll do that in two separate steps. First we’ll get all the values of the required fields:

getAllRequiredFields :: PassportEntry -> Maybe [String]
getAllRequiredFields e =
  traverse
    ( \field -> do
        v <- HM.lookup field e
        (_field, text) <- parseField (field, v)
        return text
    )
    requiredFields

The traverse magic ensures that we either get all the values we’re looking for wrapped in a Just, or Nothing if any of those fields were invalid. Ok, we’re ready to roll now!

entryToPassport :: PassportEntry -> Maybe Passport
entryToPassport entry = do
  case getAllRequiredFields entry of
    Just [byr, iyr, eyr, hgt, hcl, ecl, pid] ->
      Just $
        Passport
          { birthYear = toInt byr,
            issueYear = toInt iyr,
            expirationYear = toInt eyr,
            height = hgt,
            hairColor = hcl,
            eyeColor = ecl,
            passportId = pid,
            countryId = toInt <$> HM.lookup CountryId entry
          }
    _ ->
      Nothing

We end up having to pass String values around, which need to be parsed again into the exact types that we desire. Also we need to pass these values into a list and hope not to mess up the ordering of the fields. So it’s far from perfect, but we’re getting somewhere.

In order to use this function in our main, we replace the last line of the main fuction from:

print $ length $ filter isEntryValid entries

to:

print $ length $ mapMaybe entryToPassport entries

Running this on our test batch still returns 1, which is a good sign we haven’t broken anything.

Still, there is one thing that I’m particularly unhappy about in this code: we use an intermediate representation of the passport that has no real domain value. Nobody cares about PassportField and PassportEntry, but we need to have these types in order to build our Passport.

Not only that, but having these intermediate types means that there are bugs waiting to happen when we transform them to our desired data type:

getting the order of the fields wrong
parsing strings in inconsistent ways
forgetting to validate the presence of required fields

This is also known as shotgun parsing:

Shotgun parsing is a programming antipattern whereby parsing and input-validating code is mixed with and spread across processing code—throwing a cloud of checks at the input, and hoping, without any systematic justification, that one or another would catch all the “bad” cases.

In “Parse, don’t validate”, Alexis King goes on to describe how this is specifically related to parsing and validating:

It may not be immediately apparent what shotgun parsing has to do with validation—after all, if you do all your validation up front, you mitigate the risk of shotgun parsing. The problem is that validation-based approaches make it extremely difficult or impossible to determine if everything was actually validated up front or if some of those so-called “impossible” cases might actually happen. The entire program must assume that raising an exception anywhere is not only possible, it’s regularly necessary.

How can we avoid doing this? Let’s try something new!

A new perspective

We’re going to write the same program using parsec, a monadic parser combinator library. I’ve recently bumped into this excellent walkthrough to parser combinators, which I thouroughly recommend reading.

In short a parser is a type like this one:

type Parser a = Parser {
  runParser :: String -> (String, Either ParseError a)
}

It works by consuming input characters from the input string and returning a tuple with two values:

The first value is what’s left of the input string, so that other parsers can keep parsing the rest of the input.
The second value contains either a parse error or a properly parsed value of type a.

We’ll need to install parsec and add some imports:

import qualified Text.Parsec as P
import Text.Parsec.String (Parser)
import Control.Monad (guard)

Now let’s implement parsers for some fields:

-- byr (Birth Year) - four digits; between 1920 and 2002.
byrParser :: Parser Int
byrParser = do
  P.string "byr"
  P.char ':'
  value <- P.count 4 P.digit
  P.spaces
  let int = read value
  guard (int >= 1920 && int <= 2002)
  return int

-- iyr (Issue Year) - four digits; between 2010 and 2020.
iyrParser :: Parser Int
iyrParser = do
  P.string "iyr"
  P.char ':'
  value <- P.count 4 P.digit
  P.spaces
  let int = read value
  guard (int >= 2010 && int <= 2020)
  return int

-- eyr (Expiration Year) - four digits; between 2020 and 2030.
eyrParser :: Parser Int
eyrParser = do
  P.string "eyr"
  P.char ':'
  value <- P.count 4 P.digit
  P.spaces
  let int = read value
  guard (int >= 2020 && int <= 2030)
  return int

Here we use the guard function to introduce an assertion that will make the parser fail when the condition is not met. In general I feel that this code is quite readable, but we might want to extract a reusable helper to parse years:

yearParser :: String -> (Int, Int) -> Parser Int
yearParser value (rangeStart, rangeEnd) = do
  P.string value
  P.char ':'
  value <- P.count 4 P.digit
  P.spaces
  let int = read value
  guard (int >= rangeStart && int <= rangeEnd)
  return int

byrParser :: Parser Int
byrParser = do
  yearParser "byr" (1920, 2002)

iyrParser :: Parser Int
iyrParser = do
  yearParser "iyr" (2010, 2020)

eyrParser :: Parser Int
eyrParser = do
  yearParser "eyr" (2020, 2030)

🎉

This is the beauty of writing parser combinators. They are extremely easy to reuse and combine.

Now we want to write a parser for the height field. It might be nice to use a more specialized data type to represent that:

data Height
  = InCms Int
  | InInches Int
  deriving (Eq, Show)

Here we go!

-- hgt (Height) - a number followed by either cm or in:
-- If cm, the number must be between 150 and 193.
-- If in, the number must be between 59 and 76.
heightParser :: Parser Height
heightParser = do
  P.string "hgt"
  P.char ':'
  digits <- P.many1 P.digit
  let value = read digits
  result <- unitParser value
  case result of
    InCms _ ->
      guard (value >= 150 && value <= 193)
    InInches _ ->
      guard (value >= 59 && value <= 76)
  P.spaces
  return result

What’s that unitParser you ask?

unitParser :: Int -> Parser Height
unitParser value =
  let cmParser = do
        P.string "cm"
        return (InCms value)

      inParser = do
        P.string "in"
        return (InInches value)
   in P.choice [cmParser, inParser]

The rest of the parsers are basically a step by step translation of the requirements in English:

-- hcl (Hair Color) - a '#' followed by six chars 0-9 or a-f.
hairColorParser :: Parser String
hairColorParser = do
  P.string "hcl"
  P.char ':'
  P.char '#'
  v <- P.count 6 (P.oneOf "0123456789abcdef")
  P.spaces
  return v

-- pid (Passport ID) - a nine-digit number.
passportIdParser :: Parser String
passportIdParser = do
  P.string "pid"
  P.char ':'
  v <- P.count 9 P.digit
  P.spaces
  return v

-- cid (Country ID) - ignored, missing or not.
countryIdParser :: Parser Int
countryIdParser = do
  P.string "cid"
  P.char ':'
  value <- P.many1 P.digit
  P.spaces
  return $ read value

-- ecl (Eye Color) - one of: amb blu brn gry grn hzl oth.
eyeColorParser :: Parser String
eyeColorParser = do
  P.string "ecl"
  P.char ':'
  v <-
    P.choice $
      map
        (P.try . P.string)
        [ "amb",
          "blu",
          "brn",
          "gry",
          "grn",
          "hzl",
          "oth"
        ]
  P.spaces
  return v

You will notice we had to use this mysterious P.try function in the last snippet. This is very useful when we need to look ahead in the input string. Consider the example of blu and brn: after consuming an initial b character we land in the blu branch. If at that point we encounter a r character, we realize we need to go back and choose the brn branch instead. But by default the parsing would stop because we have already consumed the first character. P.try will make it so our parser pretends it hasn’t consumed any input so that we can keep trying other alternatives.

We have now written parsers for each individual field. So now it’s time to combine them all together…

Another problem?!

Remember that the fields can be written in any order? Uh oh.

Since our parser tries to consume input one character at a time, how the heck can we write one that has to deal with randomly ordered input?

It is impossible, right?

No, it’s possible!

The parsec library includes a wonderful Text.Parsec.Perm module:

This module implements permutation parsers. A permutation phrase is a sequence of elements (possibly of different types) in which each element occurs exactly once and the order is irrelevant. Some of the permutable elements may be optional.

Let’s import it:

import Text.Parsec.Perm (permute, (<$$>), (<|?>), (<||>))

Woah, calm down, that’s lots of operators there mate.

permute is the last call that will wrap everything up and return a parser of something.
<$$> is used to assign all the fields that we parsed to something. In our case it will be a Passport.
<||> is used to describe a required field
<|?> is used to describe an optional field

Ready for the big reveal? 🥁🥁🥁

passportParser :: Parser Passport
passportParser =
  permute $
    Passport <$$> byrParser
      <||> iyrParser
      <||> P.try eyrParser
      <||> P.try heightParser
      <||> P.try hairColorParser
      <||> P.try eyeColorParser
      <||> passportIdParser
      <|?> (Nothing, Just <$> countryIdParser)

This parser can parse passports which fields are written in any random order, as long as each required field is present once. Pretty slick, eh?

We just need to wire it up in our main:

import Data.Either (rights)

main :: IO ()
main = do
  contents <- readFile "input.txt"
  let passports =
        rights $
          map
            (P.parse passportParser "")
            (S.splitOn "\n\n" contents)
  print $ length passports

That’s all we need! We can now get rid of the intermediate PassportField and PassportEntry types and all that validation code. Welcome to the wonderful world of parsing!

If you glance over the final listing you’d be surprised to see how tidy it is. We have a description of the input that we’d like to parse and nothing more. No validations, no transformations, no other massaging of the types. That’s the beauty of parsing and expressing real world problems in terms of parsing.

I hope you enjoyed this deep dive into parsing, thanks for reading!

Complicated Haskell Words - Isomorphism

Sun, 29 Nov 2020 12:00:00 GMT

As someone who has been learning Haskell on and off for the last year, I know the struggles of just how complicated Haskell can be for beginners. You are instantly thrown off in a new world full of new terms, each one more obscure than the other. Being familiar with a statically typed language like Elm certainly helped, but I felt I struggled the most with the seemingly endless list of new words I would bump into every day… Isomorphism, functor, monoid, covariant, and so on and so forth.

So my goal here is to explain these terms using the simplest analogies I’ve come across. Some of them might not be the most complete or the most faithful, but my hope is that they will be useful lifeboats when you find yourself lost at sea. I will try to keep this series updated over time by adding new terms and simpler explanations. One word of warning: I won’t be talking about monads since there are plenty of tutorials and burritos out there.

Isomorphism

Let’s start with something that’s used all the time. If two things are isomorphic, it means that for all intents and purposes they are the same. Anytime you read isomorphic, just read more or less the same for what we care about.

This should be enough information for you to get on with your day. You can close the tab now. Thanks for reading!

The rest of this post is about building an intuitive understanding of why isomorphisms are just so damn cool.

Let’s say we want to describe if we are happy or sad.

data Mood = Happy | Sad

Now we could have a function that tells us if we are happy:

areWeHappy :: Mood -> Bool
areWeHappy Happy = True
areWeHappy Sad = False

We could come up with a way to derive a Mood from a Bool:

moodFromBool :: Bool -> Mood
moodFromBool True = Happy
moodFromBool False = Sad

If we take a look at this pair of functions we can see that we can chain them together.

mysteriousFunction =
  -- first run moodFromBool, then areWeHappy
  areWeHappy . moodFromBool

Let’s try plugging some values to this mysterious function:

if we pass True, it gets converted to Happy, then back to True
if we pass False, it gets converted to Sad, then back to False

So we basically implemented an identity function:

id :: a -> a
id x = x

This doesn’t sound too interesting, I know. What’s also not too interesting is that there is another way to combine those two functions:

otherMysteriousFunction =
  -- first run areWeHappy, then moodFromBool
  moodFromBool . areWeHappy

if we pass Happy, it gets converted to True, then back to Happy
if we pass Sad, it gets converted to False, then back to Sad

Another day, another identity function. What’s new?

Intermezzo, or why is zero cool

The ancient Greeks were thoroughly confused by the number zero and used to wonder:

How can nothing be something?

Adding zero to an existing number is a no-op, it doesn’t do anything at all. It’s a complete waste of effort!

The great thing about zero is that it helps us to define a relationship between numbers. The number 3 has now this beautiful property with itself:

3 - 3 = 0

Not only that, without zero we wouldn’t even have negative numbers! Zero stands as the legendary gatekeeper between positive and negative numbers. Now each positive number a has a friend in the nether world a', which relation is defined as:

a + a' = 0

Back to identity functions

That’s the same reason why identity functions are cool. They are a way to establish interesting relationships between other functions.

Let’s do a quick refresher on the functions we had:

areWeHappy :: Mood -> Bool
areWeHappy Happy = True
areWeHappy Sad = False

moodFromBool :: Bool -> Mood
moodFromBool True = Happy
moodFromBool False = Sad

mysteriousFunction :: Bool -> Bool
mysteriousFunction =
  areWeHappy . moodFromBool

If our mysterious function is an identity function, it means we have successfully undone what moodFromBool did in the first place. We were able to implement Control-Z for a function! We went from Bool to Mood and then back to Mood without losing any information.

otherMysteriousFunction :: Mood -> Mood
otherMysteriousFunction =
  moodFromBool . areWeHappy

In the other mysterious function, we were able to go from Mood to Bool and then back to Mood without losing information.

The definition™

Two types A and B are isomorphic if two conversion functions exist such that:

f :: A -> B
g :: B -> A

And these two properties are valid:

f . g = id
g . f = id

Having this pair of functions means that we can losslessly convert between values of type A and values of type B.

So this is what we meant at the beginning when we said that for all intents and purposes A and B are the same!

Many isomorphisms

Note that the multiple isomorphisms can exist between two types. For example we could have this:

areWeSad :: Mood -> Bool
areWeSad Sad = True
areWeSad Happy = False

moodFromBoolSad :: Bool -> Mood
moodFromBoolSad True = Sad
moodFromBoolSad False = Happy

What’s the lesson here? All isomorphisms are born equal, and no isomorphism is more equal than others.

Some examples of isomorphisms

We can easily prove that (a, b) and (b, a) are isomorphic by importing the Data.Tuple module:

f :: (a, b) -> (b, a)
f = swap

g :: (b, a) -> (a, b)
g = swap

In the same module, there’s another couple of functions:

curry :: ((a, b) -> c) -> a -> b -> c
uncurry :: (a -> b -> c) -> (a, b) -> c

Those type signatures look quite funky, right? curry lets us apply a function, originally intended to be used on a tuple, on two separate arguments:

Prelude> :t fst
fst :: (a, b) -> a

Prelude> fst (1, 2)
1

Prelude> curry fst 1 2
1

While uncurry lets us apply a function, originally intended to be used on two arguments, on a tuple:

Prelude> (+) 1 2
3

Prelude> uncurry (+) (1, 2)
3

Let’s prove that these functions create an isomorphism:

Prelude> uncurry (curry fst) (1, 2)
1

Prelude> curry (uncurry (+)) 1 2
3

After looking at the type signatures again we can say that these following functions are isomorphic:

((a, b) -> c) -> a -> b -> c
(a -> b -> c) -> (a, b) -> c

Pretty neat!

Unit

In Haskell you would sometimes see this type:

foo :: ()

It’s an empty tuple, which is referred to as the unit. It can contain a single value, which can only be:

()

It turns out to be quite useful when you need a type that contains a single value.

For example, we could prove that Bool and Either () () are isomorphic:

f :: Bool -> Either () ()
f True = Right ()
f False = Left ()

g :: Either () () -> Bool
g (Right ()) = True
g (Left ()) = False

Or that Maybe a is isomorphic to Either () a

f :: Maybe a -> Either () a
f (Just x) = Right a
f Nothing = Left ()

g :: Either () a -> Maybe a
g (Right a) = Just a
g (Left ()) = Nothing

One more thing…

Peano numbers are a simple way of representing natural numbers using a zero value and a successor function:

data PeanoNum
  = Zero
  | Succ PeanoNum
  deriving (Show)

We can construct an isomorphism between this representation and Int (let’s just pretend for a moment that negative numbers aren’t a thing):

toInt :: PeanoNum -> Int
toInt Zero = 0
toInt (Succ x) = toInt x + 1

fromInt :: Int -> PeanoNum
fromInt 0 = Zero
fromInt x = Succ (fromInt (x - 1))

Let’s take it for a spin:

Prelude> toInt Zero
0

Prelude> toInt (Succ (Succ Zero))
2

Prelude> fromInt 3
Succ (Succ (Succ Zero))

Prelude> toInt (fromInt 3)
3

🎉

Thanks for reading, I hope you have emerged from this with a newfound sense of awe for isomorphisms and identity functions!

A walkthrough of Elm Warrior

Fri, 09 Oct 2020 12:00:00 GMT

I’ve recently bumped into Elm Warrior, an Elm adaptation of Warrior JS. The premise of the game is to write a simple AI that is able to navigate through multiple levels of a dungeon. This is how the first level looks like:

You (the warrior) are represented by the @. You always start on the blue tile and your goal is to reach the green tile. I thought it would be a cool idea to write down how to approach solving a problem like this.

Robin recently wrote a great introduction to how to play Elm Warrior and I wholeheartedly recommend reading it if you’re just starting out.

Before we begin, a word of warning…

SPOILERS AHEAD!

SPOILERS AHEAD!!

SPOILERS AHEAD!!!

Now that my conscience is clear, I’ll start by copying the exposed values and functions from each top-level module and pasting them all together in one file. No docs, no type signatures, nothing, just their names. As the sage once said:

Stat rosa pristina nomine, nomina nuda tenemus.

Which roughly translates as:

The rose of old remains only in its name. Only naked names we possess.

Let’s get moving, shall we?

module Warrior exposing
    ( Warrior, Action(..)
    , id, position
    , health, maxHealth, healingPotential
    , attackDamage, inventory
    )

module Warrior.Coordinate exposing
    ( Coordinate
    , toString
    )

module Warrior.Direction exposing
    ( Direction(..)
    , all, toString
    )

module Warrior.History exposing
    ( History
    , roundsPlayed, previousStates, previousActions
    )

module Warrior.Item exposing
    ( Item(..)
    , toString
    )

module Warrior.Map exposing
    ( Map
    , look, lookDown
    )

module Warrior.Program exposing
    ( TurnFunction
    , Model, Msg
    , Config, program
    )

At this point we clearly don’t understand what is going on with the code, but just by reading this listing a couple of times we are getting familiar with the names of this particular domain. Just let them soak in.

Now I’m getting a tad more curious and will take a look at these exposed data constructors: Direction, Action, and Item. Whenever a module exposes the inner contents of a type, that’s an invitation to the world to depend on those values, so I know it’s going to be useful to take a peek.

type Direction
    = Left
    | Right
    | Up
    | Down

type Action
    = Wait
    | Heal
    | Pickup
    | Move Direction
    | Attack Direction

type Item
    = Sword
    | Potion

There is a lot of information packed in these simple names, so don’t worry if you feel a bit overwhelmed. Instead, let’s try something practical and look at the first level.

Level I

Let’s start here.

The source on the left hand side reads like this:

main : Program () Program.Model Program.Msg
main =
    Program.program
        { maps = Maps.all
        , players = [ ( "Player", takeTurn ) ]
        , msPerTurn = 500
        , progressionFunction = Progression.reachExitPoint
        }


takeTurn : Warrior -> Map -> History -> Warrior.Action
takeTurn warrior map history =
    Warrior.Wait

We can see on the right hand side that every 500 ms a Player waits message is being printed. So instead of simply waiting, we have to make our warrior move to the right. Let’s import Direction and change the takeTurn function:

import Warrior.Direction exposing (..)

takeTurn : Warrior -> Map -> History -> Warrior.Action
takeTurn warrior map history =
    Warrior.Move Right

From now on I will provide Ellie links to each step, so don’t worry about copying and pasting all the changes in the code.

Onwards and upwards!

Level II

You’ll see that after completing level I, your warrior will be automatically teleported to the next level, which looks like this:

Our warrior can only move right each step, and since the dungeon turns downwards, it gets stuck there. Poor fella.

It’s time to look more in depth at a function inside the Map module:

{-| Provides a list of everything the warrior can see in a
specific direction. The first item of the list will be the
one tile away. The second item will be two tiles away, etc.
-}
look : Direction -> Warrior -> Map -> List (Coordinate, Tile)

What’s that Tile value?

type Tile
    = Wall
    | Empty
    | SpawnPoint
    | Exit
    | Warrior String
    | Item Item

The simplest way to pass this level is:

check if we can move right; if so, proceed
check if we can move down; if so, proceed.

takeTurn : Warrior -> Map -> History -> Warrior.Action
takeTurn warrior map history =
    case Map.look Right warrior map of
        ( _, Empty ) :: _ ->
            Warrior.Move Right

        ( _, Exit ) :: _ ->
            Warrior.Move Right

        _ ->
            case Map.look Down warrior map of
                ( _, Empty ) :: _ ->
                    Warrior.Move Down

                ( _, Exit ) :: _ ->
                    Warrior.Move Down

                _ ->
                    Warrior.Wait

I know, I know, there is some duplication. Let’s resist the urge to refactor straight away and enjoy our fleeting moment of success. 🍵

Level III

Here we are.

Things are getting more complicated, and it’s clear that our nesting of case expressions won’t fit the bill any longer. We need our warrior to try to walk into any possible direction. We can do this by:

folding over a list of possible directions
checking if we found a good direction to move
if yes, we are done
if no, try with the next direction.

To represent the idea that we might have found a good move we’re going to use a Maybe Warrior.Action as the value of the accumulator.

takeTurn : Warrior -> Map -> History -> Warrior.Action
takeTurn warrior map history =
    let
        checkDirection dir =
            case Map.look dir warrior map of
                ( _, Empty ) :: _ ->
                    Just (Warrior.Move dir)

                ( _, Exit ) :: _ ->
                    Just (Warrior.Move dir)

                _ ->
                    Nothing
    in
    [ Right, Down, Left, Up ]
        |> List.foldl
            (\dir acc ->
                case acc of
                    Just v ->
                        acc

                    Nothing ->
                        checkDirection dir
            )
            Nothing
        |> Maybe.withDefault Warrior.Wait

Unfortunately this is the result…

The problem with our approach is that we’re using a stateless algorithm and we’re not checking if we already visited the same tile in the past. Now let’s take another look at the History module. We’ll find this function:

{-| Returns a list of the state of a warrior and the map,
going back to the first turn. The first element in the list
will represent the state of the warrior and the map at the
beginning of that warrior's last turn.
-}
previousStates : Warrior -> History -> List ( Warrior, Map )

What we get is a list of all previous states of our warrior, along with the state of the map at the time. So we can use the position function from the Warrior module to detect if we ever visited that tile before.

takeTurn : Warrior -> Map -> History -> Warrior.Action
takeTurn warrior map history =
    let
        alreadyVisited coords =
            History.previousStates warrior history
                |> List.any
                    (\( w, _ ) ->
                        Warrior.position w == coords
                    )

        checkDirection dir =
            case Map.look dir warrior map of
                ( newCoords, Empty ) :: _ ->
                    if alreadyVisited newCoords then
                        Nothing

                    else
                        Just (Warrior.Move dir)

                ( _, Exit ) :: _ ->
                    Just (Warrior.Move dir)

                _ ->
                    Nothing
    in
    [ Right, Down, Left, Up ]
        |> List.foldl
            (\dir acc ->
                case acc of
                    Just v ->
                        acc

                    Nothing ->
                        checkDirection dir
            )
            Nothing
        |> Maybe.withDefault Warrior.Wait

Tada! 🎉

Level IV

We are here now.

Our warrior is now stuck on the top right corner. Why is that? Well, we just added some logic that makes the warrior skip tiles which have been visited in the past. In this case, we went down a path which turned out to be a dead end, so we need to backtrack.

Instead of completely avoiding to step on a tile that has already been visited, we can think which are the possibilities when we want to explore the world in a certain direction. We can find:

the exit tile
an empty unvisited tile
an empty already visited tile
a wall tile

What we can do is to assign a value to each of these possibilities, so that we can choose the best available move every time. One great thing about Elm is how easy it is to create types for everything:

type Exploration
    = Freedom
    | CanMoveUnvisited
    | CanMoveVisited
    | NoGo

As a matter of fact, we are very interested in how many times we have already visited a tile, because we can use that information to resolve ties when deciding which cells to visit. Let’s change the type to reflect this:

type Exploration
    = Freedom
    | CanMove Int
    | NoGo

The Int that we attached to CanMove describes how many times we have visited that tile in the past. So now all we have to do is to rewrite our takeTurn function to consider each move and select the best. We can do that using List.sortWith:

{-| Sort values with a custom comparison function.

    sortWith flippedComparison [1,2,3,4,5] == [5,4,3,2,1]

    flippedComparison a b =
        case compare a b of
          LT -> GT
          EQ -> EQ
          GT -> LT

This is also the most general sort function, allowing you
to define any other: `sort == sortWith compare`
-}
sortWith : (a -> a -> Order) -> List a -> List a

But we need to provide our own comparison function…

compareExploration : Exploration -> Exploration -> Order
compareExploration first second =
    if first == second then
        EQ

    else
        case ( first, second ) of
            ( Freedom, _ ) ->
                LT

            ( CanMove x, CanMove y ) ->
                compare x y

            ( CanMove _, NoGo ) ->
                LT

            ( CanMove _, _ ) ->
                GT

            ( NoGo, _ ) ->
                GT

You will notice that better options have a “lower value”: in this way when we’re done sorting our list, the best option will be the head of the list.

Now all we have to do is to change takeTurn to explore each direction and choose the best option:

takeTurn : Warrior -> Map -> History -> Warrior.Action
takeTurn warrior map history =
    let
        visitedCount coords =
            History.previousStates warrior history
                |> List.filter
                    (\( w, _ ) ->
                        Warrior.position w == coords
                    )
                |> List.length

        exploreDirection dir =
            case Map.look dir warrior map of
                ( newCoords, Empty ) :: _ ->
                    ( CanMove (visitedCount newCoords)
                    , Warrior.Move dir
                    )

                ( newCoords, SpawnPoint ) :: _ ->
                    ( CanMove (visitedCount newCoords)
                    , Warrior.Move dir
                    )

                ( _, Exit ) :: _ ->
                    ( Freedom
                    , Warrior.Move dir
                    )

                _ ->
                    ( NoGo
                    , Warrior.Wait
                    )
    in
    [ Right, Down, Left, Up ]
        |> List.map exploreDirection
        |> List.sortWith
            (\( a, _ ) ( b, _ ) -> compareExploration a b)
        |> List.head
        |> Maybe.map Tuple.second
        |> Maybe.withDefault Warrior.Wait

At this point we can grab some popcorn and watch the progress…

Our warrior managed to complete in a swift dash levels four, five, six and seven!

Level VIII

We are stuck here now.

We have met an enemy, exciting! Let’s add another option to Exploration:

type Exploration
    = Freedom
    | Enemy
    | CanMove Int
    | NoGo

Then follow some type errors to fix a missing branch in the case statement within our compareExploration function:

compareExploration : Exploration -> Exploration -> Order
compareExploration first second =
    if first == second then
        EQ

    else
        case ( first, second ) of
            ( Freedom, _ ) ->
                LT

            ( Enemy, _ ) ->
                LT

            ( CanMove x, CanMove y ) ->
                compare x y

            ( CanMove _, NoGo ) ->
                LT

            ( CanMove _, _ ) ->
                GT

            ( NoGo, _ ) ->
                GT

Then, when we meet another warrior in our path we can attack them!

exploreDirection dir =
    case Map.look dir warrior map of
        ( newCoords, Empty ) :: _ ->
            ( CanMove (visitedCount newCoords)
            , Warrior.Move dir
            )

        ( newCoords, SpawnPoint ) :: _ ->
            ( CanMove (visitedCount newCoords)
            , Warrior.Move dir
            )

        ( newCoords, Warrior _) :: _ ->
            ( Enemy
            , Warrior.Attack dir
            )

        ( _, Exit ) :: _ ->
            ( Freedom
            , Warrior.Move dir
            )

        _ ->
            ( NoGo
            , Warrior.Wait
            )

Success! 🎉

Level IX

Let’s check this out.

Now, this is a tricky one. We’re facing a guard that’s much stronger than us. If we remain in the same spot and keep exchanging blows, we’ll die!

As the level suggests, when we notice we’re not feeling well, we can try to heal:

{-| Get the health of a warrior.
-}
health : Warrior -> Int


{-| Get the maximum health the warrior can have.
    The warrior will start a map with this much health.
-}
maxHealth : Warrior -> Int

The issue is that if we try to heal while standing near the enemy, it will keep attacking us, ultimately leading to our ghastly demise.

Let’s do a bit of refactoring first: we’re going to split up our takeTurn function into a couple of smaller functions:

allDirections : List Direction
allDirections =
    [ Right, Down, Left, Up ]


exploreDirection :
    Warrior
    -> Map
    -> History
    -> Direction
    -> ( Exploration, Warrior.Action )
exploreDirection warrior map history dir =
    let
        visitedCount coords =
            History.previousStates warrior history
                |> List.filter
                    (\( w, _ ) ->
                        Warrior.position w == coords
                    )
                |> List.length
    in
    case Map.look dir warrior map of
        ( newCoords, Empty ) :: _ ->
            ( CanMove (visitedCount newCoords)
            , Warrior.Move dir
            )

        ( newCoords, SpawnPoint ) :: _ ->
            ( CanMove (visitedCount newCoords)
            , Warrior.Move dir
            )

        ( newCoords, Warrior _ ) :: _ ->
            ( Enemy
            , Warrior.Attack dir
            )

        ( _, Exit ) :: _ ->
            ( Freedom
            , Warrior.Move dir
            )

        _ ->
            ( NoGo
            , Warrior.Wait
            )


takeTurn : Warrior -> Map -> History -> Warrior.Action
takeTurn warrior map history =
    allDirections
        |> List.map (exploreDirection warrior map history)
        |> List.sortWith
            (\( a, _ ) ( b, _ ) -> compareExploration a b)
        |> List.head
        |> Maybe.map Tuple.second
        |> Maybe.withDefault Warrior.Wait

Now we can add a function to check if an enemy is nearby:

isEnemyNearby : Warrior -> Map -> Bool
isEnemyNearby warrior map =
    allDirections
        |> List.any
            (\dir ->
                case Map.look dir warrior map of
                    ( _, Warrior _ ) :: _ ->
                        True

                    _ ->
                        False
            )

And another one to tell if our warrior has been wounded:

isWounded : Warrior -> Bool
isWounded warrior =
    Warrior.health warrior < Warrior.maxHealth warrior

In our exploreDirection function, when we meet another warrior we can decide if we’re in a fighting mood:

exploreDirection warrior map history dir =
    case Map.look dir warrior map of
        ( newCoords, Warrior _ ) :: _ ->
            if isWounded warrior then
                ( NoGo
                , Warrior.Wait
                )

            else
                ( Enemy
                , Warrior.Attack dir
                )

        -- loads of other case branches

And inside takeTurn:

takeTurn : Warrior -> Map -> History -> Warrior.Action
takeTurn warrior map history =
    if not (isEnemyNearby warrior map) && isWounded warrior then
        Warrior.Heal

    else
        allDirections
            |> List.map (exploreDirection warrior map history)
            |> List.sortWith
                (\( a, _ ) ( b, _ ) -> compareExploration a b)
            |> List.head
            |> Maybe.map Tuple.second
            |> Maybe.withDefault Warrior.Wait

What happens now?

Turns out that as soon as we move near the other warrior, we get attacked. Therefore, we immediately run away to heal, then try again, then run away again. I’m a pacifist at heart, but still I’d like to pass this level.

What’s the solution this time? To endure a little bit of pain…

Instead of running away as soon as we get slightly wounded, we can keep fighting for a bit and only run away when we’re badly wounded.

isBadlyWounded : Warrior -> Bool
isBadlyWounded warrior =
    Warrior.health warrior < Warrior.maxHealth warrior // 4

Now, we’re going to replace isWounded inside exploreDirection. We still want the original function inside takeTurn because we want to heal back to full health when we’re safe.

exploreDirection warrior map history dir =
    case Map.look dir warrior map of
        ( newCoords, Warrior _ ) :: _ ->
            if isBadlyWounded warrior then
                ( NoGo
                , Warrior.Wait
                )

            else
                ( Enemy
                , Warrior.Attack dir
                )

        -- loads of other case branches

Victory! 🎊

Level X

We’re here now.

Ooh, items! Let’s go and grab them 🏴‍☠️

We’ll need to add a new branch in the case statement in exploreDirection:

( newCoords, Item _ ) :: _ ->
    ( CanMove (visitedCount newCoords)
    , Warrior.Move dir
    )

And change our takeTurn function to pickup the item if we are standing on it:

case
    ( isEnemyNearby warrior map
    , isWounded warrior
    , Map.lookDown warrior map
    )
of
    ( False, True, _ ) ->
        Warrior.Heal

    ( _, _, Item _ ) ->
        Warrior.Pickup

    _ ->
        allDirections
            |> List.map (exploreDirection warrior map history)
            |> List.sortWith
                (\( a, _ ) ( b, _ ) -> compareExploration a b)
            |> List.head
            |> Maybe.map Tuple.second
            |> Maybe.withDefault Warrior.Wait

These changes will get us through both levels ten and eleven!

One more thing…

On the last level we can notice something a bit weird:

After we run away from the fight and we heal ourselves to full health we start doing a weird left and right dance on the first two tiles. Why is that?

It turns out we have a small bug in our visitedCount helper:

visitedCount coords =
    History.previousStates warrior history
        |> List.filter
            (\( w, _ ) ->
                Warrior.position w == coords
            )
        |> List.length

Since we spend many turns healing, we end up having a lot of states in our history where the position of the warrior is the same, which ends up inflating the number of times we have “visited” that tile. The fix is to only consider the position of the warrior when it’s moving:

visitedCount coords =
    History.previousStates warrior history
        |> List.foldr
            (\elem acc ->
                if List.head acc == Just elem then
                    acc

                else
                    elem :: acc
            )
            []
        |> List.filter
            (\( w, _ ) ->
                Warrior.position w == coords
            )
        |> List.length

With our final fix in place, it’s showtime!

Thanks for reading! 👋

Writing a program that writes itself

Mon, 04 Nov 2019 12:00:00 GMT

Recursion is one of those concepts in programming which have always been incredibly fascinating to me. What do you mean that I can call the function from the function itself?

Well, there is a class of programs that bring the whole idea of recursion to a completely different level, which is programs that are able to replicate themselves. These programs, named Quines, produce a copy of their own source code as their only output.

The name “quine” was introduced by Douglas Hofstadter in Gödel, Escher, Bach: An Eternal Golden Braid in honor of Willard Van Orman Quine, an American analytic philosopher who made extensive studies in the field of self-reference. There is a paradox named after him that goes:

“Yields falsehood when preceded by its quotation” yields falsehood when preceded by its quotation.

This is a more elaborate version of the Liar paradox, where the sentence Yields falsehood when preceded by its quotation, when preceded by itself in quotation marks, gives a new expression. If this new expression is true, then it says of itself that it must be false. But if it’s false, then what it says of itself must be true.

Pretty puzzling, huh?

Some rules

Before we start and try to write a Quine, we have to define some ground rules:

The program must produce its own source code when executed
It must not use I/O to read its own source code
It must not be an empty program

A first attempt

If we had this simple Ruby program:

puts "hello"

And we executed it, we would get:

$ ruby quine.rb
hello

So we could make our program to look like this:

puts "puts \"hello\""

But running that would only yield:

$ ruby quine.rb
puts "hello"

As we can see, this approach won’t get us closer to the solution since we’re always going to miss the outer puts.

A bit of self reflection

Luckily in Ruby there is a simple way to escape this using Object#inspect:

src = "puts src"
puts "src = " + src.inspect

$ ruby quine.rb
src = "puts src"

But we’re still missing the last line. What if we added the src again to the end of our puts statement, but this time as a normal string?

src = "puts src"
puts "src = " + src.inspect + src

$ ruby quine.rb
src = "puts src"puts src

Uh! Since the contents of the src string don’t really matter to the behaviour of our program, we can get quite creative with it!

src = "\nputs \"src = \" + src.inspect + src"
puts "src = " + src.inspect + src

$ ruby quine.rb
src = "\nputs \"src = \" + src.inspect + src"
puts "src = " + src.inspect + src

Tada! 🎉

How to write your own Quine

This simple example showed us some of the core ideas for writing a Quine:

We build the program in two parts, one that we call the code and one that we call the data. The data part of the program is derived from the code part, usually by some means of escaping.
When executed, the code first uses the data to print the data.
Then the code uses the data to print the code. This is generally easy because we intentionally derived the data from the code.

A great analogy of a Quine comes from cellular biology: if we think of the cell as the code and the cell’s DNA as the data, then we’ll see that the cell is able to replicate itself using the DNA, and this process involves replicating the DNA itself. On the other hand the DNA contains all the information for replicating the cell, but without the cell itself it would just be an inert piece of data.

Quine Quine

Douglas Hofstadter also invented the verb to quine, which means “to write something a first time, and then to write it a second time with quotation marks around it”. For example, if we quine “say”, we would get “say ‘say’“. If we were to quine “quine”, we would get “quine ‘quine’”, which itself is a quine…

Testing, testing, testing

When you’re trying to write your own Quine, you’ll get to a point when you think you’ve reached a solution, but end up having to compare two printouts which are not exactly human friendly. So you can use this bit of Bash instead:

$ diff quine.rb <(ruby quine.rb) && echo "IS QUINE"
IS QUINE

If you had make a mistake, it would print the diff like this:

$ diff wrong.rb <(ruby wrong.rb) && echo "IS QUINE"
2c2
< puts "src = " + src.inspect + src
---
> puts src = " + src.inspect + src

The weird looking <() is a Bash feature called process substitution that allows you to pass one command’s output to another program as if it were a file name.

A Quine in Haskell

Here is a very elegant solution in Haskell:

main = putStr src >> print src
  where src = "main = putStr src >> print src\n  where src = "

This works thanks to the where notation, where you can use a variable in an expression before defining it. We first print the code using the normal putStr function, then we print the data using print, which maintains the double quotes around the string. The beauty of this solution is that the data is barely escaped!

Here it is in action:

$ ghc quine.hs
[1 of 1] Compiling Main             ( quine.hs, quine.o )
Linking quine ...

$ diff quine.hs <(./quine) && echo "IS QUINE"
IS QUINE

A Quine in Elm?

We can almost directly port our Ruby code to Elm. The only challenge is that browsers don’t usually recognize newlines literally, so that makes escaping code quite daunting. But fear not, thanks to the amazing powers of CSS and the white-space property, we can make the browser behave:

span {
  white-space: pre;
  font-family: monospace;
}

The other challenge is to be able to inspect the data in a literal way: this can be achieved in Elm using the Debug#toString function.

So I challenge you to write your own implementation of a Quine in Elm. If you get stuck at any point, you can check this working example on Ellie.

Happy Quining!

References

Performant Elm, Part III, Html.Lazy

Mon, 21 Oct 2019 12:00:00 GMT

Greetings traveler! This is the third instalment of a series on how to make your Elm applications more performant. If you are into this kind of stuff, you can read about how to tell what your browser is really doing and how Html.Keyed can help you help Elm to be faster.

Today we are talk about being lazy. As an Italian, I know a thing or two about being lazy. I’m not talking “Oh I spent Sunday binge-watching Derry Girls” lazy, but more something like “I got caught with 400kg of undelivered mail in my flat” sort of lazy.

So the Elm Guide does a fantastic job at explaining what Html.Lazy does. I really recommend going there and reading about it. Anyway, the main idea of it is this function:

lazy : (a -> Html msg) -> a -> Html msg

As you can see, we pass two arguments:

a function that takes a thing
a thing

When you’re learning about functional programming, it’s common to bump into something called referential transparency (don’t open that link, that’s five minutes of your life you’re never getting back). In Elm it’s simply called:

Same input, same output

You’re always guaranteed that any function in Elm will return the same output, if you pass the same input. So in the context of web applications, by using lazy you are telling Elm that you know for sure that if the data ain’t changing, then the view shouldn’t change either.

Intermezzo, Sieve of Eratosthenes

The Sieve of Eratosthenes is a very ancient algorithm devised for finding all prime numbers up to a given limit. It works by marking each multiple of a prime number as non-prime, so that when we’re done we’re left with all the prime numbers. Its beauty can be admired in the following GIF:

A naive implementation of the algorithm in Elm would look like this:

sieve : Int -> List Int
sieve limit =
    let
        numbers =
            List.range 2 limit

        last =
            limit
                |> toFloat
                |> sqrt
                |> round

        isMultiple n m =
            n /= m && modBy m n == 0
    in
    List.range 2 last
        |> List.foldl
            (\current result ->
                List.filter
                    (\elem -> not (isMultiple elem current))
                    result
            )
            numbers

Now we can use this function to make our Elm code as slow as we want 🎉

Our testbed example

I have built a small Ellie app that calculates the number of prime numbers up to a certain limit, which you can find here. Here’s a GIF of the app in action:

We are going to focus on the view function, which looks like this:

view : Model -> Html Msg
view model =
    div []
        [ button [ onClick Toggle ] [ text "Toggle color" ]
        , div
            [ -- style stuff
            ]
            [ viewPrime 200000 ]
        ]

viewPrime : Int -> Html Msg
viewPrime limit =
    text
        ("There are "
            ++ String.fromInt (sieve limit |> List.length)
            ++ " prime numbers between between 2 and "
            ++ String.fromInt limit
        )

You can notice there is a noticeable lag in the UI after pressing the “Toggle Color” button: this happens because after receiving the Toggle event, the Elm runtime needs to rerun the view function, which triggers again the sieve computation.

Html.Lazy to the rescue

We can change this line from:

[ viewPrime 200000 ]

to this:

[ Html.Lazy.lazy viewPrime 200000 ]

and we get this result:

We can now see that after the initial call, we never really get inside the viewPrime function anymore, thus making our application blazing fast. 🤟

Happily ever after?

Unfortunately, as almost everything in else in life, nothing is as simple as it seems. If you have ever tried to use Html.Lazy and failed to see any improvements, you know how frustrating it can be. So I’m going to list some common reasons on why it might not be working.

Using anonymous functions

A very simple way to break Html.Lazy is to replace this line:

[ Html.Lazy.lazy viewPrime 200000 ]

with this:

[ Html.Lazy.lazy (\n -> viewPrime n) 200000 ]

You can check it out here. Why is that? Because Html.Lazy needs to associate the cached value with a precise function, but specifying an anonymous function forces the runtime to recreate that function every time the view is invoked.

Constructing records on the fly

A similar problem happens when we construct a record on the fly and pass it to Html.Lazy. Imagine our viewPrime looked like this:

viewPrime : { limit : Int } -> Html Msg
viewPrime { limit } =
    text
        ("There are "
            ++ String.fromInt (sieve limit |> List.length)
            ++ " prime numbers between between 2 and "
            ++ String.fromInt limit
        )

And now we called it by writing:

[ Html.Lazy.lazy viewPrime { limit = 200000 } ]

You will see the app going back to being slow again. This happens because when we build a new record we invalidate the caching mechanism of Html.Lazy.

Reference equality

The Elm Guide has this very interesting section:

Note: When are two values “the same” though? To optimize for performance, we use JavaScript’s === operator behind the scenes:

Structural equality is used for Int, Float, String, Char, and Bool.
Reference equality is used for records, lists, custom types, dictionaries, etc.

Structural equality means that 4 is the same as 4 no matter how you produced those values. Reference equality means the actual pointer in memory has to be the same. Using reference equality is always cheap O (1), even when the data structure has thousands or millions of entries. So this is mostly about making sure that using lazy will never slow your code down a bunch by accident. All the checks are super cheap!

Let’s try to reproduce this error case by changing the viewPrime function to take a list of integers:

viewPrime : List Int -> Html Msg
viewPrime limits =
    let
        line limit =
            div []
                [ text
                    ("There are "
                        ++ String.fromInt (sieve limit |> List.length)
                        ++ " prime numbers between between 2 and "
                        ++ String.fromInt limit
                    )
                ]
    in
    div [] (List.map line limits)

When you click on the toggle now, you’ll see that we’re back to being slow.

So what can we do when we need to use a record, a list, a dictionary or a custom type as a thing to pass to lazy?

The easiest solution is to store it inside our Model:

type alias Model =
    { color: Color
    , limits: List Int
    }

Now the reference won’t change as long as we don’t change our model. Check out the updated version here.

The End

Remember to always benchmark your application before and after Html.Lazy, so you can prove to your team lead that you have actually made the application faster 🤡

This article concludes the series on writing performant Elm, thanks for reading!

Performant Elm, Part II, Html.Keyed

Mon, 07 Oct 2019 12:00:00 GMT

This is the second post in a series which explores optimization techniques to build performant Elm applications. But first, let’s take a bit of a detour and talk about Saint Thomas.

Saint Thomas was one of twelve apostles of Jesus and he is also known as Doubting Thomas. Why is that? Because of one specific episode in the New Testament. In the Gospel of John it is said that when he heard that Jesus had risen from the dead and appeared to the other apostles, he said:

Except I shall see on his hands the print of the nails, and put my finger into the print of the nails, and thrust my hand into his side, I will not believe.

But when Jesus appeared later and invited Thomas to touch his wounds and behold him, Thomas believed and exclaimed “My Lord and my God”. At which point Jesus said:

Thomas, because thou hast seen me, thou hast believed: blessed are they that have not seen, and yet have believed.

Well, my point about optimization techniques in general is: be more like Saint Thomas, aka Doubting Thomas. Don’t believe what the documentation says, but instead go and see for yourself. Then, believe.

So first of all I would recommend reading the first post in this series, which shows you how to use your browser profiling tools to perform repeatable measurements.

What is Html.Keyed

The wonderful Elm Guide tells us that Html.Keyed is particularly helpful for lists of data in your interface that must support insertion, removal, and reordering.

Imagine you have a list of words in ascending order: now you change to sort order to be descending. The virtual DOM algorithm will then:

Compare the previous first word with the current first word
Compare the previous second word with the current second word
…

As you can tell, the algorithm will have to compare each and every one of our words! Instead, it could have done a much faster job and rearranged the nodes in the DOM to appease our new sort order.

So the basic idea of Html.Keyed is that you associate a String to your Html msg so that the virtual DOM diffing algorithm knows in advance which nodes have to be compared.

A working example

Let’s unleash our inner Doubting Thomas and test out these assumptions. In most UNIX systems there is a dictionary of words in /usr/share/dict/words, so I’ve taken some of those and wrote a small Ellie app to display them. It looks something like this:

To simulate a slighly more complex HTML structure, I’ve broken each word into characters and repeated them twenty times. Also, if you click the dropdown at the top of the page, it will toggle between ascending and descending sorting.

If we record a performance readout of this example, this is what we’ll see:

Let’s break down the graph (you can click on the image to see more details):

From the moment we click the button, we spend around 1000ms in scripting
Then follows 250ms of style recalculation
Then follows 2000ms of layout rendering
Which is followed by 500ms of layer tree updating

What I would like to focus on is the right hand side of the scripting portion: you can see that there are a lot of small function calls happening at the end. If you zoom in, you will see that the root function _VirtualDom_applyPatches has instantiated many thousands of times _VirtualDom_applyPatchesHelp. You can also see this in the “Bottom-Up” view:

Introducing Html.Keyed

Now let’s replace the current rendering logic with Html.Keyed. We’ll replace this line of code:

Html.ul [ Attr.class "entries" ]
    (List.map viewEntry sortedEntries)

with this line of code:

Html.Keyed.ul [ Attr.class "entries" ]
    (List.map viewKeyedEntry sortedEntries)

and implement this new view helper:

viewKeyedEntry : Entry -> ( String, Html Msg )
viewKeyedEntry entry =
    ( String.fromInt entry.id, viewEntry entry )

You can play around with the new Ellie app. Now let’s make another performance measurement!

As you can see, the general shape of the performance graph hasn’t changed. The browser is still spending most of the time recalculating styles, updating the layout and the layer tree. Unfortunately Html.Keyed is not black magic and cannot make your application faster if the bottleneck is somewhere else.

But if we zoom in the scripting section we’ll see an interesting change. First of all, it’s clear that our browser has been less busy: the usage pattern is way less fragmented. When we look at the call stack, _VirtualDom_applyPatches now calls _VirtualDom_applyPatchesHelp only once! This is even more evident in the “Bottom-Up” view:

We’ve made the code twice as fast 🤟

One More Thing

So far we have seen the benefits of Html.Keyed as far as it concerns performance. But another great use case of this library is UI consistency.

Let’s go back to the original version without Html.Keyed and add a little checkbox near each word. Now if I checked a word and then changed the sorting order, I would expect the checked box to follow the word.

Right? Wrong! See for yourself.

This happens because the virtual DOM diffing doesn’t really understand that the checkboxes belong to each word, and therefore decides that the best action to take is just to leave them there. Note that this happens because we are using the DOM to store the state of our checkboxes, so Elm has no knowledge that we care about those values.

If we apply the Html.Keyed change again by replacing viewEntry with viewKeyedEntry, we communicate our intention that these elements belong to the same unit and therefore need to be considered as a whole. Check out the updated version.

Amazing 🎉

In the next post of this series, we will move on to deconstruct the mysteries of Html.Lazy. Thanks for reading!

Performant Elm, Part I

Mon, 08 Jul 2019 12:00:00 GMT

You have written your first application in Elm. Congratulations my friend, the hard work has finally paid off. Now it’s time to relax and enjoy the pure bliss that is maintaining and refactoring Elm code.

But wait a second, someone tells you the app feels a bit sluggish to use. Mh. I thought that writing pure functions would automatically make code fast. Okay. Let’s try to stay calm.

The app looks fast at a first glance. To be fair, you’re probably using a quad-core machine with 16 GB of RAM. Maybe you should see how the app behaves on a lower end device. In Chrome, you bring up the inspector, then go to Performance tab and choose how much you want to throttle your CPU.

When you click around the app now, you can clearly see it’s quite sluggish. Mh. Time to unleash the power of the flamegraph. You can start recording a session, then click around the app as a normal user would. Easy peasy. When all is done, you should see something like this.

That looks quite exotic, but let’s not lose hope. The yellow parts of the graph represent the time your browser spent executing scripts, while the purple parts denote the time it took to render. You can also see small red corners scattered through the graph: those represent when the user has experienced noticeable delays while using the application. You can find out more information about what it means here.

You’ll also notice you can select a smaller time interval and start digging into more specific bits of what your application is doing at a certain time. For example, if you highlight the first big chunk of scripting and then look in the lower panel, you will find a tab named “Bottom-Up”.

The first entry says _Utils_eqHelp, and apparently that function took the most time to execute. This is curious, because we don’t expect our application to spend so much time comparing things. By clicking on the right hand side, we will be able to see which module is the cause of this performance problem.

As you can see, the function causing slowdown seems to be the aptly named author$project$Page$Learn$SelfReviews$Revise$Highlighter$isSpace. Let’s look at what that function does:

isSpace : Char -> Bool
isSpace char =
    List.member char
        [ ' '
        , '\t'
        , '\n'
        , '\u{000D}'

        -- what follows is a list of unicode characters used
        -- as spaces: http://jkorpela.fi/chars/spaces.html
        , '\u{00A0}'
        , '\u{1680}'
        , '\u{180E}'
        , '\u{2000}'
        , '\u{2001}'
        , '\u{2002}'
        , '\u{2003}'
        , '\u{2004}'
        , '\u{2005}'
        , '\u{2006}'
        , '\u{2007}'
        , '\u{2008}'
        , '\u{2009}'
        , '\u{200A}'
        , '\u{200B}'
        , '\u{202F}'
        , '\u{205F}'
        , '\u{3000}'
        , '\u{FEFF}'
        ]

Uh! That is really surprising. Usually iterating over a list is really fast, but in this case it seems that we are spending almost 90% of our time inside this function. As you probably know, the algorithmic complexity of iterating over a list is O(n): we could easily improve that by building a set that contains all the desired characters and checking if the current character is part of that set. The Elm standard library implements a set using a dict, which is in turn implemented using a red-black tree, which means that the time complexity will become O(log(n)).

So we can change the code that looked like this

sameToken ( _, char1 ) ( _, char2 ) =
    not (isSpace char1) && not (isSpace char2)

into something that looks like this

-- This is a very hot path in the code. So we build a set once
-- and use it to determine if a character is a space or not.

spaceSet =
    Set.fromList spaceCharacters

sameToken ( _, char1 ) ( _, char2 ) =
    not (Set.member char1 spaceSet) &&
      not (Set.member char2 spaceSet)

With this little change in place, we can repeat the performance measurement

Now our browser is spending most of its time adding elements to the DOM instead of just comparing things. We can also notice that there is no huge outlier anymore: the current top function, appendChild, uses 15% of the total time while previously _Utils_eqHelp was taking almost 90% of the total time. When we try clicking around the application, we notice an extreme performance speedup when using the application. Albeit this is still not optimal, it’s a huge improvement with a very small code change. Look at how the shape of the performance graph has changed:

I hope that after reading this you will be less intimidated to dive in performance measurements and get the most out of your browser inspector. In future blog posts we will explore what else we can do to make our Elm applications faster, in particular using Html.Keyed and Html.Lazy.

juliu.is

Tidying up your $HOME with Nix

Oh, but there is

Adding packages

Bring on the heat

I like this. What do I do next?

How do I update all my packages?

How do I manage multiple machines with different architectures?

How do I manage multiple machines with different setups?

I don’t want to use unstable software. How do I use the stable version?

I’ve changed my mind. How do I get both stable and unstable?

I need a config file which is unsupported. How do I symlink it into my home?

I’ve made a terrible mistake and screwed up my configs. Halp!

Nix is taking a lot of space on my machine.

This is bad. How do I uninstall it?

Is homebrew completely useless now?

Still reading?

Peeling zeroes in Idris

Enter Idris

Natura, artis magistra

The Meat and Potatoes

And Now for Something Completely Different

Ditch your version manager

Welcome to the jungle

Some bad news

niv

Elm at NoRedInk

Table of contents

A wild monorepo appears

Rails conventions

Our Elm programs

How we write Elm files

Nesting the Elm Architecture

When to nest TEA modules

Interacting with JavaScript

Testing

Tooling

There and back again

An interesting pair

A trip in Haskell Land

Coming back home

A simple tmux config

Mac OS install

tat

General configuration

Mouse

Command history

Window management

Keys modes

Key Bindings

Prefix

Copy mode

Quick reload

More windows management

Vim integration

Cool status bar

Fin

z-index by examples

The setup

Ordering matters

z-index doesn’t work by itself

A new mystery

Auto is not zero

The usual suspects

Permutate parsers, don't validate

Advent of Code 2020, Day 4

Advent of Code 2020, Day 4, Part II

A moment of reflection

A new perspective

Another problem?!

Complicated Haskell Words - Isomorphism

Isomorphism

Intermezzo, or why is zero cool

Back to identity functions

The definition™

Many isomorphisms

Some examples of isomorphisms

Unit

One more thing…

A walkthrough of Elm Warrior