Encrypted partition path derivation via linear search through incrementally encoded packed data

locate is a lightning-fast command line search utility. It first hit the press in the early 80s when James A. Woods proclaimed the tradeoff of nightly updates is worth it for sub-second filesystem path matches.

The proposed architecture is simple but effective: incrementally encode all paths in a purpose-built binary database and perform matches with linear search. Since nearly all matches are partial, linear search generally outperforms binary search or other optimizations. Maintainers have followed this original architecture to the present day.

The indexer is called updatedb and it generally runs nightly, as root. If you have an encrypted home partition (and you should) nothing in your $HOME will be indexed. One workaround is to index it yourself. To maintain security I recommend storing the index inside your $HOME.

I like to use anacron since it automatically performs a catch-up run if necessary. This is handy for “daily would be nice” jobs that don’t need to run at an exact hour/minute of the day.

Here’s how to do it.

Add this to your crontab (this is one long line). This fires off your own personal anacron:

@hourly /usr/sbin/anacron -s -t $HOME/.anacrontab -S $HOME/.anacron

Add this to $HOME/.anacrontab to run your indexer daily (that’s the “1”) and after a 10 minute delay (that’s the “10”):

1 10 indexhome $HOME/bin/index-encrypted-homedir

Create the executable file $HOME/bin/index-encrypted-homedir with these contents:

#!/bin/bash
 
set -o errexit
set -o nounset
set -o pipefail
 
mkdir -p "$HOME/.var" "$HOME/.anacron"
updatedb -l 0 -n '.meteor .cache' -o "$HOME/.var/locate.db"

Finally, add this to your $HOME/.bashrc:

export LOCATE_PATH="$HOME/.var/locate.db"

Free Software Claus is Coming to Town

I help organize a conference for Free Software enthusiasts called SeaGL. This year I’m proud to report that Shauna Gordon McKeon and Richard Stallman (aka “RMS”) are keynote speakers.

I first invited RMS to Seattle 13 years ago, and finally in 2015 it all came together. In his words:

My talks are not technical. The topics of free software, copyright vs community, and digital inclusion deal with ethical/political issues that concern all users of computers.

So please do come on down to Seattle Central College on October 23rd and 24th, 2015 for SeaGL!

Yes, You Should Swap

If you’ve ever set up a machine by hand, you’ve probably had to decide how much of your disk to set aside as swap.

I’ve often wondered “why swap at all”? This quote by Nick Piggin from 2004 finally helped me answer the question.

no matter how much ram you have, swap can increase performance by allowing unused anonymous memory to be paged out, thereby increasing your maximum effective RAM

Found via this post on Hacker News, where the poster raises the point that some filesystem buffers might be extremely “hot” (frequently used), but might only fit in physical RAM (where they should be) if some swap space is available to page out other “cold” information.

Update 2016-12-22: except for Kubernetes nodes, apparently.

Debugging web tests on remote servers

I run “web tests” on a remote server. I use Selenium to act like a person interacting with a website, viewing and entering data. Selenium is pretty awesome, it can drive a real web browser like Firefox.

Even better is to have these web tests run automatically every time I commit code. I use Jenkins for this. Jenkins even fires up a headless desktop so Selenium can run Firefox.

When a web test breaks (especially in some way I can’t reproduce on my local desktop), sometimes it helps to actually see what Jenkins sees as it runs the test. Here’s a quick guide for doing so on an Ubuntu GNU/Linux server.

  1. Connect to the remote server using SSH. Install VNC server:
    sudo apt-get install vnc4-server
  2. On the remote server, become the user tests run as. For example:
    sudo su - ci
  3. Set a password for the VNC server using the vncpasswd command.
  4. Start headless X server by running vncserver. Note the given display. If example.com:1 is included in the output of vncserver, the display is :1.
  5. Figure out which port the VNC server is using. I usually do something like

    sudo netstat -nape | grep '^tcp.*LISTEN.*vnc.*'

    Here’s some example output:

    tcp        0      0 0.0.0.0:6001            0.0.0.0:*               LISTEN      107        3099855     13233/Xvnc4     
    tcp6       0      0 :::5901                 :::*                    LISTEN      107        3099858     13233/Xvnc4

    By trial and error, I figured out that 5901 was the port I should use.

  6. Port-forward VNC to your local machine.

    1. Disconnect from the server.
    2. Reconnect, including -L10000:localhost:5901 on your SSH command line.
    3. Leave this connection open.
  7. On your local machine, connect a VNC client to localhost:10000. An X terminal should be displayed.

  8. In the X terminal, run your web tests.

  9. When finished debugging, kill the X server using the display noted earlier.
    vncserver -kill :1