Encrypted partition path derivation via linear search through incrementally encoded packed data

locate is a lightning-fast command line search utility. It first hit the press in the early 80s when James A. Woods proclaimed the tradeoff of nightly updates is worth it for sub-second filesystem path matches.

The proposed architecture is simple but effective: incrementally encode all paths in a purpose-built binary database and perform matches with linear search. Since nearly all matches are partial, linear search generally outperforms binary search or other optimizations. Maintainers have followed this original architecture to the present day.

The indexer is called updatedb and it generally runs nightly, as root. If you have an encrypted home partition (and you should) nothing in your $HOME will be indexed. One workaround is to index it yourself. To maintain security I recommend storing the index inside your $HOME.

I like to use anacron since it automatically performs a catch-up run if necessary. This is handy for “daily would be nice” jobs that don’t need to run at an exact hour/minute of the day.

Here’s how to do it.

Add this to your crontab (this is one long line). This fires off your own personal anacron:

@hourly /usr/sbin/anacron -s -t $HOME/.anacrontab -S $HOME/.anacron

Add this to $HOME/.anacrontab to run your indexer daily (that’s the “1”) and after a 10 minute delay (that’s the “10”):

1 10 indexhome $HOME/bin/index-encrypted-homedir

Create the executable file $HOME/bin/index-encrypted-homedir with these contents:

#!/bin/bash
 
set -o errexit
set -o nounset
set -o pipefail
 
mkdir -p "$HOME/.var" "$HOME/.anacron"
updatedb -l 0 -n '.meteor .cache' -o "$HOME/.var/locate.db"

Finally, add this to your $HOME/.bashrc:

export LOCATE_PATH="$HOME/.var/locate.db"

Visualize per-character differences in a unified diff file

Crosio Kinderstreich 1

When someone sends you a patch, it is most easily viewed with syntax highlighting. The thing you need highlighted is what changed, at a character level.

You get for free with many tools including git (with git diff --word-diff), but this doesn’t help you with a stand-alone patch (diff) file.

Luckily, git ships with diff-highlight! Send a unified diff to that script’s stdin and you get beautiful syntax highlighting, including per-character changes. Here’s a wrapper for diff-highlight for npm users. On my system I found the script at /usr/share/doc/git/contrib/diff-highlight/diff-highlight, and I just run it with the Perl interpreter that ships with my Ubuntu 16.04 desktop.

 

UDP 1, 2, 3: netcat vs. socat

TCP is handy for simple, reliable communications like this tiny toy logger. I run the server and clients in separate consoles on the same machine:
# TCP log server
nc -kl 8000 > server-log.txt
 
 
# TCP logging from netcat client
date | nc 127.0.0.1 8000
 
 
# TCP logging from socat client
date | socat STDIN TCP:localhost:8000
 
 
# TCP logging from Bash client
date > /dev/tcp/127.0.0.1/8000
The only bummer about TCP is that–in my example–other clients have to wait in line. We are logging so I want fast, one-way communication from any number of clients to the server, and reliability of every log message is probably not critical. Let’s try UDP! I could just add -u to the netcat server args to use UDP datagrams, but a netcat UDP server gets a little wonky. The easy workaround is to use socat as the server instead. socat happily accepts any datagram from multiple clients, simultaneously.
# UDP log server
socat UDP-RECV:8000 STDOUT > server-log.txt
 
 
# UDP logging from netcat client
date | nc -q1 -4 -u 127.0.0.1 8000
 
 
# UDP logging from socat client
date | socat STDIN UDP-DATAGRAM:localhost:8000
 
 
# UDP logging from Bash client
date > /dev/udp/127.0.0.1/8000
Use at your own risk. The TCP version is surely simplest, safest (ahem, still no auth – this is just a toy) and reliable. I don’t know much about what’s going on under the hood here. Insight welcome! Messages from different clients might get mangled together, too. Tested on Ubuntu 14.04.

Group chat me crazy

Group chat (IRC, Rocket.Chat, Let’s ChatMattermost, Zulip, Slack, etc) rocks! Definitely use it. But, fair warning:

My thoughts on group chat:

  1. Be available sometimes, especially when your coworkers are. Aim for healthy overlap.
  2. Be unavailable sometimes. Focus on your work and get stuff done.
  3. Managers: support your team doing both #1 and #2 above.
  4. Discuss and curate tribal knowledge in group chat, but distill often into other permanent, public, shared resources for your “knowledge base”, such as mailing lists, wikis, and (gasp) formal documentation.

Updating multiple git repositories at once

1. myrepos

http://myrepos.branchable.com – manage multiple repos (source)

2. One-liner

Assuming repo1, repo2 and repo3 are subdirs of the current dir, try:

parallel --tag -j0 git --git-dir={}/.git pull --ff-only ::: repo1 repo2 repo3

Note this assumes you’re using GNU Parallel. On Ubuntu 14.04, I had to do sudo apt-get install parallel. This uninstalled moreutils, which was a minor bummer.

Survey for my SeaGL 2015 command-line talk

I want to give my best talk ever, and I need your help. Knowing my audience will help me produce the most relevant, fun, and insightful content. Please spare a minute and take this survey. Submission does not guarantee admission. All questions are optional. Here’s more info on my talk (including date, time, and location).

Alternative to the Google Form above: all survey questions are repeated below. Email your response to [email protected]

  1. Will you attend my talk?
    • Yes/No/Maybe
  2. How experienced are you on the command line?
    • 1=Beginner,2,3,4,5,6,7,8,9,10=Expert
  3. What’s something you love or hate about the command line?
  4. What shell(s) do you use?
    • bash, zsh, ksh, fish, tcsh, other
  5. What operating system(s) do you use?
    • FreeBSD, GNU/Linux, Mac OS X, Windows, other
  6. Is there something in particular you want to learn or improve related to the command line?
    • Examples: basic stuff, piping multiple commands, find/grep/xargs/ssh/cut/paste/col/parallel/at/sed/awk/vim/git, job control (running multiple programs at once), fun/silly stuff, customizing my shell, programmable completion, shell programming, advanced/experimental stuff (like what?)
  7. Anything else you’d like to tell me?