DIY Bitcoin Hardware Wallet with a Raspberry Pi Zero

I would like to make a Bitcoin Hardware Wallet (HWW) without trusting any single HWW vendor (such as Ledger), and it should also be able to easily handle multisig setups. Ideally, it should run Electrum so that I can be reasonably assured that any wallets managed on this device will be recoverable and exportable in the future.

An easy way to do this is with a Raspberry Pi Zero.

Why the Pi Zero?

There are cost advantages to be sure, but the Pi Zero is a bare-bones computer without any networking interfaces. No wireless, no bluetooth, no Ethernet. Unless you install a network interface, the device is already airgapped. The onboard chips are so simple it also minimizes the risk of hardware vulnerabilities you'd get in a laptop or desktop.

It's also simple to setup!


You're going to need more than a Pi Zero for an Electrum-based HWW. In addition to the Pi Zero board itself (not the Pi Zero W), you'll need the following:

Camera to read QR Codes

Because the device is airgapped, you'll want a simple camera to read QR codes via Electrum. This will allow you to read unsigned transactions for signing, as well as loading public keys for multisig wallet creation. You can use an official Raspberry Pi camera. But really, a simple webcam would also work (if USB-based webcam, then obviously you'll need an adapter to connect it to one of the pi-zero micro-USB ports)

The advantage of a cheap Raspberry Pi camera is that it minimizes the risk hardware vulnerabilities that you might find with full-sized webcams. That said, the simplicity of the Pi camera and the slow processing of the pi-zero makes reading QR codes a bit slow.

Dedicated Display

Because this is an airgapped device, you will need a display. The display must be sufficient for Electrum, and in particular, for showing QR codes that can easily be read by other devices (other HWWs or smartphones).

Any portable monitor will work. The pi-zero has a mini-HDMI port, so you may need an adapter.

Input Device

I recommend a single wired mouse. Most wireless mouse + keyboard setups use an unencrypted wireless frequency and anyone with a cheap receiver in your neighborhood could be listening in on your keystrokes. I recommend a virtual keyboard (more on that later) and a single wired mouse (if USB you'll need a micro-USB adapter for the pi-zero).

Micro-SD for One-Time Software Load

This is the tricky part. The pi-zero will boot from a micro-SD card. So you'll need a micro-SD card. And because the pi-zero has no network, you won't be able to load any software onto the micro-SD card from the pi-zero itself.

Fortunately, there are MANY ways to do this. You could copy the needed files on a USB thumbdrive, but this introduces layer of vulnerability that you can avoid.

I strongly recommend a on-time software load from a separate device (a Raspberry Pi that does have Internet access), and if you ever need new software, you start the process over on a new micro-SD (and destroy the old one). See below for instructions.

If you don't already have a spare Raspberry Pi with a network interface, you'll need one (or modify this approach to use a USB drive if you're okay with those risks). You can use a Raspberry Pi Zero W, or even a Pi4; you'll only need to use it once for the initial setup (see below).

Initial Setup

You should have a Raspberry Pi Zero with a camera, display, and a wired mouse. You should also have a spare Raspberry Pi that has network access. And you should have one micro-SD card ready.

You will need a computer to burn the OS image onto the micro-SD card. I use an Ubuntu machine with Pi Imager:

Make sure to select the 32-Bit Raspberry Pi OS. Insert the micro-SD card and write the image.

One-Time Network Setup

Once you have written the OS to the micro-SD card, put the micro-SD card into the spare Raspberry Pi that does have a network adapter and a keyboard. You'll want to connect to the Internet and install the following software:

Virtual Keyboard

You can consider this step optional if you plan on using a normal keyboard with the airgapped Pi Zero. However, I would recommend avoiding wireless keyboards due to the security concerns.

sudo apt install matchbox-keyboard

Once installed, the virtual keyboard will be in the Accessories menu, as shown,


I recommend to install Electrum from the source and verify the package. At the time of this writing, this will look like,

sudo apt-get install python3-pyqt5 libsecp256k1-0 python3-cryptography

gpg --import ThomasV.asc

gpg --verify Electrum-4.0.9.tar.gz.asc

You should get the following output,

gpg: assuming signed data in 'Electrum-4.0.9.tar.gz'                                                                                              
gpg: Signature made Fri 18 Dec 2020 12:07:21 PM MST                                                                                               
gpg:                using RSA key 6694D8DE7BE8EE5631BED9502BD5824B7F9470E6                                                                        
gpg: Good signature from "Thomas Voegtlin ( <>" [unknown]                                                
gpg:                 aka "ThomasV <>" [unknown]                                                                                    
gpg:                 aka "Thomas Voegtlin <>" [unknown]                                                                            
gpg: WARNING: This key is not certified with a trusted signature!                                                                                 
gpg:          There is no indication that the signature belongs to the owner.                                                                     
Primary key fingerprint: 6694 D8DE 7BE8 EE56 31BE  D950 2BD5 824B 7F94 70E6  

Assuming things look good you can untar the package and rename it however you want,

tar -xvf Electrum-4.0.9.tar.gz
mv Electrum-4.0.9 Electrum

Shut down the Raspberry Pi and remove the micro-SD. This will be the very last time this micro-SD card should see the Internet.

Airgap the Pi Zero

Insert the micro-SD card into the Pi Zero, and physically tape over it with a tamper proof seal (such that it would be obvious if it was ever removed).

Depending on your level of paranoia, you'll want to make sure that no data can get off your pi-zero except via QR code or pen and paper. This means that the moment you generate private keys, you'll want to take precautions to never expose the device to a network or a USB thumb-drive.

I recommend attaching everything together (mouse, display, and the pi-zero) such that it looks like a Frankenstein computer. Use duct-tape if you have to. If you used a USB adapter that includes other USB ports, physically cover those with tamper proof seals. There are cases you can get for a pi-zero, I have no recommendations other than to make sure that you seal this thing such that no one can connect to it from any other device. Wrap it a faraday bag if you're truly paranoid.

Using your DIY HWW

To start Electrum on the airgapped Pi Zero, simply use the "run_electrum" command from the console,

cd Electrum

From here you'll be using Electrum in offline mode, which will be useful for doing the following,

Create Wallets

You can create as many wallets as you like, and import public keys for multisig wallets through QR codes, as well as export public keys for managing receive addresses on a separate device.

For every wallet you do create, you'll want to create a separate backup of the seed words. I recommend to use a metal backup (or a sturdy material that can withstand a house fire and still be legible).

Do not ever export the private keys or the wallet file. If you need to replace your micro-SD, simply create a new one and recover your wallets with the backup seeds. I would recommend to physically destroy the old micro-SD card. The whole point of this airgapped device is that the wallet is never exposed.

Sign Transactions

You can create transactions, and for multisig you'll need to import an unsigned transaction into your airgapped Pi Zero HWW. To do so, open the wallet in question, and then under Tools, select Load Transaction, and then select From QR code. You can scan the transaction with your Pi Zero camera, and then click Sign.

You can export a signed transaction by displaying the QR code. Scan this onto any device with Internet access and broadcast the transaction.

Verify Receive Addresses

For security, it is recommended that you verify your receive addresses (especially in a multisig setup). To do so, simply open the wallet in question and select the Addresses tab. If you don't see the addresses tab click View and then Show Addresses.

Make sure the addresses on this list match any corresponding watch-only wallets. If it is part of a multisig wallet, make sure all wallets in the group have the same receive addresses.

Posted in bitcoin


Typically, you would use mysql client cli as follows,

$ mysql -h -P 3306 -u bob -D somedb -p

You could also create a ~/.my.cnf file to set default properties, including connection info,

port = 3306
host =
database = somedb
user = bob

You could even store a password but I wouldn't recommend storing it in the clear. Also, the above approach is not useful when you have more than one MySQL database across multiple hosts, ports, etc.

Managing Multiple Connections

Unfortunately, mysql still doesn't have an easy way to do this, but if you add the following to your ~/.bashrc,

function mysqle() { mysql --defaults-group-suffix=$1 --password "${@:2}"; }

You can then setup your ~/.my.cnf as follows,

port = 3306

host =
database = somedb
user = bob

host =
port = 63306
database = superdb
user = bob

As long as you prefix the section with "client", then this will allow you to create as many different sections as you need (for different apps, different environments, whatever).

To connect, just use the shell function from earlier,

$ mysqle superdb
Posted in bash, mysql, shell tips

FP Error Handling

I would like to handle errors in pure FP (functional programming), something more idiomatic and easier to read than nested try/catch blocks.

Let's say we have the following procedure,

def sumCirclesDiv(rs: List[String], d: Int = 1): Int = {
    val areas = => {math.Pi * r.toDouble * r.toDouble})
    val divs = / d)

This is pretty straightforward. Given a list of numbers (represented as strings), compute the areas of these as circles, and then divide by a common divisor (default to 1) and return an Integer sum. But what happens if one of the input strings cannot be parsed? What happens if the divisor is 0?

We could use try/catch exception handling, and end up with something like this,

def safeSumCirclesDiv(rs: List[String], d: Int = 1): Int = {
    try {
      val areas = => {math.Pi * r.toDouble * r.toDouble})
      val divs =
    } catch {
      case x: ArithmeticException => {
        println("you cannot divide by " + d)
      case x: NumberFormatException => {
        println("you must provide parseable numbers")
      case unknown: Throwable => {
        println("Exception: " + unknown)

This works, but presents numerous problems. For starters we return Int 0 in the case of an exception. But what if we want to ignore invalid strings and return a value for the valid input strings? We could do something like this,

def saferSumCirclesDiv(rs: List[String], d: Int = 1): Int = {
    var acc = 0
    for (r <- rs) {
      try {
        val area = math.Pi * r.toDouble * r.toDouble
        acc += area.toInt / d
      } catch {
        case x: ArithmeticException => {
          println("you cannot divide by " + d)
        case x: NumberFormatException => {
          println("you must provide parseable numbers")
        case unknown: Throwable => {
          println("Exception: " + unknown)

This will work for lists that contain a mix of valid and invalid string inputs, but it's not exactly idiomatic, the function is not pure, and it's an unwarranted assumption that invalid inputs should result in a zero. A result of zero should not be indicative of input errors.


In Scala (as well as in Functional Java libraries) we can use the Option monad. This is built-in to Scala, although as we'll see, it is fairly easy to implement directly.

For starters, our function signature should not lie (that is, if we can't guarantee an Int, we shouldn't promise an Int). We should instead return an Option of an Int, e.g.,

def optionSumCirclesDiv(rs: List[String], d: Int = 1): Option[Int] = ...

In this monadic approach, we should get the following results,

scala> optionSumCirclesDiv(List("1.5", "4.2"))
res0: Option[Int] = Some(62)

scala> optionSumCirclesDiv(List("1.5", "4.2"), 0)
res1: Option[Int] = None

scala> optionSumCirclesDiv(List())
res2: Option[Int] = None

scala> optionSumCirclesDiv(List("0.0"))
res3: Option[Int] = Some(0)

scala> optionSumCirclesDiv(List("1.0", "foobar"))
res4: Option[Int] = Some(3)

When using Option in Scala, we have either Some or None, and importantly, Some(0) is not the same as None. Let's implement this in pieces, and do so in a way that keeps our functions pure. For starters, we have an input List of strings that could contain invalid inputs (like "foobar" in the above example). We'll want to filter out all entries that cannot be cast to a number. You can do this explicitly with a List.filter expression, but we can also do so by using Option.

def optionDouble(s: String): Option[Double] =
  try { Some(s.toDouble) } catch { case _: Throwable => None }

With this function, we can map over a list and then flatten it (removing the Some and None wrappers), which is the same as using a flatMap(),

scala> List("1.1", "foo", "4.2").map(optionDouble)
res0: List[Option[Double]] = List(Some(1.1), None, Some(4.2))

scala> List("1.1", "foo", "4.2").map(optionDouble).flatten
res1: List[Double] = List(1.1, 4.2)

scala> List("1.1", "foo", "4.2").flatMap(optionDouble)
res2: List[Double] = List(1.1, 4.2)

We can now filter out all invalid inputs, and be guaranteed a List with zero or more valid inputs. Using this same approach, we can define other functions that use Some and None,

def circled(r: Double) = Some(math.Pi * r * r)

def divN(y: Int)(x: Double): Option[Int] =
  try { Some(x.toInt / y) } catch { case _: Throwable => None }

The circled function is obvious, but why create a divN? In this case, we can curry divN (within a flatMap) as follows,

scala> List(2.0, 4.0, 6.0).flatMap(divN(2))
res0: List[Int] = List(1, 2, 3)

scala> List(2.0, 4.0, 6.0).flatMap(divN(3))
res1: List[Int] = List(0, 1, 2)

If we chain together these functions in flatMaps, we'll have exactly what we need to implement optionSumCirclesDiv, i.e.,

def optionSumCirclesDiv(rs: List[String], d: Int = 1): Option[Int] = {
  rs.flatMap(optionDouble).flatMap(circled).flatMap(divN(d)) match {
    case Nil => None
    case x => Some(x.sum)

And because we have a chain of flatMaps, we can use a for-comprehension, e.g.,


// is equivalent to..

for {
      s  <- rs
      r  <- optionDouble(s)
      a  <- circled(r)
      ad <- divN(d)(a)
    } yield ad

Either / Try

The only potential problem with using the Option monad is that we're swallowing our errors. Maybe we want to know what specific exception occurred in our input list. A useful (albeit ugly) approach is to use the Either data type. Rather than Some or None, Either can be Left or Right. By convention, Left is used for errors and Right is used for success, e.g.,

def eitherDouble(s: String) = 
  try { Right(s.toDouble) } 
  catch { case _: Throwable => Left("Cannot parse " + s) }
def circled(r: Double) = Right(math.Pi * r * r)
def divN(y: Int)(x: Double) =
  try { Right(x.toInt / y) } 
  catch { case _: Throwable => Left("Cannot divide by " + y.toString) }

The return type for these functions is Either[String, Int], and we can use them as follows,

def eitherSumCirclesDiv(rs: List[String], d: Int = 1): Either[String, Int] = {
    val list =
    list.collect{ case Left(x) => x }.map(println) //side effect!
    list.collect{ case Right(x) => x } match {
      case Nil => Left("No parseable numbers to compute")
      case x => Right(x.sum)

A more idiomatic approach to this problem is to use the Try monad. This combines traditional exceptions with a data structure that is similar to Either, but instead of Left/Right we use the more intuitive Failure/Success, e.g.,

import scala.util.{Try,Success,Failure}
def tryDivArea(d: Int = 1)(s: String) =
    val r = s.toDouble        
    val a = math.Pi * r * r
    a.toInt / d

By simply returning a Try() structure, we'll get either Success() or a Failure() object that wraps the corresponding exception (e.g., NumberFormatException). We can use this function as follows,

def trySumCirclesDiv(rs: List[String], d: Int = 1): Try[Int] = {    
    val all =
    all.filter(_.isFailure).map(println) //side effect!
    all.filter(_.isSuccess).map(_.get) match {
      case Nil => Failure(new Exception("Empty List"))
      case x => Try(x.sum)

In the above cases we're simply printing the exception messages (a side effect to what is otherwise a pure function), and in practice we would not want to do that; keeping our functions pure will allow for proper exception logging as well as parallel streams.

All of the above code can be found on my github, and should run correctly in Scastie.

Posted in scala, software eng.

How to Find a Job as a Software Engineer

Software Engineering as changed a lot since the 1990s: we've seen the rise of the Internet, smartphones, machine learning, and a golden age of programming languages. Yet despite the technological advances, the actual process of finding a well-paying job as a software engineer hasn’t changed much.

The buzzwords have changed, the amount of money has skyrocketed, the volume of people and recruiters has definitely changed. But the specific frustrations of job seekers and hiring managers has remained unchanged. We spend the same amount of time, if not more, finding a job as we ever did.

Even with all the advanced machine learning -- algorithms promising to find the right candidates for the right job -- the actual job hunt is as clumsy as it ever was. As a job seeker, you’ll fill out the same information (address, veteran status, etc) over-and-over and wade through various recruiters and headhunters. Meanwhile, hiring managers are still combing through resumes desperate to find competent applicants.

In the end, third-party recruiters and third-party tools almost always get in the way and slow you down. None of them, not the recruiters, not the job sites, not the advanced job-matching algorithms; none of them are actually working for you; they exist only to profit from the company that is trying to hire you. Third-party recruiters are not interested in building an algorithm that actually finds you a good job. You as the job seeker are the product that third-party recruiters are actively selling to companies. No one is trying to find you a good job, except you.

Now, to be fair, there are some decent recruiters out there, but most third-party recruiters will spend their time calling you when you’re NOT looking for a job.

So, what is the best strategy for finding a job? Well, it turns out it’s pretty simple. It's a strategy I would recommend to everyone. And if some of this seems obvious and old-fashioned, that’s because it is.

Ask, Seek, Knock

Before you apply for anything, ask about the companies that you'd consider working for. Do they pay competitive market salaries? Are they working on important problems? Do people like working there?

Vet companies (on glassdoor or similar sites). Don't start applying right away, and don't talk to recruiters. Always start with vetting! Do not apply to any company that you haven’t properly vetted. It should be a great company to work for, a company working on important problems. Ask yourself: would working at this company be good for you, for your family, and your community?

For each vetted company, go to their jobs website, and/or find the hiring manager. Find all of the open jobs that you qualify for. Be honest about this. In some cases, the company may have no jobs that you qualify for. This is why doing your research up front is so useful; you’ll save yourself and all the hiring managers a lot of time by doing this kind of research.

If instead you went with recruiters and were using the latest job matching algorithm, be prepared to waste a lot of time talking to people about jobs that you are not a good match for (based on the easy to research questions listed above).

Finally, apply to each of the jobs that you are qualified for, and apply directly on the company website or directly to a hiring manager. And unless they specifically tell you to only apply once, apply to each of the job-postings that you are interested in. Most companies (especially large companies) have separate in-house recruiters working with different hiring managers and they may not go out of their way to find if you are a match for a job in a different department. It’s not uncommon to get rejected on some positions while getting calls from other positions, all at the same company (and in some cases it might be from the same in-house recruiter. They deal with lots of applicants, don’t take it personally).

It's timeless advice: Ask (the right vetting questions), Seek (the right companies), and Knock! You'll save yourself countless hours. Although a point of caution:

Beware of Scams

Sometimes you'll see a posting with an awesome salary, a great job description, but no company, just a recruiting agency. This is likely a bait and switch. Don’t fall for it. Remember, research the company first, before applying. Do not apply at a company that you haven’t vetted. If the recruiter doesn’t want to give you details, it means they’re obviously not the exclusive recruiter for that job and you’ve very likely already seen the real job posting elsewhere. Remember, the only person that’s trying to find the best job for you, is you!

And then What?

Unless your resume is an absolute mess, you'll likely start receiving calls from prospective companies. From here, you'll be asked to do online coding tests, phone screens, and possibly an in-person interview. Much has been written (arguably too much) about how to pass technical interviews -- the best advice I've learned: Interviews go both ways. While they are testing you (to determine if you'd be a good match), you should be testing them. Are they a good match for you and your life goals? Do they like working at this company? What would they change at the company? Be as up-front and honest as possible about what you're looking for and don't just try to impress them. Save yourself the pains of future stress by asking behavioral interview questions back at the interviewers.

Posted in software eng.

Extremely Large Numeric Bases with Unicode

Previously, we discussed using Punycode for non-ASCII domain names with internationalized URLs, e.g., https://去.cc/叼

I would like to use this approach to create a URL Shortening Service, where we can create shortened URLs that use UTF-8 characters in addition to the normal ASCII characters.

Most URL shortening services use a base-62 alphanumeric key to map to a long-url. Typically, the base-62 characters include 26-uppercase letters (ABCD…), 26 lowercase letters (abcd…), and 10 digits (0123…), for a total of 62 characters. Occasionally they will include an underscore or dash, bringing you to base-64. This is all perfectly reasonable when using ASCII and trying to avoid non-printable characters.

For example, using base-62 encoding, you may have a typical short URL that looks like this:


However, nowadays with modern browsers supporting UTF-8, and offering basic rendering of popular Unicode character sets (中文, 한글, etc), we can leverage a global set of symbols!

One of the larger contiguous Unicode ranges that has decent support on modern browsers is the initial CJK block as well as Korean Hangul syllables. Why are these interesting? Well, rather than a base-62 or base-64, we can use CJK and Hangul syllables to create extremely large numeric bases.

The CJK range of 4e00 to 9fea seems to be adequately supported, as well as the Hangul syllable range of ac00 to d7a3, this would give us a base-20971 and a base-11172 respectively. Rather than base-62, we can offer numeric bases into the tens of thousands!

This would allow shortened URLs to look like this:


Taken to extremes, let's consider a really large number, like 9,223,372,036,854,775,807 (nine quintillion two hundred twenty-three quadrillion three hundred seventy-two trillion thirty-six billion eight hundred fifty-four million seven hundred seventy-five thousand eight hundred seven). This is the largest signed-64-bit integer on most systems. Let's see what happens when we encode this number in extremely large bases:

= 7M85y0N8lZa
= 瑙稃瑰蜣丯
= 셁뻾뮋럣깐

The CJK and Hangul encodings are 6-characters shorter than their base-62 counterpart. For a URL Shortening Service, I'm not sure this will ever be useful. I'm not sure anyone will ever need to map nine quintillion URLs. There aren't that many URLs, but there are billions of URLs. Let's say we're dealing with 88-billion URLs. In that case let's look at a more reasonable large number.

= 3CG2Fy1
= 執洪仉
= 닁읛껅

NOTE: while the character-length of the Chinese string is less than the base-62 string, each of the Chinese characters represents 3-bytes in UTF-8. This will not save you bandwidth, although technically neither does ASCII, but it's worth mentioning nonetheless.

To convert a number to one of these Unicode ranges, you an use the following Python,

def encode_urange(n, ustart, uend):
    chars = []
    while n > 0:
        n, d = divmod(n, uend-ustart)
    return ''.join(chars)

def decode_urange(nstr, ustart, uend):
    base = uend-ustart
    basem = 1
    n = 0
    for c in unicode(nstr):
        if uend > ord(c) < ustart:
            raise ValueError("{!r}, {!r} out of bounds".format(nstr, c))
        n += (ord(c)-ustart) * basem
        basem = basem*base
    return n

The CJK range is 4e00 to 9fea, and you can map arbitrary CJK to base-10 as follows,

>>> decode_urange(u'你好世界', int('4e00',16), int('9fea',16))
>>> print encode_urange(92766958466352922, int('4e00',16), int('9fea',16))

Unicode is full of fun and interesting character sets, here are some examples that I have built into x404:

# base-10   922,111
base62:     LSR3
top16:      eewwt
CJK:        鶱丫
hangul:     쏉걒
braille:    ⣚⡴⡙
anglosaxon: ᛡᛇᛞᚻᚢ
greek:      οΒΦδ
yijing:     ䷫䷔䷫䷃
Posted in python

PseudoForm, a Bot-resistant Web Form

I would like to create an HTML form that is resistant to bots. This could be a classic comment form or really any web-accessible form. In most cases, requiring authentication and an authorization service (such as OAuth) would be sufficient to protect a back-end web service. But what if you have a publicly accessible web form, and you want to keep it open and anonymous, but provide some measure of protection against automated bots?

For example, I would like to protect my URL Shortening Service such that only humans using the web front-end can access the back-end service layer, and I would like this layer of protection to be bot resistant.

Most spam-bots will crawl through webpages looking for HTML forms, and will seek too exploit any unprotected form handlers (i.e., the backend service). A more advanced bot can even leverage common authentication and authorization services. Other than annoying our users with CAPTCHA tests, what are some approaches that we can take?

Honeypot Form

A simple and effective approach is to create a dummy HTML form that is not rendered in the actual browser, but will be be discovered by most spam-bots. This could be as simple as,

<div id="__honey__">
  <form action="post-comments.php" method="POST">
  <input type="text" name="Name">

The "post-comments.php" will be discovered by most spam-bots, yet in our case this is a misdirect. If you don't want it to return a 404, you can add a dummy file named "post-comments.php" that does nothing but will always return a 200. In fact, anyone trying to access this form handler is likely a bot, so feel free to do whatever you want with this. A simple but fun trick is to include the request IP address (the IP address of the bot itself) as the form action,

<form action="http://{{request.remote_addr}}/post.php" method="POST">

A naive spam-bot may start scanning its own IP address and may even send HTTP POST messages to itself.

Meanwhile, we would add Javascript in <head> with a "defer" attribute,

<script src="PseudoForm.js" defer></script>

The "defer" attribute will execute the "PseudoForm.js" after the user-agent has parsed the DOM but before it has been rendered to the user. Inside this script, you can replace the fake honeypot form.

document.getElementById('__honey__').innerHTML = actual_form;

The "actual_form" can be anything you want, and to further confound spam-bots, it does not even need to be an actual HTML form.


In the case of my URL Shortening Service, the actual form does not use an HTML <form> element, and instead uses a simple <button>. In fact, using proper CSS, you can use any element you want, and style it as a button, even something like this,

<div id="__honey__">
  <input type="text" id="add_url" placeholder="Enter long URL...">

There is no form handler in this case, and the back-end service is known only to the Javascript (which can add onclick event handlers to our CSS-styled button). This is sufficient to protect against naive spam-bots, but a more advanced spam-bot can easily parse our Javascript and determine actual service endpoints, such as looking for calls to XMLHttpRequest.

PseudoForm + formkey

In order to ensure the back-end service is accessed via our front-end, you can create a time-sensitive token. For my URL Shortening Service, all back-end service calls require a formkey. That is,

this.getformkey = function() {
    function(err, data) {
      if (err !== null) {
        console.log('Something went wrong: ' + err);
      } else {
        if ("formkey" in data) {
          _self.formkey = data.formkey;

"getJSON" is a wrapper around XMLHttpRequest, and all this does is fetch a valid formkey and save it within the PseudoForm Javascript object. We can then keep formkey up to date in the webapp, like so,

setInterval( PseudoForm.getformkey, 60*1000 );

In the above case, PseudoForm.formkey will be refreshed every 60 seconds, and any calls to the back-end service must use this formkey. The value of formkey will appear as a nonsensical hex string, yet in-reality it is a time-sensitive hash that can only be verified by the back-end (which generated the formkeys in the first place), i.e.,

def getFormKey(self, ts=None, divsec=300):
    ''' return a time-sensitive hash that can be 
        calculated at a later date to determine if it matches
        the hash will change every divsec seconds (on the clock)
        e.g., divsec=300, every 5-minutes the hash will change
    if ts is None:
        ts = int(time.time())
    return hashlib.sha1((

def isValidFormKey(self, testkey, lookback=2, divsec=300):
    ''' return True IFF
        the testkey matches a hash for this or previous {lookback} iterations
        e.g., divsec=300, 
        a formkey will be valid for at least 5 minutes (no less than 10)
        e.g., lookback=6, 
        a formkey will valid for at least 25 minutes (no less than 30)
    for i in range(lookback):
        if testkey == self.getFormKey(int(time.time())-divsec*i):
            return True
    return False

I included a secret key (by default a server generated GUID), to ensure that a bot would never be able to calculate a valid formkey on its own. All back-end services require a valid formkey, which means that any automated bot that accesses one of our back-end services must first fetch a valid formkey, and can only use that formkey within the time-limit set by the back-end itself (default is 5-minutes).

PseudoForm + handshake

Theoretically, an automated bot could be programmed to utilize a valid formkey, giving the appearance of a real human user accessing the front-end, and it could then spam the back-end service. This was exactly what I wanted to avoid in my URL Shortening Service. To further strengthen the PseudoForm, all back-end services require a time-sensitive handshake.

In other words, when you make a request to the URL Shortening Service, the response is a unique time-sensitive handshake. The handshake must be returned within a narrow time-window. If the response is too soon, the request is denied. If the response is late, the request is also denied. This has the added benefit of throttling the number of real transactions (roughly one per second) per IP address.

Therefore, in order for an automated bot to exploit the PseudoForm, it not only needs to bypass the honeypot, it needs to carefully time its HTTP calls to the back-end, throttling itself, and perfectly mimic what would happen seamlessly by a human on the front-end.

It's certainly not impossible, and any software engineer of reasonable talent could build such a bot, but that bot would effectively be using the front-end. In the end, PseudoForm makes it very difficult for bots, throttling any bot to human speeds, but keeping everything very simple for a human (try it yourself, there are no CAPTCHAs or any nonsense).

PseudoForm example

My URL Shortening Service, x404, leverages exactly this kind of PseudoForm (including some of the code in this page). It's a simple URL shortening service that was created as a novelty, but it provides a protected back-end service that is far more difficult to exploit than just manually using the front-end. In other words, it's extremely easy for a human to use, and incredibly difficult for a bot.

The full process of the x404 PseudoForm is as follows,

1. create honeypot HTML form (misdirect naive spam-bots)

2. create a PseudoForm that appears as a normal web form in the browser (also prevent the honeypot form from rendering)

3. user enters information and clicks "add"

The PseudoForm front-end will display a "processing" message to the user. The PseudoForm front-end will also do an HTTP POST to the PseudoForm back-end, and if it includes a valid formkey, then the PseudoForm back-end will respond with a unique handshake string.

4. wait a second...

After a short wait (roughly 1s, but configurable however you want), the PseudoForm front-end will do an HTTP PUT to the PseudoForm back-end. If valid formkey, valid return handshake, and it is within the valid time window, then the back-end will commit the transaction and return success.

Literally, the back-end will reject any return handshake (valid or not) that appears too soon (or too late).

5. success! user sees the results
Posted in css, html, javascript, python

multiple git remotes

I would like to manage a project across multiple remote git repositories, specifically, a public github repository and my own private repositories.

Fortunately, git supports as many remote repositories as you need. When you clone a repository, there will be a default remote called "origin", i.e.,

$ git clone
$ cd dotfiles
$ git remote -v
origin (fetch)
origin (push)

Adding additional remotes is trivial, i.e.,

$ git remote add bob
$ git remote add ann
$ git remote add pat

Now, when we look at the remotes for our repository we'll see the new entries,

$ git remote -v
origin (fetch)
origin (push)
bob (fetch)
bob (push)
ann (fetch)
ann (push)
pat (fetch)
pat (push)

If we want to pull from Ann's repository and merge it into our local master branch,

$ git pull ann master

And then if we wanted to push those changes to Bob's repository,

$ git push bob master

We can also rename and remove remotes, i.e.,

$ git remote rename bob bobbie
$ git remote remove bobbie

In practice, we may not want to be constantly merging everything into a local master, instead, we may want to investigate the code before any merges. This can be done easily. I prefer to use tracked branches, as follows,

$ git checkout -b bob-master
$ git remote add bob
$ git fetch bob master
$ git --set-upstream-to=bob/master

We can now inspect the bob-master branch and merge manually as we see fit,

$ git checkout bob-master
$ git pull
$ git checkout master
$ git diff bob-master
Posted in git, software eng.

OOP or Procedural?

I would like to know when it is best to use object-oriented programming, and when it is best to use procedural programming.

tl;dr: neither, go with functional programming

By procedural programming, I mean the kind of code you'd find programming in C; imperative control flow, functions, data structures, and algorithms. For example,

#include <stdio.h>

float f_to_c(float f) {
    return (f - 32) * 5 / 9;

int main() {
    float fahrenheit;
    printf("Please enter the temperature in Fahrenheit: ");
    scanf("%f", &fahrenheit);
    printf("Temperature in Celsius = %.2f\n", f_to_c(fahrenheit));
    return 0;

And by object-oriented programming, I mean the kind of code with abstraction, inheritance, polymorphism, and encapsulation. For example,

import java.util.*;

interface TemperatureConverter {
    public float convert();

class Temperature {
    float degrees;
    Temperature(float t) {
        degrees = t;

class Fahrenheit extends Temperature implements TemperatureConverter {

    Fahrenheit(float t) {

    public float convert() {
        return ((degrees - 32)*5)/9;


class FahrenheitToCelsius {

    public static void main(String[] args) {
        Fahrenheit fahrenheit;
        Scanner in = new Scanner(;
        System.out.print("Enter temperature in Fahrenheit: ");
        fahrenheit = new Fahrenheit( in.nextFloat() );

        System.out.println("temperature in Celsius = " 
            + fahrenheit.convert());


I admittedly forced some inheritance and polymorphism into the above code, but it's arguably just as easy (if not easier) to read than the C example (despite being considerably longer).

In both cases we hid the implementation details (the specific formula that converts Fahrenheit to Celsius) from the main(). However, the OOP example also hides (encapsulates) the data structure as well. In the Java example we encapsulate the float within the Temperature base class, which the Fahrenheit class inherits. And since the Fahrenheit class implements the TemperatureConverter interface, then we're guaranteed to have a convert() method. There is still some implicit typecasting (a float to string within the println), but the idea is that the main() function doesn't care about the underlying data structure.

As Robert Martin (Uncle Bob) put it, "Objects expose behavior and hide data." The Fahrenheit class exposed a convert() behavior and hid the underlying data structure. This, according to Uncle Bob, makes it easy to add new objects without changing existing behaviors. For example,

class Celsius extends Temperature implements TemperatureConverter {

    Celsius(float t) {

    public float convert() {
        return 9*degrees/5 + 32;


This code has no impact on the existing Fahrenheit class, and we can safely call convert() on both Fahrenheit and Celsius objects. Additionally, if we use generics on the Temperature class, then we could allow for different data structures (such as double or BigDecimal) on something like a Kelvin class. In OOP, adding new classes is generally easy.

That said, what if we wanted to add new behavior? Maybe we want to add an isRoomTemperature() method. If so, we could add a new interface and then implement it in Celsius and Fahrenheit, but what if we had also implemented that new Kelvin class? Or several other Temperature classes? And shouldn't the convert() method return a Temperature class? This could get messy and will lead us into DRY problems. In fact, this is an area where OOP is not ideal. Even Uncle Bob admits that if we're adding new behaviors then "we prefer data types and procedures."

This seemingly obvious and innocuous statement in Clean Code is actually very profound, especially considering the fact that OOP and classic procedural programming do not mix well in a single code-base. Whether you go with OOP or not, if Uncle Bob is correct, depends on whether or not you will be adding and managing lots of behavior, or whether you will be adding and managing lots of data types. If the behavior will be relatively unchanged, then OOP would be beneficial, but if we're planning to add or change behavior, then procedural programming would be preferred. I honestly don't know what kind of software projects aren't primarily adding new behaviors (new features).

For reference, adding a room temperature check is easy in the C code,

#include <stdio.h>
#include <stdbool.h>

bool is_c_room_temperature(float c) {
    return c >= 20 && c <= 25 ? 1 : 0;

float f_to_c(float f) {
    return (f - 32) * 5 / 9;

bool is_f_room_temperature(float f) {
    return is_c_room_temperature(f_to_c(f));

int main() {
    float fahrenheit;
    printf("Please enter the temperature in Fahrenheit: ");
    scanf("%f", &fahrenheit);
    printf("Temperature in Celsius = %.2f\n", f_to_c(fahrenheit));
    if (is_f_room_temperature(fahrenheit)) {
        printf("%.2f is room temperature\n", fahrenheit);
    return 0;

Classic procedural code does not concern itself with adding behaviors to objects. Instead, it treats data types as data types and isolates the "procedural" behaviors into functions that are performed on those data types. If we stick to pure functions (no side effects, and all inputs map to unique outputs), then we'll have highly testable code that can run in highly-concurrent environments.

For example, adding a Kelvin conversion would look like this,

float c_to_k(float c) {
    return c + 273.15;

Likewise, adding a Fahrenheit to Kelvin conversion would simply chain together two pure functions,

float f_to_k(float f) {
    return c_to_k(f_to_c(f));

Procedural code focuses entirely on behavior. Adding this functionality in a pure OOP style would result a laundry list of classes, interfaces, and methods. It can get out of hand quickly, and we'd soon be researching design patterns to try to regain some sense of code quality.

In practice, most developers tend to treat OOP and procedural programming with a sort of religious devotion, zealously adhering to their preferred programming style and feeling that the alternative is sacrilege. I think Uncle Bob was onto something when he said that "good software developers understand these issues without prejudice and choose the approach that is best for the job at hand." That's also from Clean Code, a book that should be read at least as often as it's referenced (it's a bit like George Orwell's 1984, most people reference it without ever having read it).

Uncle Bob is certainly more diplomatic than Joe Armstrong, the creator of Erlang, who had famously said,

"The problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle."

To date, I've never heard a reasonable counter-argument to this objection to OOP, namely, that objects bind data structures and functions together (which inevitably leads to an explosion of side-effects). Even as you try to decouple the banana from the gorilla, you end up creating even more classes, more side effects, and most likely an even worse problem. I'm not sure I'd go so far as to say OO Sucks, but I am hard pressed to defend OOP in light of decades of hard learned lessons.

Obviously, good code is preferable to bad code in any language. There is plenty of bad procedural code out in the world. But honestly, in OOP you often find good programmers writing bad code. Let's go back to some of the earliest lessons in software engineering, specifically, Fred Brook's essay, No Silver Bullet, and ask ourselves how much accidental complexity has been created by OOP? How much code in an average OOP project is tackling the essential complexity of a problem versus the accidental complexity?

In fairness, OOP was popularized by Java, which solved many problems from the early days of C and C++ (such as garbage collection and platform independence). In the decades since, Java has added capabilities found in modern languages (such as lambda expressions, collections, stream api, higher-order functions, etc). Most of the new capabilities come from the world of functional programming, and exactly zero of these capabilities come from OOP.

Whether we like it or not, the future may not be kind to OOP. Multi-core architectures and distributed computing are pushing software into high-concurrency asynchronous environments. Even worse, the push to cloud computing and microservices leads us to an increase in latency within a highly concurrent asynchronous world. This is an ideal environment for a separation of data structures from functions (pure functions). This is a great environment for Haskell and Erlang (or coding pure functions using Scala, Python, or Go), but regardless of the language, you couldn't ask for a worse environment for OOP.

Posted in c, java, software eng.

Trie or Set

Given a grid or input stream of characters, I would like to discover all words according to a given dictionary. This could be a dictionary of all English words or phrases (say, for an autocomplete service), or for any language. This is especially useful for languages where words are not clearly separated (e.g., Japanese, Chinese, Thai).

Typically, this is done with a Trie or a DAWG (Directed Acyclic Word Graph). A Trie can be implemented in Python using a nested dict, i.e.,

def _make_trie(wordict):
    trie = {}
    for word in wordict:
        current_trie = trie
        for letter in word:
            current_trie = current_trie.setdefault(letter, {})
        current_trie['$'] = '$'
    return trie

def _in_trie(trie, word):
    ''' True IFF prefix or word in trie
    current_trie = trie
    for letter in word:
        if letter in current_trie:
            current_trie = current_trie[letter]
            return False
    return True

Using this approach, we can scan through a large stream of characters for potential words. Imagine a classic matching game where you are looking for words within a grid of characters. Programmatically, you would scan through the grid with combinations of characters. The advantage of a Trie (or DAWG) is that it allows for efficient pruning. In other words, if a character combination is not in the Trie, then you can cease pruning.

An alternative approach is to create a Set of word prefixes, i.e.,

almost_words = set([])
for word in wordict:
    for i in range(len(word)-1):
        almost_words.add( word[0:i+1] )

If the dictionary contains ['apple', 'are'] then the Set almost_words would contain the following,

{'a', 'ap', 'app', 'appl', 'ar'}

In other words, rather than test if a character string exists in the Trie, one can simply check the Set almost_words. If there is no match then that particular path can be pruned. Here is a simple RTL (right-to-left) character scanner that uses this approach:

def _setscan_rtl(grid, wordict):
    ''' generator yielding word candidates
    almost_words = set([])
    maxlen = 0
    for word in wordict:
        if len(word) > maxlen:
            maxlen = len(word)
        for i in range(len(word)-1):
            almost_words.add( word[0:i+1] )
    for line in grid:
        for i in range(max(len(line),maxlen) - maxlen):
            candidate_word = ''
            for c in range(min(len(line),maxlen)):
                candidate_word += line[i+c]
                if candidate_word not in almost_words:
                yield candidate_word

I created a simple test case to determine if a Set was truly faster, and whether or not it was as memory efficient. There was a noticeable increase in performance using Set over Trie (for both large and small data sets). Interestingly, the performance difference was even more pronounced when using Japanese characters, indicating that language parsers can use a simple Set (or hashmap) as opposed to a Trie or a DAWG.

$ /usr/bin/time ./
177.84user 0.42system 2:58.54elapsed 99%CPU (0avgtext+0avgdata 507412maxresident)k
0inputs+0outputs (0major+145801minor)pagefaults 0swaps

$ /usr/bin/time ./
250.44user 0.56system 4:11.86elapsed 99%CPU (0avgtext+0avgdata 680960maxresident)k
0inputs+0outputs (0major+184571minor)pagefaults 0swaps

Full results and code are available on my github.

Posted in data arch., python


I would like to iterate over a stream of words, say, from STDIN or a file (or any random input stream). Typically, this is done like this,

def iter_words(f):
    for line in f:
        for word in line.split():
            yield word

And then one can simply,

for word in iter_words(sys.stdin):
    # do something

For a more concrete example, let's say we need to keep a count of every unique word in an input stream, something like this,

from collections import Counter
c = Counter

for word in iter_words(sys.stdin):

The only problem with this approach is that it will read data in line-by-line, which in most cases is exactly what we want, however, in some cases we don't have line-breaks. For extremely large data streams we will simply run out of memory if we use the above generator.

Instead, we can use the read() method to read in one-byte at a time, and manually construct the words as we go, like this,

def iter_words(sfile):
    chlist = []
    for ch in iter(lambda:, ''):
        if str.isspace(ch):
            if len(chlist) > 0:
                yield ''.join(chlist)
            chlist = []

This approach is memory efficient, but extremely slow. If you absolutely need to get the speed while still being memory efficient, you'll have to do a buffered read, which is kind of an ugly hybrid of these two approaches.

def iter_words(sfile, buffer=1024):
    lastchunk = ''
    for chunk in iter(lambda:, ''):
        words = chunk.split()
        lastchunk = words[-1]
        for word in words[:-1]:
            yield word
        newchunk = []
        for ch in
            if str.isspace(ch):
                yield lastchunk + ''.join(newchunk)
Posted in python


I would like a webapp that supports UTF-8 URLs. For example, https://去.cc/叼, where both the path and the server name contain non-ASCII characters.

The path /叼 can be handled easily with %-encodings, e.g.,

>>> import urllib
>>> urllib.parse.quote('/叼')

Note: this is similar to the raw byte representation of the unicode string:

>>> bytes('/叼', 'utf8')

However, the domain name, "去.cc" cannot be usefully %-encoded (that is, "%" is not a valid character in a hostname). The standard encoding for international domain names (IDN) is punycode; such that "去.cc' will look like "".

The "xn--" prefix is the ASCII Compatible Encoding that essentially identifies this hostname as a punycode-encoded name. Most modern web-browsers and http libraries can decode this kind of name, although just in case, you can do something like this:

>>> '去'.encode('punycode')

In practice, we can use the built-in "idna" encoding and decoding in python, i.e., IRI to URI:

>>> p = urllib.parse.urlparse('https://去.cc/叼')
>>> p.netloc.encode('idna')
>>> urllib.parse.quote(p.path)

And going the other direction, i.e., URI to IRI:

>>> a = urllib.parse.urlparse('')
>>> a.netloc.encode('utf8').decode('idna')
>>> urllib.parse.unquote(a.path)
Posted in python

Using getattr in Python

I would like to execute a named function on a python object by variable name. For example, let's say I'm reading in input that looks something like this:

enqueue 1
enqueue 12
enqueue 5
enqueue 9

Afterwards, we should see:


Let's say we need to implement a data structure that consumes this input. Fortunately, all of this behavior already exists within the built-in list datatype. What we can do is extend the built-in list to map the appropriate methods, like so:

class qlist(list):
    def enqueue(self, v):

    def dequeue(self):
        return self.pop()

    def print(self):

The sort and reverse methods are already built-in to list, so we don't need to map them. Now, we simply need a driver program that reads and processes commands to our new qlist class. Rather than map out the different commands in if/else blocks, or use eval(), we can simply use getattr, for example:

if __name__ == '__main__':
    thelist = qlist()
    while line in sys.stdin:
        cmd = line.split()
        params = (int(x) for x in cmd[1:])
        getattr(thelist, cmd[0])(*params)
Posted in shell tips

Graph Search

I would like to discover paths between two nodes on a graph. Let's say we have a graph that looks something like this:

graph = {1: set([2, 3]),
         2: set([1, 4, 5, 7]),
         3: set([1, 6]),
         N: set([...] }

The graph object contains a collection of nodes and their corresponding connections. If it's a bi-directional graph, those connections would have to appear in the corresponding sets (e.g., 1: set([2]) and 2: set([1])).

Traversing this kind of data structure can be done through recursion, usually something like this:

def find_paths(from_node, to_node, graph, path=None):
    ''' DFS search of graph, return all paths between
        from_node and to_node
    if path is None:
        path = [from_node]
    if to_node == from_node:
        return [path]
    paths = []
    for next_node in graph[from_node] - set(path):
        paths += find_paths(next_node, to_node, graph, path + [next_node])
    return paths

Unfortunately, for large graphs, this can be pretty inefficient, requiring a full depth-first search (DFS), and storing the entire graph in memory. This does have the advantage of being exhaustive, finding all unique paths between two nodes.

That said, let's say we want to find the shortest possible path between two nodes. In those cases, you want a breadth-first search (BFS). Whenever you hear the words "shortest path", think BFS. You'll want to avoid recursion (as those result in a DFS), and instead rely on a queue, which in Python can be implemented with a simple list.

def find_shortest_path(from_node, to_node, graph):
    ''' BFS search of graph, return shortest path between
        from_node and to_node
    queue = [(from_node, [from_node])]
    while queue:
        (qnode, path) = queue.pop(0) #deque
        for next_node in graph[qnode] - set(path):
            if next_node == to_node:
                return path + [next_node]
                queue.append((next_node, path + [next_node]))

Because a BFS is guaranteed to find the shortest path, we can return the moment we find a path between to_node and from_node. Easy!

In some cases, we may have an extremely large graph. Let's say you're searching the Internet for a path between two unrelated web pages, and the graph is constructed dynamically based on scraping the links from each explored page. Obviously, a DFS is out of the question for something like that, as it would spiral into an infinite chain of recursion (and probably on the first link).

As a reasonable constraint, let's say we want to explore all the links up to a specific depth. This could be done easily. Simply add a depth_limit, as follows:

def find_shortest_path(from_node, to_node, graph, depth_limit=3):
    queue = [(from_node, [from_node])]
    while queue and depth_limit > 0:
        depth_limit -= 1
        (qnode, path) = queue.pop(0) #deque
        for next_node in graph[qnode] - set(path):
            if next_node == to_node:
                return path + [next_node]
                queue.append((next_node, path + [next_node]))
Posted in python, software eng.

python unittest

I would like to setup unit tests for a python application. There are many ways to do this, including doctest and unittest, as well as 3rd-party frameworks that leverage python's unittest, such as pytest and nose.

I found the plain-old unittest framework to be the easiest to work with, although I often run into questions about how best to organize tests for various sized projects. Regardless of the size of the projects, I want to be able to easily run all of the tests, as well as run specific tests for a module.

The standard naming convention is "", which would include all tests for the named module. This file can be located in the same directory (package) as the module, although I prefer to keep the tests in their own subdirectory (which can easily be excluded from production deployments).

In other words, I end up with the following:

 - test/

Each of the test_*.py files looks something like this:

#!/usr/bin/env python
# vim: set tabstop=4 shiftwidth=4 autoindent smartindent:
import os, sys, unittest

## parent directory
sys.path.insert(0, os.path.join( os.path.dirname(__file__), '..' ))
import ModuleName

class test_ModuleName(unittest.TestCase):

    def setUp(self):
        ''' setup testing artifacts (per-test) '''
        self.moduledb = ModuleName.DB()

    def tearDown(self):
        ''' clear testing artifacts (per-test) '''

    def test_whatever(self):
        self.assertEqual( len(, 16 )

if __name__ == '__main__':

With this approach, the tests can be run by, or I can run the individual

The script also must add the parent directory on the path, i.e.,

#!/usr/bin/env python
# vim: set tabstop=4 shiftwidth=4 autoindent smartindent:
import sys, os
import unittest

## set the path to include parent directory
sys.path.insert(0, os.path.join( os.path.dirname(__file__), '..' ))

## run all tests
loader = unittest.TestLoader()
testSuite =".")
text_runner = unittest.TextTestRunner().run(testSuite)
Posted in python

HTML + CSS + JavaScript Lessons

I would like a very simple introduction to web development, from the basics of HTML and CSS, to the proper use of JavaScript; and all without getting bogged down in complicated textbooks.

I've been working with HTML, CSS, and JavaScript (as well as dozens of programming languages in more environments than I can remember) for over 20 years. While there are some excellent resources online (I recommend w3schools), I believe web development is a very simple topic that is often unnecessarily complicated.

I created a simple set of 9 lessons for learning basic web development. This includes HTML, CSS, and some simple JavaScript (including callback functions to JSONP APIs), everything you need to make and maintain websites.

You can find the lessons here

It's also available on Github

Posted in css, html, javascript