IMMENSE.LY

Thisisfine

Thu, 17 Dec 2020 03:17:29 +0000

After a long period of not posting, I present my latest doodle, just in time to see off 2020.

You can try it out on kubernetes:

kubectl run -it --rm --image=ghcr.io/pdevine/thisisfine tif

or for the docker inclined:

docker run -it --rm ghcr.io/pdevine/thisisfine

… and it should look something like:

This marks a return to using block mode in Go-AsciiSprite, and it’s the first time I’ve ever used one for doing a doodle. The first project I did with block mode was Text Invaders!, and I also put together a demo for something that resembled Super Mario Bros, but I was never really happy with the way either worked.

The biggest problem was movement. When I created Go-AsciiSprite originally I had only been thinking of ways to move normal text characters around the screen. That worked great, but with block mode we effectively doubled the resolution of the terminal. If you moved a block mode sprite, it would always move an even number of pixels around the screen. You could never move it by an odd number of pixels.

This ended up being fine for Text Invaders!, but I felt like the movement looked really jerky. For a while this seemed like a bit of a dead end. Eventually I realized I could solve the problem by introducing a new screen buffer. Instead of rendering each block of a sprite and then moving the entire sprite, I could instead delay rendering by creating a really big surface and then rendering the blocks as a whole frame.

Surfaces were an idea that I borrowed from Pygame. I’m still struggling with whether these should be a unique feature of Go-AsciiSprite, or whether I should just use Golang Images. I’m pretty torn here between using something which is a standard and used in a lot of different applications and scenarios, or crafting something which is easy to understand with only the features needed to easily render cheezy graphics in the terminal.

I guess we’ll see. This is fine, right?

Mastermind

Sun, 12 Apr 2020 20:53:12 +0000

A friend recently posted this puzzle that I thought was pretty awesome:

This is one of those brain teasers that I think looks somewhat daunting at first, but if you have played the game Mastermind before, it’s pretty straightforward to solve.

I’ve been giving Mastermind as a software engineering programming interview exercise for more than a decade. What I like about it is that it’s easy to describe to someone who has never played the game before, and also that it has three parts to it which get progressively more difficult. This first starts off with a simple recursive algorithm (or you can solve it with binary math for those inclined) for generating a search space. It then requires you to write a slightly tricky “scoring” function for determining which digits are in the correct or incorrect places. If you’ve made it this far and there’s time left, the third part is about writing an algorithm for “solving” the code.

Here’s the code I used to solve this particular puzzle:

import itertools
  
combos = itertools.product(range(0, 10), repeat=3)
 
def score(guess, secret):
    red = 0
    g_nums = [0] * 10
    s_nums = [0] * 10

    for x in range(0, len(guess)):
        if guess[x] == secret[x]:
            red += 1
        else:
            g_nums[guess[x]] += 1
            s_nums[secret[x]] += 1
 
    white = sum([min(g_nums[x], s_nums[x]) for x in range(0, len(g_nums))])
    return red, white

def filter(guess, g_score):
    global combos
    n_combos = []
    for c in combos:
        if score(c, guess) == g_score:
            n_combos.append(c)

    combos = n_combos

def main():
    filter((7, 1, 3), (1, 0))
    filter((7, 0, 2), (0, 1))
    filter((3, 5, 7), (0, 2))
    filter((8, 9, 1), (0, 0))
    filter((9, 1, 5), (0, 1))

    print combos

if __name__ == "__main__":
    main()

Let’s start off with the first part, which is generating the search space. Here I’m being extremely lazy. There are 10³ (i.e. 1,000) potential “combinations” to choose from to find the correct “combination” to open the lock. This is where I hate the word “combinations” because it’s not really technically correct. A combination implies that order of the digits doesn’t really matter, but in this case, that’s not true. It’s also not technically correct to say that there are 1,000 “permutations” for our “combination” because a permutation implies that we can’t use the same value more than once (i.e. there are not 10!/7! or 720 permutations). In this case, we’re using a permutation with repetition, or more specifically the cartesian product of the three sets of 10 digits. What ‘itertools.product()’ is doing is creating a generator with 1,000 tuples which look like: (0, 0, 0), (0, 0, 1), (0, 0, 2), … (9, 9, 7), (9, 9, 8), (9, 9, 9).

For the interview I give, I usually use colours instead of digits (at least for the examples), and I use 8 possible colours, and 4 positions instead of 3. Using powers of two makes the math a lot tighter for software engineers since we’re used to working that way. Typically the candidate also writes a recursive function here to solve this, although often things go off the rails here either because recursion can be tricky, or because they try to write it iteratively and don’t want to use four nested for loops. Key to doing well here is really understanding how many times the inner part of your for loop is being called. Also, I don’t mind if candidates use things like itertools() because it just shows that they understand how to use a language, so if they want to write a one liner, that’s just perfectly fine.

After generating the search space, the scoring function comes next. The scoring function takes two codes and compares them against each other to determine two things: How many digits are in the right place, and how many are correct but in the wrong place. In the game this is given by “red” and “white” markers respectively, hence why I chose those as variable names in the code. The biggest thing to remember is that there is an order of operations where we need to get the “red” values before we get the “white” values. What I mean by this is that we should check for the right colour/digit being in the right place before we check to see if it’s the write colour/digit in the wrong place.

For any values which are not in the correct place, we’re going to create a tally for both the guess and the secret number. Let’s say we had two numbers, (5, 2, 1) and (1, 1, 3). We would put (5, 2, 1) into the tally array so that it would look like:

[0, 1, 1, 0, 0, 1, 0, 0, 0, 0]

That is, we have one 1, one 2, and one 5. If we take (1, 1, 3), it would look like:

[0, 2, 0, 1, 0, 0, 0, 0, 0, 0]

We have two 1’s, and one 3.

To get the number of numbers which were in the incorrect place, we can compare the tallys for each number and take the sum of the minimum values. Most of the time we’re just comparing 0’s, so we don’t increase the total sum, but for the number 1, we have two 1’s in the guess, and one 1 in the secret, so min(2, 1) gives us 1, which is the total of correct numbers in the incorrect place.

The third and final part of the code is writing the filter function. For this we take the search space we already generated (the 1,000 tuples), and we iterate through each of them comparing them against the number we’re guessing and then checking to see if the result of the scoring function matches our guess score. The cool thing about the scoring function is you can feed numbers in either way and they will always give us the same score. For our example we could give (5, 2, 1) and (1, 1, 3) for either the guess or the secret or switch them around and we would always get back the same score. For each of the two numbers that we’re going to check (our guess number and each number in our search space), we only keep ones which match our guess score.

This technique actually throws out a lot of numbers really quickly. If you look at the size of the search space each time we run it against a filter, we get the correct answer after just three guesses. That’s because the score gives us a lot of information about what we’re looking for, even if we guessed something that was entirely incorrect.

Levenshtein Distance

Wed, 27 Nov 2019 18:53:49 +0000

A few weeks ago I posted about One Way Changes, which was one of the coding challenges in Cracking the Coding Interview. A One Way Change would check to see if two words, such as “flood” and “floor” are one change away from each other by removing, inserting, or substituting one character for another one.

I’ve been thinking a lot about search recently, and learning about where the industry is at with NLP. One of the first things I ran in to was an exercise for calculating how many inserted, removed, or substituted characters it takes to get from one word to another. This is the minimum edit distance or the Levenshtein distance between two words, and it seemed strange to me that CtCI didn’t include this problem in the Recursion and Dynamic Programming chapter, given that it superficially feels really close to the One Way Change exercise.

Surprisingly though, the easiest way you can calculate the Levenshtein distance is unrelated to actually inserting, removing, or substituting characters into your first string. I guess that shouldn’t be so suprising given how many possible characters there are that you could potentially try, but my first intuition was to try to think about how I could reuse my solution to the One Way Changes problem.

A far better solution involves calculating how close each potential substring of each string is to each other substring. The algorithm I ended up implementing was essentially the Wagner-Fisher algorithm which iteratively builds up a matrix of all possible substrings. Here was my solution:

def lev_distance(s1, s2, lev_cost=1):
    def lev_sub(c1, c2):
        if c1 != c2:
            return lev_cost
        return 0

    s1 = " " + s1
    s2 = " " + s2

    n = len(s1)
    m = len(s2)

    matrix = [[0 for x in range(n)] for y in range(m)]

    for x in range(n):
        matrix[0][x] = x
    for y in range(m):
        matrix[y][0] = y

    for y in range(1, m):
        for x in range(1, n):
            matrix[y][x] = min(
	        matrix[y][x-1]+1,
		matrix[y-1][x]+1,
		matrix[y-1][x-1] + lev_sub(s1[x], s2[y]))

    return matrix[m-1][n-1]

The first part of the function:

    matrix = [[0 for x in range(n)] for y in range(m)]

    for x in range(n):
        matrix[0][x] = x
    for y in range(m):
        matrix[y][0] = y

creates a zero matrix and then adds an index of both words into the first row and column of the matrix. If we had the two words bitten and knitting, we would end up with:

[[0, 1, 2, 3, 4, 5, 6],
 [1, 0, 0, 0, 0, 0, 0],
 [2, 0, 0, 0, 0, 0, 0],
 [3, 0, 0, 0, 0, 0, 0],
 [4, 0, 0, 0, 0, 0, 0],
 [5, 0, 0, 0, 0, 0, 0],
 [6, 0, 0, 0, 0, 0, 0]]
 [7, 0, 0, 0, 0, 0, 0]]
 [8, 0, 0, 0, 0, 0, 0]]

We now walk through each of the cells and take the minimum of three potential values:

The cell to the left in this row plus one
The cell above in this column plus one
The cell to the left and above this cell plus the value from our levenshtein cost for the two characters at this place

This is done with the code:

    matrix[y][x] = min(
        matrix[y][x-1]+1,
        matrix[y-1][x]+1,
        matrix[y-1][x-1] + lev_sub(s1[x], s2[y]))

In the case of [1,1], we look at [0,1] and [1,0] we get a value of 2 and 2 for each respectively, and we look at [0,0] and add whatever lev_sub() returns, which in this case is a 1. The minimum of (2, 2, 1) is 1, which we then put back into our matrix at [1, 1].

Once we’ve gone through each character in both strings, we’ll end up with a matrix which looks like:

[[0, 1, 2, 3, 4, 5, 6],
 [1, 1, 2, 3, 4, 5, 6],
 [2, 2, 2, 3, 4, 5, 5],
 [3, 3, 2, 3, 4, 5, 6],
 [4, 4, 3, 2, 3, 4, 5],
 [5, 5, 4, 3, 2, 3, 4],
 [6, 6, 5, 4, 3, 3, 4],
 [7, 7, 6, 5, 4, 4, 3],
 [8, 8, 7, 6, 5, 5, 4]]

Back to our example of changing bitten to knitting, the shortest way to transform it is in four steps:

bitten -> kbitten (insert a ‘k’)
kbitten -> knitten (substitute ‘n’ for ‘b’)
knitten -> knittin (substitute ‘i’ for ‘e’)
knittin -> knitting (insert a ‘g’)

and if we look in our matrix in the last position [8,6], we get our result for the shortest number of steps.

Is Rotation

Wed, 06 Nov 2019 00:36:21 +0000

I’m not certain how practical this next problem is, but it seems to be a staple of interviewing questions. The problem is to write a function to check to see if one string is a rotation of another string. By rotation, they’re asking if the string is offset by some amount of characters, and then has excess characters prepended to the beginning of the string. For example, “erwat” is a rotation of the string “water”.

I ended up solving this two different ways, the first was to just brute force the problem by writing a function which would rotate one of the strings until it matched the other string. Here’s the code:

def isRotation1(s1, s2):
    if len(s1) != len(s2):
        return False

    def rotateStr(mystr, count):
        return mystr[count:] + mystr[0:count]

    for n in range(len(s1)):
        if s2 == rotateStr(s1, n):
            return True

    return False

This is pretty ineffecient. The string copy is super expensive, so the runtime of this is O(N²) which is pretty ugly.

A smarter way of solving this problem is to concatenate one of the strings against itself and then searching for the other string inside of the larger string. With our “erwater” example, we end up with a larger string which is “erwaterwat” and then we just search in that string for “water”. If it exists, then this must be a rotated string. Here’s the code to do that:

def isRotation2(s1, s2):
    if len(s1) != len(s2):
        return False

    if s1 in s2 + s2:
        return True

    return False

Set Matrix Zeros

Mon, 04 Nov 2019 19:14:37 +0000

Much like the Rotation Matrix, the “Set Matrix Zeros” problem involves being careful about not destroying the existing matrix which is being operated on. The idea is that given a cell in the matrix which is set to zero, set all of the other cells in the same row or column to zero.

I tackled this in two ways which are almost identical. First up was just making a copy of the initial matrix and working on that instead. Here’s the code:

import copy

def zeroMatrix1(mtrx):
    newMtrx = copy.deepcopy(mtrx)

    def zeroRowCol(r, c):
        for n in range(len(mtrx)):
            newMtrx[n][c] = 0
        for n in range(len(mtrx[0])):
            newMtrx[r][n] = 0

    for r in range(len(mtrx)):
        for c in range(len(mtrx[0])):
            if not mtrx[r][c]:
                zeroRowCol(r, c)
    return newMtrx

The deepcopy is important here otherwise you end up with a copy of the original matrix and you’ll be operating on that instead. There’s not really a “trick” here per se, but the key is if we hadn’t made the copy and zeroRowCol() operated on the original matrix, it would fill in everything with lots of zeros.

Since making a copy of the matrix is expensive, there is a way of doing this by just keeping track of any cells we found that had a zero, and calling zeroRowCol() later. That looks like this:

``` def zeroMatrix2(mtrx): def zeroRowCol(r, c): for n in range(len(mtrx)): mtrx[n][c] = 0 for n in range(len(mtrx[0])): mtrx[r][n] = 0

points = list()

for r in range(len(mtrx)):
    for c in range(len(mtrx[0])):
        if not mtrx[r][c]:
            points.append((r, c))

for p in points:
    zeroRowCol(*p)

return mtrx
```

Rotation Matrix

Sun, 03 Nov 2019 16:33:46 +0000

Next up is “Rotate Matrix” which, unsurprisingly, rotates an X by X matrix by 90 degrees. There are two ways of tackling this problem, the first which is creating a new matrix and copying over each of the cells, and the second is to be able to swap elements in place. The second is harder because it means you have to iterate around the matrix with each of the values because you don’t have a separate space to put them.

Let’s go over the solution which copies elements to a new matrix:

def rotateMatrix(mtrx):
    N = len(mtrx)
    newMtrx = [[0 for x in range(N)] for y in range(N)]
    for r in range(N):
        for c in range(N):
            newMtrx[r][c] = mtrx[N-1-c][r]
    return newMtrx

and let’s say we have a 4x4 matrix which looks like:

[['A','B','C','D'],
 ['E','F','G','H'],
 ['I','J','K','L'],
 ['M','N','O','P']]

We want to rotate that matrix so that it looks like:

[['M','I','E','A'],
 ['N','J','F','B'],
 ['O','K','G','C'],
 ['P','L','H','D']]

In rotateMatrix() we create a new blank 4x4 matrix initially filled with zeros. We then walk through each of the cells of the matrix and take each cell in each column from the bottom up. The runtime here is O(N) even though we have the two for loops because we’re only ever touching each cell once. The biggest problem is that we’re using twice the amount of space because of our new matrix.

Now for the second solution. To rotate without creating a new matrix, we instead will need to carry each cell around the matrix. So in the case of our above matrix, we save the ‘A’ from (0,0) and then carry the ’M’ from (3,0). We then take the ‘P’ from (3,3) and put it in (3,0), and then the ’D’ from (0,3) and put it in (3,3). Repeat that process for ‘B’ and ‘C’, and then rotate the cells at the center of the matrix.

Here’s the code:

def rotateMatrixInPlace(mtrx):
    N = len(mtrx)
    for r in range(N/2):
        for c in range(r, N-1-r):
            tmp = mtrx[r][c]
            mtrx[r][c] = mtrx[N-1-c][r]
            mtrx[N-1-c][r] = mtrx[N-1-r][N-1-c]
            mtrx[N-1-r][N-1-c] = mtrx[c][N-1-r]
            mtrx[c][N-1-r] = tmp

This again works out to O(N) since at most we’re going to 20 swaps (16 for copying around the cells, and 4 for the swaps).

Run Length Encoding

Fri, 01 Nov 2019 15:23:38 +0000

I was actually somewhat surprised that Run Length Encoding (they call it “String Compression”) was included in Cracking the Coding Interview, because it’s not particularly a difficult algorithm to write. I would have thought that writing Huffman Encoding would have been more interesting in interviews given the entire chapter on graphs and trees in the book.

The key to RLE is to remember the last character as you’re iterating through the string. If the next character is different, write out the current character and append a count of the number of times you have seen it.

Here’s the code:

def runlength(s):
    counter = 0
    lastChar = ''
    newStr = ""

    for currChar in s:
        if currChar == lastChar:
            counter += 1
        else:
            if lastChar != '':
                newStr += lastChar + str(counter)
            counter = 1
            lastChar = currChar

    newStr += lastChar + str(counter)

    return newStr

I didn’t bother with the check for whether the returned string is shorter than the string which was passed in, however, it’s easy enough to do.

But, there is a slight problem here, and I’m guessing that they’re expecting interviewers to play this up. The runtime complexity for iterating through the string is going to be O(N), but the string copy, in theory, is going to be O(N²) every time we concatenate the new string. Or rather, it would be in anything prior to Python 2.4 which changed the behaviour of how concatenation works. And I guess the question is do you really want to be fighting with the interviewer in the middle of a technical interview over the subtleties of how Python string concatenation works? The answer to that is “maybe” depending on the interviewer, but this is starting to feel like a test of wits with iocane powder.

Anyway, long story short, if we used a list() and appended to it instead of appending to the string, we could ensure that the function remains O(N). Use a "".join(myStr) at the end and pass the new resultant string back.

Happy Halloween!

Thu, 31 Oct 2019 04:00:08 +0000

Happy Halloween everyone! To celebrate, I’ve released a new Docker Doodle!

$ docker run -it --rm docker/doodle:halloween2019

A fun fact: I made the ASCII moon follow the actual phases of the moon, so in theory it should change to a “first quarter” moon on November 4th. Halloween this year gets a waxing crescent moon.

One Way Changes

Tue, 29 Oct 2019 22:07:13 +0000

Next up is writing a program which can compare two strings to see if they’ve been changed by having had one character removed, one extra character inserted, or one character replaced with a different character.

I tackled this by breaking it into three separate problems. I think the point of this question is to see whether you can decompose a problem into smaller steps and then see how you can glue them back together again.

First up is writing the function which can remove a character from one string and see if it matches the other string.

def removeChar(s1, s2):
    for n in range(len(s1)):
        if s2 == s1[:n] + s1[n+1:]
            return True
    return False

This is just walking through each character of the string and then using slicing techniques to construct new possible strings. If we match our second string, return True, otherwise just return False.

Next we need to check if a character was inserted into the first string. We could write a routine which adds each letter of the alphabet into each position of our first string, but that is needlessly fussy and would take a lot of time and complexity when there is a far better option. We’ve already got the string we’re trying to match against, so instead, let’s just reuse our removeChar() function and flip the two strings around. That way we’ll remove characters from our second string, and if the two strings match, we’ll know that a character was inserted.

def insertChar(s1, s2):
    return removeChar(s2, s1)

We then need to check if only one character is different in each of the strings. The easiest way to do this is just walk through one of the strings and compare it to the other one.

def replaceChar(s1, s2):
    if len(s1) != len(s2):
        return False

    oneChanged = False
    for n in range(len(s1)):
        if s1[n] != s2[n]:
            if oneChanged:
                return False
            oneChanged = True
    return oneChanged

Now that we’ve got all of our functions, we just glue them together.

def oneway(s1, s2):
    return insertChar(s1, s2) or removeChar(s1, s2) or replaceChar(s1, s2)

In terms of runtime complexity, we haven’t done anything super fancy here. We’re mostly just iterating through copies of one of the strings. The kicker though are the string slices. Those are making full copies of the string, which means we’re unfortunately at O(n²) time complexity. We might be able to use some other data structure for creating the slices, but unless there was some requirement for cutting that down, I think this solution is still sufficient.

Palindrome Permutations

Mon, 28 Oct 2019 19:18:46 +0000

This was kind of a weird problem, but the solution was kind of fun to write. Given a string, determine if it’s a palindrome and then write out all the possible permutations of palindromes that string potentially would have.

Here’s my solution:

import collections
import itertools

def palinpermute(s):
    s = s.replace(" ", "")
    c = collections.Counter(s)

    pivot = ''
    for k, v in c.iteritems():
        if v % 2 != 0:
            if not pivot:
                pivot = k
            else:
                return (False, [])

    newStr = ""
    for k, v in c.iteritems():
        if k != pivot:
            newStr += k*(v/2)

    palins = list()
    for c in itertools.permutations(newStr):
        palins.append("".join(c) + pivot + "".join(reversed(c)))

    return (True, palins)

Like a lot of the other solutions to string problems, we’re back to using the Counter again. In this case we use it to make certain that for each character, there has to be a matching one. If there isn’t a matching one, choose that character as a pivot point for our palindrome. Once we’ve selected a pivot point, there can’t be another one, so just return False and an empty list if we find one.

This next part is probably not obvious:

    newStr = ""
    for k, v in c.iteritems():
        if k != pivot:
            newStr += k*(v/2)

This generates a half of a palindrome which we’re going to use to find all of the permutations. If we had the palindrome tacocat, we would end up with a string like act here. When we call itertools.permutations(), that’s going to come up with a list which looks like [('a', 'c', 't'), ('a', 'c', 't'), ('c', 'a', 't'), ('t', 'a', 'c'), ('c', 't', 'a'), ('t', 'c', 'a')]. Now it’s just a simple matter of joining each of these lists into new strings around our pivot point (if we have one), and we’ve found each of the potential palindromes.

The runtime complexity here comes down to how itertools.permutations() works. The permutations come out in lexigraphical sort order, so that’s at least n*log(n) average. Behind the scenes it’s doing something more akin to itertools.product() and filtering out repeated results, which means this can get kind of ugly really quickly from a performance standpoint.

I’m sure there are some interviewers out there who might think using itertools.permutations() feels like cheating. My take on it when I have interviewed candidates is that with very few exceptions it’s fair game to use parts of any standard library for any given programming language. Regardless, it’s not that hard to write some kind of recursive function which generates the same list if the interviewer is demanding it.

String Permutations

Mon, 28 Oct 2019 16:28:02 +0000

Up next is checking whether one string is a permutation of another string. When I originally wrote my solution I was thinking the question was asking to find a permutation which is also a substring of the other string. My solution for that problem looks like:

from collections import Counter
def isPermutation(s1, s2):
    c = Counter(s1)
    for n in s2:
        c[n] -= 1

    for v in c.itervalues():
        if v < 0:
            return False

    return True

This again makes use of a Counter from collections. I think the only real takeaway here is you should definitely clarify with the interviewer up front what’s expected by the question. In this case it would have been better to have just written:

def isPermutation(s1, s2):
    c1 = Counter(s1)
    c2 = Counter(s2)
    return c1 == c2

Which is really just an exercise in comparing two dictionaries. Behind the scenes python is going to take O(n) to construct the two counters, but it’s unclear to me exactly how it will compare them against each other. My suspicion would be that it would compare the keys of the two dictionaries in alphabetical order, which would mean n*log(n) for runtime complexity, but you could have constructed the counter as an array of 26 or so integers for all of the lowercase characters in the string, and then compared the two arrays. For that case you’re still in O(n).

The other solution the book mentions is to sort the arrays themselves and compare, which would look something like:

def isPermutation(s1, s2):
    return sorted(s1) == sorted(s2)

That saves you the extra step of using the Counter, and you end up in n*log(n) land again thanks to the timsort. There’s also the temporary storage for holding the two ordered strings which gets chucked away once the function returns.

Unique Strings

Mon, 28 Oct 2019 04:09:45 +0000

I’ve been perusing Cracking the Coding Interview lately, and thought I’d share some of my solutions to a few of the problems. If you’re planning on doing a coding interview, I’d definitely recommend attemping to come up with your own solutions, but I figured these might be useful as I wrote them in Python and the book is pretty heavy on Java.

I’m actually not a huge Java fan for doing interviews. Most of the Java programmers that I’ve interviewed over the years tended to get caught up in syntax or fighting the compiler on CoderPad and things have a habit of not turning out well. I think this is exacerbated by a lot of engineers being used to writing in an IDE and then struggling with an unfamiliar interface. When I’m conducting interviews, I actually don’t really care what language a person is using, although I give the interviewee that caveat that if I don’t know the language I’m probably going to ask a lot more questions.

Anyway, first up is the Unique Strings problem, which I guess is kind of like FizzBuzz. I think the point is really to just get you to warmed up to move on to more difficult problems. In Python I cooked up a bunch of different ways to do this.

First up is using a set:

def allUnique1(s):
    return len(set(s)) == len(s)

This uses the fact that sets will omit duplicates, so if you pass it a string with the same character twice, you’re going to end up with a shorter length set. In terms of runtime complexity, this is O(n), but it does require making an extra copy of the string inside of the set, however, if the string was just ASCII characters, the set would be pretty tiny. If it was in Unicode, then it could in theory be a lot longer, but if you’re feeding in a huge string, the set is still going to be pretty tiny.

The second solution relies on sorting:

def allUnique2(s):
    s = sorted(s)
    for cnt in range(len(s)-1):
        if s[cnt] == s[cnt+1]:
            return False
    return True

Python uses “Timsort” which is a hybrid between Mergesort and Quicksort. After sorting the list in place, we iterate through it checking adjacent cells to see if they’re the same. Time complexity here is n*log(n) on average for the sorting. I’m actually not 100% sure of the space required behind the scenes with sorted(), although ultimately we’re assigning over the top of our string, so any used memory would be freed up. Also, I decided to count down here instead of counting up, but you could go either way.

The third and fourth solutions involve using collections.Counter().

from collections import Counter
def allUnique3(s):
    c = Counter(s)
    for v in c.itervalues():
        if v > 1:
            return False
    return True

and:

from collections import Counter
def allUnique4(s):
    c = Counter(s)
    return len([v for v in c.iterkeys()]) == len(s)

These are both similiar but allUnique3() will perform slightly better because it doesn’t have to count the all of the values inside of the counter if it detects a duplicate. allUnique4() is essentially the same thing as allUnique1(), just done with a Counter instead of a set.

Were I a gambling man, I would guess that an interviewer is expecting an O(n²) solution which iterates through the string multiple times to find the duplicate and then asking to refine on that. That’s fine for a lower level language like C or Assembly, but in Python I think allUnique1() would probably be your best bet in almost every case.

Switching to Hugo

Sat, 26 Oct 2019 22:09:15 +0000

Ever since I first created this blog I’ve had reservations about using Wordpress. It’s not that I have anything particularly against Wordpress, it’s just that as someone who is more comfortable on the command line than in a browser, I always felt a little out of my element.

It also felt a bit like overkill running a small, fairly static blog with a backend database to hold all of the content, even if that database was just a tiny MariaDB instance. It was running in a docker container, but I’d neglected it for almost two years because, honestly, who really has time to pay attention to that stuff for a tiny blog with about 0.3 viewers? Plus, even that “tiny” database image was weighing in at over 400MB (side note, newer versions are down to around 100ish MBs compressed, so I guess that’s good?).

All of this had been weighing on my psyche for quite some time. I’d been thinking about using Hugo for a while, but I knew converting everything over was going to be a huge pain in the neck. It got to the point where I had stopped writing blog posts because I didn’t want to use the clunky Wordpress interface and knew I’d just be compounding my misery when I finally did switch everything over. The problem though was at the same time didn’t want to deal with the cognitive burden to figure out how to actually do the conversion.

But that’s all behind me now, as the new site is up and running. Switching everything over required:

Dumping all of the posts into XML
Converting the XML into a series of markdown blog posts (I think I used blog2md in a container, but this got lost in a container somewhere)
Figuring out how to use Hugo to generate static sites and trying to work out a similar theme
Fixing up each of the blog posts to be formatted correctly (apparently Wordpress and Hugo markdown isn’t quite the same)
Figuring out how to display galleries in Hugo (I still don’t have the old photo captions ported over, but maybe one day)
Converting all of the old galleries
Upgrading the Digital Ocean droplet to run Ubuntu 18.04 (it was still on 16.04)
Upgrading Docker and docker-compose to 19.03 (they were on versions from 2017!)
Getting nginx to work properly with the new static site and the old TLS certs from Let’s Encrypt (the site now has an A+ from Qualys. Huzzah!)
Switching Let’s Encrypt to properly upgrade the certs (the jury is still out on if this is actually working, but the cert is still valid for another month and a half)

So all in all, not the easiest process, but it’s alive! The best part now is I can easily write blog posts in markdown with vim and test them out locally using Docker Desktop, push them to GitHub when they’re ready, and then easily refresh and rebuild the site. There’s still some things to do, like minifying everything, but the site load times are already blazingly fast with just static content and nginx.

Radical Revisited

Mon, 04 Mar 2019 05:47:06 +0000

Way back when I was in college in the mid-90’s, I took a year off to professionally write video games. This was in Vancouver, BC, and I was writing (at least attempting to write) in 65816 assembly for the Super Nintendo at a company called Radical Entertainment. At the time I had just finished my second year of college, and I had taken a semester on the basics of machine computation. Everything seemed pretty foreign though after having only programmed at that time in Basic, Pascal, Modula-2, and C.

C is often described as a “low level language”, and being “close to the hardware”, which is definitely true, but at the time it felt a heck of a lot closer to Pascal than it did to writing in assembly. The Super Nintendo doesn’t have a lot of memory, either on the stack, or in ROM, and it wasn’t really possible to do things like have proper subroutines, at least on the code base that I was working on. Most of the code was a mess, and there wasn’t really any formal training; you were expected to just figure it out, and figure it out quickly. This isn’t easy to do when your development system is a cobbled together Super Nintendo with a development board hooked up through the parallel port to a 486. Assembling and loading the whole game could take 10-15 minutes, so you would quickly have to figure out tricks to speed things up like only loading portions of the game into memory.

Radical had had a lot of clunky titles for the SNES in the 90’s including Bebe’s Kids, Brett Hull Hockey, and Power Piggs of the Dark Age. There were a lot of great, super smart, talented people who worked there, but for whatever reason we just didn’t develop really great games. I suppose this mostly had to do with release pressure as well as the sheer difficulty of making magic come out of the Super Nintendo. It’s possible, but even with the blood, sweat, and tears we were pouring in, it was still awfully hard.

The code base I was working on, however, wasn’t terrible because I was developing the title from scratch, or using some clunky library routines, but was due to our team writing a port of two existing games to work with exercise bicycles. Specifically Speed Racer, and Mountain Bike Rally. Both titles, particularly MBR, were kind of a mess. Speed Racer had fantastic artwork, and a really great sound track, but it was a victim of its own ambition. It combined a Mode-7 (faux 3D) racing game with a 2D platformer, and neither part was properly tuned or worked well. The racing parts suffered from brutal yo-yo AI and kind of bizarre track layouts. The platformer had absolutely awful collision detection and unresponsive controls, probably due to having a 7 frame walk cycle animation. MBR was also a Mode-7 game, but it had ridiculously poor controls and made you (honest to god) repeatedly button mash the gamepad in order to pedal the mountain bike. Not exactly a super fun game.

There was something about using a real exercise bike that made both games work though. The feedback you got from the pedal tension increasing when you rode/drove up a Mode-7 hill and then decreased when you were coming back down the other side was really compelling. You were getting a workout, and actually having fun while doing it. The interaction with the bike pulled something out of each game and made up for all of the weird deficiencies and idiosyncrasies.

That said, the bikes, to my knowledge, never got sold. Spin classes weren’t really a thing yet, and there just wasn’t a market in 1995 for a $1000+ (in 1995 dollars no less) fitness bike that hooked up to a kids computer game system. On the flip side, if you can get your hands on one now, the bike/game is so extremely rare that it’s now (as of 2018) the fourth most expensive Super Nintendo game out there!

Tetromino Hard Drops

Fri, 19 Oct 2018 01:01:22 +0000

A friend of mine (and former co-worker) had been bugging me about adding hard drops into Tetrominosince soft drops are kind of annoying, particularly when you’re at lower levels. I had actually been really reluctant to do that, because I was trying to base the game play as closely as possible on NES Tetris, but with the skin of the original Tetris. It turns out that the original Electronika 60 (nee PDP/11) version of Tetris did have hard drops, so I have finally relented.

My 10yo daughter is somewhat responsible for this. She pulled out the two Nintendo DS’s that my wife and I used to play on all the time, but which have sat in the cupboard for the better part of a decade. After changing out the batteries with some cheap Chinese knock-offs from Amazon, they’re both working great and I got some quality DS and GBA Tetris time. I realized pretty quickly how much it sucks not having hard drops.

If you want to check it out, try running docker run -it --rm pdevine/tetromino. If you have an older version, you’ll probably have to re-pull the image. Alternatively, just check out the video below.

A Blast from the Past

Sun, 30 Sep 2018 06:01:41 +0000

Way back in 2006 when the Iraq War was in full swing and Arrested Development was finishing up its first run, I made an agreement with my (at the time) girlfriend (now wife) that if I got rid of a bunch of my old PCs, I could get a brand spanking, shiny, new Mac Pro. The first of the new aluminum ones which was rocking four cores at 2.66 GHz, a big pile of RAM, and even a SATA hard drive.

That was twelve years ago, and although I had upgraded the machine before, it was looking a little long in the tooth. It was clear that I was going to need to do some major surgery on this beast if I was going to keep it going in the future.

A glance on Ebay shows they now sell for somewhere in the $80 to $150 range, which is a long way from the $2500+ (about $3100 in 2018 dollars) that I paid for it. This is for one of the first 64 bit machines in one of the nicest looking chassis of any system that I’ve ever owned. So what can we do with this machine?

The first time I upgraded it, in 2009, was to replace the original terrible nVidia card that was causing a lot of issues. I ended up getting an ATI Radeon HD 4870 with 512 MB which seemed to work well. In 2011, I’d upgraded the RAM to 16 GB, and added a stonking fast (for the time) 240 GB SSD drive. The onboard SATA controller is only SATA-2, which is great for spinning disks, but ideally one would want it to be faster for SSD. This particular SSD drive was only SATA-2, so that wasn’t really a problem. Somewhere along the way I also shoved in a couple of 750 GB SATA drives which I was holding copies of all of our home movies and photos.

So what could we upgrade in 2018?

On the operating system side, the machine was stuck at OSX Lion because Apple had decided to stop supporting the first gen Mac Pros in 2012. This had to do with the original EFI bootloader being 32 bit instead of 64 bit. There wasn’t really anything wrong with the bootloader though, and along the way the community had managed to patch it to make it still work. That meant I could upgrade all the way to OSX El Capitan.

Upgrading it, however, ended up being quite the chore. The 240 GB drive had quite a bit of data on it, and it required putting a copy of El Capitan on the disk before running through the hacky upgrade process. Somewhere along the way it ended up running at of disk space, but because of the hacked bootloader, it didn’t want to boot back into Lion. I ended up putting the machine in Target Disk mode and connected to it through my old MacBook Pro on the Firewire 800 port, but that took a couple of days due to having to order a new cable from Amazon. With some space freed up, I was able finish the installation and El Capitan works like a charm.

Unfortunately with macOS Sierra, High Sierra, and Mojave, Apple has yet again abandoned its old machines and now requires an extension to the Intel chipset called SSE 4.1. This is basically several new instruction codes that make things like floating point work more quickly. Unfortunately older processors, like the ones in the once swanky 2.66 GHz quad core Xeon CPUs, weren’t going to cut the mustard anymore. But no matter, is it even possible to upgrade the processors?

The answer is yes, you can upgrade them, but unfortunately not to the Penryn architecture which included the SSE 4.1 instructions. Even though there are processors that will fit into the same CPU sockets (LGA 771), they aren’t compatible. A pair of quad-core 3.0 GHz Clovertown Xeon processors can be had for $50 on Ebay though, so might as well upgrade to those (twice the cores!). To get them to work, you have to flash the firmware to make the machine think that it’s a Mac Pro 2,1 system, but flashing the firmware is pretty straight forward.

Changing the CPUs, on the other hand, was probably the most difficult part of the rebuild. This requires removing the memory bay as well as the CPU shroud. This isn’t as easy as it sounds because the memory bay has these extremely annoying little plastic clips which require you to depress all four of them simultaneously and work them out. Getting the CPU heatsinks off also turned into a bit of a chore. They use hex screws to hold them on, and unfortunately one of the 8 screws that hold the two heatsinks in place is extremely awkward to get at and requires a really long screwdriver. I ended up going over to Fry’s Electronics to buy a new screwdriver to get the last screw out, but unfortunately it didn’t fit either. In the end, it turned out that I had a really long Torx screwdriver lying around, and apparently you can use Torx bits with hex bolts, which I hadn’t realized before.

The Fry’s trip wasn’t a complete waste, because I also picked up some CPU thermal paste. With the heatsinks off and cleaned with some rubbing alcohol, it was time to put on some new thermal paste. Spreading the thermal paste is probably the most finicky part of the upgrade, but if you’ve done it a few times, it’s really not that difficult. Changing most other things on this machine, unlike in newer Apple systems, is really easy. Adding a new drive into one of the four exposed drive bays is even trivially easy. Just pop the sled out, connect a 3.5” SATA drive into the sled, and push it back in. You’ll need an adapter if you’re going to slot in a 2.5” drive, which is what I had originally done when I added the SSD drive. One really nice thing about the original motherboard, is that it also included extra SATA connectors for hooking up additional drives in the 5.25” bays. The system originally shipped with an Apple Super Drive (which would read and write both CDs and DVDs), and included room for a second drive in the bay, although it never was populated. The Super Drive uses IDE, so there is also a connector for it on the motherboard.

I actually have stacks of old 4TB Western Digital Red drives that are lying around from my old startup, so I figured why not upgrade the old 750 GB drives, and move the 240 GB SSD drive into the 5.25” bay. I had some old 3.5” to 5.25” converters lying around from the 4U chassis build, so I attached a pair of those to the 3.5” to 2.5” converter that the SSD drive was already sitting in. It’s not super pretty, but you can’t really even see the 5.25” bay once everything is put back together. The trickiest bit here was accessing the extra SATA-2 jacks and threading through the SATA cable. This requires removing the front fan assembly from the machine which first requires removing the plastic shroud over the CPUs as well as partially removing the memory bay. Since I had already done this to replace the CPUs it was easier to do both at the same time. The fan assembly is also really wedged into the system and required a bit of elbow grease and determination to get out, but once it is out, it’s pretty simple to thread the SATA cable from the 5.25” bay and connect it to the extra jacks.

For the 4x4TB drives, I originally figured I could run it in software RAID-5 (just like you can in Linux), however I discovered after going through all this work that macOS doesn’t actually support software RAID-5. This has left me in a bit of a quandary. You can purchase SoftRAID from One World Computing which does support it without having to add any more hardware, or, I could just buy a hardware based RAID controller. So far I haven’t decided which way to go, however I’m leaning toward the hardware solution since in theory it might actually be a little cheaper than shelling out full retail price for SoftRAID. Apple did produce a Mac Pro RAID card back in the day, so it might be worth picking one of these up second hand, but I don’t know if they are any good or not.

Much like swapping out the drives in the four main bays, upgrading the system memory is also trivially easy to do.It doesn’t require anything like removing the memory bay itself. You just pull out the memory boards (there are two of them), push in the new DIMMs, and power the machine on again. The boards themselves have LEDs on them which will signal if there is a memory parity error or not. Originally Apple claimed that you could only put 16GB of RAM in the machine, but in reality it can easily address up to 32GB of RAM, and if you upgrade past OSX Lion, you can apparently even get 64GB of RAM to work in some configurations. I already had 16GB of RAM, so figured upgrading to 32GB of RAM would be sufficient. This actually ended up being the most expensive part of the upgrade, and the chips that I bought from Other World Computing were not exactly matched to each other, and I was initially a little worried because the parity error LED lit up. I ended up reseating the chips and the error went away, and so far it looks like the everything is behaving correctly.

One part of the upgrade did not go so well though. I ordered an ATI Radeon 5770 GPU off of Ebay to replace the older Radeon 4870, but the card unfortunately didn’t work and the machine wouldn’t boot when I powered it up. I also accidentally knocked off one of the little clips on the end of PCI-e slots that holds the card in place. This isn’t really a huge deal as the GPU still fits in and is pretty snug. I’m not going to use this machine for any heavy gaming, so I think I’ll just stick with the Radeon 4870 for now.

So was all this work worth it? It was really fun rebuilding the machine, but there are a few things that I’m not super thrilled about though, like not being able to get it to work with Sierra, High Sierra, or Mojave. The Xeon CPUs are also too old to support Extended Page Tables which means that it can’t run Docker Desktop, so I’ll have to see how well the newer versions of VMware Fusion work to be able to use Docker in a Linux VM. That said, I spent less than $200 on the upgrade, although I did have a lot of the parts on hand. Even without going all out like I did on some things like the 16TB of drives, and only putting 16 GB of RAM in it instead of 32 GB, you could build one of these machines for around $250, which is pretty decent if you just want a Mac to futz around with. For a little extra money though you could get a Mac Pro 4,1 is also very upgradeable, and can work with much more modern software. If you want a cheap Mac, that would be my recommendation.

I m g 20180908 112456

I m g 20180908 112619 1

I m g 20180908 115644

I m g 20180908 122917 1

I m g 20180908 123200 1

I m g 20180910 192417

I m g 20180910 192428

I m g 20180915 173019 1

I m g 20180915 173109 1

I m g 20180915 173528 1

I m g 20180915 214424 1

I m g 20180916 105101

I m g 20180916 105116 1

I m g 20180916 114023 1

I m g 20180916 114744 1

I m g 20180916 145648

I m g 20180916 145655

I m g 20180916 151718

I m g 20180916 152633

I m g 20180916 163755

Tetromino Elektronika

Wed, 04 Jul 2018 00:53:11 +0000

TL;DR - A falling tetromino block game based upon NES Tetris in the style of the original 1984 Tetris by Alexey Pajitnov

Try it out with the command: $ docker run -it --rm pdevine/tetromino

It should look something like:

This game was written because of an offhand remark my boss made to me in 1994. I had just finished up two years of college in Vancouver, British Columbia when I had taken a job at Radical Entertainment to write Super Nintendo games. I wasn’t a particularly great programmer at the time (which is arguably still true), however, I found writing games in 65816 Assembly (the language of choice for the Super Nintendo) extremely daunting. There was sparse documentation, few code samples, and the assembly that I had learned in college was fairly different than the code I was writing at Radical. That coupled with a slow build process on an Intel 486DX2-66 computer, and an even slower data transfer of the object code across a parallel port onto our homebuilt development kits, meant a lot of 80 hour work weeks.

I remember mentioning to my boss, Jack Rebbetoy, about how difficult I was finding things, and he made a remark about how we had engineers on-staff who could knock out a copy of Tetris in a weekend. This really blew my mind. We did have a lot of really talented engineers at Radical, but at the time I was having a hard time even getting a sprite to render correctly on the screen.

Fast forward 24 years, and I’m now working as a Product Manager at Docker. I still love writing games, but haven’t done it professionally since that job at Radical. The problem with using Docker to run and distribute games though, is that there isn’t an easy to use video subsystem. You can use it with X11 and using a remote display, or just passing in the X socket, but that’s not exactly easy to do. So given the limitation to really only use ASCII text, can you still create meaningful games and applications?

I ended up putting together a very rudimentary ASCII sprite library (github.com/pdevine/go-asciisprite), written in Go. Go was chosen since I wanted the compiled binary to be really small. I think the entire Tetromino docker image weighs in at about 2.8MB, which is shameful by 1994 standards (two whole floppy disks!), but pretty tiny in 2018. After putting together the sprite library, the choice for the first game to write seemed pretty natural; Could I actually write Tetris in a weekend?

The answer is a pretty definitive “yes”. Even though I’ve spent more than a weekend writing Tetromino Elektronika, the core of the game was written in about that amount of time, mostly spent riding Caltrain up and down the peninsula in the SF Bay Area. I tried to keep things as close as possible in terms of timings with the original Classic NES Tetris game (sorry, no hard drops!), and tried to make the look-and-feel similar to the original Tetris from 1984 on the old Soviet based Electronika 60.

So with that, hopefully you enjoy the game, and maybe even get inspired to create something with go-asciisprite. Code is available at

github.com/pdevine/tetromino

More Ascii Sprites

Sat, 14 Apr 2018 19:08:06 +0000

A while back I posted about the pdevine/whale-test container you can pull from Docker Hub. When I wrote that I had been trying to write a sprite library in golang which was based upon the gizak/termuilibrary, but I kept running into weird issues with the way that it was buffering data before flushing it out. I guess that’s the problem with trying to graft something like a sprite library on top of a library designed for creating text widgets.

As time went by the whale-test code kind of bit rotted, and it looks like termui isn’t being supported anymore at all. Most of the posts in the github repo are asking if the project is dead, and none of the pull requests have been added in years (actually, one was, and that now causes a panic). So, instead of using termui, I decided to just use nfs/termbox-go directly, which is the low level library that termui is based upon.

If you do a docker run -it --rm pdevine/whale-test, it’s now written with the newly revamped, nascent sprite library called pdevine/go-asciisprite. I’ve also added a new demo which you can run with docker run -it --rm pdevine/pants which is very trouser related. I work for Docker after all.

All source code can be found at http://github.com/pdevine/go-asciisprite

Solving for E and O

Thu, 01 Feb 2018 05:47:22 +0000

I was helping my 10 year old out with her math homework this evening, and she had been given an interesting series of problems which made you find the difference between two numbers. Pretty straight forward stuff, except that they specified that each individual digit in each of the three separate numbers (the minuend, the subtrahend, and the difference) should be either even or odd.

This is a lot more challenging!

There was one particular question which everyone was struggling to find an answer for which looked something like this:

EEOE - OOEE = EEEE

‘E’ being an even digit, and ‘O’ being an odd digit. It turns out there are actually 9,000 answers to that, but how do you prove it? Like any normal (former) software engineer, I decided to just brute force it and hack together a solver. You can save this and run it by passing it three strings of E’s and O’s.

import sys
import itertools

odd = [1, 3, 5, 7, 9]
even = [0, 2, 4, 6, 8]

def compute(minuend, subtrahend, difference):
    def get_nums(template):
        nums = list()
        factor = 1
        for x in reversed(template):
            if x.lower() == 'o':
                nums.append([n*factor for n in odd])
            if x.lower() == 'e':
                nums.append([n*factor for n in even])
            factor \*= 10
        return [sum(n) for n in itertools.product(*nums)
            if len(template) == len('%d' % sum(n))]

     num_a = get_nums(minuend)
     num_b = get_nums(subtrahend)

     nums = list()
     for x in num_a:
         for y in num_b:
             if x < y:
                 continue
             diff = x - y
             if len('%d' % diff) != len(difference):
                 continue

             found = True
             for c in difference:
                 if c == 'o' and diff % 2 == 0:
                     found = False
                     break
                 elif c == 'e' and diff % 2 == 1:
                     found = False
                     break
                 diff /= 10

             if found:
                 nums.append([x, y, x - y])

    for n in nums:
        print "%d - %d = %d" % (n[0], n[1], n[2])

if __name__ == '__main__':
    compute(sys.argv[1], sys.argv[2], sys.argv[3])

2017 Aerospace Re-cap

Thu, 04 Jan 2018 17:24:08 +0000

2017 was a relatively boring year for space

SpaceX stole the show this past year, by launching an unprecedented 18 rockets. The last one was fittingly with another ten Iridium NEXT satellites, for a total of 40 Iridium satellites in 2017. They also had a couple of other firsts, including launching the first “flight proven” first stage for the SES-10 mission last March, as well as sending a reused Dragon and first stage to the International Space Station in December. The last Iridium flight also reused a first stage booster, so in total they reused five first stage boosters. What’s remarkable is how routine they’re making recovery and reuse look, and it’s going to go a long way into making space affordable.

With all of those successes, however, there still were a few misses. Neither the Falcon Heavy, nor the un-crewed Dragon 2 ended up flying this year. Elon is quoted saying that they didn’t realize how difficult it would be to get the Falcon Heavy to work, and that ”strapping three Falcon 9 cores together is tougher than it sounds”. Who would have thought that rocket science would be difficult? The Dragon 2 also didn’t take flight this year, however, there’s plenty of action for it, along with the Falcon Heavy, on the launch manifest for 2018.

2017 also came and went without any paying passengers riding on a Virgin Galactic or a Blue Origin sub-orbital flight. Blue Origin got a little closer to that goal, doing a flight in December (just a week or two ago) which made it all the way up to 322,000 feet. They also did take a passenger up on the flight, named “Mannequin Skywalker”, who seemed pretty stoic about the entire affair.

I’m going to notch this prediction up as correct, but given the glacial pace of aerospace in general, it wasn’t that far of a stretch.

There were a lot fewer 747s, A340s and A380s

The trend to get rid of airplanes with greater than two engines continued through 2017, and as I’m writing this, there aren’t any left flying for any major airlines in North America. There are a few holdouts flying in Europe (I did end up flying an SAS A340 earlier last year to Copenhagen), and the Asian super carriers still have plenty, but most of these planes are being replaced by 777s, 787s, A330s, and A350s. Airbus said that if a new order doesn’t come in from Emirates that they’re going to cancel the A380 altogether. The old Hub-and-Spoke airline model is pretty much dead at this point, and thank goodness. Being able to fly smaller planes, more frequently, direct to your destination is always going to be more preferable than being stuck in a terminal waiting for a connecting flight.

… and there also were a lot more narrow bodies

It was a mixed year for the beleaguered Bombardier CSeries. Delta gave it its vote of confidence with an order for 75 planes, and options for 50 more. After which, Boeing turned around and had the US Commerce Department slap a 300% fine on the plane accusing Bombardier of dumping planes below cost on the US market. After which, Bombardier gave away 50.1% of the CSeries program to Airbus which is going to turn around and build planes in Alabama. Oh, and the US International Trade Commission may drop the tariff after all because Bombardier built a total of 17 planes last year (off of their 22 plane target), and hey, it’s now like Boeing isn’t heavily subsidized, right? Phew.

Meanwhile, Boeing keeps racking up more orders for the 737 MAX (another 175 are slated to go to flydubai. Southwest just started taking deliveries of its 737 MAX planes (it was the US launch partner) and Boeing delivered a total of 49 planes this year. Airbus delivered 134 A320s after delivering 68 in 2016.

Orders seem to be slowing down though, with the 737 MAX topping out at 420 new orders this last year, compared to 540 orders the year before. Airbus fared even worse in this regard with only 185 orders this year vs. 711 orders in 2016. Both programs have huge backlogs though, with Boeing needing to deliver 4,016 planes and Airbus backlogged a whopping 5,022.

So I’m not certain if I can fully claim this one. There definitely were more narrow bodied planes this year, but I’m baffled why Boeing decided to try and squeeze Bombardier. I guess they didn’t predict that Bombardier would counter with giving away half of the CSeries program. I’m not sure anyone could have predicted that.

The Everdrive N8

Sat, 15 Jul 2017 17:16:48 +0000

I’ve recently found myself hooked on watching the Classic World Tetris Championships on YouTube. Here’s the CTWC 2016 Final between Jonas and Jeff. It’s completely mesmerizing, and it’s amazing to me that anyone can play Tetris consistently at that level.

The Championship is played on an NES, and it inspired me to dust off my old consoles and bust out a copy of Tetris. Sure, I could run it in OpenEMU, which is a fantastic piece of software, but there’s something to be said about running on a real console. The problem that I found though was two of my NES consoles didn’t want to boot (yay red blinky light) with either of the copies of Tetris that I have. I did finally manage to get one to boot on my Retro Duo, but it got me thinking that I wanted something better than to have to keep cleaning out all of my old cartridges.

One solution would have been to just break down and buy one of those bootleg multicarts that you see all over Amazon and Ebay. At least I wouldn’t have to keep switching out the cartridge every time I wanted to play another game. The problem from what I’ve read though is that many of them are of dubious quality, and ethically they seem even more dubious. I suppose I could also have bought one of the new NES Classic Editions, but they don’t even come with Tetris!

Instead, I opted for the Everdrive N8, which is essentially a Nintendo cartridge with an SD card reader on it that can play ROMs directly off of an SD card. It also adds some neat features like being able to save your games, and allows you to use Game Genie cheats. It’s essentially like OpenEMU except that you get to play on real hardware. Pretty slick!

Getting the Everdrive N8 to work was, admittedly, a bit of a challenge. The first problem is that it doesn’t ship with any kind of instructions. You need to prepare an SD card (one between 4GB and 32GB apparently. I had an extra 8GB SD card lying around), and it has to be formatted with FAT/FAT16/FAT32 with a copy of the Everdrive N8 OS unpacked on it.

For whatever reason the SD card reader on my ancient Macbook Pro didn’t initially want to let me write to the card. It turns out you need to set the silly plastic lock about halfway between locked and unlocked positions (!) in order to get diskutil to work correctly. I managed to repartition and format the SD card after a few dozen attempts, and found v16 of the NesOS on the Interwebs. My first attempt at loading the N8 on the Retro Duo ended up with some strange behaviour with it ultimately hanging on a screen saying “OS init”, but after reformatting the SD card yet again everything worked.

Using the N8 is pretty straight forward, but the various buttons used for selecting things is counter intuitive. Once you do select a ROM, load times with my SD card were blisteringly fast. It really only takes a half second or so for a game to load up. NES and Famicom ROMs are pretty tiny, so my 8GB SD card can easily hold my entire collection of games, plus probably every game ever created for the NES ever. I tried out about a dozen or so and all of them loaded without any issues. That’s pretty amazing given the flakiness that the NES can have with dirty cartridge contacts.

So why use the N8 over the new NES Classic Edition? Well, if you can manage to get a Classic Edition, it suffers from some real problems. The controller cables are short and you’re locked into the 30 pack-in games. The most egregious thing for me though is that it runs an emulator instead of running the ROMs on real hardware. For that, you either need one of those janky knockoff multi-carts, or something slick like the Everdrive N8.

Dwarf Fortress

Fri, 24 Mar 2017 17:25:31 +0000

TIL the world’s most complex (text) game Dwarf Fortress requires OpenGL and ALSA. I guess there’s not as much point to stuffing it into a docker image, although when has that stopped me before?

Aerospace Predictions for 2017

Tue, 17 Jan 2017 00:17:01 +0000

If there’s one thing I’m even less qualified to predict than my automotive predictions, it has to be on what’s happening in the aerospace industry. But, that didn’t stop me before, so why stop now?

2017 is going to be a relatively boring year for space

I say this right after SpaceX has just launched for the first time since the September _anomaly_ and there are now 10 Iridium Next satellites in orbit. Both the launch and landing on “Just Read the Instructions” went off without a hitch. SpaceX is definitely going to steal the show this year, and _hopefully_ they’ll be able to resume, and even accelerate, their torrid pace of launches. The most exciting launch will be the oft delayed Falcon Heavy demonstration flight from the old Space Shuttle and Saturn V launch pad at the Kennedy Space Center, followed by the first “Crew Dragon” demonstration late this summer.

Neither the “Crew Dragon” or Boeing’s CST-100 spacecraft are going to carry passengers this year though.

For that we’ve got to wait until 2018. In fact, 2018 looks pretty promising for aerospace in general, with the James Webb and the wheel-less Mars InSight lander.

The big question marks are Virgin Galactic and Blue Origin. Virgin Galactic has resumed testing of SpaceShip 2, but I’m guessing that 2017 is going to come and go before any paying passengers get to ride on it. As for Blue Origin? Who knows. They probably have the best shot of launching passengers on sub-orbital flights this year, but Jeff Bezos is so tight lipped that it’s difficult to say with any real confidence.

We’ll see a lot fewer 747s, A340s and A380s

Over in the aeronautical side of the things, airlines will continue to downsize from their super-jumbos in favour of 777s, 787s, A330s and A350s. There’s pretty much no reason to fly with more than two engines at this point, and three engine jumbos like the MD-11 and L-1011 are already ancient history. That leaves Boeing with the “Queen of the Skies” and Airbus stuck with the A340 and the A380. Even with fuel prices staying at relatively low levels, it still makes more economic sense to fly longer haul routes with smaller planes than it does to fly a plane with twice as many engines. That includes the price of additional crew, which is still cheaper than all of the fuel which gets consumed.

United already said it’d accelerate phasing out of their 747s this year, and Delta has said the same thing. That means there won’t be a single US carrier flying 747s by the end 2017. Meanwhile Airbus is souring on the A380neo (new engine option) which means there aren’t going to be a lot of fuel savings any time soon.

… and a lot more narrow bodies

For smaller planes, it’s still unclear whether the Bombardier CSeries will take off. Embraer is taking their fight against Bombardier to the WTO, however it’s not like Bombardier is making any money at this point. The real turning point could be when Delta starts to fly the CSeries and if its cost competitive and reliable. That’s not expected until spring of 2018, so it could be another long year for Bombardier.

The 737 MAX on the other hand will see its first delivery this year, and presumably we’ll start seeing them in the US shortly afterward, starting with Southwest. The MAX isn’t a new airframe though, and it just replaces the engines to be something around 14% more efficient. That’s not much different than the A320neo which Airbus introduced last year.

Automotive Predictions for 2017

Sat, 07 Jan 2017 18:57:18 +0000

I’m probably the least qualified person to comment on the automotive industry. As a family, we only have one car, and we’re more likely to ride a bicycle or take a train than drive the car. That said, we live in the Silicon Valley, and it seems like there has been a tectonic shift in the automotive zeitgeist from Detroit to Northern California.

Driverless cars will not quite be ready for prime-time.

I see Waymo (Google) cars on the street in Palo Alto pretty much on a daily basis at this point. I saw the Uber cars in SF, and even the Otto semi-truck driving around. Tesla is releasing a highly anticipated upgrade to Autopilot, and it seems like every day another auto manufacturer announces that they’re working on driverless cars. So, we’re getting close. Really close. But I don’t think we’re quite there yet.

Even if Tesla completely nails Autopilot, there just aren’t enough Tesla’s on the road to bring driverless cars into the mainstream. When I travel to places which are not the Silicon Valley everyone thinks driverless cars are a pipe dream. And they have a point. How do you make a car drive down a freeway in Alberta with drifting snow where you can barely see the lane markers? I’m sure we’ll get there, but it’s a lot easier making your car work in the Bay Area where we barely even have “weather” than it is in a place with monsoons, dense fog, or real winter driving conditions.

Electric cars will still not take off, but probably will in 2018.

Speaking of Tesla, electric cars will still be niche in 2017. The Tesla Model 3 is slated to be released late in the year, but given that Elon tends to be wildly optimistic with his release dates, I would be surprised if it came before 2018. And really, Tesla is the only contender at this point. There are a number of car companies that have electric cars (like BMW, GM, Ford, Fiat, VW, Nissan), but only Tesla has cracked the nut of making a cool looking, high performing car without the range anxiety. If you can drive 4.5 hours at 70-75 mph you probably don’t care that it takes 45 minutes at a Super Charger to fill your car up to 80% capacity.

The other contenders just don’t have enough cool electric vehicles in the pipeline to make any kind of meaningful impact on the market this year. They all know with ever increasing fuel efficiency standards that they need to get into the electric car business, but it takes time to develop the technology and build out a charging station network. The Chevy Bolt may do well, but cheap fuel prices will hinder adoption in places which are not the Silicon Valley. Oh, and just about everyone would rather have a Tesla Model 3 than a Bolt.

Hydrogen Fuel Cell cars will continue to stagnate.

Toyota and Honda seem to be doubling down on fuel cells, and if you’re a hydrocarbon manufacturer like Saudi Aramco you’re probably rooting for them to succeed. But fuel cell cars aren’t going anywhere fast. Refining oil is a dirty business, even if you get “free” hydrogen as a byproduct, and driving trucks of hydrogen around to a non-existent network of hydrogen “Gas” stations just isn’t going to happen any time soon, if ever. On top of that, both companies seem to be taking their styling cues from Michael Bay, and I’m not sure most people, at least in North America, want to be driving in a car that looks like a Transformer. As such, the Honda Clarity and the Toyota Mirai aren’t going to sell particularly well.

Uber and Lyft will continue to put a lot of pressure on the Taxi Industry, but will be more expensive.

Both Uber and Lyft need to start making money. Pretty much no one wants to ride in a taxi at this point, and it feels like the only way taxi cab companies are going to survive is through legislation to preserve their monopolies and keep Uber and Lyft out of their markets. The strategy for 2017 will be to continue to bunker down and hope that neither Uber or Lyft starts to make a profit. This is not a winning strategy.

There are some other head winds that ride hailing services are going to face in 2017. Cities are starting to realize that their traditional transit services like busses are being eroded, and traffic caused by all of the additional cars is going to continue to fill downtown cores. Expect cities like NY and SF to start imposing ride hailing surcharges which go towards paying for more mass transit.

On Docker Storage and NFS

Tue, 18 Oct 2016 02:09:43 +0000

We’re just about to release a new version of Docker Trusted Registry (DTR) at work, and I thought I’d highlight one of the cool features that we did for NFS storage.

In the past, we told people who wanted to use NFS to bind mount their NFS directories directly into a directory which was storing their registry volume. This is really a pain to configure, particularly if you want to automate the process. It requires creating the volume, inspecting it, mounting the NFS directory and then bind mounting that directory in the freshly inspected volume. Oh, and also you need to make certain the NFS client options are correct if you want it to work in your HA cluster. This is less than ideal.

With DTR 2.1, you can just specify a new flag during install or reconfigure called --nfs-storage-url which will automatically take care of everything for you. The best part is when you join a new DTR replica to the cluster, the new node will automatically get the NFS mount point and everything will just work like magic.

So how does this work?

We’re taking use of a little known feature in the local storage volume plugin which allows you to back docker volumes with NFS. Instead of the docker volume being backed with a local file system, the docker daemon instead mounts the configured NFS directory when the volume is attached to a container and that container starts up.

Let’s say we’ve created an NFS server called nfs-srvr and we’ve exported the directory “/exports”. We’ll use this exported directory to back a new volume on a different docker host which we’ll call dckr1. First, on our dckr1 host, let’s make certain that we can access the exported directory

root@dckr1:~# showmount -e nfs-srvr
Export list for nfs-srvr:
/exports \*

Alright, that looks good, so let’s go ahead and create a new volume which is backed by the “/exports” directory.

root@dckr1:~# docker volume create --driver local --opt type=nfs \\
--opt o=addr=<IP ADDRESS>,rw --opt device=:/exports --name=test-vol

And finally, let’s attach it to a new container running alpine

root@dckr1:~# docker run -it -v test-vol:/storage --rm alpine:3.4 sh
/ # ls /storage
...

And that’s pretty much it. There are a couple of caveats to using it right now. The first is that there is (as of Docker 1.12) a bug which you can’t use the host address of the NFS server, and instead have to use the IP address. Also, make certain that you have the nfs client kernel library installed on the docker host (usually packaged in nfs-common or nfs-utils), otherwise your container will zombie when it starts because it won’t be able to back the volume correctly.

Party Parrot Time!!

Wed, 12 Oct 2016 22:41:08 +0000

$ docker run -it --rm pdevine/partyparrot

Need I say more?

SpaceX's Interplanetary Transport System

Tue, 04 Oct 2016 16:06:56 +0000

OK, I think I sorta called this one a couple of years ago when SpaceX was experimenting with their Grasshopper demonstrator.

I still don’t know how difficult it will be to refine methane on Mars, but at least Elon has a somewhat reasonable answer. Here’s my post from Hacker News:

That said, the thing I’m struggling with the most right now is some kind of economic reason for why you would build the ITS. From an “awesome” factor, I’m all on board, but Mars doesn’t really have anything that we don’t have on Earth (that we know of). In terms of how to finance this beast though, I feel like maybe setting sight on something closer to home like orbiting space hotels or trips to the moon make a lot more economic sense. With something closer to home you can have a lot of efficiency gains in terms of reusability which you don’t get by only having your ship work every two years.

ASCII Sprites with Golang

Mon, 29 Aug 2016 04:41:27 +0000

I’ve been trying to think of a way to spruce up the Docker container which we print on all of the t-shirts for Dockercon. At Dockercon 16 in Seattle we told everyone to run this command:

$ docker run -it --rm hello-seattle

That displays some data about how containers work from a really superficial level. Somewhat useful for novices, I guess, but it doesn’t exactly have a ton of pizzazz.

When I first started working at Docker a year and half ago, I converted some of my ancient Pyweek video games which I’ve written to run from Docker containers. These tend to be a lot more interesting, but getting them running from the command line can be a huge pain in the neck. You have to pass in the X11 socket, give it access to the sound card and do all sorts of nastiness which I wouldn’t want to print on the back of a t-shirt. This would be fine if the docker engine had support for showing graphical style applications, but for now, I’ve ruled it out for a quick and dirty demo.

Instead, I’ve been thinking about the limitations of ASCII and have been wondering if there were a way to use it with typical gaming primitives like sprites. I wanted to keep the size of the container image really small, so that ruled out writing anything in python where I’d have to package up the interpreter. Instead I went with a statically compiled golang application which meant it could live by itself inside of a container.

I’m not actually finished yet, but I managed to cobble together a bit of a demo. It just bounces around a bunch of ASCII Docker whales, but even that is a step up from the old container. Here’s a screenshot:

You can give it a try with this command:

$ docker run -it --rm pdevine/whale-test

Use ‘a’ to add more whales, and ‘z’ to take them away. ‘q’ will quit out of the program. Oh yeah, and I guess it goes without saying that you need Docker to run it. I would imagine it works just fine with Docker for Mac and Docker for Windows, but I haven’t tried them out.

Anyway, it’s a start. I did run in to some interesting problems writing it which maybe I’ll write up in a future post.

The Rift Has Arrived

Sat, 30 Jul 2016 17:46:11 +0000

It’s here! It only took six (!) months to get here, but the Rift has finally arrived. There was a little bit of a snafu in delivering it (I had to be at home to sign), but after waiting so long, what’s an extra weekend? Initial impressions:

Nothing else comes close to how immersive this thing can be. You have to really try it to appreciate it. I know everyone says that, but it’s the truth. You literally try touching stuff in mid-air and try ducking out of the way of things.
My friend Chris’s game Dead Secret is truly terrifying. I’m not sure if this is just me due to not having really played suspense/horror games before, but it’s hard for me to not want to rip the Rift off of my head while playing it.
Oculus gets most things right in terms of quality. The packaging was great, although things slipped in shipping. I can’t seem to get the wireless Xbox One controller to work wirelessly, but I don’t know if that’s because I already have a wired Xbox 360 controller and an old Logitech Dual-Shock controller. I ended up using my wired Xbox 360 which works great.
EVE: Valkyrie makes me want to barf. Literally. I assumed that since I almost never get motion sickness, and that I can actually fly a plane for realsies, I wouldn’t have any problems playing this game. Which is sorta true, except I then went to bed and woke up a few hours later wanting to hurl.
Oculus makes pretty games that aren’t particularly fun to play. Lucky’s tale looks amazeballs, but it’s nowhere near as fun as something like Super Mario 3D World. Same goes for Farlands, which is also gorgeous but I just couldn’t get into the game and the voice acting was driving me crazy.
Steam’s VR experience is kind of terrible, at least with a Rift. At first it was confused and thought I had an HTC Vive, and now it just shows me in a big open place with no menus in it. I finally, sorta, got DCS World to work through it, but I don’t know how to make input work, so I can’t get past the first screen.

I m g 20160718 191956611

I m g 20160718 192015981

I m g 20160718 192207611

The Video Card Has Arrived

Sat, 09 Jul 2016 17:56:22 +0000

After attempting to get an nVidia GTX 1080 for a few months, I realized it probably wasn’t worth the extra $250, so opted for a GTX 1070 instead. What’s been crazy about this rollout of GPUs is that it’s still next to impossible to get any of the cards, and the places that do have them in stock are mostly marking them up 80-100%. Oh, and in the interim to fight off new, cheaper Radeon RX 480 cards from AMD (at half the performance), nVidia just released a paired down GTX 1060 for $250.

That said, the GTX 1070 should be fine for driving the Oculus Rift… when it finally gets here. The shipping window for the Rift has come and gone and no sign of the thing is in sight, despite having pre-ordered at the beginning of January. Apparently you can just go buy one at Best Buy though, which just seems wrong. Good job, Facebook.

In terms of performance, it seems like you can crank all of the settings up in any game and play at 1920x1200 and they work great. The problem here being that I don’t really have any of the new block buster triple-A titles which can really take advantage of the GPU. I fired up Portal 2 and Rocket League and both were great at max settings. The problem though is that both ran fine on the old GTX 660 (although not maxed out), and Portal 2 came out more than five years ago. I also tried out Elite Dangerous, which I’m told is really awesome on the Rift. At high settings the GTX 1070 didn’t seem to even break a sweat. I’ll have to do some more bench marking and get concrete numbers.

I m g 20160708 165059515 1

I m g 20160708 165443956

I m g 20160708 170154228

I m g 20160718 191820842

I Love It When a Plan Comes Together

Sun, 26 Jun 2016 04:28:43 +0000

One of the last things to do to get this 4U beast finished was to figure out how to get some of the data off of the old Ultra-wide SCSI drives. Originally the machine was used to house four separate 9GB virtual machines, and the whole thing was stuffed into a co-lo down on San Antonio road in Palo Alto back in the early 2000s. I’d used one of the VMs as a general purpose Linux host which did double duty as a web server and an email server and the other ones were used by friends for pretty much the same purpose. I really wanted to get some of the emails back though because I’d lost touch with a friend in Japan, and knew I had his email and snail mail addresses buried somewhere on one of the drives.

There were, however, several problems I needed to tackle to get the data back. Not only did I not have a way of getting the Ultra SCSI drives to connect to anything since the old motherboard was dead, I also needed to figure out how to read the file system, since they were partitioned as VMFS2 although the virtual partitions were primarily ext2.

The SCSI problem I solved by buying a cheapo $45 LSI Logic card on Amazon Marketplace which was being sold as a tape backup adapter. It looks like someone just plucked it out of an old HP machine, but it was cheap and did the trick. Four of the five drives spun up just fine, although the years haven’t been particularly kind to them as the whine from them was pretty much unbearable. I can only imagine how loud they would have been had I had left the old fans in as well.

To get the data off, I just “dd’d” each of the drives into files on the SSD drive, since I figure I’ll never use them again (also, does anyone need a slightly used LSI Logic Ultrawide SCSI card?). I can attach the files as loopback devices in linux, however I still don’t have anything which will directly mount the VMFS2 partition. I’m fairly certain ESXi can auto-convert from VMFS2 to VMFS3, but I’m not sure how I’d do that since I have no idea how to loopback mount each of the files.

Anyway, it’s a moot point. I just used “strings” on the drive I wanted and was able to pull out my friend’s email address and it turns out he still has the same one after 12 years! I’m going to call it mission accomplished.

There were still a few remaining things to do though before I could re-rack the machine. I used one of the 5.25” to 3.5” adapters from the old drives to attach it to the 3.5” to 2.5” adapter I had bought for the SSD drive. I also needed to get a cheap video card to hold me over until the GTX 660 was freed up from the gaming rig, so I bought a GT 730 since I figured I might as well get one which was quasi-useful.

Here are some pics…

I m g 20160609 193546276

I m g 20160614 201306892

I m g 20160614 201521169

I m g 20160614 202335336

I m g 20160616 174438648

I m g 20160616 174456698

I m g 20160616 202319867

Wordpress with TLS

Sun, 05 Jun 2016 21:55:49 +0000

When I got Wordpress working for the site, I ended up modifying the base container to make Let’s Encrypt work. Originally I was thinking I would just set up a reverse proxy in front of it to do the TLS termination and then pass everything unencrypted between the Wordpress container and the secure web container. It turns out that the default configuration for Wordpress has a php routine which attempts to figure out whether SSL is enabled or not, so a reverse proxy won’t actually work unless the connection between the front-end to the Wordpress container is also encrypted. That defeats the purpose though, since we’d have to also modify the Wordpress container and set up our own self-signed certs. Anyway, for the site I did end up modifying the Wordpress container, but I also figured I’d fix things to make it so other people could use a reverse-proxy. I need to bug some people on the team here at Docker to get the fix reviewed/accepted, but you can find the change here. One last thing about Let’s Encrypt. It’s a pretty awesome service, but the certs it issues expire every three months. Unless you automate some way of refreshing the certs, you’re going to be in for a rude awakening every 90 days or so. I ended up using certbot (EFF’s Let’s Encrypt service) and shoving a script into cron.daily which checks for a new cert and then if there is one, brings the website down and replaces it. It looks kinda like this:

$ /usr/local/letsencrypt/certbot-auto renew --pre-hook "docker-compose -f /path/to/docker-compose.yaml stop" --post-hook="docker-compose -f /path/to/docker-compose.yaml up -d"

It seems to work, but 90 days haven’t come up yet. In theory it will replace the certs if it’s close to the 90 day cutoff, so I should probably check to see if it worked in mid-August.

Getting a GeForce 1080

Sat, 28 May 2016 03:41:58 +0000

So by accident this morning, my 8 year old’s new digital watch’s alarm went off at 6am. Which also happened to be exactly the same time when the new nVidia GeForce 1080 GPUs went on sale. Unfortunately, in the 5 minutes that it took me to fumble for my credit card and open the laptop, they’d completely sold out. It’s still a few more weeks until the Oculus Rift shows up, but I’d really like to finish the 4U rackmount build soon.

Slow Progress

Mon, 23 May 2016 00:17:30 +0000

I worked a little more on fixing up the 4U rackmount machine. Despite how big this puppy is, I keep running into limitations with space. To accommodate the 256GB Samsung SSD drive, I ended up having to buy a 2.5” to 3.5” adapter which I put into a tray which used to be mounted on the side of the chassis. It turns out the placement of the RAM on the AMD motherboard meant the tray wouldn’t fit, so I’ll have to mount the drive with a 3.5” to 5.25” adapter. It’s adapters all the way down. One big problem though, was that originally I was thinking of just leaving the stack of 9GB Seagate Barracuda drives in the 5.25” bay in there until I could work out some way to get the data off. I’m not even 100% sure what’s on there, other than I use to run some ancient version of VMware ESX back when I used to work for Virtzilla. I guess the one big problem is really two problems. The first is I need to figure out how to get the Ultra SCSI drives working again, and the second is that they’re formatted with VMFS 2. Yay for ancient connectors and ancient file formats. One thing I did get done, was both of the old fans have now been swapped out for much quieter Cooler Master 92mm units. I ended up snipping off the 3pin power connectors and splicing them with the old molex connector from the old fans. Cooler Master was nice enough to include a 3pin to molex adapter, but one of them wouldn’t fit into the Corsair PSU’s molex connector, and the other I had already cut off before realizing there was an included adapter. Oops. The end result though is the machine is now quiet enough to put in my rack without my wife wanting to kill me, so it’s smiles all around.

2016 05 17 182312

2016 05 17 201703

2016 05 22 122633

2016 05 22 122647

2016 05 22 131834

2016 05 22 141830

2016 05 22 141842

Something New About Sockets

Fri, 20 May 2016 15:56:47 +0000

I just found out something really cool about curl. Actually, it’s so cool that at first I couldn’t believe that I didn’t know about it, but apparently it’s a new thing. There is now a “–unix-socket” command which will allow you to pass HTTP through a unix socket. This might not initially seem like a big deal, but it’s extremely handy when you’ve set up HTTP pass-through a unix socket to something like Gunicorn if you’re in Django land. It’s also particularly handy for debugging the Docker API which you can get to at /var/run/docker.sock. Here’s an example:

$ curl -s --unix-socket /var/run/docker.sock http://localhost/images/json | jq -r "."
[
  {
    "Id": "sha256:aeff5a9860a391e0313f68c37ed368e524f53d6ecb6be0f2b70ef3922ebcda28",
    "ParentId": "",
    "RepoTags": [
      "gcr.io/tensorflow/tensorflow:latest"
    ],
    "RepoDigests": null,
    "Created": 1461181967,
    "Size": 714212803,
    "VirtualSize": 714212803,
    "Labels": null,
  }
]

That gives you a lot more data than if you were trying to directly parse “docker images”, plus makes it super easy to debug what’s going on under the covers.

The Gaming Build

Thu, 19 May 2016 06:35:52 +0000

While waiting for the parts to finish off the rackmount build, I put together the new machine for the Oculus. It’s amazing to me how nice things are these days compared to even 10 years ago. Everything just snaps together like Legos. I think the hardest part is probably spreading the Arctic Silver onto the top of the CPU. The new GeForce 1080s still haven’t shipped yet, so in the mean time, I’ve borrowed the GeForce 660 from the rackmount machine. You can’t really tell that much of a difference from the the AMD mobo, but I guess now it’s technically “Oculus ready”, or at least, will be with a new GPU. Here’s the part’s list, and here are some pics.

2016 05 16 19 34 54

2016 05 16 19 39 19

2016 05 16 20 23 23

2016 05 16 20 32 44

2016 05 16 21 01 31

2016 05 16 21 43 21

2016 05 16 21 43 34

Off to the Races

Thu, 19 May 2016 06:03:42 +0000

After a night of banging my head against the wall, I seem to have the blog more or less up and running. When I was originally setting everything up internally, I made a simple docker-compose.yaml file which used the official WordPress and Mariadb images. The problem though is that the docker images don’t support TLS out of the box, and the documentation on the Docker’s website doesn’t explain how to set it up (mea culpa, I should ask someone at work about this). My assumption was this wouldn’t be a big deal, because I could just slap an nginx container onto the front to terminate the TLS, and then just use HTTP pass-through to the WordPress container.

Unfortunately, that didn’t work out so well.

WordPress assumes ignores the WordPress Address and Site URL Address schemas when it serves up static content. That meant that some of the content was getting served up with HTTPS and some of it was still HTTP. To make things worse, the admin console completely ignores the settings and relies relative links. Unfortunately because I was passing things through to a private container, this meant that it tried hitting the container name. I ended up ditching the Nginx container, and digging through the Apache configuration in the WordPress container, and finally figured out how to make TLS work. So, lots of mucking around with Dockerfiles, and a fresh new cert from Let’s Encrypt and we’re finally running with a modicum of security. Now I just need to figure out how to automate getting the certs, because they expire every three months.

Forward Progress

Mon, 16 May 2016 04:42:00 +0000

Well, I’m a little further ahead.

A new EVGA PSU and a cooler which actually fits into the 4U chassis have arrived today. I’ve now swapped out the old Cooler Master Hyper 212 EVO (say that five times quickly) with a more svelte Cooler Master GeminII M4 heat sink and fan. It doesn’t seem to cool quite as much, but it still works fine. It’s doubtful that I’m going to do any crazy over clocking on this box now that it’s going to be hidden away inside of a server closet. I also have installed the newly minted Ubuntu Server 16.04 onto an extra 256GB Samsung Pro SSD drive that I had kicking around from Netkine days. This is replacing the 5 x 3.5” Ultra SCSI drives which totaled a whopping 45GB worth of storage. Four of them were 9GB Seagate Barracuda drives, and the last one was a 9GB IBM drive. Do you remember when IBM made disk drives? Me neither.

Oh, funny thing about installing Ubuntu. The only spare USB drive that I could find happened to be in the form of an Apple Camp Wrist Bracelet that had been given to the kids by a family friend who works for Apple. Weird thing about it though, was that in El Capitan I couldn’t seem to get UNetbootin (the thing which writes ISO images to USB sticks) to work. I ended up having to use:

$ sudo dd if=ubuntu-16.04-server-amd64.iso of=/dev/disk2 bs=1024

That worked like a charm, but first I’d fiddled around with Disk Utility and even fdisk and couldn’t seem to manage to get UNetbootin to recognize the USB drive. Not sure what’s up with that. Two problems are still left. The two 92mm fans in the chassis sound like an F-18 taking off, which isn’t going to work out well in my server rack. Also, the four of the drives were sitting in a 5.25” drive bay and the last one was bolted onto the side of the server. Unfortunately the machine doesn’t have anything that can handle the SSD drive, so I’m going to need to get an adapter.

2016 05 15 14 53 33

2016 05 15 15 12 59

2016 05 15 15 13 07

2016 05 15 15 52 10

2016 05 15 15 58 50

2016 05 15 16 25 53

2016 05 15 17 37 59

2016 05 15 17 38 28

The Big Swap

Sun, 15 May 2016 06:22:33 +0000

I’m kind of stuck.

I’d wanted to take the 6 core AMD FX-6300 I had and stuff it into an old 4U rackmount chassis that I had lying around from my first mini-start-up (way back in 1999). Actually, the chassis itself may have come from a different start-up, but such is life in the Silicon Valley.

One thing that’s somewhat nice about PCs is that even though the standards change, the footprint for ATX motherboards is roughly the same as it was 20 years ago. I pulled the old dual-core Pentium III board out of the machine, and popped the new(-er) AMD board in, however, one thing I hadn’t anticipated was that a Cooler Master Hyper 212 EVO heat sink will not fit into a 4U (!) chassis. I guess I should have thought of this before, given it feels like the machine is really a heat sink with a motherboard strapped to it.

A quasi interesting historical footnote about the Pentium III vs the AMD FX6300. The P3 ran at 450 MHz with a 250 nm die size. The AMD runs at 3.5 GHz with a 32 nm die size. Not quite 10x the speed with 1/10th the size, but pretty close.

Oh, I also gutted a 350W PSU out of another machine I had lying around only to realize it didn’t have a PCIe cable on it to power the older nVidia GTX 660 I was planning on stuffing in to it. So… more parts from Amazon tomorrow. Yay 1-day shipping!

2016 05 14 13 41 06

2016 05 14 13 41 13

2016 05 14 13 41 23

2016 05 14 13 43 42

2016 05 14 13 44 19

2016 05 14 14 07 50

2016 05 14 14 16 10

2016 05 14 14 16 18

The New Rig

Sat, 14 May 2016 17:46:08 +0000

So back in January I pre-ordered an Oculus Rift. It’s now mid-May and I still haven’t gotten it yet, however, I think it’s probably imminent. When I ordered it, I actually didn’t want to get it right away, because my current PC gaming rig just isn’t going to cut the mustard. Plus, I figured that nVidia was going to release a new GPU which would probably knock the socks off of the current generation. So now it’s getting close to when the Rift is going to ship, and, wouldn’t-you-know-it, nVidia just announced the new GeForce 1080s are going to ship in a few weeks.

So, it’s time to build the new machine.

I figure I probably shouldn’t go overboard; my last machine I built I ended up getting an AMD CPU with 6-cores just because I thought it’d be interesting for Netkine (my now defunct startup). This time I’m dialing back the cores to 4 with an Intel i5-6600k, but overall it should be pretty smoking fast (or at least fast enough for driving the Rift).

2016 05 13 08 13 07

Working the Kinks Out

Sat, 14 May 2016 16:56:38 +0000

OKAY. I think I’ve finally worked the kinks out getting a basic WordPress site deployed. I’ve been wanting to set up a blog for a bit now about some of the things I’ve been working on, but my first attempt with Docker Compose resulted in things going missing. I’ve since tweaked the compose file to use named volumes, which hopefully will mean that revisions to the compose file won’t cause data to disappear. As an aside, I have to say the world of WordPress is really foreign for me. The ecosystem is just so massive that it’s really hard to figure out how to do basic things display a code snippet. I’ll see if I can figure out how to do that later, plus it would be great to figure out how to secure this beast with TLS.