What I Read

Posted 12 Aug 2019 to media and has Comments

Based on a few requests, I’ve finally put together a list of what I read every week. Let me know in the comments if I’ve missed something I should be reading.

Newsletters

Most of these are daily, and I tend to read all of each email.

Regular Site Visits

Most of these get a visit, most days.

Presence on this list doesn’t imply endorsement or political alignment, obviously (and in full disclosure, I’ve worked with or for a number of publications on the list).

Smaller Elixir Docker Images

Posted 22 Feb 2019 to elixir, docker and has Comments

When running Elixir in production, I was surprised how large the Docker images were and how long they took to build. In an effort to keep those images smaller, and reduce the build time for those images, I spent a while testing different options. The solution I came up with seems obvious in retrospect - but took a long while to test all the different options. I’ve made liberal use of multi-stage builds, which allow you to pull artifacts from intermediary images into later stages. The final image can then be significantly smaller. These steps assume you’re using Distillery for packaging your Elixir application for deployment using OTP releases.

I’ll go through all of the three stages in order below, each Dockerfile snippet is a part of a single file that I put together in the last section.

Dependency Stage

The first stage (which I name base) just consists of adding the mix file and lock, fetching dependencies, and building them. This will happen irrespective of your application files, meaning your app files can be updated and this stage will remain the same. Changing your dependencies list in mix.exs will result in this whole stage being rebuilt, but hopefully that’s not something that happens frequently. In my experience, building the dependencies was the primary cause for how long it took to build new images, so I wanted this time penalty to be rare.

Here’s the first part that just builds the deps into a base intermediate image:

# First, download and compile all dependencies in an intermediate base image
FROM elixir:1.8.1-alpine as base
# Your dependencies for the OS may differ
RUN apk update && apk --no-cache add --virtual builds-deps build-base libressl libressl-dev
WORKDIR /app
ADD mix.exs .
ADD mix.lock .
ENV MIX_ENV prod
RUN mix local.rebar --force
RUN mix local.hex --if-missing --force
RUN mix deps.get --only prod
RUN mix deps.compile

Release Build

The next stage builds the actual release using Distillery. It compiles your application code, creates a release, and then ungzip’s the release package into a folder to be picked up by the subsequent stage. For the smallest image, you should include the Erlang Runtime System via Distillery (instead of using a docker image that includes the Erlang VM). Based on testing, this option can help reduce the size of the final image significantly. Just make sure your rel/config.exs has the following options:

environment :prod do
  set include_erts: true
  set include_src: false
  ...
end

Here’s the next section in the Dockerfile to build the release and ungzip it into a folder for pickup in the next stage:

# Now build a release from our app files
FROM elixir:1.8.1-alpine as intermediate
COPY --from=base /app /app
ENV MIX_ENV prod
RUN mix local.hex --if-missing --force
WORKDIR /app
ADD . .
RUN mix compile
RUN mix release
WORKDIR /release
# replace 'my_app' below with your application name, and possibly your app version
RUN tar -zxf /app/_build/prod/rel/my_app/releases/0.1.0/my_app.tar.gz -C /release

Release Artifact

The last step just copies the release to a base alpine image to run, and then runs the result in the foreground.

# now run the release.  Make sure the alpine version below matches the alpine version
# included by erlang included by elixir:1.8-alpine
FROM alpine:3.9
RUN apk update && apk add --no-cache bash openssl
ENV MIX_ENV prod
ENV LANG C.UTF-8
COPY --from=intermediate /release /app
EXPOSE 4001
CMD ["/app/bin/my_app", "foreground"]

Putting It All Together

And here’s the final Dockerfile with all of the parts in order. My final release is ~50M, which is significantly smaller than the ~200MB when not using intermediate stages.

# First, download and compile all dependencies in an intermediate base image
FROM elixir:1.8.1-alpine as base
# Your dependencies for the OS may differ
RUN apk update && apk --no-cache add --virtual builds-deps build-base libressl libressl-dev
WORKDIR /app
ADD mix.exs .
ADD mix.lock .
ENV MIX_ENV prod
RUN mix local.rebar --force
RUN mix local.hex --if-missing --force
RUN mix deps.get --only prod
RUN mix deps.compile

# Now build a release from our app files
FROM elixir:1.8.1-alpine as intermediate
COPY --from=base /app /app
ENV MIX_ENV prod
RUN mix local.hex --if-missing --force
WORKDIR /app
ADD . .
RUN mix compile
RUN mix release
WORKDIR /release
# replace 'my_app' below with your application name, and possibly your app version
RUN tar -zxf /app/_build/prod/rel/my_app/releases/0.1.0/my_app.tar.gz -C /release

# now run the release.  Make sure the alpine version below matches the alpine version
# included by erlang included by elixir:1.8-alpine
FROM alpine:3.9
RUN apk update && apk add --no-cache bash openssl
ENV MIX_ENV prod
ENV LANG C.UTF-8
COPY --from=intermediate /release /app
EXPOSE 4001
CMD ["/app/bin/my_app", "foreground"]

Raspberry Pi Home Router + VPN

Posted 11 Oct 2018 to raspberry pi, vpn, networking and has Comments

Internet Service Proviers (ISPs) are constantly trying to get away with some icky things, and rarely get caught or punished. A prime example is Verizon’s tracking of mobile users with “perma-cookies”, which resulted in only a paltry $1.35M fine. You can get around your ISP’s prying eyes by using a Virtual Private Network (your computer connects directly to a third party, and all of your traffic is forwarded from there to the rest of the internet). If you want more background on VPN’s, the Electronic Frontier Foundation has a great guide (and list to VPN vendor comparisons) here.

I’ve been wanting to build a fail-safe wireless access point for a while, to provide a wireless network in my apartment that is only up if it’s connected to a VPN. Any device connected to the wireless network can know for sure that it is bypassing inspection from my ISP. The only downside is that services that rely on your IP address to locate you will think you’re in the same location as the VPN server (which seems like a small price for added privacy).

The hardware/software are so cheap/easy at this point, it’s pretty easy to set up. So here are some instructions for what I did!

VPN Selection

I ended up using Private Internet Access (PIA), which admittedly sounds like a total scam site. In terms of protocols/encryption levels supported and the number of server locations, I think they’re a true winner. The one caveat is that they do operate out of London, which is under Five Eyes jurisdiction. They claim that they don’t keep any logs at all, but it’s probably safest to assume that they will protect you from the prying eyes of your ISP (but probably not the curiosity of the Five Eyes).

Hardware

For the device, I went with the Rasberry Pi 3 model B+. Paired with an external antenna, my whole apartment is pretty decently covered.

For some visual insight into whether the device was “connected” to my VPN of choice or not, I added some LED’s via the 8 LET Blinkt. When the Access Point (AP) is connected to the VPN, there’s a beautiful rainbow pattern that shifts to a random arrangement every minute. When the connection is down, it’s all red.

In terms of hardware setup, just plug the external antenna into one of the USB ports and the Blinkt strip into the top of the Pi (though make sure the curves on the top that match the corners of your Raspberry Pi).

Software

If you install the base Raspbian Lite distribution (no desktop necessary) then you should be good to go. Just use the installation guide for getting the OS onto your micro-SD card, and connect your Pi’s ethernet port to your modem or router.

Once you’re booted up, the steps are simple:

Set up the wireless access point
Set up the VPN connection
Make sure all traffic is routed from the wireless network to the VPN
Turn on the pretty indicator lights

Install Packages

Upgrade everything and install necessary packages.

$ sudo su -
$ apt-get update
$ apt-get upgrade
$ apt-get install hostapd openvpn dnsmasq python3-blinkt

Wireless Access Point

Now we’re going to edit some configuration files. I found editing using the default installed nano to be the easiest to edit these files.

Assuming your wireless antenna is showing up as wlan1 (since onboard wifi would be wlan0) - edit your file () /etc/dnsmasq.conf to be:

no-resolv
server=209.222.18.222
server=209.222.18.218
interface=wlan1
cache-size=2000
dhcp-range=192.168.42.10,192.168.42.50,24h

This sets up your DHCP server and DNS server (which forwards all requests to Private Internet Access’ DNS servers at 209.222.18.222 and 209.222.18.218).

Next, we can configure our access point. Set the contents of /etc/hostapd/hostapd.conf to:

interface=wlan1
ssid=My-AP
hw_mode=g
channel=11
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=password
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP
ieee80211n=1

Make sure to replace ssid with the name of the access point you’d like, and set wpa_passphrase to be the wireless password you’d like for your network. Now we can start these services:

$ update-rc.d dnsmasq enable
$ service dnsmasq restart
$ update-rc.d hostapd enable
$ service hostapd restart

VPN Setup

We’ll be using the OpenVPN client to connect to the VPN. PIA provides a handy online tool for generating a config file, which you should definitely use. Choose “OpenVPN 2.4 or newer”, then “Linux”, then choose a VPN location closest to you physically. The recommended “UDP/1198” choice is fine (though, if you’re really concerned about security, full RSA-4096 could also be selected, but your little Raspberry Pi may have trouble with the computational power necessary for the increased encryption). Check the box “Use IP” (to avoid any DNS issues) and then click “Generate”. The resulting file should be copied to your Pi at /etc/openvpn/pia.conf. Then, add the following line at the end:

auth-user-pass /etc/openvpn/login.txt

And create a file at /etc/openvpn/login.txt with your PIA username and password:

username
password

Then, we can start the OpenVPN connection:

$ update-rc.d openvpn enable
$ service openvpn restart

Traffic Routing

Now we can make sure we’re routing all traffic from the wireless network to our VPN connection. Edit /etc/sysctl.conf and un-comment or add to the bottom:

net.ipv4.ip_forward=1

Now we can set up routing internally. Run the following commands:

$ sysctl -p /etc/sysctl.conf
$ iptables -P FORWARD DROP
$ iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE
$ iptables -A FORWARD -i tun0 -o wlan1 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ iptables -A FORWARD -i wlan1 -o tun0 -j ACCEPT
$ iptables-save > /etc/iptables.ipv4.nat

Then, edit the file /etc/rc.local add before the line exit 0 add this line (so that settings are applied after each restart):

iptables-restore < /etc/iptables.ipv4.nat

Indicator Lights

As the regular pi user, create a file named update in your home directory:

#!/usr/bin/env python3
import os
import colorsys
import time
from urllib.request import urlopen
from blinkt import set_pixel, set_brightness, show, clear, set_clear_on_exit, set_all

set_clear_on_exit(False)
set_brightness(0.04)

with open('/etc/openvpn/pia.conf', 'r') as f:
    line = [l for l in f.readlines() if l.startswith('remote')][0]
    vpnip = line.split()[1]

vpn = 'tun0' in os.listdir('/sys/class/net/')
try:
    with urlopen('http://ipinfo.io/ip') as r:
        ip = r.read().decode('utf-8').strip()
        vpn = vpn and ip == vpnip
except:
    vpn = False

clear()
if vpn:
    spacing = 360.0 / 16.0
    hue = int(time.time() * 100) % 360
    for x in range(8):
        offset = x * spacing
        h = ((hue + offset) % 360) / 360.0
        r, g, b = [int(c * 255) for c in colorsys.hsv_to_rgb(h, 1.0, 1.0)]
        set_pixel(x, r, g, b)
else:
    set_all(255, 0, 0, 1)
show()

This script will test your VPN connection and ensure that the internet visible IP you’re connecting from is the PIA VPN IP. You can set this script to run every minute by running:

$ crontab -e

And add this line after all of the comments:

* * * * * /home/pi/update

That’s all!

Testing

It’s always good to test to make sure that you’re not only connected to the internet, but that you’re not leaking. PIA provides a test page to ensure you’re connected, which is a great first place to start. You should also run a leak test to ensure your DNS isn’t leaking.

Collect All Seven

Posted 22 Apr 2017 to travel and has Comments

Finally collected all 7 continents with a trip to Antarctica in March! It took a while, but cleaned up pics are now online.

Circling the World in 18 Days

Posted 25 Sep 2016 to travel and has Comments

I’m going on a trip in October to further the namesake of this blog; I’m going to find some science as I make a circle on the great flat plane of the earth. What follows is some background for the reasoning behind my trip, followed by a description of my plan.

Background: The Untaught Controversy

Like most people who are products of the American education system, I once believed the “round Earth” theory as a fact. No alternative theories were given, the robust scientific controversy was hidden, and the idea of a flat Earth was presented as an “Archaic concept” from our distant ancestors. Robert J. Schadewald, a former president of the National Center for Science Education, even spoke out about the failures of the South Carolina school system in their supression of a valid alternative theory:

Only spherical-earth science is presented to students in virtually all of those courses that discuss the shape and origin of the earth. Public schools generally censor flat-earth science and evidence contrary to the spinning ball theory … Public school instruction in only the spherical theory also violates the principle of academic freedom, because it denies students a choice between scientific models and instead indoctrinates them in spherical-earth science alone.

Schadewald wrote this as a part of a bill written for the SC legislature demanding equal time in classrooms for flat-earth science. As head of the National Center for Science Education, he acknowledged the pressure from so-called “experts” to silence alternative theories and ignore the scientific controversy. Round-earthers teach their theory as fact rather than the hypothesis that it is.

It’s not just scientists and educators, though, who have advocated for greater transparency in alternative theories. None other than President George W Bush provided his support for teaching the scientific controversy wherever it exists:

I felt like both sides ought to be properly taught so people can understand what the debate is about. I think that part of education is to expose people to different schools of thought… you’re asking me whether or not people ought to be exposed to different ideas, the answer is yes.

When you look at the support that exists for competing theories to spherical-earth “science” it’s hard to believe that so many people still dismiss it outright. Everyone from musicians like BoB to the political leader Mohammed Yusuf (the former leader of Boko Haram in Nigeria) have spoken out in support of what so many plainly see is true: there is no curvature to the earth, the ground we walk on is flat.

The cities in the background are approx. 16miles apart... where is the curve ? please explain this pic.twitter.com/YCJVBdOWX7
— B.o.B (@bobatl) January 25, 2016

The opposition is strong, though. The famously outspoken biologist Richard Dawkins (who hasn’t published a scientific paper since 1990) had this to say:

If you give the idea that there are two schools of thought within science, one that says the earth is round and one that says the earth is flat, you are misleading children.

Coming from a biologist without any training in physics or geography, that’s fairly rich. Perhaps he has been too busy spreading militant atheism to see the clear scientific experiments dating back to 1838 that show the Earth can’t be round.

It’s clear that the science is far from settled - but many still can’t bring themselves to admit that the flat surface they see with their eyes should be evidence enough. It’s tough to buck the trend and stand against what many portray as “consensus” (which, clearly, it isn’t). Daniel Shenton (the president of The Flat Earth Society) said this to The Guardian:

I haven’t taken this position just to be difficult. To look around, the world does appear to be flat, so I think it is incumbent on others to prove decisively that it isn’t. And I don’t think that burden of proof has been met yet.

Mr. Shenton leads one of the largest and best organized groups trying to spread awareness of the flat Earth theory. The very first member of the group was the famous musician Thomas Dolby who joined the Flat Earth Society in 2009. Thomas Dolby understands the power of science - having written a number of songs about it’s power to enlighten and to blind, like this gem:

The group of folks who question spherical-earth “science” are not some fringe group of backwards conspiracy theorists. Eric Oliver, a political scientists at the University of Chicago, acknowledges this openly:

If they were like other conspiracy theorists, they should be exhibiting a tendency toward a lot of magical thinking, such as believing in UFOs, ESP, ghosts, the Devil, or other unseen, intentional forces. It doesn’t sound like they do, which makes them very anomalous relative to most Americans who believe in conspiracy theories.

One final note: There’s one very large, worldwide, internationally recognized group that appears to know the truth, too. Here are two logos: The one on the left is the logo for The Flat Earth Society, and the one on the right is for the United Nations:

Both show exactly the same thing: the earth is in the form of a disk with the North Pole in the center and Antarctica as a wall around the edge. No globe, no sphere - just a flat disk.

The Trip

Starting on October 12, I’ll be making my way across the top of the disk in a giant circle. I’ll leave from NYC and return on October 30th, making stops at these locations:

San Diego, California
Tijuana, Baja California
Los Angeles, California
San Francisco, California
Sydney, Australia
Singapore
Johor Bahru, Malaysia
Moscow, Russia
Berlin, Germany
London, England
Reykjavik, Iceland

As I travel, I will spreading knowledge of the flat-earth alternative theory, hopefully helping to change hearts and minds. I’ll also have a compass and be taking readings as I travel. As some have suggested, if the earth is round then I should see a compass pointing into the earth (through the earth to the north pole), especially from such locals as Sydney outside of the equator. I’ll be measuring the angle of the compass needle as compared with the level horizon.

I welcome any additional ideas for experiments in the comments, and look forward to posting my results. I’ve got a list of folks I’m meeting with along my way, but if anyone will be in any of the cities above during my travels I’d love to meet up, talk through the science, and find ways to support the academic freedom of inquiring minds all over the great earth plane!

Bitboot: P2P Network Bootstrapping

Posted 05 Aug 2016 to bitboot, p2p, tint-foundation, dht and has Comments

TL;DR: Bitboot allows arbitrary groups of computers on the internet to find each other without any local dependencies and without setting up a central server. It can be used to bootstrap new nodes on a p2p network or to simply find your other computer whose IP changes all the time.

I’ve been working with a few custom peer-to-peer (p2p) networks lately, and one issue that I kept running into was figuring out how to bootstrap the networks. If only a few nodes are out there running, and their IP addresses may change without warning, how could I reliably find those nodes when starting up a local node?

On most “distributed” p2p networks, clients are initially bootstrapped by connecting to a handful of hard-coded, centralized nodes (yes, even Bitcoin and BitTorrent). Every new peer-to-peer network must solve this same challenge, usually by hardcoding centralized bootstrap servers. I really didn’t want to have to do that - especially for a small test network.

After hitting my head against a wall for a few hours, I worked out the framework for a proof of concept (with an anonymous friend) that would solve the issue by relying on an existing network - namely, the friggin’ enormous BitTorrent distributed hash table. The particular characteristics of a Kademlia DHT (the type of DHT BitTorrent uses) allow you to create nodes and with IDs and a concept of distance between those IDs. This means that you can select certain IDs when you join the network that are near particular points - what I call “rally points.” By choosing IDs that have special characteristics (preset byte values in the ID) and close distances to the rally point, you can find other “special” nodes in the massive BitTorrent DHT rather quickly. Once you’ve identified them, you can then try to connect to them on whatever port your other network runs on. Since nodes on the BitTorrent DHT are running as full nodes, they don’t harm the existing network in any way (and, in fact, they strengthen the network by adding additional storage space and redundancy).

We decided to call this particular method bitboot and created an implementation on github. It’s in NodeJS because the best BitTorrent DHT library we could find was also in NodeJS. When you run bitboot, you give it a magic name to uniquely identify the network you’d like to join. Bitboot then joins the existing BitTorrent DHT and finds other nodes with the same magic name. It does this by selecting a rally point to hang out near based on the magic name where it will meet other nodes with the same magic name value. Also, the ID it uses is carefully selected so other nodes can pick it out as a bitboot peer based on the value of the magic name (in case other non-member nodes are hanging out around the rally point).

There’s a command line program to give it a try - check it out at github.com/tintfoundation/bitboot.

Capistrano for Python

Posted 29 Jul 2016 to python, fabric and has Comments

In the Python world, Fabric seems like the de facto method for deploying code. At its core, though, Fabric is just a way to streamline the use of SSH, much like Ruby’s SSHKit. Reliable deployments, however, typically involve more than just running a few commands on a remote server. For instance, Ruby’s Capistrano builds on it’s underlying ssh library SSHKit and provides a lot of useful functionality, like:

the ability to inject tasks in a dependency chain (before you run TaskA, always run TaskB)
configuration variables with values that are role specific, where each server has one or more roles
configuration values that are evaluated at run time (i.e., the ability to have values that are callable at runtime)
the ability to define a sane flow convention for deployment and then give people the ability to override/inject only what they need. Convention over configuration!

I feel like there should be a way to reliably do these things, and Fabric isn’t enough. So I made a project called Fake - for Fabric + Make (in the same way that Rake is Ruby + Make). It includes all of this functionality, including Capistrano’s default deploy tasks for reliable code checkouts (and super fast rollbacks!).

Check it out at https://github.com/bmuller/fake for documentation and examples.

via Geir Kiste

Going Off Grid

Posted 06 Feb 2016 to deep thoughts, off grid, media and has Comments

Since I joined Vox Media back in April in the OpBandit acquisition, Vox has raised a little money, acquired Re/code, and turned into a unicorn 🦄. Over the last 10 months, I’ve had the pleasure of working with some of the best and most talented people in the publishing space. It has been an amazing experience, and I’m very grateful to Trei Brundrett and the rest of the Vox Media Product Team for creating such a great environment for product development. After integrating the OpBandit platform into Vox’s core CMS, building out solid data infrastructure and data science teams, and an incredible amount of self-reflection, I’ve decided to take some time away from online publishing.

While I think online publishing clearly fills a part of the necessary role of a free press in a democracy, at this point the immediate technological challenges of putting words and pictures on the internet are solved. Most of the tech and data science efforts are now focused on increasingly marginal gains in engagement (and more importantly, ad views). Audience growth is no longer a direct function only of the quality of the content produced; instead, it is completely captive to the whims of black box algorithms controlled by third party platforms. Online publishing is just about as far away from the simplicity of “We make X and people pay us for it” as you can get, and things are getting even more complicated. As much as I love the environment, the people, and the new challenges of the industry - I’d like to spend some time doing something positive with a more immediate and direct impact to people’s lives.

There are some amazing technological advances going on in the world today with the potential for that sort of impact - including efforts to make humanity interplanetary, eradicate diseases, and redefine human computer interaction. Another example comes via a former (LivingSocial) coworker of mine Dan Mayer who wrote a great blog post about his new job at a company providing solar power to households in Africa. The company, Off Grid Electric, provides clean, affordable energy via solar power as a service. They do this in a part of the world where 90% of people have no access to an electrical grid, and they’re on track to light a million homes in Africa over the next couple of years. They’re not only bringing power, but thousands of jobs in the process. What’s more, this access to energy goes beyond just convenience and modernization, as the Washington Post points out:

Across sub-Saharan Africa, fully 590 million people lack access to power. And it’s a life-or-death issue: Indoor air pollution from wood stoves now kills 3.5 million people per year, more than AIDS and malaria combined.

When Dan recently reached out about an open position working on these problems, I couldn’t help but apply, and I couldn’t say no to their subsequent offer. I’m excited to be joining Off Grid Electric this week and will be immediately heading to their headquarters in Arusha, Tanzania for three months. Not only will I be able to travel to one of the most beautiful parts of Africa, but the work I’ll be doing has the potential to directly impact the quality of people’s lives. I don’t know if my future will include an eventual return to the online publishing world; either way, I look forward to seeing my old colleagues at Vox Media continue to provide a stellar example of what a modern media company can accomplish.

Now I’m off to start packing…

It's Go Time: Better Time in Golang

Posted 20 Jun 2015 to golang, arrow and has Comments

Dealing with time in Go is a pain. The built-in time package doesn’t include much in the way of helper functions. Formatting time is especially difficult; unlike many other languages in the C-family, Go has no support for strftime-based formatting. Instead, you have to remember a specific date (1/2 3:04:05 2006 -0700) and format that date’s values in the form you’d like to mimic. For instance, if I wanted to format a date into the format mm/dd/yyyy HH:MM:SS, here’s what you have to do:

fmt.Println("Today's date:", time.Now().Format("01/02/2006 03:04:05"))

Of course, this typically require having to look up the magic date in the docs to make sure you’ve got the right month/day/year. I’m lazy, so naturally I’d rather just have the same strftime format I’m used to.

Time Flies Like an Arrow; Fruit Flies Like a Banana

The Arrow library provides a C-family style strftime-based formatting and parsing in Golang (among other helpful date/time functions). Here’s an example of formatting and parsing:

// formatting
fmt.Println("Current date: ", arrow.Now().CFormat("%Y-%m-%d %H:%M:%S"))

// parsing
parsed, _ := arrow.CParse("%Y-%m-%d", "2015-06-03")
fmt.Println("Some other date: ", parsed)

You can also get the time at the beginning of the minute / hour / day / week / month / year.

t := arrow.Now().AtBeginningOfHour().CFormat("%Y-%m-%d %H:%M:%S")
fmt.Println("The first second of this hour was at:", t)

t = arrow.Now().AtBeginningOfWeek().CFormat("%Y-%m-%d %H:%M:%S")
fmt.Println("The first second of the week was at:", t)

You can also more easily sleep until specific times:

// sleep until the next minute starts
arrow.SleepUntil(arrow.NextMinute())
fmt.Println(arrow.Now().CFormat("%H:%M:%S"))

There are also helpers to get today, yesterday, and UTC times:

day := arrow.Yesterday().CFormat("%Y-%m-%d")
fmt.Println("Yesterday: ", day)

dayutc := arrow.UTC().Yesterday().CFormat("%Y-%m-%d %H:%M")
fmt.Println("Yesterday, UTC: ", dayutc)

newyork := arrow.InTimezone("America/New_York").CFormat("%H:%M:%s")
fmt.Println("Time in New York: ", newyork)

And for generating ranges when you need to iterate:

// Print every minute from now until 24 hours from now
for _, a := range arrow.Now().UpTo(arrow.Tomorrow(), arrow.Minute) {
     fmt.Println(a.CFormat("%Y-%m-%d %H:%M:%S"))
}

Much easier. There’s more magic not listed here; check out the docs on godoc.org for the full list.

OpBandit: An Exit, and Retrospective Lessons

Posted 05 Apr 2015 to opbandit, deep thoughts and has Comments

Blaine and I started OpBandit back in 2012 with the idea that there was a real need for better optimization tools for online publishers. We quit our jobs, built a company, and eventually had the pleasure of working with some of the top publishers in the world. We built a product that helped customers across seven countries serve hundreds of millions of optimized page views per month (in four languages!). We’ve worked out of some of the best real estate in the publishing world and had the pleasure of growing close to some other fantastic startups in the media space. It has been, without a doubt, one of the most educational experiences of my life (like an incredibly expensive, montessori style MBA).

After two years, right as we were getting into the swing of our first round of external financing, we were approached by Vox Media. Vox is one of the fastest growing media companies out there, and we have been huge fans of their work for a while. They were serious, we listened, and eventually we came to this:

Vox buys datascience company OpBandit http://t.co/qH6R4NaipM
— Financial Times (@FT) April 6, 2015

Over the past two and a half years, we’ve learned countles lessons along the way. Many of our assumptions and decisions turned out to be correct (from pure luck or the benevolence of the startup gods) - and some were way off. Just so I never forget, here’s a brief list of a few of them.

1. Don’t Underestimate the Sales Role

Unfortunately, I think I’ve historically looked down on the business development / sales sides of companies. It was easy to think something like:

I'm doing crazy math and writing complex code, how hard could it be to talk to a few folk on the phone and send lots of email. BD is just the group of people who sign agreements with other companies to provide a technically impossible, non-existent feature as if it already exists. Sales is just persistent calling and emailing until you eventually annoy a potential customer into buying your product.

I now know that I was full of shit. It takes a very unique set of skills to maintain interest and close deals, and those are skills that cannot be undervalued if a company is to succeed. In my opinion, technical and product proficiency is necessary - but not sufficient - for rapid scaling. While I think founders make great salespeople, if I could do it all over again, I would want to have a BD person on the team on day one.

2. Enterprise Sales Sucks

This is a grab bag of sales lessons.

Many times we would email a potential customer / partner / investor and not hear back for days, send a few follow ups over the coming weeks, and eventually give up. Then, they’d respond out of the blue and mention they’d been working behind the scenes and had great news and we’d be working together on whatever. Non-response isn’t the same as never going to respond.
No matter what email client you use, figure out how to create templates. 80% of the millions of emails I sent over the last 2 years fit into one of a handful of templates. Use email templates.
Maintain relationships. Even if a sale / partnership doesn’t work out, the people you talked to will probably end up somewhere else at some point and might be able to make a sale happen there instead. Keep in touch.

3. Build for Scale

It felt incredibly silly spending tons of time ensuring that our system could handle 10 million hits / hour when there were no customers and no hits so far. It takes some weird combination of hubris and faith to spend a weekend working on automating massive server deployments when you have a single tiny instance running on AWS’s free tier. It seems ridiculous, but I think there’s an immense value to developing with an expectation of success. We didn’t have to rewrite a “prototype” to handle increasing loads - since all of the code from day one was designed as if massive growth were inevitable. It definitely costs more up front (prototypes obviously take less time to develop) - but there’s a way to at least design for growth without having the expectation of needing rewrites. For instance, choose a DB that can scale with you as you grow, instead of “We’ll use DB X since it’s easiest now - until we get to big for it, and then rewrite everything to handle DB Y when we get to that point.”

Automate all the things. 5 minutes a week restarting that service that always needs to be restarted is absolutely worth 30 minutes right now to automate that restart.

Log all of the things. If you aren’t logging/tracking it, you can’t fix it. Track system stats, app stats, user stats, and everything else in between. Track response times, query times, and how many cups of coffee you drink per day. Track it all. Store the stats - storage is cheap. Realizing just now that your response time has gone up 10x over the last month is expensive.

Alert on all bad things. Automated alerts are best when they warn about bad things that will happen soon, rather than alerts that occur when it’s too late. For instance, set up an alert to go off when your SSL certificate expires in one week, not when it has already expired. If you’re getting too many alerts, then they’re either things that should be fixed or your system is too sensitive. If I get a text message from a monitoring service, it means only one thing - I need to take some action. Useless alerts are like the boy that cried shitwolf. Never cry shitwolf.

4. Time / Expected Return Tradeoff: Beware the Time Suck

There will always be more things on your TODO list than you will ever complete, so constant reassessment of prioritization is necessary. Implicitly, it’s some sort of ranking for all possible tasks $X$ based on the probability of success $P(x_i = success)$ times the utility value of the successful outcome $u(x_i)$ divided by the commitment of time necessary $t(x_i)$, or

\[ \max_X \frac{P(x_i) u(x_i)}{t(x_i)} \]

Since neither $P$ nor $t$ are known, this equation is almost meaningless. Good luck.

That said - there was one time in particular when we vastly overestimated the expected value of a successful outcome $u(x_i)$ and spent way too much time (weeks) working on a particular opportunity. It was a failure and a massive time suck. You won’t know exactly what the payoff will be - but the earlier you can nail that down the more you can understand what’s potentially worth your time.

5. Get Some Office Space

Get some office space ASAP. The bump in productivity is worth the expense.

6. Your Time Estimate is Too Low

I can’t remember a single time any of our time estimates for anything weren’t at least half of what they should have been (if not less). This goes for everything from the big ones like time to close a round and time until the next sale to the little ones like time to features being completed and how long it takes checks to clear.

I don’t remember ever saying, “Boy, that was fast.” Ever.

7. Find Others to Join Your Folie à Plusieurs

We really lucked out with our first adviser. He was there from day one with a strong belief in us and our mission (and I do mean day one, when I said things like “What’s a one-pager?”). Find others (early!) who will share in your vision and provide constant encouragement, because at many points you will feel uncertain about whether what you have is actual vision or mad delusion. It’s incredibly encouraging to have an external voice reminding you of what you’ve accomplished so far and expressing an expectation for future success. It’s not just nice things, though; having someone who is also willing to give you a solid kick if you need to refocus is also helpful. What’s important is that you have that external moral support before there are any major proof points.

Later, once the vision starts turning into reality (and you have happy customers and proof points and actual success stories) it’s easier to find people who can share the vision (and who can help with the doubts you’ll feel even after the successes). It’s critical, however, to find those early (blind) believers - and find them as early as possible.

Those are the big ones. Let me know what I’m missing in the comments!

« Older Posts »