It's Go Time: Better Time in Golang

Posted 20 Jun 2015 to golang, arrow and has Comments

Dealing with time in Go is a pain. The built-in time package doesn’t include much in the way of helper functions. Formatting time is especially difficult; unlike many other languages in the C-family, Go has no support for strftime-based formatting. Instead, you have to remember a specific date (1/2 3:04:05 2006 -0700) and format that date’s values in the form you’d like to mimic. For instance, if I wanted to format a date into the format mm/dd/yyyy HH:MM:SS, here’s what you have to do:

fmt.Println("Today's date:", time.Now().Format("01/02/2006 03:04:05"))

Of course, this typically require having to look up the magic date in the docs to make sure you’ve got the right month/day/year. I’m lazy, so naturally I’d rather just have the same strftime format I’m used to.

Time Flies Like an Arrow; Fruit Flies Like a Banana

The Arrow library provides a C-family style strftime-based formatting and parsing in Golang (among other helpful date/time functions). Here’s an example of formatting and parsing:

// formatting
fmt.Println("Current date: ", arrow.Now().CFormat("%Y-%m-%d %H:%M:%S"))

// parsing
parsed, _ := arrow.CParse("%Y-%m-%d", "2015-06-03")
fmt.Println("Some other date: ", parsed)

You can also get the time at the beginning of the minute / hour / day / week / month / year.

t := arrow.Now().AtBeginningOfHour().CFormat("%Y-%m-%d %H:%M:%S")
fmt.Println("The first second of this hour was at:", t)

t = arrow.Now().AtBeginningOfWeek().CFormat("%Y-%m-%d %H:%M:%S")
fmt.Println("The first second of the week was at:", t)

You can also more easily sleep until specific times:

// sleep until the next minute starts

There are also helpers to get today, yesterday, and UTC times:

day := arrow.Yesterday().CFormat("%Y-%m-%d")
fmt.Println("Yesterday: ", day)

dayutc := arrow.UTC().Yesterday().CFormat("%Y-%m-%d %H:%M")
fmt.Println("Yesterday, UTC: ", dayutc)

newyork := arrow.InTimezone("America/New_York").CFormat("%H:%M:%s")
fmt.Println("Time in New York: ", newyork)

And for generating ranges when you need to iterate:

// Print every minute from now until 24 hours from now
for _, a := range arrow.Now().UpTo(arrow.Tomorrow(), arrow.Minute) {
     fmt.Println(a.CFormat("%Y-%m-%d %H:%M:%S"))

Much easier. There’s more magic not listed here; check out the docs on for the full list.

OpBandit: An Exit, and Retrospective Lessons

Posted 05 Apr 2015 to opbandit, deep thoughts and has Comments

Blaine and I started OpBandit back in 2012 with the idea that there was a real need for better optimization tools for online publishers. We quit our jobs, built a company, and eventually had the pleasure of working with some of the top publishers in the world. We built a product that helped customers across seven countries serve hundreds of millions of optimized page views per month (in four languages!). We’ve worked out of some of the best real estate in the publishing world and had the pleasure of growing close to some other fantastic startups in the media space. It has been, without a doubt, one of the most educational experiences of my life (like an incredibly expensive, montessori style MBA).

After two years, right as we were getting into the swing of our first round of external financing, we were approached by Vox Media. Vox is one of the fastest growing media companies out there, and we have been huge fans of their work for a while. They were serious, we listened, and eventually we came to this:

Over the past two and a half years, we’ve learned countles lessons along the way. Many of our assumptions and decisions turned out to be correct (from pure luck or the benevolence of the startup gods) - and some were way off. Just so I never forget, here’s a brief list of a few of them.

1. Don’t Underestimate the Sales Role

Unfortunately, I think I’ve historically looked down on the business development / sales sides of companies. It was easy to think something like:

I'm doing crazy math and writing complex code, how hard could it be to talk to a few folk on the phone and send lots of email. BD is just the group of people who sign agreements with other companies to provide a technically impossible, non-existent feature as if it already exists. Sales is just persistent calling and emailing until you eventually annoy a potential customer into buying your product.

I now know that I was full of shit. It takes a very unique set of skills to maintain interest and close deals, and those are skills that cannot be undervalued if a company is to succeed. In my opinion, technical and product proficiency is necessary - but not sufficient - for rapid scaling. While I think founders make great salespeople, if I could do it all over again, I would want to have a BD person on the team on day one.

2. Enterprise Sales Sucks

This is a grab bag of sales lessons.

  1. Many times we would email a potential customer / partner / investor and not hear back for days, send a few follow ups over the coming weeks, and eventually give up. Then, they’d respond out of the blue and mention they’d been working behind the scenes and had great news and we’d be working together on whatever. Non-response isn’t the same as never going to respond.
  2. No matter what email client you use, figure out how to create templates. 80% of the millions of emails I sent over the last 2 years fit into one of a handful of templates. Use email templates.
  3. Maintain relationships. Even if a sale / partnership doesn’t work out, the people you talked to will probably end up somewhere else at some point and might be able to make a sale happen there instead. Keep in touch.

3. Build for Scale

It felt incredibly silly spending tons of time ensuring that our system could handle 10 million hits / hour when there were no customers and no hits so far. It takes some weird combination of hubris and faith to spend a weekend working on automating massive server deployments when you have a single tiny instance running on AWS’s free tier. It seems ridiculous, but I think there’s an immense value to developing with an expectation of success. We didn’t have to rewrite a “prototype” to handle increasing loads - since all of the code from day one was designed as if massive growth were inevitable. It definitely costs more up front (prototypes obviously take less time to develop) - but there’s a way to at least design for growth without having the expectation of needing rewrites. For instance, choose a DB that can scale with you as you grow, instead of “We’ll use DB X since it’s easiest now - until we get to big for it, and then rewrite everything to handle DB Y when we get to that point.”

Automate all the things. 5 minutes a week restarting that service that always needs to be restarted is absolutely worth 30 minutes right now to automate that restart.

Log all of the things. If you aren’t logging/tracking it, you can’t fix it. Track system stats, app stats, user stats, and everything else in between. Track response times, query times, and how many cups of coffee you drink per day. Track it all. Store the stats - storage is cheap. Realizing just now that your response time has gone up 10x over the last month is expensive.

Alert on all bad things. Automated alerts are best when they warn about bad things that will happen soon, rather than alerts that occur when it’s too late. For instance, set up an alert to go off when your SSL certificate expires in one week, not when it has already expired. If you’re getting too many alerts, then they’re either things that should be fixed or your system is too sensitive. If I get a text message from a monitoring service, it means only one thing - I need to take some action. Useless alerts are like the boy that cried shitwolf. Never cry shitwolf.

4. Time / Expected Return Tradeoff: Beware the Time Suck

There will always be more things on your TODO list than you will ever complete, so constant reassessment of prioritization is necessary. Implicitly, it’s some sort of ranking for all possible tasks \(X\) based on the probability of success \(P(x_i = success)\) times the utility value of the successful outcome \(u(x_i)\) divided by the commitment of time necessary \(t(x_i)\), or

\[ \max_X \frac{P(x_i) u(x_i)}{t(x_i)} \]

Since neither \(P\) nor \(t\) are known, this equation is almost meaningless. Good luck.

That said - there was one time in particular when we vastly overestimated the expected value of a successful outcome \(u(x_i)\) and spent way too much time (weeks) working on a particular opportunity. It was a failure and a massive time suck. You won’t know exactly what the payoff will be - but the earlier you can nail that down the more you can understand what’s potentially worth your time.

5. Get Some Office Space

Get some office space ASAP. The bump in productivity is worth the expense.

6. Your Time Estimate is Too Low

I can’t remember a single time any of our time estimates for anything weren’t at least half of what they should have been (if not less). This goes for everything from the big ones like time to close a round and time until the next sale to the little ones like time to features being completed and how long it takes checks to clear.

I don’t remember ever saying, “Boy, that was fast.” Ever.

7. Find Others to Join Your Folie à Plusieurs

We really lucked out with our first adviser. He was there from day one with a strong belief in us and our mission (and I do mean day one, when I said things like “What’s a one-pager?”). Find others (early!) who will share in your vision and provide constant encouragement, because at many points you will feel uncertain about whether what you have is actual vision or mad delusion. It’s incredibly encouraging to have an external voice reminding you of what you’ve accomplished so far and expressing an expectation for future success. It’s not just nice things, though; having someone who is also willing to give you a solid kick if you need to refocus is also helpful. What’s important is that you have that external moral support before there are any major proof points.

Later, once the vision starts turning into reality (and you have happy customers and proof points and actual success stories) it’s easier to find people who can share the vision (and who can help with the doubts you’ll feel even after the successes). It’s critical, however, to find those early (blind) believers - and find them as early as possible.

Those are the big ones. Let me know what I’m missing in the comments!

txque: (Real) Async Background Jobs for Python

Posted 21 Mar 2015 to python, twisted, txque and has Comments

There are a number of Python projects I’m working on (many of them utilizing Twisted) that all would benefit from the ability to enqueue background jobs for workers to chew on. While there are a number of existing background job options for Python (Gearman, RQ, and Celery to name a few), I wanted the ability to seamlessly utilize the batteries of Twisted. Most importantly, I don’t want a worker tied up if a job involves a lot of IO. I want the ability to dispatch jobs asynchronously to multiple machines each running some number of multi-threaded workers that can all handle asynchronous IO.

Since Twisted already handles thread pooling, there really wasn’t that much left to do except create some wrappers for the concepts of jobs, workers, and dispatchers (things that enqueue jobs). The result is named txque - and is dead simple to use.

Here’s the definition of a job:

from import Job

class MyBackgroundJob(Job):
    def run(self, anArgument, akeyword=avalue):
        # Now do some work.  This work can be synchronous, or it can return a deferred.
	someExpensiveFunction(anArgument, akeyword)

Here’s what it looks like to enqueue a job:

from txque.dispatchers.sql import MySQLDispatcher

# create a dispatcher
dispatcher = MySQLDispatcher(user="username", passwd="password", db="txque")

# queue the job, then stop the reactor once queued
dispatcher.queue(MyBackgroundJob('hi there'), queue='default', priority=10)

And then you can run as many workers as you’d like:

twistd -noy worker.tac

Check it out at

Gender and Data: Survey Results

Posted 10 Dec 2014 to ruby and has Comments

As a followup to the post describing my thoughts around renaming the gender_detector gem, I wanted to release the results of the survey at the end (if you haven’t read the original post, you should definitely go back to get a sense of context).

First, there were way more responses then I hoped for (545 total responses!). 15% were female (83), 81% male (444), and 3% (18) provided another gender (which were mostly antagonistic, like “squirrel”). Just over half (283) provided additional comments.

I’ve got some summary stats below - and while I’d love to provide some inferences based on the responses, I’d much rather see what analysis others may do with the data. Leave a comment below if you do a write up, and I’ll add a link in this post.

Question Summaries

When I first heard the name of the gem SexMachine I felt…

For the male respondents, 80% mentioned indifference or happiness (355) and 11% mentioned feeling uncomfortable (50). For females, 77% mentioned indifference or happiness (64) and 14% mentioned feeling uncomfortable (12).

On a scale of 0 to 5, how strongly do you feel that the name should have been changed.

Average for the males was 1.1 and females was 0.7.

Additional comments

About half of the males (226) and half the females (44) provided additional comments. 3% of the males (12) and 2% of females (2) used some form of “thanks”.


  1. The responses are self-reported. There’s no way to tell how many people provided honest responses.
  2. I didn’t clean up the data. For instance, duplicates and invalid responses were not removed (I’m assuming the few squirrels were actually the same squirrel responding multiple times).
  3. While 545 responses was about 500 higher than I was expecting, it’s still a miniscule representation of the entire coding world. Typical warnings about a small n apply.


You can download the response data from google docs. Leave a comment below if you do a write up, and I’ll add a link in this post.

Why I'm Renaming A Gem

Posted 17 Nov 2014 to ruby and has Comments

Almost two and a half years ago, I made a gem that determines someone’s most likely gender based on their first name. I named it SexMachine. It’s a machine that detects the gender (most often, the sex) of a person. The name seemed to fit, and it’s also the name of a song everyone has heard.

Two and a half years later, someone stepped forward to say that the name I chose was harmful.

While the comments on the Github pull request have inevitably spiraled into fatuous inanity, I do think there are a few items that are absolutely worth consideration.

First, to clarify, I don’t think it’s healthy to view the word “sex” with the prudishness that most Americans still seem to posses. Sex is not shameful, and it is not degrading. As to how appropriate of a name it is, I think that depends completely on the context. The question of whether or not the name is appropriate for a work environment isn’t one I think matters that much in this case. I’m not your coworker and my personal projects (made in my spare time with my personal resources) should not be conflated with anyone’s workplace. Github and rubygems are more like flea markets full of free art where you are welcome to take whatever you think is useful, but there’s no guarantee that you’re not going to stumble across a tasteless painting of someone doing something naughty with a banana. That’s part of what makes open source so wonderful - because someone may actually be looking for that exact painting, even if it’s not something you’re willing to hang in your corporate boardroom.

That said, I think there’s a much more convincing argument (that unfortunately has become conflated with workplace fit). I think the best argument basically boils down to a question of whether or not I, as an open source developer, want to be as welcoming as possible to a group that has long felt marginalized and uncomfortable. This is the one that I believe is worth substantial and thoughtful consideration. If enough people from that group claim that some words that I chose make it harder for them, who am I to question their feelings. I believe there is genuine sincerity in the request, and this means that I need to listen.

This is why I’m going to rename the gem. I’ve received a ton of emails and comments from both sides, and I want to make it clear that my decision has nothing to do with feeling the pressure of one group over the other. It’s because of comments like this, from a woman engineer that I corresponded with:

While the idealist in me would love to aim for a world where sex was treated more equally and openly, the unfortunate reality of tech is that it has been a haven for misogynistic men and the environment is heavily male dominated. While in an ideal world the name SexMachine would be something that both genders could joke about, the reality is that the tech community is not ready or capable of that today.

I want to bring about that day as quickly as possible. This is my contribution.

PS - If you have a moment, please take a second to fill out this short, anonymous survey. I’m a scientist and can’t help try to collect some data.

Hubot as an AWS Gatekeeper

Posted 20 Aug 2014 to hubot, aws and has Comments


On AWS, it’s easy to fall into a situation when you’re using the equivalent of a poor man’s VPN. Each time you want to access a server, you go into the AWS console online and authorize some ports in a security group for your current IP address. Then you forget to remove those rules, naturally. What if there was a way to just externally monitor your online presence, and then open access when you show up online and remove that access when you sign off… Like some sort of presence protocol, perhaps with messaging… That’s extensible too.. Like some sort of Extensible Messaging and Presence Protocol

But wait, such a thing exists. And it’s not constrained to XMPP - most other chat protocols have a way to track presence, and most of them can be accessed by Github’s Hubot. So I decided to just build a Hubot script that will open ports in security groups when I sign online, and close them when I leave. Easy.

Installation and usage is easy - see the hubot-aws-sesame github page for more info. The way Hubot can get your IP when you sign online is a bit tricky, but it works. When you sign online, Hubot sends you an image URL for an image serviced by Hubot’s built in web server. Most chat clients will automaticaly load the image, giving Hubot access to your IP. Wooo - easy poort man’s auto-connecting VPN!

Github repo is here, NPM package here.

Stunning: Determining Your Public IP

Posted 17 May 2014 to internet, ruby and has Comments

Programmatically fetching your public IP address (aka, internet visible IP) can be tough. Most often, I’ve done something silly like fetching and then parsing the page. That introduces a pretty bad dependency.

Fortunately, there’s actually a protocol for asking for your internet visible IP called Session Traversal Utilities for NAT (or STUN). It’s used by services that require peer-to-peer connection negotiation (like Skype, Google hangouts, etc).

Here’s an example Ruby script that will hit up a few public STUN servers (starting with Google’s) and return your found IP:

The Internet, 3.0

Posted 18 Mar 2014 to internet, the future and has Comments

So, you think you have a great idea for an online product? Nice. In the brave new world of the new Internet - the steps are easy:

Get a Permit

Submit an application to the governance board. You’ll need a permit, so make sure you describe what the service is / will do in detail. Also, you’ll need to guarantee that you won’t allow hateful language, harassment, or intimidation on your site. Government issued photo id will, of course, be required. Also, it’s a big investment (you must purchase access for 5 year chunks at a time), so make sure your domain is a good one! Finally, you should feel good that a huge chunk of the $10K you’re paying for the domain is going to a Universal Service Fund to help pay for internet infrastructure in remote regions of the world.

Find a Hosting Company

If your application is approved, and the domain you requested isn’t taken, then you’ll be able to pay your fee and will be assigned the domain. Here’s where the fun begins! You now need to choose a hosting provider. Good news though! You have lots of choices - Verizon or Comcast/Time Warner. You could go with an independent host (like AWS) if you really want to, but they will tack on steep fees to cover their costs to connect to the ISPs. Those fees, though, don’t cover your actual bandwidth costs - just the rent that the ISP’s charge so the tubes are actually connected to AWS. The other problem is that there really aren’t too many of them - the ISP’s aren’t very keen on the idea of letting AWS take away it’s own cloud computing (this already exists, no joke), so they try to recoup via “access charges” that are passed on to you the customer.

Submit your domain ownership certifications along with your application and application fees. Once you’re accepted, you’ll be online in no time!

Get Connected

You now need to pick an ISP to provide access to your domain. If you went with a ISP as a host already, then the decision has been made. If you went with an independent hosting provider, the great news is that you have lots of choices! You can choose from Verizon or Comcast/Time Warner. Actually, your choice may be more limited, because ISP’s really prefer exclusive contracts with their hosting providers, so that decision may have already been made for you too (wasn’t that easy?).

Submit your hosting provider’s ID (HPID) along with your unniversal connection number (UCN) to the ISP, along with your access application and associated fees. You’ll need to select your bandwidth class (make sure you pick a big enough pipe - especially if you’re idea is as great as you think!) and access period. Be careful of overage fees if you end up with a huge burst of traffic - this is where the ISP makes a lot of their money (just like overage minutes on your phone bill) - but it’s your fault for not choosing a high enough bandwidth class in the first place. You’ll also need to choose a delivery preference level - this is what affects connection speed and is based on the number of other clients in higher preference levels and the traffic they’re getting. With limited resources - it makes sense that sites that pay more should have less latency.

Just select which contries you’d like to be able to access your site (some may be off-limits depending on the type of content you’re providing), and you’re almost done.

If you want to serve up images, video, or sound - good news! Your ISP has invested in some great media hosting services that you get to utilize (usage is mandatory for all “rich media assets”). Just make sure you picked a good bandwidth class and delivery preference level - media files can be big and really expensive to deliver.

Finally - if you want to accept in-site payments, there’s good news for that too! Your ISP contract gives you the ability (requirement, really) to utilize some great payment gateway infrastructure. There’s a tiered fee structure, but you’ll never pay more than 10% per transaction, and you’ll never have to worry about complicated payment gateways.


Finally, you’re online! Your site won’t load as fast as Facebook’s, but with your awesome idea, you may one day have enough money to kick that speed up a few tiers.

The number of websites will, thankfully, shrink down to a manageable number. Free content sites will be a thing of the past, but with publishing platforms available online (who better to host your content than the same service that delivers it?) you can find something within your price range.

Some public service sites (non-profits, etc.) will get reduced tier access for free or small administrative costs (if this is you, just be warned that application approval times range between 6 months to 2 years).

It’s a brave new world, but if you follow these easy steps, your site will be online in no time.

So You Think You Found a Technical Co-founder

Posted 18 Feb 2014 to business, startups, opbandit, not science and has Comments

A friend of mine is about to launch a new startup, and thinks he found a potential technical co-founder. There’s lots of advice for what to do if you’re a non-technical person looking for a technical co-founder (earn one, stop looking, date, pay, give up and learn to code, etc.). It’s not clear, though, what you should do when you think you’ve found that special someone to share in your folie à deux.

I think there are some basic questions that you need to ask yourself and a different set of questions you should ask your new potential co-founder. As the co-founder of a current startup, these are the questions I asked of my fellow founder and that I asked myself; it’s been almost 2 years and we still haven’t killed each other, so I figure these may be a good place to start.

For the Potential Tech Co-founder

  1. How much time would he be able to devote to something he has a major stake in?
  2. How long can he go at that rate without taking a paycheck?
  3. What do his current obligations look like - kids? wife? parents he takes care of? Obviously, the fewer the better, though one of the most prolific engineers I know has 9 kids.
  4. Will you be physically working together in the same space? If not, how often can you skype/hangout/etc, and do you already have a good working relationship? While distributed teams can certainly succeed, the process of brainstorming / creating can be much easier if you’re face to face.
  5. Does he think your idea is fucking awesome and brilliant and so cool that he can’t wait to start building?
  6. Can he give examples of projects (either in a company or solo) that he thought were brilliant in the start and that he eventually worked on for more than a year? How did he feel at the end of a year? Enthusiasm certainly fades, but excitement needs to have sustainable cycles to keep him (and you!) motivated in the long term.
  7. How does he feel about uncertainty? Could he work on a project for a year without knowing for sure that it will ever be successful (and maintain his sanity)?
  8. Has he ever had a job that lasted more than 6 months/a year/2 years where he worked with the same small group of people every day?
  9. Has he failed at a startup before? A “yes” answer is better than “no” - but it should come with thoughful reasoning about why the company failed.

For Yourself

  1. Do you get along with this guy well enough that you would trust him with the details of your bank account? Your passwords to every service online? If not, what would it take to get to that point?
  2. Can you communicate well enough that you both clearly understand each other (at least most of the time)?
  3. Is he able to hack things together (done right now is better than perfect later) and JSIO? Bascially, is he a motherfucking programmer? Does his github account have more than 10 repositories? If not, then maybe this person would be better at a later stage (when you need a manager who knows a little about engineering).
  4. Does your potential tech co-foudner have the sense to know when he’s out of his depth? Many skillsets are needed when you only have a handful (or 2!) people, but eventually specialization will be necessary (if you’re successful). Will he know when it’s time to bring in someone else to help scale your systems, for instance?

These are, of course, not exhaustive, but hopefully they provide a good starting point.

Good luck!

Kademlia: A DHT in Python

Posted 14 Feb 2014 to python, kademlia, dht and has Comments

A distributed hash table (DHT) is a decentralized dictionary that is comprised of many nodes that each store a portion of a key/value lookup table. Any participating node can write to and read from the entire hash table.

The Kademlia distributed hash table is one of the better known DHT descriptions, and it’s used by BitTorrent for trackerless torrents and by the Gnutella network (originally “LimeWire”).

I couldn’t find a good implementation in Python (that followed the paper and wasn’t buggy), so I wrote one. Naturally, it uses Twisted to provide asynchronous communication. The nodes communicate using RPC over UDP to communiate, meaning that it is capable of working behind a NAT.

The library aims to be as close to a reference implementation of the Kademlia paper as possible.

Check out the code and examples here -

« Older Posts »