On AWS, it’s easy to fall into a situation when you’re using the equivalent of a poor man’s VPN. Each time you want to access a server, you go into the AWS console online and authorize some ports in a security group for your current IP address. Then you forget to remove those rules, naturally. What if there was a way to just externally monitor your online presence, and then open access when you show up online and remove that access when you sign off… Like some sort of presence protocol, perhaps with messaging… That’s extensible too.. Like some sort of Extensible Messaging and Presence Protocol…
But wait, such a thing exists. And it’s not constrained to XMPP - most other chat protocols have a way to track presence, and most of them can be accessed by Github’s Hubot. So I decided to just build a Hubot script that will open ports in security groups when I sign online, and close them when I leave. Easy.
Installation and usage is easy - see the hubot-aws-sesame github page for more info. The way Hubot can get your IP when you sign online is a bit tricky, but it works. When you sign online, Hubot sends you an image URL for an image serviced by Hubot’s built in web server. Most chat clients will automaticaly load the image, giving Hubot access to your IP. Wooo - easy poort man’s auto-connecting VPN!
Programmatically fetching your public IP address (aka, internet visible IP) can be tough. Most often, I’ve done something silly like fetching whatismyip.com and then parsing the page. That introduces a pretty bad dependency.
Fortunately, there’s actually a protocol for asking for your internet visible IP called Session Traversal Utilities for NAT (or STUN). It’s used by services that require peer-to-peer connection negotiation (like Skype, Google hangouts, etc).
Here’s an example Ruby script that will hit up a few public STUN servers (starting with Google’s) and return your found IP:
So, you think you have a great idea for an online product? Nice. In the brave new world of the new Internet - the steps are easy:
Get a Permit
Submit an application to the governance board. You’ll need a permit, so make sure you describe what the service is / will do in detail. Also, you’ll need to guarantee that you won’t allow hateful language, harassment, or intimidation on your site. Government issued photo id will, of course, be required. Also, it’s a big investment (you must purchase access for 5 year chunks at a time), so make sure your domain is a good one! Finally, you should feel good that a huge chunk of the $10K you’re paying for the domain is going to a Universal Service Fund to help pay for internet infrastructure in remote regions of the world.
Find a Hosting Company
If your application is approved, and the domain you requested isn’t taken, then you’ll be able to pay your fee and will be assigned the domain. Here’s where the fun begins! You now need to choose a hosting provider. Good news though! You have lots of choices - Verizon or Comcast/Time Warner. You could go with an independent host (like AWS) if you really want to, but they will tack on steep fees to cover their costs to connect to the ISPs. Those fees, though, don’t cover your actual bandwidth costs - just the rent that the ISP’s charge so the tubes are actually connected to AWS. The other problem is that there really aren’t too many of them - the ISP’s aren’t very keen on the idea of letting AWS take away it’s own cloud computing (this already exists, no joke), so they try to recoup via “access charges” that are passed on to you the customer.
Submit your domain ownership certifications along with your application and application fees. Once you’re accepted, you’ll be online in no time!
You now need to pick an ISP to provide access to your domain. If you went with a ISP as a host already, then the decision has been made. If you went with an independent hosting provider, the great news is that you have lots of choices! You can choose from Verizon or Comcast/Time Warner. Actually, your choice may be more limited, because ISP’s really prefer exclusive contracts with their hosting providers, so that decision may have already been made for you too (wasn’t that easy?).
Submit your hosting provider’s ID (HPID) along with your unniversal connection number (UCN) to the ISP, along with your access application and associated fees. You’ll need to select your bandwidth class (make sure you pick a big enough pipe - especially if you’re idea is as great as you think!) and access period. Be careful of overage fees if you end up with a huge burst of traffic - this is where the ISP makes a lot of their money (just like overage minutes on your phone bill) - but it’s your fault for not choosing a high enough bandwidth class in the first place. You’ll also need to choose a delivery preference level - this is what affects connection speed and is based on the number of other clients in higher preference levels and the traffic they’re getting. With limited resources - it makes sense that sites that pay more should have less latency.
Just select which contries you’d like to be able to access your site (some may be off-limits depending on the type of content you’re providing), and you’re almost done.
If you want to serve up images, video, or sound - good news! Your ISP has invested in some great media hosting services that you get to utilize (usage is mandatory for all “rich media assets”). Just make sure you picked a good bandwidth class and delivery preference level - media files can be big and really expensive to deliver.
Finally - if you want to accept in-site payments, there’s good news for that too! Your ISP contract gives you the ability (requirement, really) to utilize some great payment gateway infrastructure. There’s a tiered fee structure, but you’ll never pay more than 10% per transaction, and you’ll never have to worry about complicated payment gateways.
Finally, you’re online! Your site won’t load as fast as Facebook’s, but with your awesome idea, you may one day have enough money to kick that speed up a few tiers.
The number of websites will, thankfully, shrink down to a manageable number. Free content sites will be a thing of the past, but with publishing platforms available online (who better to host your content than the same service that delivers it?) you can find something within your price range.
Some public service sites (non-profits, etc.) will get reduced tier access for free or small administrative costs (if this is you, just be warned that application approval times range between 6 months to 2 years).
It’s a brave new world, but if you follow these easy steps, your site will be online in no time.
A friend of mine is about to launch a new startup, and thinks he found a potential technical co-founder. There’s lots of advice for what to do if you’re a non-technical person looking for a technical co-founder (earn one, stop looking, date, pay, give up and learn to code, etc.). It’s not clear, though, what you should do when you think you’ve found that special someone to share in your folie à deux.
I think there are some basic questions that you need to ask yourself and a different set of questions you should ask your new potential co-founder. As the co-founder of a current startup, these are the questions I asked of my fellow founder and that I asked myself; it’s been almost 2 years and we still haven’t killed each other, so I figure these may be a good place to start.
For the Potential Tech Co-founder
- How much time would he be able to devote to something he has a major stake in?
- How long can he go at that rate without taking a paycheck?
- What do his current obligations look like - kids? wife? parents he takes care of? Obviously, the fewer the better, though one of the most prolific engineers I know has 9 kids.
- Will you be physically working together in the same space? If not, how often can you skype/hangout/etc, and do you already have a good working relationship? While distributed teams can certainly succeed, the process of brainstorming / creating can be much easier if you’re face to face.
- Does he think your idea is fucking awesome and brilliant and so cool that he can’t wait to start building?
- Can he give examples of projects (either in a company or solo) that he thought were brilliant in the start and that he eventually worked on for more than a year? How did he feel at the end of a year? Enthusiasm certainly fades, but excitement needs to have sustainable cycles to keep him (and you!) motivated in the long term.
- How does he feel about uncertainty? Could he work on a project for a year without knowing for sure that it will ever be successful (and maintain his sanity)?
- Has he ever had a job that lasted more than 6 months/a year/2 years where he worked with the same small group of people every day?
- Has he failed at a startup before? A “yes” answer is better than “no” - but it should come with thoughful reasoning about why the company failed.
- Do you get along with this guy well enough that you would trust him with the details of your bank account? Your passwords to every service online? If not, what would it take to get to that point?
- Can you communicate well enough that you both clearly understand each other (at least most of the time)?
- Is he able to hack things together (done right now is better than perfect later) and JSIO? Bascially, is he a motherfucking programmer? Does his github account have more than 10 repositories? If not, then maybe this person would be better at a later stage (when you need a manager who knows a little about engineering).
- Does your potential tech co-foudner have the sense to know when he’s out of his depth? Many skillsets are needed when you only have a handful (or 2!) people, but eventually specialization will be necessary (if you’re successful). Will he know when it’s time to bring in someone else to help scale your systems, for instance?
These are, of course, not exhaustive, but hopefully they provide a good starting point.
A distributed hash table (DHT) is a decentralized dictionary that is comprised of many nodes that each store a portion of a key/value lookup table. Any participating node can write to and read from the entire hash table.
I couldn’t find a good implementation in Python (that followed the paper and wasn’t buggy), so I wrote one. Naturally, it uses Twisted to provide asynchronous communication. The nodes communicate using RPC over UDP to communiate, meaning that it is capable of working behind a NAT.
The library aims to be as close to a reference implementation of the Kademlia paper as possible.
Check out the code and examples here - github.com/bmuller/kademlia.
Gurbanguli (President of Turkmenistan)
Toquen combines Capistrano 3, Chef, and AWS instance tags into one bundle of joy. Instance roles are stored in AWS tags and Toquen can suck those out, put them into data bags for chef, and create stages in capistrano. You can then selectively run chef on individual servers or whole roles that contain many servers with simple commands.
A Toque is a chef’s cap. Chef + cap = Toque.
Toquen is a gem - and simply extends capistrano with tasks to make AWS tag information (roles, names, etc) available both within Chef as well as in capistrano as stages. You can then run chef-solo on single servers, all servers with a given role, or on all servers. For instance, the following command will create all relevant stages for capistrano as well as create a servers data bag:
And then will run chef-solo on all machines:
cap all cook
There’s also a bootstrapping feature that can be run to initialize a server. The process will:
- Update all packages
- Sets the hostname to be whatever you set as value for the Name tag in AWS
- Set ruby 1.9.3 as the default ruby
- Install rubygems
- Install the chef and bundler gems
Right now the only supported distributions are Ubuntu and Debian, but alternatives like Redhat could easily be added by creating additional templates for the bootstrapping script.
Before beginning, you should already understand how chef-solo works and have some cookbooks, roles defined, and at least a folder for data_bags (even if it’s empty). The rest of this guide assumes you have these ready as well as an AWS PEM key and access credentials.
Generally, it’s easiest if you start off in an empty directory. First, create a file named Gemfile that contains these lines:
source 'http://rubygems.org' gem 'toquen'
Then, create a file named Capfile that contains the following line:
And then on the command line execute:
bundle cap toquen_install
This will create a config directory with a file named deploy.rb. Edit this file, setting the location of your AWS key, AWS credentials, and chef cookbooks/data bags/roles.
Then, in AWS, create an AWS instance tag named “Roles” for each instance, using a space separated list of chef roles as the value. The “Name” tag must also be set or the instance will be ignored.
This will create a data_bag named servers in your data_bags path that contains one item per server name, as well as create stages per server and role for use in capistrano.
At this point you can run chef-solo using the cook task:
# one server cap server-<server name> cook # Or a all the servers with a given role cap <role name> cook # Or on all servers cap all cook
There are a few alternatives (including toque and other toque) out there - but most haven’t yet moved to the magic available in capistrano 3 and none can pull roles out of AWS. Toquen is small and delightful and will play nice if you already have a ton of cap tasks.
To see the rest of the docs, check out the toquen github page.
It’s been such a long time, and I finally got around to releasing a new version of mod_auth_openid (downloadable here). It has dual support for both Apache 2.2 and 2.4. Apache 2.4 support was a long time coming (thanks to osteenbergen for the help).