Skip to content

…it’s not a Unit Test

Riffing on Jeff Foxworthy’s “…you might be a redneck”, I present “…it’s not a Unit Test”:

  • If it requires manual setup… it’s not a Unit Test.
  • If it requires manual intervention… it’s not a Unit Test.
  • If it requires a network connection… it’s not a Unit Test.
  • If it requires a container… it’s not a Unit Test.
  • If it requires a database… it’s not a Unit Test.
  • If it accesses a file system… it’s not a Unit Test.
  • If it requires a service to be available… it’s not a Unit Test.
  • If it takes longer than a millisecond to run… it’s not a Unit Test.
  • If it lacks assertions… it’s not a Unit Test.
  • If it breaks on certain dates or at certain times… it’s not a Unit Test.
  • If it requires a UI… it’s not a Unit Test.
  • If it fails intermittently for an unknown reason… it’s not a Unit Test.
  • If it passes intermittently for an unknown reason… it’s not a Unit Test.
  • If it fails when you look at it funny… it’s not a Unit Test.
  • If the mock setup code dwarfs the actual test code… it’s not a Unit Test.
  • If you have to mock beyond immediate dependencies… it’s not a Unit Test.

I think that’s a pretty good start. What have I missed? Are there any that make you cringe? Let your voice be heard: leave a comment!

Backpacking Blunders

The Cascades are a very photogenic mountain range: a choice location for a first backpacking trip (you can see full resolution versions of these photos and more here). I have done plenty of camping and hiking in my lifetime, but until several weeks ago I had never combined the two. It’s something I’ve always wanted to do. Work has me traveling to Seattle currently and when a couple colleagues started talking about taking some weekend backpacking trips I jumped at the opportunity.

At the trailhead (from left to right): Greg "Hippie" Warren, myself, and Pat

I’m not in terrible shape, I know plenty about camping and hiking, and I’m a pretty smart guy. I was sure I had things figured out. Oh, but don’t worry: this didn’t go as bad as you might now be thinking. By comparison, Hippie is in great shape (long distance runner), and both Hippie and Pat have been on many excursions: they were fine.

Heavy forest canopy and lush ground foliage made for magnificent views.

We got a later start than I had expected and we decided the new guy should set the pace (that would be me, and normally a good idea). I was a man on a mission, knowing we wanted to make 7 – 9 miles in the first day and it was already 3:30 PM. We made the nine miles, but at the expense of my feet.

I'm pretty sure the moss in this forest had plans for total world domination!

You see, I knew (yet ignored) the four most important pieces of backpacking gear: footwear, footwear, footwear, footwear:

  • Shoes: the heel should lock, the toe-box should have ample room for your foot (fail, and fail)
  • Socks: moisture wicking is key to minimizing chances of blisters. This means NO COTTON (MAJOR fail)
  • Lacing: can greatly improve the shoe’s fit. I have narrow heels yet my foot is wide in front. Loosening the laces at the bottom and tightening them at the top would have worked well for me (see the first point)… and, fail
  • Camp shoes: I thought my Vibram’s would be a good idea (so did Hippie). Swollen feet and gloves for your feet are NOT a good combo. Flip flops are great for wearing around camp in the summer and they let your feet breathe.

All of those cascades add up!

I also made what I am sure is every backpackers first blunder: I brought way too much. I bought a great pack, then filled it with a bunch of heavy stuff. This has led me to my fifth backpacking axiom (after The Four Footwears):

When preparing for a backpacking trip lay everything out that you think you need, and eliminate everything you are not certain you’ll need. Then, get rid of half of what’s left.

Pat has since introduced me to the backpacking big three. I already had an ultralight pack (2 lbs, 3 oz.). I have since purchased an ultralight tent (2 lbs. 10 oz. with footprint) and sleeping bag (3 lbs.). Add to that eliminating a lot of extra stuff and my pack will be significantly lighter next time. Oh, and I bought some new boots that actually fit, so no bruises and blisters next time. Much of the gear I got on clearance, though I would have bought most of it anyway. I learned my lesson the hard way! I estimate I was carrying a total of 270 lbs. (body weight, plus 50 lbs. of gear, food and water). I’m pretty sure I’ll weigh in at 250 lbs. this weekend when we head into the William O. Douglas Wilderness near Mt. Rainier.

I really enjoyed my first backpacking trip despite barely being able to walk by the end (quite literally), and I learned a LOT. I’m sure this next trip will be even more enjoyable… and I’ll probably learn a ton again, which will make it even better!

Focused on Fast

I love the counterintuitive. Fortunately software development provides many opportunities to observe the phenomena. Of course it can be frustrating at times, but it is fascinating once you realize what is really going on.

A while ago I blogged about Jerry Weinberg’s Pickle Principle. The very next section of his book containing that anecdote tells the story of two brothers, Roamer and Homer. The basic premise is that Roamer wanted a quiet life at home and Homer wanted to see the world, but both of them ended up with the exact opposite of what they strove for. Jerry boils down the essence of this story in a paradox: “the best way to lose something is to struggle to keep it.” Thinking about this, I realized that software development projects experience this when it comes to delivery dates.

The only time I can remember having an indefinite period of time to finish a project was on the amortization application my friend and I worked on for a local accountant. We were in the 9th and 10th grade at the time and we had approached him about giving us an interesting problem to solve. In every project since that one, though, there has been time pressure and very often the team has succumbed to the temptation to focus on going fast. Invariably this ended poorly. In the worst cases (yes, more than one) the project was canceled with dozens of man-years expended and millions of dollars lost.

Think about your own experiences. How many times have you experienced situations where the push was to go fast? How many of those undertakings succeeded at going fast? How many of them succeeded at all? Probably not many. Yet the focus on going fast remains.

Recently I wrote the following on a team Retrospective card: “We are focused on going fast when we should be focused on going well.” I’m not sure where I first heard this phrase but it has stuck with me. Focusing on going fast will, at best, slow things down. At worst the focus on speed will derail the effort entirely.

Paradoxically, a focus on going well means regularly taking time to stop, step back and reflect. Whenever I succumb to the rush to make progress I find that I am slowed by missteps, miscalculations and misunderstandings. Whenever I focus on going slow and methodically, taking time to learn in the moment, I find that progress occurs apace.

So the next time you hear the crack of a whip and feel the pressure to redouble your efforts, consider stepping back and taking stock of the situation instead of running headlong into the unknown. I think you’ll find, as I have, that going fast occurs as a happy side effect of remaining circumspect.

EC2 Automation

A new colleague and friend of mine, Cosmin and I were talking at lunch the other day about EC2. He has been playing with AWS and EC2 for longer than I have and had some great suggestions. One of them was to prefer automating EC2 Instance creation and setup with shell scripts over using Instance Snapshots, because:

  • You don’t have to remember all the nit-picky details of getting the new instance configured exactly the way you like it
  • Every minute detail will happen the same every time
  • The script serves as a reminder of every step taken to create, start and configure the instance
  • You don’t have to pay the AWS charges for Snapshot storage!

So I started building an EC2 instance creator script. So far it successfully spawns a new instance, assigns the instance a static IP address, and SSH’es into the new instance: all by typing a single command at the command prompt! Here is the script:

#!/bin/bash

# Authorize TCP, SSH & ICMP for default Security Group
#ec2-authorize default -P icmp -t -1:-1 -s 0.0.0.0/0
#ec2-authorize default -P tcp -p 22 -s 0.0.0.0/0

# The Static IP Address for this instance:
IP_ADDRESS=$(cat ~/.ec2/ip_address)

# Create new t1.micro instance using ami-cef405a7 (64 bit Ubuntu Server 10.10 Maverick Meerkat)
# using the default security group and a 16GB EBS datastore as /dev/sda1.
# EC2_INSTANCE_KEY is an environment variable containing the name of the instance key.
# --block-device-mapping ...:false to leave the disk image around after terminating instance
EC2_RUN_RESULT=$(ec2-run-instances --instance-type t1.micro --group default --region us-east-1 --key $EC2_INSTANCE_KEY --block-device-mapping "/dev/sda1=:16:true" --instance-initiated-shutdown-behavior stop --user-data-file instance_installs.sh ami-cef405a7)

INSTANCE_NAME=$(echo ${EC2_RUN_RESULT} | sed 's/RESERVATION.*INSTANCE //' | sed 's/ .*//')

times=0
echo
while [ 5 -gt $times ] && ! ec2-describe-instances $INSTANCE_NAME | grep -q "running"
do
times=$(( $times + 1 ))
  echo Attempt $times at verifying $INSTANCE_NAME is running...
done

echo

if [ 5 -eq $times ]; then
echo Instance $INSTANCE_NAME is not running. Exiting...
  exit
fi

ec2-associate-address $IP_ADDRESS -i $INSTANCE_NAME

echo
echo Instance $INSTANCE_NAME has been created and assigned static IP Address $IP_ADDRESS
echo

# Since the server signature changes each time, remove the server's entry from ~/.ssh/known_hosts
# Maybe you don't need to do this if you're using a Reserved Instance?
ssh-keygen -R $IP_ADDRESS

# SSH into my BRAND NEW EC2 INSTANCE! WooHoo!!!
ssh -i $EC2_HOME/$EC2_INSTANCE_KEY.pem ubuntu@$IP_ADDRESS

I’ve been spending a lot of time creating and terminating EC2 instances, as well as having instances up and running for hours at a time (30 hours total over the last couple weeks if I’m reading the report correctly. The cost for all this experimenting? Ninety-Seven CENTS!

You’ll notice this script uses ec2-run-instances with the –user-data-file parameter, and that is where the magic happens. Once the instance is running, the script you pass in that parameter is automatically run. I intend to use that script to:

  • Setup my home directory structure
  • Run a package installer (in this case, yum) to add everything I need from the repository
  • Any other miscellaneous setup like creating users, assigning permissions, setting up daemons and starting them, etc.

My intention is to add a public project to github to accompany the gist cited above once I have this second script in decent shape. For now, feel free to poke holes in the script above. I do a couple quirky things, but I have reasons for doing them the way I did. Feel free to add your questions or suggestions for improvements in the comments section below!

nodeJS: Beginnings

So two posts ago I documented my first foray into nodeJS. The result was a simple HTTP server that served one line of static HTML. All I really accomplished was to prove to myself that nodeJS was installed correctly, I could copy/paste six lines of code, and that the copied code would actually create a working HTTP server. Of course I was ecstatic when it worked… a bit giddy, even! :)

With the aroma (stench?) of success still lingering I plunged in again, this time following along with Ryan Dahl‘s 2009 JSConf presentation. Of course, nodeJS has changed a lot in the last 14 months so getting his presentation examples to work took a bit more than copy/paste skills. It made for an excellent playground in which to learn nodeJS basics.

Having finished the video and finding success in getting each of Ryan’s demos to work in the latest nodeJS (v0.3.6-pre) I ended up with an excellent beginner’s showcase: 7 example nodeJS scripts showing off non-blocking, long-poll, file IO, streaming IO, TCP servers and HTTP servers. Each one of these scripts can be run with the simple command “node <script_name>”. The gist of my experience (sorry… I couldn’t resist) can be found below: well, at least 2 of the seven examples. You’ll have to visit this gist on github if you’re interested in seeing all 7 examples along with comments containing directions and expected response details for each script.

My next step will be to drive nodeJS services through tests, ideally using the excellent BDD framework… Jasmine!

var sys = require('sys');

setTimeout(function () {
  sys.puts('nodeJS!');
}, 2000);

sys.puts('Hello');

var puts = require('sys').puts;

setInterval(function () {
  puts('Hello!');
}, 500);

process.addListener('SIGINT', function () {
  puts('\nGoodbye!');
  process.exit(0)
});

var net = require('net');

var server = net.createServer(function(socket) {
  socket.write('Hello... and Goodbye!\n');
  socket.end();
});

server.listen(8000);

var stat = require('fs').stat,
    puts = require('sys').puts;

var promise = stat('/etc/passwd', function(error, stats) {
  if (error) throw error;
  puts('\n/etc/passwd last modified ' + stats.mtime + '\n');
});

var http = require('http');

http.createServer(function (request, response) {
  response.writeHead(200, {'Content-Type': 'text/plain'});
  response.end('Hello World!\n');
}).listen(8124, "127.0.0.1");

var http = require('http');

http.createServer(function (request, response) {
  response.writeHead(200, {'Content-Type': 'text/plain'});
  response.write('Hello\n');

  setTimeout(function() {
    response.write('World!\n');
    response.end();
  }, 2000);
}).listen(8124, "127.0.0.1");

var puts = require('sys').puts;
var spawn = require('child_process').spawn;

var cat = spawn('cat');

cat.stdout.on('data', function(data) {
  if (data) puts(data);
});

cat.stdin.write('Hello ');

setTimeout(function() {
  cat.stdin.write('nodeJS!');
  cat.stdin.end();
}, 2000);

EC2: Important Safety Tip

As I was doing research and working to get my EC2 instance up and running, I ran across a thread (I can’t remember where) that outlined a potentially frustrating scenario. Fortunately I escaped learning this lesson in my normal mode and learned it from someone else’s mistake!

There are two ways to shut down a running EC2 instance: “Stop” and “Terminate”. I had only used “Stop”, which leaves the instance in tact but not using any CPU cycles. That way you can easily start the instance later while avoiding the hourly charge for an instance if you aren’t using it.

Here is the important safety tip: do not use “Terminate” until you’ve created a snapshot of your instance. “Terminate” means… terminate, as in “with extreme prejudice.” If you terminate your instance, it’s gone. If you didn’t create a snapshot of your instance, you will have to rebuild it from scratch.

So, prefer “Stop” over “Terminate” and, once you have a working instance, take a snapshot of it so you can get it back when your instance inadvertently gets terminated.

’nuff said.

nodeJS

While my last post cast a wide net, this post will start the process of focusing in on just a couple things: first nodeJS and later, Angular.

So what is nodeJS? Based on my (very little) experience, it is a tool for building lightweight, highly-concurrent, non-blocking, asynchronous services in JavaScript. It supports many common protocols like TCP, HTTP, HTTPS, TLS(SSL), DNS, IP/UDP, and TTY. Its elegance is in “hiding” the event loop and threading, and in it’s lack of support for parallel processing. Node’s philosophy is to handle parallelism by spawning new OS processes. A single node process generally handles many hundreds or even thousands of connections (depending on how much work they do). Once a node process is saturated, scaling is achieved through starting another process. In Ryan Dahl’s words, “a single node handles many concurrent connections, but it does no parallel work. Multiple Nodes are required to do parallel processing, effectively forcing a shared-nothing model with process boundaries.”

This approach appeals to me. I have always felt that a loose confederation of small, simple services that can be aggregated to provide more complex offerings is a much better approach than creating a single, monolithic web server that handles every service request an enterprise can imagine (and creates a single point of failure). Using nodeJS, services can be independent which is more robust, easily scales by spooling up more processes either on the same machine or different machines, and makes the overall system much easier to extend and maintain. This also makes nodeJS an excellent tool for cloud-based services, where scaling is accomplished by spooling up new instances of virtual machines on an as-needed basis.

Having said all of that, I’m not proposing nodeJS is the perfect fit for any situation. However, I do think that just about any organization would find it a valuable tool, even if just used for a couple simple services. I can envision a migration path where some of the simpler enterprise services are rewritten as individual node processes and removed from their monolithic web server.

Here is the list of nodeJS resources I promised in my last post:

In my last post, I also promised the code for my first HTTP server written in nodeJS (it is very close to the example provided on the nodeJS web site):

var http = require('http');

http.createServer(function (request, response) {
  response.writeHead(200, {'Content-Type': 'text/plain'});
  response.end('Hello NodeJS!');
}).listen(80);

console.log('Server running on port 80');

If you’re running node on an EC2 instance (as I outlined in my last post) and want to use port 80, you’ll have to “sudo node <filename>”. A non super-user doesn’t seem to have permissions to start a service on that port. If you’d rather run the server locally you’ll need to change line 6 above to:

}).listen(8000, '127.0.0.1');

Of course you’ll also need to append the port number to the URL in your browser: http://127.0.0.1:8000.

In my next post I’ll start exploring some of the details of nodeJS, demonstrating non-blocking, TCP, HTTP, File IO, and more!

Shiny New Tech

I have been playing with some new toys in the JavaScript space. It started over a year ago with my interest in using BDD to design and code JavaScript for the browser. I started with JSpec and made the transition to Jasmine. Along the way I puttered around in JQuery and even a bit in extJS.

I then briefly experimented with Cappuccino and SproutCore, but found JavaScript on the server and it’s continued acceptance as a first-class language more intriguing. Mozilla’s Rhino hit my radar, then envJS. My latest find on the server side is nodeJS.

This week at Geeknight Dallas I was introduced to yet another JavaScript tool: Angular. Add to all of this a long-standing desire to dive into Amazon’s AWS (specifically wanting to set up an EC2 instance) and I’ve created a pretty nasty technology soup! It’s high time to narrow my focus.

Over the last couple days I’ve set up an EC2 Linux Micro instance to start playing with a nodeJS server. Over the next several posts I’ll be chronicling my journey, starting with this list of things I did to get my EC2 instance set up:

  • I used the AWS Console to create my “t1.micro” instance:
    • Selected the default security group, which didn’t have SSH enabled
    • Added SSH through the Security Groups section of the AWS Console
    • Copied my public key to my .ssh directory
    • Selected “Connect” under Instance Actions in AWS Console to get an example of the command I needed.
    • Found out that the example shows logging in as root which doesn’t work: use ec2-user instead.
  • Once I was able to SSH into my EC2 instance it was time for some installs. Fortunately the Micro instance is based on RedHat and has yum installed, as well as Ruby, Python and other things. I added the following:
    • sudo yum install git
    • sudo yum install gcc
    • sudo yum install gcc-c++
    • sudo yum install openssl-devel
    • git clone git://github.com/ry/node.git ~/src/node
    • cd ~/src/node
    • ./configure
    • make
    • sudo make install

At this point I was able to type “node -v” in my ssh instance and see “v0.3.6-pre.” Success!

Coming up next time: a nice list of nodeJS resources to study, as well as a quick and dirty http server written in nodeJS (from their web site) which serves one page: “Hello World!”

Yes, I spent two days to put a web site up on the internet that only serves one static page. Ah, but it’s only the beginning!

Agile Blind Spots, Part 3

or Prematurely Optimizing Development Teams

It has been a while (almost a year) since I started my “Agile Blind Spots” series. If you’re like me, you may need a memory refresher: here is the first article, and here is the second article. If you’re also lazy like me, the basic gist of these articles was that, as of a year or so ago, I was finding Agile adoption:

  • strong in individual development teams
  • starting to push upstream, dealing with how work gets identified, sized and prioritized before being handed to a development team
  • weak in pushing downstream to improve the substantive amount of work that occurs (at least in larger organizations) to get a project into production after the development team is “done”

I dealt with the first two points in the first two articles, and left the third point unaddressed. I had made peace with the fact that this series would be left unfinished… until the last couple weeks. The impetus for this change of heart? A relatively new buzzword: DevOps.

I may follow up this article with some of my ideas around DevOps, but something else occurred to me while having this particular line of thought re-awakened. In my experience, Agile adoption in many organizations has taken a myopic view of success: make the development teams better and everything will be better.

One of the worst disconnects I’ve personally witnessed ended up creating a six month backlog of production deployments: the development teams were cranking out high quality releases only to have them sit and rot because Operations couldn’t keep up with the demand. The irony here is deep: while Agile proponents trumpeted the triumph of teamwork over individualism, the actual project was a failure: it wasn’t getting promoted to production for months after the victory was claimed! Then it hit me: this is a case of Premature Optimization.

Yes, I know this is normally applied at the coding level, but the premise works equally well for the process of getting a gleam in one’s eye to a tangible, usable product. The tie-in with Lean and the TPS are equally obvious: if you’re not working on solving your biggest bottleneck, it’s Muda. To my mind, the development teams became the biggest bottleneck for many organizations sometime in the early-to-mid 1990′s. The Snowbird meeting was an acknowledgment that too many software projects were ending in failure. Even though there was probably no discussion of Lean or the TPS, I think those involved in the discussion believed they were seeking a solution to the biggest problem: seeking to remove the biggest bottleneck. (EDIT: Please see my note in the comments for clarification on this paragraph)

So your organization has turned your rag-tag group of software developers into high performing Agile teams. Congratulations! You’ve removed a bottleneck. Now where is your organization’s biggest bottleneck? I’m guessing it’s not the development team anymore. One of the prime candidates is Operations, but it could by anything. If you’re still focused on improving yesterday’s bottleneck, it’s Premature Optimization and it’s waste.

So what are your organization’s biggest bottlenecks? How do you know? Are they prioritized? Are you focusing your primary efforts on the worst bottleneck? If you’re not, maybe it’s time that you did.

BDD & Javascript, Feedback Needed

Lately, the teams I’ve been coaching have been looking to get Unit Test coverage around their Javascript code. This has resulted in the following approach:

  1. Extract the Javascript code from a JSP page and save it in its own .js file
  2. Add test coverage using Jasmine
  3. Remove said Javascript from the JSP, replacing the embedded code with an import of the new .js file
  4. Refactor the .js file with impunity

Having repeated this exercise a few times now, I’ve found adding test coverage to be very easy due to the dynamic nature of Javascript: very easy, and a bit disconcerting.

Let me share an example. Here is the function that needed test coverage, followed by the Jasmine test:

function resetForm() {
  document.getElementById("searchValue").disabled=true;
  document.getElementById("filterValue").disabled=true;
  document.someComplexForm.reset();
}

view raw reset_form.js This Gist brought to you by GitHub.
describe('Reset Form to Default Values', function() {

  it('returns the form to the way it was when first loaded', function() {
    addTagToHtmlBody('<div id="searchValue"></div>');
    addTagToHtmlBody('<div id="filterValue"></div>');

    document.someComplexForm = new MockComplexForm();
    spyOn(document.someComplexForm, 'reset');

    resetForm();

    expect(document.getElementById("searchValue").disabled).toBeTruthy();
    expect(document.getElementById("filterValue").disabled).toBeTruthy();
    expect(document.someComplexForm.reset).toHaveBeenCalled();

    function addTagToHtmlBody(new_tag) {
      $(new_tag).appendTo('body');
    }

    function MockComplexForm() {
      this.reset = function() { /* Does something interesting that is tested elsewhere */ }
    }
  });
});

Whether or not reset_form.js is “good” Javascript is not my primary concern. It is more about my approach to getting test coverage around the code. On the actual html page, searchValue and searchFilter are Text Boxes, and someComplexForm is… well, a form. However, in the test I just make sure the DOM has these elements defined without being explicit about what they are.

Here is my thinking: I don’t really care what goes on with those other entities. My only concern is that whenever resetForm() is called, searchValue and searchFilter have their “disabled” property set to “true” and the “reset” function is called on the object someComplexForm. The dynamic nature of Javascript allows me to define those things as simply existing, not caring what type they are or even what properties they have (like the “disabled” property, which is never defined in the setup of the test).

Is this a good approach to testing Javascript? It seems like a good test to me, but maybe I’m taking too many liberties. Am I relying too much on the dynamic nature of Javascript? Or is this a test that’s just silly (i.e. completely unnecessary)?

I’m not confident I have the right answer to any of these questions. I’m hoping my readership will have some helpful hints now that I’ve made my ignorance a matter of public record.

So, what am I missing? What would make this a better test?