Add some more extra redundancy again

It’s the season for coughs and sniffles, and last week I took my turn. I went to bed one night with a stuffy nose, and it got me thinking about software.

What’s the connection between sniffles and software, you ask?

Let’s talk redundancy. It’s a familiar technique in software design, but I believe we compartmentalize it too much under the special topic of “high availability”–as if only when that’s an explicit requirement do we need to pay any attention.

Redundancy can be a big deal. Image credit: ydant (Flickr)

Redundancy in nature

Mother Nature’s use of redundancy is so pervasive that we may not even realize it’s at work. We could learn a thing or two from how she weaves it–seamlessly, consistently, tenaciously–into the tapestry of life.

Redundancy had everything to do with the fact that I didn’t asphyxiate as I slept with a cold. People have more than one sinus, so sleeping with a few of them plugged up isn’t life-threatening. If nose breathing isn’t an option, we can always open our mouths. We have two lungs, not one–and each consists of huge numbers of alveoli that does part of the work of exchanging oxygen and carbon dioxide. Continue reading

Features are not chunks of code

photo credit: nandadevieast (Flickr)

Before the industrial age, “features” were noteworthy aspects of a face or a geography: a patch of color, abundant wrinkles, a scar… The human brain is stunningly good at identifying and comparing such features–perhaps because that ability has been central to our nurture as children, our bonding into family units, and our survival as a species.

When we want to say what a face looks like, we describe its features. They are an entrée into experience with it.

At the dawn of the computer age, the advertising and publishing industries were already talking about how products–or aspects of them–could be “featured” in media. Highlighted characteristics were called “features”, and this metaphor transferred seamlessly into digital language. Software product managers now traffic in “features” and “feature sets.”

We use the term so comfortably that we sometimes forget what it has to teach us.

Continue reading

3 Commandments of Performance Optimization

In my experience, most programmer attitudes on speed fall into one of these categories:


Programmers with this mindset think about performance on occasion, but it’s not a big focus. Occasionally they’re forced to tackle problems because a particular design is too slow, a customer is unhappy, or new scaling requirements materialize. In such cases, they experiment until behavior improves, and then go back to the work they really care about.


Programmers with this mindset have a hard time not thinking about performance. Every design they do reflects elaborate consideration of how to minimize footprint and/or how to complete a task in the shortest possible time. (Note that those two priorities often conflict.) Programmers who are passionate about performance often feel like their laissez-faire counterparts are derelict in their duty.

I don’t think either of these extremes is healthy in all cases. I have seen programmers who chronically think about performance too late,  creating large refactoring burdens and sabotaging their company’s success. Sometimes when you go from “make it work” to “make it fast” you find that all your original work is a waste, because a totally different design (even different tests, conceivably) is the only way forward; I wrote about this in “A Quibble with Martin’s ‘Optimize Later’ Notion“.

On the other hand, it is possible to be too passionate about performance; optimizing the performance of the dev team (by decreasing coding and testing time) is often a better business choice than optimizing execution speed in ways that make code more complex and harder to verify. I have encountered performance zealots disqualifying a perfectly good design on the grounds that it’s not performant enough in a use case that only 2 customers on the entire planet would ever care about. Not smart. As I’ve said many times, good code is balanced.

ThrustSSC — the first car to break the sound barrier. Sometimes speed is the ultimate criterion. However, most money is made on cars with more modest performance requirements. Photo credit: cmglee (Wikimedia Commons)

Let’s assume you buy my criticism of the extremes, and you’re willing to apply the “it depends” doctrine. Continue reading

Metrics, Plumb Lines, and System Thinking

Friday morning I was at a seminar taught by Jason Taylor, CTO at Allegiance. We were discussing how dev team velocity and product quality can compete for our attention; sometimes we trade one for the other. Jason mentioned that he’s a fan of competing metrics, and some neurons connected in my brain.

Plumb line suspended from the center point of multiple balancing legs. Photo credit: suttonhoo (Flickr)

I’m a big believer in measurement. As the old adage goes, you can’t improve what you don’t measure. Next time someone urges you to change a behavior, or tells you she’s going to, ask what measurement of change is being proposed. If you get an unsatisfying answer, I predict you’ll also get an unsatisfying outcome.

I’m also a big believer in balance, as I’ve written about before. Good software balances many considerations.

Besides these existing predispositions, I’d recently read a blog post by Seth Godin, cautioning about the need to choose wisely what we measure. And I’ve been digesting The Fifth Discipline, by Peter Senge, which advocates wholistic, systemic thinking, where we recognize interrelationships that go well beyond simplistic, direct cause-and-effect.

All of these mental ingredients Continue reading

Ken Ebert: Kill three birds.

(A post in my “Role Models” series…)

Killing birds is just a metaphor; who’d want to hurt something this cute? Photo credit: Valerian Gaudeau (Flickr).

Most people optimize for a single outcome.

Ken Ebert is made of smarter stuff.

When I began my career, I was like many engineers–keenly aware of issues like code quality and performance, but not so clued in to customer experience or business priorities. I’d go to team meetings, get new assignments, and head back to my cube to code away. If I was unsure how to accomplish my goals, I’d ask a senior engineer, and do my best to reflect the values they cared about. After all, they were the experts.

It gradually dawned on me that some of my engineer mentors (not many; geeks are better-rounded than popular stereotypes make them out to be…) suffered from a kind of myopia. They could build complicated and highly functional things, but they had little interest in how their constructions mapped into sales, revenue, and company success. I went through a revolution in my perspective, got religion about user experience, and spent a lot of time learning from product management.

In 2003, Symantec acquired several companies in rapid succession, and threw five sites together into a new business unit with marching orders to build a suite and conquer the world. I was tasked with architecture across the picture, and my perspective expanded again. Now it wasn’t just a single product’s success I was worrying about–I had to make or shepherd design decisions that enhanced the profitability of multiple independent products as well as the amalgam of all of them. Was it more important to make codebase X localizable, or to invest in a better install for codebase Y? If team A moved to a newer version of tomcat, would it break the integration story for our suite or impose undue burdens on team B? Resources were constrained (they always are!), time was short…

I began to sense that the way I made decisions needed to change, but it wasn’t until I worked intensely with Ken Ebert, years later, that I could put into words exactly how.

Making wise decisions about architecture, SOP on dev teams, product features, investment in R&D versus marketing–it’s like trying to solve an equation with multiple variables. Profit maximization requires that you consider more than one input at a time. This is partly what I had in mind when I blogged about the need for balance in good code, and when I blogged about optimization.

Many times I’ve heard Ken say something like this: “I think we should make feature N the centerpiece of our next release.” And my first response has been, “That’s doable, but kinda boring. Why not focus on M instead?”

When Ken nods sagely at my question, I know what’s coming.

N is glamorous, and low-risk, but it’s only useful to 3% of our customers. If we build M, we’ll address a sales inhibitor for half the funnel, plus we’ll be positioned to do features P and Q on our long-term roadmap, plus I’ll be able to get an analyst to write about it, plus …, plus …”

Ken doesn’t just kill two birds with one stone. He kills three, or five, or six.

I’m not sure how Ken manages to be so good at this, but here’s what I’ve done to try to approximate his success:

  • Evaluate in multiple time horizons.
  • Evaluate for the channel as well as direct sales.
  • Evaluate for support and professional services as well as dev.
  • Evaluate for marketing value.
  • Evaluate for opportunity cost.
  • Evaluate with an eye to domino effects.
  • Evaluate for morale and momentum.
  • Evaluate for technical debt.

Of course, nobody can do all of these, exhaustively, all the time. Don Kleinschnitz’s wisdom about putting a stake in the ground is an important counterbalance. But I find that I can think, at least briefly, about ramifications of a choice from multiple perspectives–and when I do, I often make better choices.

Thanks for the lesson, Ken.

Action Item

Look at the bulleted list of perspectives above; find one you’ve been neglecting. Pick a significant decision that’s on the horizon, and spend some time thinking seriously about that decision in the context of the neglected perspective. What did you learn?