Big Crud Isn’t Big Data

“Big Data” is another one of those buzz words that seems to be everywhere these days. We hear stories regularly about how fast the world’s data grows and how big it’s going to be by 20xx. Vendors then reason that we should buy their wares to cope. This infographic is typical:


I have several deep professional connections to big data[1], going back decades, so when I say I think a lot of it is manufactured silliness, I’m hoping you’ll pause before laughing me off.

The fact is, most of the “data” that’s exploding is not hard-won intellectual treasure for the ages; it’s marginal stuff like the viewing history on Fred Flintstone’s deleted Netflix account. More than big data, we’re experiencing a “big crud” wave, because we’re pack rats. This comic has it right: Continue reading

Do Androids Browse (For Electric Sheep)?

The movie Blade Runner is based on a Philip K. Dick short story entitled “Do Androids Dream of Electric Sheep?

Perhaps some new questions should be added to this classic…

In an interesting example of science fiction becoming reality, a group of researchers is now creating a sort of world wide web for the robots of the world. Whether or not androids dream, they may soon be able to use social networks for robots, and use public, internet-accessible resources to get their day-to-day work done. The initiative is called “RoboEarth“:

I believe this sort of technological evolution is the wave of the future. It represents a promising confluence of cloud computing, distributed architecture, big data, hadoop-like map-reduce, supercomputing, ubiquitous internet connectivity, and the every-device-has-an-IP-address promise of IPv6. It would be nice if my next Roomba didn’t have to relearn the floorplan of my house, but could simply download knowledge that the older model has laboriously developed. I’ll bet over the next decade, the market will discover hundreds of variations on that theme.

I just hope we’re smart enough to stop before robots start frittering away their time clicking cows on Facebook… :-)

The Power of Simplicity

Most stories about zen masters, gurus, or other paragons of wisdom follow a similar pattern. The pupil discovers a problem. He or she struggles with it. The problem gets more and more overwhelming. Solutions are elusive. Finally the pupil goes to the master and pours out his heart, whereupon the master offers a pearl of insight that radically reinterprets the problem.

Seek the simple... Photo credit: departing(YYZ) (Flickr)

Seek the simple… Photo credit: departing(YYZ) (Flickr)

There’s a reason why this narrative exists in every culture: human beings need the insight that comes from synthesis, pared down to its essence. We crave the simple but profound:

  • Less is more.
  • Do unto others as you’d have others do unto you.
  • A watched pot never boils.
  • Freedom isn’t free.

The software industry desperately needs this sort of insight, but far too often I see us operate in the stage of the narrative where the pupil misunderstands the problem, struggles, and makes things worse. Continue reading


Recently I’ve been digesting Start With Why, by Simon Sinek (Another nice find, Trev!) For an overview, watch his TED talk.

Maslow’s hierarchy of needs. A person’s “why” can derive from any of these levels, but I think I’ll be happiest if I can map mine to the top of the pyramid.

I don’t buy everything in the book, and I think many of the author’s assertions would be stronger if he offered detailed evidence. For example, he asserts that Apple is special because it has a driving “why” that shapes company decisions in fundamental and pervasive ways, and glosses over how Apple lost its vision during the non-Steve-Jobs years.

Why “why”?

Nonetheless, his central premise is insightful and important: starting with “why” we do things will create greater satisfaction, wisdom, and success. Once we have a why that’s correct in our hearts, many of the whats and hows of life flow naturally. Sacrifice, patience, and creativity blossom. This is true of our software architectures, product requirements, general business activities, personal relationships, and other endeavors. People who hate their jobs often feel that way because they find their days filled with activities that they deem empty or soulless. Human beings can’t be passionate about stuff that doesn’t resonate with their deep values to some degree.

I find interesting resonance between Sinek’s thesis and other important ideas such as Jim Collins’ hedgehog concept, the idea of laddering in marketing theory, Maslow’s hierarchy of needs, and Covey’s “begin with the end in mind.” I also find it interesting how congruent this is with a central tenet of Bridging the Communication Gap, by Godjko Adzic; he argues that the “why” of a product requirement, rather than the “what”, is the most important thing for product management to articulate. (Thanks, Shawn, for the great book recommendation.)

My Why

Which leads me to this: I need to articulate and then be true to my own “why.” Why, exactly, am I in the software industry? What do I hope to accomplish? Why do I work on enterprise software, as opposed to mobile apps or cool web mashups? Why did I start a blog? Why do I pick certain topics for my posts, and then spend significant time articulating my thoughts?

Here’s my current answer:

I believe in making complex computing simple so the world can innovate and improve.

Testing My Answer

What I like about this answer is that it ties back to my core values. I believe we have opportunities to do lots of good in the world–eradicating poverty and disease, advancing science, learning to understand and value one another better. And I believe an impediment to that is all the complexity we create and discover. Big data is hiding a lot of the insight that would let companies serve customers better. Number crunching supercomputers are needed to predict the weather better, to help predict crop yields in Bangladesh. And so on.

Thus, if I can do my part to make really tough computing problems more tractable, I’m helping make the world a better place.

I also like that this answer tells me when I’m wandering. If I’m building systems that don’t hide/manage/solve complexity, I’m probably off track. If I am building software that’s frivolous (“Angry Birds” comes to mind), I probably won’t be very happy. If I work for a company that aspires to nothing more grandiose than making money, I should either change the company or change my job. If my blog posts don’t help me understand or communicate on topics that reinforce that goal, I’m wasting my time.


What is your “why”? Struggle with it until you come up with a satisfying answer. Then use it to test your current work assignments. Does this exercise give you any insight or point to ways to rebalance your priorities?

The Scaling Fallacy

If X works for 1 ___ [minute | user | computer | customer | …], then 100X ought to work for 100, right? And 1000X for 1000?

Sorry, Charlie. No dice.

One of my favorite books, Universal Principles of Design, includes a fascinating discussion of our tendency to succumb to scaling fallacies. The book makes its case using the strength of ants and winged flight as examples.

Have you ever heard that an ant can lift many times its own weight–and that if that if one were the size of a human, it could hoist a car over its head with ease? The first part of that assertion is true, but the conclusion folks draw is completely bogus. Exoskeletons cease to be a viable structure on which to anchor muscle and tissue at sizes much smaller than your average grown-up; the strength-to-weight ratio just isn’t good enough. Chitin is only about as tough as fingernails.

Tough little bugger — but not an olympic champion at human scale. Image credit: D.A.Otee (Flickr)

I’d long understood the flaws in the big-ant-lifting-cars idea, but the flight example from the book was virgin territory for me.

Humans are familiar with birds and insects that fly. We know they have wings that beat the air. We naively assume that at much larger and much smaller scales, the same principles apply. But it turns out Continue reading