Big Data In Motion

I’ve been at Cloud Expo this week, listening to lots of industry hoopla about building cloud-centric apps, managing clouds, purchasing hardware for clouds, buying private clouds from public cloud providers, and so forth.

Photo credit: aquababe (Flickr)

One interesting decision made by the organizers of the conference was to bring “big data” under the same conference umbrella. There’s a whole track here about big data, and it gets mentioned in almost every presentation.

And I’ve sensed a shift in the wind.

Years and months ago, “big data” was all about mining assets in a data warehouse. You accumulated your big data over time. It sat in a big archive, and you planned to analyze it. You spun up hadoop or used some other map-reduce-style tool to crunch for days or weeks until you achieved some analytical goal.

What I’m hearing now is an acknowledgement that an important use case for big data–perhaps the most important use case–has little to do with data at rest. Instead, it recognizes that you’ll never have time to go back and sift through a vast archive; you have to notice trends by analyzing data as it streams past and disappears into the bit bucket. The data is still big, but the bigness has more to do with volume/throughput, and less to do with cumulative size.

This has interesting implications. Algorithms that were written on the assumption that you can corral the data set under analysis need to be replaced by ones based on statistical sampling; exactness needs to give way to fuzziness.

Interestingly, I think this will make computer-driven data analysis much more similar to the way humans process information. As I’ve said elsewhere, when faced with a difficult design problem, a smart question to ask is: how does Mother Nature solve it?

// Comments on Comments


We spend little or no time teaching programmers how to write good comments.

This is surprising, when you consider how often “total lack of comments” or “poor comments” are cited as evidence that certain modules (or the programmer who wrote them) are the worst thing that ever happened to the technoverse.

Comments shouldn’t leave you–or anybody else–mystified. :-) Image credit: xkcd

I happen to think that there are much yuckier tech things than poor or missing comments in code. But I still think our general level of comment proficiency is lower than it should be.

Here is my attempt to raise the bar a little.

Why We Comment

Sooner or later, most interesting programming problems require a sophisticated mental model of a problem. Building these models is hard work, and once we have them, we are paid to share with our team (or our future selves).

The best way to share mental models with other engineers is Continue reading

Baby Steps

Film poster, displayed under fair use as documented on Wikimedia Commons.

If you’ve seen the delightful comedy movie, What About Bob?, you are no doubt smiling at my title.

Bob is a neurotic and thoroughly irritating patient who depends on his psychotherapist for lots of emotional strokes and life coaching. He ingratiates himself with the therapist’s family and gets himself invited to be their guest on a weekend getaway, against the protests of the therapist. He then proceeds to drive the therapist crazy.

One of Bob’s favorite phrases is “baby steps,” which captures the therapist’s advice to solve problems a little bit at a time, rather than in overwhelming chunks.

“Baby steps” is surprisingly good advice for many questions in software design. It doesn’t apply in all cases, but it applies far more often than it ends up being used.

The Purpose of Design

We create UML diagrams, personas, design docs, lo-fi mockups, and other artifacts to capture our architectural thinking because they provide us with a roadmap of sorts. We need to identify and steer to a consistent point of the compass if we want to produce complex artifacts that meet customer needs. The bigger and more diversified our teams get, and the more moving parts we’re orchestrating into our software, the more important this is.

However, all of these artifacts are means to an end. To whit: Continue reading

Six Learning Tips For Tech Folks

In Seven Habits of Highly Effective People, Stephen Covey reminds readers to periodically suspend their lumberjacking long enough to sharpen the metaphorical saw. In other words — take time to renew, to learn, to get better, to work smarter.

Sharpen the saw. Photo credit: ATsawyer (Wikimedia Commons).

Chances are, you’re nodding your head. We all recognize the truth of the principle, but we struggle to put it into practice. My friend who’s a voracious learner is an example we all need to emulate better.

Here are a few tried-and-true habits that help me.

1. Look it up.

If you’re like me, you constantly encounter new acronyms, technoslang, APIs, languages, buzz words… Years ago, the effort to track down the meaning of such items was substantial–a trip to the library, maybe. Now you can usually define a term with 10 seconds of googletime, and you can get the skinny on fatter topics with a 30-second scan of a wikipedia topic. We spend so much of our time in front of screens; resolve now to make a habit out of looking up what you don’t know. It will cost your schedule nothing, and keep you learning.

2. Read.

I’m not suggesting that you should spend hours every day devouring deep content. I subscribe to a number of blogs, newsletters, and industry publications, and I usually rip through all of them in just a few minutes each day. For subscriptions that arrive in my email, I often read subject lines or headlines only, then press Delete. This keeps me aware, and I can always dive in if something truly interesting comes up. I also use StumbleUpon, which has led me to delightful discoveries a few times.

Besides the digital equivalent of sound bites, it’s important to expose yourself to richer content. Ask people you respect for book recommendations. Set a goal to read an in-depth magazine article once a month…

3. Follow thought leaders.

All voices in the blogosphere are not created equal. Twenty or thirty years ago, the only way to get inside the mind of luminaries was to hear them present at conferences, and to buy their books. Now, barriers to silent up-close study (or even one-on-one dialog) are much lower. People like Linus Torvalds, Scott Meyers, Bjarne Stroustrup, Guido van Rossum, Yukihiro Matsumoto, Martin Fowler, RMS, Kent Beck, Don Box, Erich Gamma, and Grady Booch have web sites, podcasts, twitter accounts, google profiles, and the like. Connect and learn.

4. Keep track of questions.

I have a google doc where I record topics on my to-study list, things I want to memorize, experiments I want to try. I also take plain old pen and paper to meetings, and take notes not so much about content but about loose ends or questions that arise in my mind.

Of course, you need to revisit your questions and answer them now and then. Maybe while a build is running and you’re eating a sandwich…

5. Teach others.

The conventional wisdom that the teacher learns more than the students is not just trite sentimentality. Put yourself on the hook to teach something that you’ve learned. If you’re comfortable presenting, volunteer to train team members. If that makes you nervous, write up something you learned in a blog. Or write an email to someone at work, sharing a cool blog post that made you think. Or just capture your learning in insightful comments in your code. But get the ideas out to others, which will deepen and enrich their potency in your own mind.

6. Find a foil.

In literature, a foil is a character that, when placed beside the protagonist, provides significant contrast in personality, style, or motivation. Foils challenge tidy assumptions and deepen the character that the reader cares about the most.

We all need to be challenged. We need to learn about other viewpoints–not at some surface level, so we’ll have talking points to contradict, but in depth, so we appreciate the contributions that other human beings can offer. Nothing is more certain than that the aggregate wisdom of others will always exceed our private insight.

Chances are, you’ll regularly encounter viewpoints different from your own, with no particular effort. So set yourself the task of thoroughly understanding and articulating such a viewpoint now and then, to the point where its proponent would nod with satisfaction and agree that you’ve nailed their thinking. Good things will happen.

Action Item

Pick an item or two from the suggestions above–or find some ideas of your own, that work for you–and start today to turn them into habits. Slow and steady is better than inconsistent bursts…

Extra credit: Tell others about the goals you set. This commits you, and might stimulate some worthwhile conversations. Or add a comment here, sharing other ideas you have. (Remember the value of teaching… :-)

Julie Jones: Learn voraciously.

(A post in my “Role Models” series…)

When you’re a twenty-something computer geek, the pace of the software industry doesn’t worry you much. You’re full of energy and enthusiasm for your chosen career, and you’re confident you’ll quickly absorb whatever wasn’t covered during college.

Add a decade or so, and you may see the world a bit differently.

Expertise and Moving Targets

Is mastery possible when your instrument keeps changing? Photo credit: Wikimedia Commons

The rate at which new technologies burst on the scene, acquire mindshare, and demand the attention of engineers can be overwhelming. A journeyman’s knowledge on a given subject may be obsolete long before it matures into mastery. Gartner’s hype cycle can make your head hurt.

If you aspire to excellence, the dawning realization that you will never learn it all (you can’t be perfect on breadth), and that even in a narrower problem domain where you might specialize, flux will erode your skills (you can’t be perfect on depth) might cause some angst.

Maybe you’ve read Malcom Gladwell’s Outliers. Yo Yo Ma put in 10,000 hours to become great at the cello; what are you supposed to do if your instrument changes every year or two?

My friend Julie has an answer.

Learning to Learn

I first met Julie Jones when I was in my early 30’s. We worked together on the “new engine” initiative at PowerQuest (a pivotal experience I’ve blogged about before). Julie had much more experience than I did; she taught me abstract factories and dependency injection, loose coupling, declarative programming, streams, unit testing, XP, and many other key concepts.

At first I thought Julie just plucked these ideas out of an impressive mental archive. She had rich and varied experience, and she seemed to know so much.

As I came to know her better, however, I realized that knowledge was not her great talent. Sure, she knows a lot–but more importantly, she learns a lot. The XP she taught me–that came from a book by Kent Beck that she’d just finished. The loose coupling? Scott Meyers, Effective C++. Again, recent reading. Fancy template techniques? Alexandrescu, Modern C++ Design, just barely published.

Julie was constantly, steadily learning. She subscribed to the C++ User’s Journal. She played around with new boost libraries. When she got an assignment to improve the build system, she tinkered with Perl, then decided she wanted to learn Python and threw herself into the effort with gusto. One weekend, she wrote a Python module to manage advancement in her foosball tournament; within a month or two, she was teaching the rest of us. She was always adding new books to her bookshelf. She went to conferences. When we had to have support for LDM and dynamic disks in our disk management layer, Julie volunteered, studied the relevant specs, and quickly rolled out library extensions.

What I learned from Julie’s example is that a superb software engineer isn’t so much an expert on a technology; a superb software engineer is an expert student of technology.

Learn to learn. In the long run, the engineer who’s mastered that skill will deliver far more business value than a narrow subject matter expert. And she or he will have a lot more fun surfing the turbulence of the tech industry.

Thanks for the lesson, Julie.

Action Item

Identify a few habits you could form to help you learn more, more often, and more easily. (Check out this post for some ideas.) Start working on them.