It’s the start of another school year, and my seventh-grade son is learning algebra. As I sat beside him to coach him through some homework the other night, I shared my favorite bit of wisdom about how to make math problems—even complex ones—simple and error-free:

Write the progression from known to unknown, one step at a time.

In my experience, the surest recipe for disaster is to short-circuit this rule. Collapse a few steps in your head in the name of efficiency, and you’ll forget a minus sign, or you’ll group incorrectly, or you’ll lose track of an exponent or an absolute value—and you’ll end up with a mess. You’ll have to debug your solution by slogging back through the problem from the beginning until you figure out where you went wrong.

It’s interesting—and maybe, profound—how nicely this piece of advice maps onto the design principle of progressive disclosure. The human mind is simply wired to perceive in broad outlines, and then to gradually clarify, a few details at a time.

Don’t believe me? Try a short experiment: draw this fractal.

I’ll bet that instead of laying down every pixel, like a printer, you immediately produce a simplification that captures the general shape as lines, with a lot of detail suppressed. You did this as a kid, when you drew stick figures and triangle+half-circle sailboats.

Artists sometimes squint to blur out what they don’t want to see, leaving only general patterns and colors. But coders never do, because we don’t expect code to work that way.

Drowning in details

I think this is one of the flaws in most programming languages I know: they immediately plunge you waist-deep into implementation details that hide the forest among the trees. To see what I mean, try this experiment, which I did a couple weekends ago:

1. Download an interesting and complex codebase that you haven’t worked with before.
2. Try to add a new feature that sounds conceptually simple to your naive ears. For example, just add a command-line switch or make the program crash in a common code path.

I did this with the Julia programming language, but you could download Mozilla or Apache or Hadoop or anything else that sound interesting, and I think you’d have similar results: like me, I predict that you’ll quickly be overwhelmed with questions. Which of 3 likely entry points is the true top of the call graph? What do all these #ifdefs mean, and which ones are going to be active in my build? Which parts of the code have dependencies I need to understand?

This is the equivalent of trying to solve the entire algebra problem by holding all the transformations of an equation in your head at the same time. It can be paralyzing. None of the techniques that are popular in programming communities today give satisfactory answers to this problem—not BDD or TDD, not golden threads, not design docs, not javadoc/doxygen, not aggressive commenting, not ER or UML diagrams, not architecture description languages.

This need for context, for a high-level picture, for a sketch that gives you a useful skeleton of a mental model, is the reason why any new hire into a team with a complex codebase gets a whiteboard-ish orientation from smart teammates. We intuitively know we need it, and we’ll never get it from the code itself. On a codebase that I currently own, it is a major reason why the team hates the code and tells horror stories about its learning curve.

There’s just no easy entrée.

What’s in a book?

You could claim that all is for the best in this best of all possible worlds (nod to Pangloss). After all, teams do provide overviews and walkthroughs and design docs, and sooner or later, we get through the learning curve. But I think code could do a much better job of communicating, if we raised the bar.

Think about books for a minute. Hopefully you bought a print book recently enough to remember what it was like to pick it up and consider reading. How did you decide? If you’re like me, you probably read a blurb on the front or back cover. Did you glance at the foreword or preface? Did you read the first page to see if it felt interesting? Did you scan a table of contents? Did you flip to an index to see everywhere that your favorite subtopic was referenced?

Code has only weak parallels for these broad-brushstroke mechanisms.  In some sense, the main() routine is like chapter 1—but what comes after that may quickly become indecipherable, especially if you’re doing OOP or AOP or event-driven programming. You might liken java packages to a rough structure, but I think that in practice, they only deliver mediocre value because they tend to group by topic, not by code flow or by structural role. Headers insulate you from some details of an implementation, as I’ve noted—but finding the “important” classes out of a sea of thousands is not particularly easy. Interfaces allow you to encapsulate and suppress details—but they don’t tell you how they fit into a gestalt. Tests as a form of documentation are helpful, but they make it even harder to distinguish major themes from trivia. IDEs give you tree views, but trivia is intermixed with overarching concepts. Nowhere in a codebase do you typically find an explicit discussion about which constructs matter at install time, or which ones are important during startup but not during the later lifetime of the program.

Toward utopia

I don’t think this lack-of-a-big-picture problem can be solved with a single silver bullet. But here are a few ways that a better programming language/environment/ecosystem might make things better:

• Imagine a “granularity slider” in a code view, analogous to the zoom in/zoom out slider in google maps. What if a language understood the distinction between vital/domain/first-class objects and trivia, and could suppress details in a treeview or a code editor depending on your current zoom level? (IDEs already support collapsing class and file/folder views, and collapsing or expanding blocks of code. But to my knowledge, they don’t analyze a callgraph and decide that some functions are unimportant at a particular level of detail, and they don’t collapse file boundaries to make holistic program flow visible at a single glance.)
• Imagine that an IDE could generate, from some combination of static code analysis, profiling, and annotations in the language, a picture of how an application’s object model evolves through its lifecycle. And imagine that this diagram’s granularity could also be dialed in or out: In the startup phase, you might have a hierarchy of objects that looks like X. In the config phase, it looks like X’. In the collaboration phase, it looks like X”…
• Imagine that a programming language supported—or even required—a mapping between a customer-facing use case (a “story” or “golden thread”) and a theory of operation (and/or maybe a set of UML diagrams). And a further mapping between a theory of operation and a particular flow through the code. This would mean that you could explore the code at the highest level by saying “show me what happens when the ‘Admin cancels a job’ story is handled” — and correlate that to object model state, call graphs, and architectural views at arbitrary levels of detail.
• Imagine that instead of working on an algorithm one function at a time, you could work one subset of a callgraph at a time. So the code for the whole callgraph is displayed as a tree (with adjustable detail). Using a REPL, you feed in a piece of input to a top-level function, and immediately see how those parameters change and flow into all the functions that the top-level function depends on.  (This is a feature of a cool IDE called LightTable; I’d love to see it become more mainstream.)
• Imagine a language that lets you stub in rough broad-brush strokes, and just generates stubs automatically. This is two or three steps beyond what IDEs do today, where you can write code and highlight half a line and hit a keystroke that generates a stub, whereupon the IDE switches you to the generated stub and forces you to write an implementation then and there. I want to be able to write a line of pseudocode and have the compiler recognize that it’s pseudocode, and just silently generate a stub that will compile, with a default impl that A) writes/logs something telling me the name of the stub I just hit, where it’s called from, and what parameters it received; B) matches it to a stubbed unit test; C) lets me assign a hard-coded value; D) embeds a breakpoint in the impl such that when I run in a debugger, and hit the stub, I am prompted to supply more impl or more hard-coded values. I want to do all of that without breaking the flow of the higher-level algorithm I’m working on. Something like this:

// normal function, not stubbed
int x = do_something(); 
// stub declared, assigned a placeholder return value,
// and expected to compile with a working unit test
// without me ever leaving my flow
int y = do_something_else(a, b) stubbed with return 25;

• Imagine that you could create classes without declaring their methods, and then start a debugger session where you (and teammates) could role play different class interactions to model what you expect to have happen–and that as you role played, the IDE recorded your choices as new stubs, methods, and workflows, so that by “playing” the system, you gradually build it. (See my posts about role-play centered design for more on this.)

I have a few other ideas about how progressive disclosure might work in a better programming language, but I think I’ll stop there. I’m very curious to see if other smart people out there have good suggestions of their own.

Action Item

Tell me what you think would make it easier to perceive the rough behavior and structure of a big, complicated codebase in more efficient ways.

Related articles

• Working with UML class diagrams (jetbrains.com)
• Abandoning PHP for Python (techblog.stickyworld.com)