The bridge-building metaphor.

So much talk throughout the eons about how software development ought to be "like building bridges", and it turned out that perhaps it is far closer than we ever thought although not in the way we always intended:

Somewhere, someone's fist is pounding the table again. Why can't we build software the way we build bridges?

Well, maybe we already do.

As OSAF's programmers labored to construct their tower of code, piling bug fixes atop Python objects atop the data repository, I watched the work proceed on the new eastern span of the Bay Bridge. The project, replacing half of the 4.5-mile, double-decker bridge that several hundred thousand vehicles cross each day, was born in the 1990s and first budgeted at a little over a billion dollars. The original design called for a low-slung, unadorned causeway, but political rivalries and local pride led to the adoption of a more ambitious and unique design. The new span, a "self-anchored suspension bridge", would hang from a single tower. A web of cables would stretch down from that lone spire, underneath the roadway and back up to the tower top, in a picturesque array of gargantuan loops. It was going to be not only a beautiful bridge to look at but a conceptually daring bridge, a bootstrapped bridge —a self-referential bridge to warm Douglas Hofstadter's heart.

There was only one problem: Nothing like it had ever been built before. And nobody was eager to tackle it. When the State of California put it out to bid, the lone contractor to throw its hat in the ring came in much higher than expected.

In December 2004, California governor Arnold Scwarzenegger stepped in and suspended the project, declaring that the Bay Area region would have to should more of the ballooning cost of the project and calling for a second look at the bridge design. Never mind that work on half of the bridge, the water-hugging viaduct that would carry motorists for more than a mile on a slow climb up to the future main span, was already very far along, and every morning you could see vehicles swarming up a temporary ramp onto the new roadbed. Schwarzenegger wanted the project scaled back to a less novel and cheaper design. The governor, the state legislature, the state's transportation agency, and local governments spent months bickering and horse-trading. The transportation agency claimed that each day of delay was costing the state $400,000. Finally, in July 2005, a new compromise reaffirmed the fancier single-tower design, to be paid for with bridge toll hikes and other measures, and projected a new finish date for the bridge: 2012 —almost a quarter century after the Loma Prieta earthquake had shaken a chunk of the old bridge deck loose.

As I read about the controversy, I couldn't help thinking of all the software management manuals that used the rigorous procedures and time-tested standards of civil engineering as a cudgel to whack the fickle dreamers of the programming profession over the head. "Software development needs more discipline," they would say. "Nobody ever tried to change the design of a bridge after it was already half-built!"

(Rosenberg: pp. 346-347)

Sounds like a horror story. As a matter of fact, it doesn't sound as different from the story about the development of Chandler that Rosenberg narrates in his book: changes of heart, disputes, political controversy, worries about the final cost, delays... In other words, as soon as civil engineers try to build something that has never been done before —something truly innovative— they run into the very same problems software engineers do.

Two different roles of the software programmer, according to Simonyi.

Charles Simony, the famous Hungarian-American programmer who oversaw the creation of Microsoft Office, distinguished between two separate roles of software design:

In Simonyi's view, one source of the trouble is that we expect programmers to do too much. We ask them to be experts at extracting knowledge from nonprogrammers in order to figure out what to build; then we demand that they be experts at communicating those wishes to the computer. "There are two meanings to software design," he says. "One is designing the artifact we're trying to implement. The other is the sheer software engineering to make that artifact come into being. I believe these are two separate roles -the subject matter expert and the software engineer."

(Rosenberg: pp. 278-279)

I imagine he is referring to the same sort of separation of roles we see between the architect (or the engineer) and those who lay the bricks and finally shapes the original plans in real life. If that's what he is referring to, it definitely is a true distinction. On the one hand, you have the programmer as a designer, the professional who makes the long-term decisions about the overall design of the application, the language to be used, the resources thay will be needed, etc. Then, on the other hand, you have the programmer as coder, the one who actually writes the code and implements the plans laid out by the designer. We're definitely talking about two separate roles, two different phases of the software development. However, what's not so clear to me is that they need to be done by different people too. Perhaps that should be the case, ideally. However, programming has not reached the level of reliability and abstraction that's so prevalent in other fields, such as the construction business, for instance. It's not so clear to me that we can safely divide software development teams in groups of designers and coders, the latter just implementing whatever the former lay out on paper. There is a good amount of wishful thinking in that idea, I'm afraid. As a matter of fact, it could very well be that you could end up with a worst case scenario: software designers who have little experience in the nitty-gritty business of actual programming (this is not, obviously, Charles Simonyi's case) putting together their pie-in-the-sky schemes, while programmers who have never been asked to take a bird's view of their projects continue working on spiritless line of code after line of code without having a clue how the whole thing fits together. I seriously doubt this would represent a step forward for the world of software development as we know it.

The physics of software.

Rosenberg brings up yet another interesting point when it comes to software development:

" 'Software engineering' is something of an oxymoron," L. Peter Deutsch, a software veteran who worked at the fabled Xerox Palo Alto Research Center in the seventies and eighties, has said. "It's very difficult to have real engineering before you have physics, and there isn't anything even close to a physics for software."

(Rosenberg: p. 276)

However, is it even possible to find out the "physics of software"? Is there one? The natural answer to that question is that, obviously, there is none. Software doesn't have a physical existence of any type whatsoever. It only exists in the form of bits and code and, prior to that, in a programmer's mind or, at the very best, some well documented specs. In that case, we could assume that Deutsch is right on the money: there is (and there will never be) any true software engineering discipline. And yet, is Deutsch's premise correct? Is it true that only physical things can be "engineered"? I suppose it all depends on the approach one takes to the issue. If we start from a mechanistic point of view, the assumption that only something physical can be engineered appears to be true. However, if we take a more systemic approach, it would appear as if everything is information, bits that flow around and can be channeled in one form or another. If his latter vision were true, then the problem is not so much that software engineering would be impossible but rather that we'd have to modify our own concept of what engineering means.

The invisibility of software.

Rosenberg speculates that the invisibility of software makes it more difficult to design it in a reliable manner as it (supposedly) happens in the physical world:

As we have seen, software sometimes feels intractable because it is invisible. Things that we can't see are difficult to imagine, and highly complex things that we can't see are very difficult to communicate . But invisibility isn't the only issue. We can't see electricity or magnetism or gravity, but we can reliably predict their behavior for most practical purposes. Yet simply getting a piece of software to behave consistently so that you can diagnose a problem often remains beyond our reach.

(Rosenberg: p. 275)

Since he discusses this in the context of GUI programming, it automatically triggered an idea in my mind: would it help if we were to build Shockwave Flash mockups? Would that make software more and visible and, at the very least, help understand the way the different GUI components interact with eachother? It may be worht a try. Obviously, I don't think it would be of any help when it comes to the internal logic of a program, but it may be of some use in the design of the graphical front-end, especially when it comes to obtaining some quick up-front input from the customers.

A public infrastructure of servers.

Somewhere during the development process of Chandler, Kapor rethinks the peer-to-peer foundations:

Kapor was rethinking the merits of peer-to-peer anyway. At a June meeting where he formally announced the adoption of a server-based sharing design to his staff, he explained, "I've had a significant change of point of view. There was a kind of frontier idealism that was well intentioned but not practical on my part. The issue is about empowering people. It's not about the infrastructure. Maybe we need a robust public infrastructure of servers to let people do what they want to do. My and OSAF's original position was, electricity is good, therefore everyone should have their own power plant! Unconsciously, I always imagined that user empowerment somehow meant a server-free or server-light environment. Now I think that's actually wrong."

(Rosenberg: p. 214)

Could that "public infrastructure of servers" be what Google and others have been building in the past few years? The time when individuals can have their files "on the cloud" is here already. Millions of people have their music, pictures and opinions hosted on sites such as Google Documents, Blogger, MySpace, Facebook, Picassa, Shutterfly, Flickr, YouTube and many others. Right now they access all this information via snappy web applications written in AJAX but there is nothing stopping us from adding these services to regular, old-style rich apps. Actually, the folks at the Mozilla Labs are already working on some Firefox extensions to do exactly that. In this respect, it could very well be that the project to build Chandler started too early for its own sake.

WebDAV, peer-to-peer and the influence of hype.

Who doesn't remember the big peer-to-peer trend that happened a few years back? Napster showed up in the radar, it quickly became the next big thing and... well, it quickly went away too. Yes, it also served a purpose. Who thinks that we'd have the iTunes store today without that little experiment? Still, it's also true that peer-to-peer became fashionable, trendy, cool, the thing to do and, all of a sudden, everybody wanted to fit the framework into whichever project they were working on. According to what we can read in Dreaming in Code, that's precisely what happened to Kapor and the team behind Chandler:

If WebDAV could do it, why was it so har for Chandler? Chandler's peer-to-peer approach meant there was no central server to be what developers call, with a kind of flip reverence, "the source of truth". WebDAV's server stored the document, knew what was happening to it, and could coordinate messages about its status to multiple users. Under a decentralized peer-to-peer approach, multiple copies of a document can proliferate with no master copy to rely on, no authority to turn to.

Life is harder witout a "source of truth". For programmers as for other human beings, canonical authority can be convenient. It rescues you from having to figure out how to adjudicate dilemmas on your own. After just a few weeks at OSAF, Dusseault became convinced that the peer-to-peer road to Chandler sharing was likely to prove a dead end. The project had little to show for its efforts to date anyways. "But it was like, we're doing peer-to-peer. We have to. We said we would. We decided to."

(Rosenberg: p. 213)

WebDAV was certainly much better suited to the project they had in mind that peer-to-peer. The same could be said of the traditional client-server approach. So, what's the morale of the story? Even the best hackers make mistakes when they let themselves be influenced by hype.

An innovative feature for a PIM: stamping.

Interesting feaure:

Yin explained the latest thinking on items. Most PIM programs required users to decide up front, when they created a new item, what it was: Were you creating a new email? A calendar event? A to-do task? Chandler would instead let you sit down, start typing a note, and decide later what kind of item it was. Like the human body's undifferentiated stem cells, notes would begin life with the potential to grow in different directions. This design aimed to liberate the basic act of entering information into the program from the imprisoning silos. It also made room for Yin's proposed solution to the item mutability problem: The mechanism users would employ to specify the "kind-ness" of an item would be called stamping.

Say you had typed a note —a couple of sentences about a meeting— and then wanted to put the meeting on the calendar. You would stamp the note as an event. Chandler's detail view would add fields that let you specify a date and a time; your generic note was now an event. Later, if you wanted to invite a colleague to the meeting, you could take the same note and stamp it as an email. A "to" field and a subject line would appear in the detail view. You would fill it out and click on a "send" button.

(Rosenberg: pp. 189-190)

I don't think any other existing PIM software allows you to do that. Apparently, many of these innovative features in Chandler are actually carried from Lotus Agenda, an old piece of software that Kapor wrote back in the Lotus days. It's a pity that today's PIM software is, for the most part, quite run-of-the-mill. When it comes to doing simple things (i.e., dealing with email or taking care of our appointments on a digital calendar) they are too complex and prone to corrupt our (valuable) data. As a matter of fact, it wouldn't be the first time I try to use one of these fancy applications and always end up reverting to good old time-tested mutt. No, mutt is not cool and fancy with all its bells and whistles, but it is extremely flexible, it lets me configure it precisely the way I want it and, above all, I can trust it with my email. I won't corrupt the data as soon as the email folders are too large, forcing me to deal with an emergency crisis at the worst possible moment. So, if I am going to use one of these newflangled applications, it'd better be something different, creative, innovative. It'd better be something that seemlessly links my email to my calendar, my tasks and my contacts. Otherwise, why even bother with a new app?

Tree structures vs. semi-lattices (and the advantage of organically grown, dynamic structures).

I found the following two paragraphs quite inspiring:

One day in April 2004, Chao Lam sent Mimi Yin a link to an article that he had found in a blog posting by a writer named Clay Shirky, a veteran commentator on the dynamics of online communication. Shirky had written about his rediscovery of an old article by Christopher Alexander, the philosopher-architect whose concept of "patterns" had inspired ferment in the programming world. The 1965 article titled A City Is Not a Tree analyzed the failing of planned communities by observing that typically they have been designed as "tree structures". "Whenever we have a tree structure, it means that within this structure no piece of any unit is ever connected to other units, except through the medium of that unit as a whole. The enormity of this restriction is difficult to grasp. It is a little as though the members of a family were not free to make friends outside the family, except when the family as a whole made a friendship."

Real cities that have grown organically —and real structures of human relationships— are instead laid out as "semi-lattices", in Alexander's terminology. A semi-lattice is a looser structure than a tree; it is still hierarchical to a degree but allows subsets to overlap. Why do architectural designs and planned communities always end up as "tree structures"? Alexander suggests that the semi-lattice is more complex and harder to visualize and that we inevitably gravitate toward the more easily graspable tree. But this "mania every simpleminded person has for putting things with the same name into the same basket" results in urban designs that are artificially constrained and deadening. "When we think in terms of trees, we are trading the humanity and richness of the living city for a conceptual simplicity which benefits only designers, planners, administrators, and developers. Every time a piece of a city is torn out, and a tree made to replace the semi-lattice that was there before, the city takes a further step toward dissociation."

(Rosenberg: pp. 184-185)

Aside from the obvious implications all this has for urban planning, there are also some consequences for software development. In this sense, the traditional approach to programming that viewed it as something akin to building a bridge, a huge work of engineering that needed to be planned way ahead, discussed, negotiated, documented and supervised prior to even laying out the first stone ought to give way to a different, more dynamic approach. This is precisely what has been happening in the last decade or so with the rise of agile methodologies of software development and, in general, the open source approach. Things are now to be understood in a more dynamic manner, as a flow, as something more amorphous, difficult to comprehend and control, an ever-changing set of specifications and features that needs to be managed in the sense of being navigated through —surfed even—, more than controlled. Reality is too rich and diverse to shoehorn it into a predefined shape.

Now, what I find quite interesting in all this is that the above description is quite compatible with the works of Gilles Deleuze, Félix Guattari and Manuel de Landa. Reality is to be described as a rhizome instead of a tree. Hierarchies may exist, but in a completely different form. They are not absolute anymore. They are highly dependent on the position they (and us) occupy in the whole landscape of things. I find it difficult to believe that a philosophy like this may not rise to take center stage in our century, replacing the outdated mechanistic approach of the past.

Linus Torvalds on avoiding large projects.

In accordance with the principles stated in a previous post about developing in Internet time, open source projects tend to stay away from large and ambitious plans. This is not to say that they never reach the status of large projects, of course. However, it does mean that they usually start as small, little projects that try to "scratch an itch" and, given enough interest and a good amount of contributions, may grow to something as large as the Linux kernel, GNOME or KDE. These ideas were pretty well expressed by Linus Torvalds in an interview published by Linux Times June 2004 where they asked him if he had any advice for people starting large open source projects.

"Nobody should start to undertake a large project," Torvalds snapped. "You start with a small trivial project, and you should never expect it to get large. If you do, you'll just overdesign and generally think it is more important than it likely is at that stage. Or, worse, you might be scared away by the sheer size of the work you envision. So start small and think about the details. Don't think about some big picture and fancy design. If it doesn't solve some fairly immediate need, it's almost certainly overdesigned".

(Rosenberg: p. 174)

Once more, the emphasis is on a project that's small, releases quick and nimbly responds to the users' feedback. This mindset is at the very core of the open source development model. Commercial software companies may stress that their approach is quite different, refusing to treat customers as beta testers. However, this is quite disingenous. Anybody who has gone through the experience of running the first public release of any software product knows what I mean.

Vitruvius applied to software design.

Inspiration can truly come from anywhere, but when it comes to software design it is clear that there is a connection between it and architecture, as Kapor noted:

Software design, Kapor argued, was not simply a matter of hanging attractive graphics atop the programmer's code. It was the fundamental creative act of imagining the user's needs and devising structures in software to fulfill those needs. As the architect draws up blueprints for the construction team, so the software designer should create the floor plans and elevation views from which software engineers would work. Reaching back to ancient Rome, Kapor proposed applying to software the architecture theorist Vitruvius' principles of good design: firmness —sound structure, no bugs; commodity —"A program should be suitable for the purposes for which it was intended"; delight —"The experience of using the program should be a pleasurable one".

(Rosenberg: p. 149)

The terms may sound outdated, but the idea truly still applies: a well designed software should be solid and consistent, able to meet the requirements and a joy to use. Apple's designers have always put this advice to good use.