“Software has been and will remain fundamentally hard,” says Grady Booch, in a Computerworld interview October 29. Booch is “Chief Scientist” at IBM's Rational software division.

The interview struck a chord with me. Many things that affect what we at Database Designs do have gotten easier, cheaper. Computers are cheaper, networking is easier, web hosting is cheaper, Open Source avoids licensing costs.

Producing mission critical software applications for our clients remains hard. Not hard in the same way as it was five, ten, or fifteen years ago, but still hard.

What does Booch say? “In every era, we find that there is a certain level of complexity we face. Today, a typical system tends to be continuously evolving. You never turn it off, [and] it tends to be distributed, multiplatform. That is a very different set of problems and forces than we faced five years ago...Traditionally — we’re talking a few decades ago — you could think of software as something that IT guys did, and nobody else worried about it. Today, our civilization relies upon software.”

Translating this into our own experiences, I would list five crucial elements you should consider as your organization plans to commission a new web site or database project.

First, external communication requirements make things hard. The good news is that most kinds of organizational processes folks like us need to model in a data management system are familiar to us at this point. Data input web pages or database forms come together pretty well.

On the other hand, once you start adding in layers of external reporting requirements, evaluation systems, data exports, complexity grows. We can't ignore these requirements, because they are part of the connected world Booch alludes to, but they are a big part of what makes software hard today.

Second, “the continuously evolving” part of things. If an organization commissions a system and imagines we can go off and build it by ourselves--without your steady participation in strategy, planning, design, review--time and costs grow. Organizations who are not used to managing software projects underestimate what they will need to put into the process. They may have done excellent strategic technology planning a year ago and RFP creation six months ago, and believe their work is done. Our work depends on our clients' full engagement.

Third, legacy data systems. I used to think of legacy systems as anything dependent on really old tools, like Cobol or pre-relational databases. A legacy system today can be any existing piece of software that you cannot redo as part of new planning and yet needs to be integrated into the new. Everyone has legacy systems and many things that really don't want to work together have to work together. Booch reminded me of this when he says:

“Most of the interesting systems today are no longer just systems by themselves, but they tend to be systems of systems. It is the set of them working in harmony. We don’t have a lot of good processes or analysis tools to really understand how those things behave. Many systems look dangerously fragile. The bad news is they are fragile. This is another force that will lead us to the next era of how we build software systems.”

Fourth, a corollary of the third point is if we DO come to replace the old, data conversion can easily become the most taxing element of a project. The old just doesn't pour into the new, as everyone from line staff to program managers to senior managers discover layers of the old that can't or won't go away easily.

Fifth, the “never turn it off” element. It is fundamentally hard to carry out an update to an existing working system or launch something new, especially an always-on website, without affecting someone's daily routine. Testing, training, data checking before something new goes live becomes more crucial than ever, and yet it is so hard—understandably so—for client project teams to make enough time to do this.

Booch says, “All of a sudden, you wake up and say, 'I can’t live without my cell phone.' We, as software developers, build systems of incredible complexity, and yet our end users don’t want to see that software.” Even as hardware and even broadband Internet becomes part of the appliance part of our lives, we have to educate each other—developers and everyday users—to see the software. Maybe not all the time, but at least during periods of software change, we all have to see the software and see what it means for it to remain working. Software adds more and more value to what we depend on today, and a lot of it still needs to be customized to make the most difference. The process of creating it just has to be taken more seriously than ever.

The rest of Booch's interview is worth a read. He sheds more light on IBM's commitment to Open Source as part of building the software development infrastructure for the next generation of software development. It is important to understand Chief B is writing from the point of view of the large, well funded enterprise projects for presumably well organized teams. If he sees these issues at that level, one can imagine them even more at the level of nonprofit and small business systems. I compiled these five points from reflecting on several projects of the last few years. Maybe these will help you in planning for your encounter with web or software developers. Maybe you have others. Love to hear.