Why software engineering is not similar to other engineering disciplines and how it is changing the game

It is estimated that there are more than 11 million professional software developers worldwide as of 2014. When I started working as a programmer in 1973, a gray beard at the first company I worked for gave me some advice. He said: Learn the things that do not change.

When I started college six years ago in 1967, the school I attended didn’t have a major called computer science, and so I did my undergraduate and graduate work in mathematics while taking a few computer programming courses along the way. This was how many of us as software developers got their start in the 70’s.

The term software engineering was new at the time, as it was coined at the NATO Conference on Software Engineering in 1968. The thinking at the time was that we needed to apply existing engineering methods of software development to address common budget, schedule, and quality problems that were referred to at the time as the “software crisis.” As a result, what most people think of as software engineering involves activities much like other engineering disciplines including civil, mechanical, and electrical engineering.

On the surface, this idea makes sense. When you build something using other engineering disciplines (such as a bridge, a building, a specialized piece of hardware, a circuit board), you need to know the requirements, design a solution, implement it, and test it. All of these steps make sense for the program, too. So one can certainly argue from this perspective that software engineering should be similar to these other engineering disciplines. However, when you look closely at what we’ve learned about software development over the past 40 years, as well as how we teach it to software developers today, this analogy quickly breaks down.

By the time the 1990s began, as computer programming had become a large part of what used to be called computer science, many universities had added a course titled “Software Engineering” to their computer science curricula. Popular textbooks that were used at the time to teach these courses included Ian Somerville’s book “Software Engineering”. From 1992 to 1994, I used the fourth edition of this textbook to teach software engineering at Binghamton University. Today, Ian Sommerville’s book is still in use at many universities around the world – now in its ninth edition. This leads us to a question:

Why do we need to review a textbook about every 3-4 years that is supposed to teach our students the basics of software engineering?

If you look at textbooks used in civil engineering, mechanical engineering, and electrical engineering, the vast majority of these do not require revisions very often. To understand why this is the case, we need to take a closer look at what is taught in most universities around the world under the name “Software Engineering”.

When you look closely, you’ll find that we teach the next generation of software professionals everything that is currently popular in terms of software practices and methods. Popular software practices and methods today are known by buzzwords like Agile, Use Cases, User Stories, RUP, XP, Scrum Lean, PSP, TSP, the list goes on and on…

The problem with this approach to teaching software engineering is that software practices and methods come and go a lot and will keep coming and going which is why Sommerville has to constantly update his textbook. This leads us to another question:

What about the gray beard at my first company in 1973 that told me to learn things that never change? Did he give me bad advice? If not, what are we teaching our next generation of software professionals regarding the things that never change in software engineering?

Before answering these questions, let’s first step back and ask a few different questions:

Is there really a set of things that never change in software engineering?

If they exist, do we know what they are?

If we know what they are, are we teaching them in a consistent way to the next generation of software professionals, so when they come out of college they’re ready to act like software professionals?

This set of fundamentals of software engineering is already in place. This belief prompted an international group of volunteers to take on the task of jotting down the basics. The intent is to teach these basics to our next generation of software developers to help prepare them as true software professionals.

The volunteers involved in this initiative (known as SEMAT – Software Engineering Method and Theory) have been working on this task since 2010. Last year, SEMAT achieved a major milestone when the Object Management Group, an international standards group, announced that it had adopted “Essence” as the official OMG standard.

This leads us to a few more questions:

How different is the Essence standard from what our software developers are taught today, and how different is the Essence standard from what has been taught for the past 40 years under the name of software engineering?

And the:

Will the differences really help solve the problems that many believe still plague the software industry with regard to shared budgeting, scheduling overruns, and poor software quality?

From one perspective, what Essence picks up is nothing new. The Essence standard includes common words such as stakeholder, opportunity, requirement, program system, team, work, and way of working. But from another perspective, what Essence picks up is completely new. In fact, some call it a “paradigm shift” that many of the “old guard” would have great difficulty even understanding.

To give you an idea of ​​the changes made when using Essence, I’m thinking back to my early days as a programmer in the late 70’s. In those days I worked in the field of flight simulators developing software systems to train pilots to fly high performance aircraft. One of my areas of expertise has been writing software to provide recording/playback capabilities to assist instructors in training young aircraft pilots in flying skills.

I remember one specific project I worked on and a client lead trainer I worked with. After showing him how he could use my recording/playback software to help him show student pilots who had made mistakes, he excitedly wrote a number of faults asking for changes to be made to my software.

I argued vehemently with my program manager that none of these issues were actually defects. As I took the time to explain what was possible with my recording/playback software, the demo trainer began to envision additional features that could make his job easier. He wrote his ideas as a bug even though they were all enhanced capabilities that we never planned to introduce and weren’t part of the requirements.

But my project manager doesn’t want to discuss with the client whether these requests are in scope or out of scope. His point was—as with many programs offered at the time and still offered today—that it was easier to change programs than to engage the client in a discussion.

Because programs are soft, we tend to see them as easy to change. It’s not like hardware. The metal is not easily bent. This perspective changes the whole game when it comes to software.

This ability to change software code quickly and in endless ways completely changes the dynamics that exist between software developers and stakeholders including program managers and customers. One way this difference manifests itself is that as users become familiar with the software, they often see new ways changes to the software can make their job easier as my experimental mentor did in the late 1970s.

We now know from experience that there are other dimensions of software engineering that are critical to effective professional software engineering practices. These other dimensions take us beyond just the ease of changing the code. So far, these extra dimensions haven’t received anywhere near the attention they deserve.

When you change the code, it may also affect the requirements, and it may also affect other capabilities in the previously tested software system. Changing the code means additional work, additional testing, possibly changes to the supporting user manuals etc… All of this affects budget and schedule, and introduces additional risks to the quality of the software.

While the ability to quickly change software code brings great power to the software industry, it also means that software professionals must be more attuned to their agreed-upon way of working, the impact and time it takes to do overtime, and the risks when making quick, unplanned changes. The agile movement over the past 10 years has done a great service in helping the software community understand this huge difference in software engineering including the importance of early and continuous interaction with stakeholders and the importance of software developers estimating the cost of their labor.

While the software engineering community has learned a lot from other engineering disciplines, they have also learned the critical importance of these other dimensions that make differences from previous engineering experiences. These differences mean that software developers need to be trained in new and different ways to be effective software professionals.

Shortly after starting the SEMAT initiative in March 2010, an early SEMAT signatory sent me a draft copy of a book he was working on revising. Watts Humphrey who was planning to be very active in the early SEMAT work fell ill just as he was preparing the SEMAT work and I was asked to help him continue his planned efforts. In late August of the same year, Watts sent me the following email a few months before his death. Agree that I can share this email with others:

Paul, from your comments, it’s as if you get the point of my book, for which I’m grateful… The right answer, and the one I’ve been interested in pursuing with SEMAT, is about how to ensure that software professionals are properly trained and have an appropriate set of professional attitudes and skills before they get into industry. It is my hope that the SEMAT effort will eventually be able to drive the impulse to have the academic community refocus their programs on education program professionals to work as professionals and manage themselves.

When they do, their graduates will be able to negotiate with their management and do a superior job…. This is what professionals should do… A good start in this direction is to convince them of the need for software that measures people’s work. Since software work, as we said, is cognitive work, any really precise measures must be taken by software professionals themselves. … Watts Humphrey

Watts Humphrey has been referred to as the father of software quality. After completing his distinguished career at IBM, he became a Software Engineering Institute Founding Fellow of the Software Process Program. In 2003 he was awarded the National Medal of Technology.

Today, Watts would have been heartened by the ongoing SEMAT work in the academic community. Developed and offered to students this year by Dr. Carlos Zapata at Universidad Nacional de Columbia in Medellín, Colombia, the first full-fledged undergraduate course based on the new Essence standard was developed and offered to students, and Essence is being used in first- and second-year software engineering courses at KTH Royal Institute of Technology in Sweden under the direction of Dr. Mira Kajko-Matson. There have also been field studies of Essence conducted by Dr. Cecile Perrier at Carnegie Mellon University West in the US with students. The next step for the SEMAT community is to show how Essence can help the industry by publishing case studies of real use and measured results in industrial projects.