The Broken Window Theory of Software Development

In the late 1980s, the crime rate in New York City reached an all-time peak.  Attempts were made to understand this crime wave and fight it better. The "Broken Window Theory" was advanced to explain how New York and other major urban cities in the United States have reached this sad situation. The theory gives as an analogy a house with a broken window. Passerby looking at it will deduce that its owners don't care much about it (they didn't fix the broken window) or are absent, and those with malicious intents will tend to damage the house. It might start by breaking another window and drawing graffiti on a sidewall. It will end with the total vandalism of the house and kicking out its inhabitants. 

Several experienced software developers can attest to having witnessed a metaphorical broken code window in some of their projects. It might have started with some code copied a few times.  A golden rule here holds that twice is ok, but three times and you should refactor. So, the code was copied three times. A method would have been written that is too long. A rule here is that from 20 to 50 lines, it's ok. So maybe it was 80 lines. Perhaps it grows to 100 lines. Maybe 200. Then there's coupling. Master classes with references to most other classes in the code. A service that is too dependent on the ten other services. Variable names that are not informative. Maybe some tests are missing. Maybe testing is scant. Maybe there are no unit or integration tests. What starts with a few lines triplicated, ends with a spaghetti bowl of code where the only solid rule that the developer team abides by is copy-pasta.

Now the company's business might not care much about these technical details, but it should. Bad code translates to either bad performance and bugs. Or if the company has invested an enormous budget in fixing all the issues, it will mean software that is too brittle and is not maintainable. Whenever the business needs to adapt to some new market requirement, it will face the nightmare of the bad code.

You can count yourself lucky if you never been or seen or heard of such projects. But they are out there, they are plenty, and they are the bane of software development in the enterprise and start-ups alike. For years these problems have been far too common. Therefore the practice of continual refactoring has been so important for the development process. In the domain-driven design world, refactoring to gain further insight has been sung like an old mantra.

The sheriff lays down the law

But could we do better? Former New York Mayor Rudy Giuliani was a big fan of the "Broken Window Theory." The story popularized by Malcolm Gladwell in his book "The Tipping Point" goes that in the early 90s, he, along with police commissioner William Bratton gave the directive for NYC cops to issue extreme penalties for minor criminal offenses such as graffiti on the subway and other acts of vandalism. The story goes that with this extreme application of justice and control the crime wave in New York City went down that today it is no higher than anywhere else in America.

The same method is sometimes applied in the software world. With the addition of a new development lead, her role becomes to control and monitor the developers. With frequent code reviews, the Senior Developer and the Developer Lead can detect quality defects and force the staff to fix and learn from their bad coding habits. If you find code that is repeated more than twice, then you make sure the developer refactors her code.

Automated tools are also there to help. SonarQube has been invaluable to detect the infamous cyclometric complexities and code smells that plague amateur code. And even the most senior developers with the time constraints of modern projects can be guilty of bad code.

But does this practice actually work? Can the sheriff always lay down the law?

Freakocoding

Around the time that Malcolm Gladwell published his Tipping Point, another book made the rounds on the New York Times bestseller list. Written by Economist Stephen Dubner and journalist Steven Levitt, "Freakonomics" is based on the econometrics research of Dubner. If you don't know econometrics, it is data science before data science was cool and data was big. Technically it is the application of statistical methods to data to find reasons for regressions and explain economic phenomena. Dubner was not interested in explaining regular economic phenomena such as what leads to recessions and how full employment can lead to inflation. Instead, one of his pet projects was to try to explain the lowering of the crime rate in New York City that took place in the 90s. His conclusion was shocking. He found a correlation between the number of aborted births with the landmark Rowe vs. Wade's decision in the United States that legalized abortions and the number of would-be criminals in NYC. In short, his theory goes that there is less crime in New York because would-be criminals born to lower-income single women were aborted a long time ago. So, it was nothing that New York Mayor and Police commission did that brought down crime; only better quality people were born who are not so inclined to commit crimes.

Off the bat, the narrative is very different in the software development world. With the last decades’ boom in the industry, the high pay and demand in the job market, a lot of wouldn't be software developers picked up the profession. And yet no demand glut seems to be on the horizon. Even if the current economic climate results in a recession, companies will seek to automate their operations to lay-off more workers and thus, they will need more software. Demand for programmers is best seen in the interview process. The top 5, as well as the unicorns, have the most demanding interviews to get the best and greatest, but for most companies, basic software development skills will probably get you an entry-level job that pays more than the average salary of your region.

That is not to say that for those companies seeking to produce great software, weeding out the bad apples, and hiring the best and brightest is not a way to go. It's just that the market is competitive, and not all companies can pay top dollar.

Moneyball it?

Maybe a ragtag team of misfits can solve it, Moneyball style? Another great book and movie, this time by the famous Michael Lewis, tells the story of how the Oakland A’s managed to recruit a top team of A-players by focusing on what traditional baseball recruiters miss like actual batting averages. This, too, might be a way to go. To find those rough diamonds in the coding world that have been rejected by the Top 5. A famous story is that the inventor of the Mac operating system's brew package manager was rejected by Google because he couldn't invert a binary tree. Brew is of course used by a lot of Google engineers using Macs. The classical top 5 interview process focuses on algorithms and neglects other aspects of coding that might be valuable. This said the Top 5 can also apply these processes, and some odd anecdotal stories have been told of people getting recruited by google just by scanning their google searches.

Quality over quantity => Programmer < Software Developer < Software Engineer && Local Resource > Outsource Resource

Some people just want to code. They want to code so badly that they don't care about design. They want to code so badly they don't want to think. "Just tell me what to do, and I will code it for you." Ah, the beauty of horse racing, or is it? But development quality increases in proportion to the amount of thinking required from the developer to figure things out. So on one end of the spectrum, you have programmers. These are not necessarily Computer Science graduates, but they can be. An army of functional designers and other businesspeople are recruited to tell these guys what to do and how they should do it. They lack abstract thinking or motivation for it. On the other end of the spectrum, you have software Engineers who are coming up with fancy designs and blueprints of what needs to be done and can sure as hell code it for you. In between, you have the software developer.

At the same time, the trend in the 00s and 10s was to hire armies of outsourced programmers to cut costs. However, quality problems due to education levels and communication skills of these programmers mean that their productivity is low. If we remember the bible of outsourcing in the early 21^st century – Thomas Friedman's The World is Flat – recommended to move the tedious low challenging work to outsource centers and keep value-added work handled by local developers.

There are times when outsource developers can be great, world-class even. Classical economics explains the wage difference between people and countries based on productivity. Someone who earns more for the same job does it better, faster, and more efficiently. Yet there are cases in the global labor market where this is not true. These present opportunities for wage arbitrage. For instance, a Middle Eastern country ravaged by war, where the local market doesn't provide many opportunities but with a highly educated workforce. Or some land locked former Soviet Republic that inherited an excellent educational system in the STEMs but has a dearth of resources. Another example is the genius developer in some poor country. And we know the saying that there are more above average IQ people in China and India than there are people in America. Yet such opportunities are few and far in between.

Furthermore, For the case of the genius-developer odds are they will be able to immigrate to the west and gain good employment with their tech skills. For the case of war-ravaged countries or poor resources countries, communication and infrastructure are bound to pose a challenge. For those above-average IQ developers in China and India, well they are earning above-average wages. For instance, 35K USD salaries for developers in some Chinese provinces are not uncommon (competitive, but not by much as the average developer salary for example in Canada is around 50K USD).

The existence of wage arbitrage is still a mystery. An Economist Magazine article explained the puzzle of why laborers who immigrate to western countries end up earning higher wages by productivity increases that result from benefiting from the western country's technology and infrastructure. Something that is hard to square with today's communication technology advances.

But it depends on the company's budget and operating culture. Some business enterprises of the legacy loving kind will tend to recruit the former for lack of budget or not perceive any distinction between the three categories. Thinking about what needs to be done is relegated to other roles. The coder just needs to implement the functionality written in the user story that is further documented in extensive functional designs. Everything is supposed to be laid out for her, and if there are any challenges, she can ask the local sheriff – ehem the development lead for some help. She is spoon-fed, and this is how they want her to be. In the end an army of Quality Assurance professionals are going to test the code and make sure that it does what it is supposed to do.

However, spaghetti spaghetti, copy-pasta, mangiare… Good code this does not make.

Good Architecture to the rescue

Well, if you can't recruit the best and brightest and can't Moneyball your way to them, what other options you as a software development manager have? It was told that in the early days of Facebook, the slogan was "move fast and break things." This lead to an unmanageable jumble of code that finally, they dropped "break things" from that slogan. Many companies have realized the need for proper software architecture. This architecture is, for example, a house with proper security measures that make it hard to vandalize.

The rise of Microservices in recent years has allowed teams to focus on writing their best code possible by making the task more manageable. A smaller service means that developers only focus on the part of the code they can understand and better manage. Bad code is mostly the effect of working in a huge code base that you can't understand akin to living in a city that is run down by crime. An old Egyptian saying goes "Would the guest not dance if the host is playing the tambourine?"

Yet architecture is not a panacea. We all know of architectures that are not respected or are so rigid that they don't keep up with the current business requirements. At the same time, a single sheriff can't monitor a large enough software team. What can be done?

Two heads are better than one?

The "eXtreme Programming" movement has a straightforward premise: take good software development practices to the extreme. This is where pair programming has been mostly successful. Put two developers together on tasks and continuously rotate them. XP is about method and philosophy and jazz, but mostly it is about test-first development, code quality, and pair programming. If you never did pair programming it can be intimidating at first or you might fail to see the value. One programmer at the prospect of pairing was mostly concerned with his Spotify subscription "Should I cancel it since I need to talk to my peer. It's such a waste of time". Two mice. Two keyboards. Two screens mirroring the same thing. And four sets of hands. The pair would swing between one writing the test case and the other coding the functionality. Between discussing the design and implementing it. Between telling a joke and laughing at it. "Go for it," the operative three words for the first time peer to take over the keyboard and implement the little bit of functionality. A short function. Straight to the point because the test was written beforehand, and it forced her to have a small manageable function.

At the same time, the test-driven nature means that you are making code that will document itself with tests. Code that will not regress. Code that is clean and pure. Dare I say code that doesn’t need the invisible QA army?

A panacea of having the best and greatest. A race to the top, instead of a race to the bottom. The two heads are reviewing the code. So you don't need a code reviewer. You no longer need a sheriff. The top coder on the team – selected by the team – will end up with a leadership role. But this is a servant leader. Making sure the team excels.

And the COVID-19 can’t stop XP. There's remote pairing with tools like Teams, Zoom, and others.

Developers, Developers, Developers, Developers, Developers…

We come to close the loop with one word: Developers. Who doesn't remember the hilarious videos of Steve Ballmer shouting that he needs Developers to use Microsoft software? The same Steve Ballmer who halved the value of Microsoft. Yet he is right in a way, what is needed is developers. Good developers. Great Developers. The problem with all these management schemes is to manage an army of mediocrity. If your army is excellent and strives for excellence, they will be self-organizing. They won't need huge layers of management. The managers' job would be just "Get out of the way and let the workers do their best." When you have more managers on a project than developers, it is a telling sign that something is grossly wrong. And indeed, this is what plagues a lot of Enterprises today.

Yeah, but it will cost me

You might retort that it will cost you more. Yet all software costs money. The question is, do you want to pay the full price now, or do you want to get stingy, penny pinche, and in the end pay more when it comes to fix the bugs and restore order in the neighborhood.

XP has a moment of glory recently, mainly because it works. You don't need a sheriff to lay down the law in town. Your developers will monitor one another and work under the umbrella of the common law. You will still need architecture, but it has more chances of being applied when the Architect has her hands in the code a well.

Ways forward

We have exposed the Broken Window Theory of software development of how the craft of code writing can be endangered by run-away bad development and letting things slide. We examined whether the solution had a Development Lead act as a sheriff and constantly monitor a team of developers and have proposed that the best solution is to have pair programming in an XP setting. We challenged some old-timers' wisdom that XP will cost more money and suggested that better code quality will reduce the total amount spent on the project.

References

The economist magazine, to make the world richer let people move

Migrants who move from lower- to higher-income countries typically earn three to six times more than they did at home, according to the World Bank. The simple act of moving makes them more productive, because rich countries have better institutions, the rule of law, efficient capital markets and modern companies. Construction workers in rich countries put up better buildings because they have better tools, reliable electricity and their employer does not have to pay off corrupt local officials. Scientists in rich countries make more breakthroughs because they have better laboratories and a wider selection of other scientists to work with.

Thomas Friedman the world is flat

Instead of trying to defend low-wage assembly jobs, Mexico and other middle-income countries should focus on creating jobs that add higher value. Only if more productive companies with higher-value-added activities replace less productive ones can middle-income economies continue down the development path

a result, added Watanabe, Japanese companies, to remain globally competitive, have had to shift some production and a lot of assembly of middle-range products to China, while shifting at home to making "even higher value-added products." So China and Japan "are becoming part of the same supply chain."

Educating more people at the tertiary level has two effects. First, it produces more people with the skills to claim higher-value-added work in the new niches that require more pattern recognition, synthesizing,