Coding within company constraints

Organising for microservices

Conway’s law states that any organisation that designs a system will produce a design whose structure is a copy of the organisation’s communications structure. According to Perry Krug, director for customer success at Couchbase, Conway’s law cannot be fought, but he believes that by understanding its influence, it is possible to work within its constraints.

Its influence extends to the realms of software development, which means modern software development practices need to be cognisant of the organisational structure that exists within a business.

At one level, this may seem to contradict the more collaborative working practices that define modern software development. “You need to collaborate to successfully develop applications,” says Krug. “If new ways of working make close collaboration difficult, collaborate loosely. If shared resources are causing constraints and bottlenecks, couple software more loosely to its foundations. Trends like microservices are an explicit recognition of these constraints and should now be coming into their own.”

Some industry experts believe that microservices architecture empowers software innovation.

One of these is Arup Chakrabarti, senior director of engineering at PagerDuty. “Microservices enable a far greater focus on customer experience than was previously the case,” he says. “Isolating areas of the overall code base enables the creation of mini innovation factories across the business – each operating at its own pace. With less to integrate, there’s less stepping on toes.”

However, in Chakrabarti’s experience, the use of microservices can itself lead to an explosion in complexity and all the challenges that come with that.

Bloomberg, for instance, has split some of its monolithic applications into containerised microservices that run as part of a service mesh. But this architecture has brought its own set of challenges, particularly as it can sometimes be hard for developers to fully understand what is actually going on.

Peter Wainwright, senior engineer on the developer experience (DevX) team at Bloomberg, says distributed trace can help, since it allows an engineer to see all of a service’s dependencies. Downstream services can also see which upstream services rely on them.

“A prime example is our company-wide securities field computation service. Calculations around the universe of securities can take place in many places. Knowing where to route requests for such computations is non-trivial, so we use a sort of smart proxy that becomes a ‘black box’ router for requests,” he says.

“Services provide data for requests without needing to know who’s asking. Unfortunately, this presents an obstacle when there are performance problems. Distributed trace restores visibility into ‘who I am calling’ and ‘who is calling me’ that the monolith previously gave engineers implicitly.”

To ensure its engineers could focus on their applications, Bloomberg has built scalable platforms using open source products like Kubernetes, Redis, Kafka and Chef. This, says Wainwright, enables developers to use turnkey infrastructure for the heavy lifting and drop in their application code.

Testing in modern software development

In terms of reducing bugs, there is plenty that can be gleaned from how open source code is tested.

“Successful open source projects, like the GNU Compiler Collection (GCC) that builds large parts of a Linux distribution, for decades have required running a test suite before submitting a patch for inclusion,” says Gerald Pfeifer, chief technology officer (CTO) at SUSE.

He says open source projects like LibreOffice use tools such as Gerrit to track changes and tightly integrate those with automation tools such as Jenkins that conduct builds and tests.

“For such integration testing to be effective long term, we need to take it seriously and neither skip it, nor ignore regressions introduced,” says Pfeifer.

He believes extensive automation is a key success factor, as is enforcing policies, to ensure code quality. “If your workflow involves code reviews and approvals, doing automated testing before the review process even starts is a good approach,” he says.

According to Pfeifer, LibreOffice and GCC share an important trait with other successful projects focusing on quality. “Whenever a bug is fixed, a new test is added to their ever-growing and evolving regression test suites,” he says. “That ensures the net which is cast constantly becomes better at catching and hence avoiding, not only old issues creeping in again, but also new ones from permeating. The same should apply to new features, though when the same developers contribute both new code and test coverage for those, there tends to be a risk of having a blind eye.”

Describing how its service mesh architecture is tested, Bloomberg’s Wainwright says: “Instead of bundling changes on a fixed schedule, better testing enables us to make changes more rapidly. Changes are smaller, so there’s less that can go wrong and mitigation is simpler.”

While code quality is often measured in terms of defect rates, error budgets and the like, Wainwright belies the real benefit of easier testing is psychological. “Teams confident that their tools have their back will make more progress in a shorter time,” he says. “That means we deliver more value to our customers faster.”

However, as PagerDuty’s Chakrabarti points out, one of the biggest changes of recent years is customers’ intolerance for anything less than perfection when it comes to performance. “Many believe engineers have their own ‘hundred millisecond rule’ to contend with. ‘Come back later’ is no longer an acceptable response,” he says.

According to Chakrabarti, the idea of engineering web-scale applications is pretty much taken as given these days, particularly in the wake of the coronavirus, which has seen companies scrambling to support new ways of working. Data from PagerDuty shows that new code and heavier volumes of traffic have resulted in more incidents – as much as 11 times more in some sectors.

“As an industry, we’ve got better at fixing things without affecting customer experience, including through an automated approach to digital operations management. It’s still early days for auto-remediation, but we are starting to hand over more control to technology. With time and further advances in machine learning, we ought to be able to teach some of our systems to self-heal based on past events – even in circumstances that only occur once in a blue moon,” says Chakrabarti.

The pandemic’s influence on software development

Beyond the technical aspects of developing bug-free code, built in a modern way, such as the self-contained IT architectures and microservices approach discussed earlier, the coronavirus pandemic has made team communications a top priority.

“In a socially distanced world, product managers are more important than ever. Developers will know what needs to be done to create a solution – if they know what they’re supposed to solve,” says Couchbase’s Krug. “However, solving these issues means getting inside the customer’s head. Expecting developers to become psychologists is a step too far. Instead, there needs to be clear communication between product managers and developers over what customers’ issues are and ideally what they want their end state to be.”

This means fast communication with distributed teams is essential, adds Krug. “If it isn’t in place, a priority for all teams in a business should be making sure it is.”