Software development practices in the enterprise have traditionally focused on delivery high-quality code built on proven platforms. But the web and the emergence of apps, built on web scale infrastructure, are rewriting the rulebook.
In fact, businesses struggle to compete with startups that can somehow maximise the value of the new economy, and are able to undermine traditional business models.
“Over the past 20 years, IT has been set up for efficiency, cost reduction and doing things as safely as possible,” says Benjamin Wootton, co-founder of Contino.
He says companies are now driven by the need to work faster and are becoming more agile in order to improve the customer experience.
“This is applying pressure on IT and how we develop software,” he says.
Whereas IT heads previously implemented heavyweight internal IT processes and used outsourcing to reduce cost and maintain quality, in Wootton’s experience, this style of running IT slowed down IT departments. “DevOps and continuous delivery allow organisations to operate faster, which is what enterprises want to do today,” he says.
But Wootton argues that among the challenges for IT leaders is the fact that big enterprises are risk adverse and tend to stick with a tried and trusted formula, often contrary to contemporary best practices.
Shifting the enterprise mindset
Kingsley Davis, a partner at Underscore Consulting, adds: “You want to deliver quickly and at pace, which means having a clear strategy about what things are not important for the product.”
Technology such as Docker, to enable developers to create code that can run in their own containers, along with the ability to have short feedback loops, helps businesses to adapt more quickly. Such technology and techniques form the basis of the cultural shift that companies of all sizes need to make to enable their developer teams to become more adept at delivering software quickly, says Davis.
“Culture is very easy to instil when there is a small group of people,” he says. “Hiring is key.”
Davis recommends that IT leaders plan in advance, and hire people appropriate to the direction the IT strategy is taking.
Russ Miles, lead engineer at Atomist, believes IT leaders can learn much from the way webscale organisations approach software development. “Organisations of any size have to compete,” he says.
The speed of change is such that IT leaders cannot afford not to adapt their business processes. “People look at what Netflix is doing and the thing to take away is that agile software development will only get you so far,” says Miles. “The software itself needs to be as adaptable as the process.”
What this boils down to, says Miles, is that IT leaders need to figure out how to adapt systems and the work IT departments need to do, to achieve the speed and flexibility required by the business.
A case for smarter analytics
If they cannot meet the needs of the business, business users will go elsewhere, or even develop the systems themselves.
“Business users are driving software development,” says Frank Ketelaars, big data technical leader for Europe at IBM. This is a form of shadow IT, he adds. “They use spreadsheet data warehouses as their own analytics platform.”
Given the need for developers to be productive and create applications quickly, Ketelaars says technologies such as Apache Spark make it possible for businesses to develop machine learning capabilities more easily than before.
Also, the availability of deep learning services is pushing the boundaries of analytics in terms of the massive computational intelligence such algorithms can bring to bear on hard-to-solve problems. But with such technological developments comes new challenges.
Frank Ketelaars, IBM
Ketelaars says the plethora of analytics tools available makes it hard to validate data models. “It is extremely difficult working with the variety of analytics libraries and tools that are available,” he says.
What developers need, says Ketelaars, is a route to analytics services via a common programming interface, giving them the features of an analytics tool within their own applications.
Another challenge facing analytics applications is how to make sense of the data. “Deep learning is here, but one thing that is missing is context,” says Ketelaars. “If I have a picture with two people running after each other, and one has a frisbee, I know this image is about playing.”
Deep learning algorithms can instantly recognise the image of two people running, he says, but adds: “The context changes dramatically if one of them has a chainsaw. This is where context controls what you should do with the image.”
For Ketelaars, understanding context will be a key requirement in applications, to understand the meaning behind the analytics. “You have to start thinking about what data you have to control the behaviour of your application,” he says.
Essentially, applications become smarter, providing users with the information they need based on a deep contextual understanding of what it deems relevant or important.
Arguably, there is room for improvement among the array of tools, building blocks and techniques that developers use to create software, says Phil Trelford, founding member of #F Foundation. “A general-purpose programming language is a bit like a spanner and we are all trying to build large systems with spanners,” he says. “What I would like to see is precision tools.”
In fact Trelford goes further, saying the industry needs better “meta tools”, in other words software tools to help the developers of programming tools build precision instruments, rather than generic spanners.
While, as the saying goes, “a bad workman blame his tools”, software developers are keen to see improvements in the tooling they use. In part, this helps them cope with the added complexities when coding and operations become one, as in DevOps, says Trelford.
“Personally, I would like to be able to say, ‘I need to make this thing happen’, then make it as quickly as possible,” says Miles. But today, developers need to draw together a lot of threads to create and run applications successfully, and complexity is increasing all the time, he says. “Smart tooling will help them handle the cognitive overhead,” he adds.
Finding out more about our CW@50 panel discussions
- In this video, we speak to a panel of CIOs about best practices in working with IT startups.
- Leading technology experts discuss the trends in enterprise software and how this will affect future applications.
- In this panel debate filmed at our 50th anniversary, we contemplate how IT could evolve over the next 50 years.
Enterprises appear to be following smaller companies in adopting new, more productive ways to code. For instance, WhatsApp, which was developed by a handful of Erlang developers, was sold to Facebook for $19bn, while Walmart recently acquired the F#-based Jet.com e-commerce platform for $3bn.
While procedural programming languages such as Java and C have been the bread and butter of enterprise software development, what has been particularly interesting for software development is the uptake of functional programming in recent years, particularly among companies that need to support large numbers of internet customers.
As Trelford explains: “Apart from Java, which has huge user groups, the biggest programmer user groups in London are those of the functional programming languages. I run the functional London meeting here. We have been meeting here for about six years and we have over 1,000 members. Scala, I believe, has 1,500-2,000 members and Clojure has been growing, as has Erlang.”
Many proponents of these programming languages talk about how little code they need compared with using a procedural language, which makes them attractive for writing code quickly, says Davis.
Malcolm Sparks, Juxt
But among the reasons for the interest in functional programming is reliability. Malcolm Sparks, director and founder of Juxt, says: “One of the issues we still face as developers is how to build really big software systems. The bigger the project, the more likely it is to fail. It is easier to build small software systems.”
Sparks argues that, ideally, software developers should look at architecting systems by integrating many small software components, each of which has been developed to be highly reliable. “We are moving to a world where individual software systems are becoming so critical and so important that we had better build them using the best tools and the best languages, and this is why we are seeing a rise in interest in functional programming,” he says. “Functional programming is a better approach for writing highly reliable systems.”
Changing software development landscape
Among the changes Miles is seeing is that software development is no longer a factory floor to churn out new products. Rather, he believes software development is evolving into a continuous R&D practice.
“Companies that regard software as a driver for them are the ones that will win and one of the pieces of advice I give to company boards is that they should not think of software development as a general problem that we can solve by throwing more people at it,” says Miles. “Think of software development as a place where you might be surprised what comes out.”
Enterprises do not often have the luxury of greenfield development, but as Trelford points out, where enterprises need a new system, there is the opportunity to experiment with the least risk.
Russ Miles, Atomist
Wootton says: “Everyone is always excited about the new greenfield stuff, but there is a real business case with legacy.” Often, the real business case is actually the J2EE or .Net code that has been running for a decade or more and requires a big support team.
“You might do something crazy like services on your mainframe, but it turns out this may be where you get the biggest return on investment,” he says.
“Legacy is a bad thing,” says Sparks. While it is exciting to create new code based on microservices and perhaps functional programming, the biggest challenge faced by corporate IT is often how to handle a growing legacy of old stuff.
“It can be the millstone that drags you to the bottom of the sea,” says Sparks, who urges CIOs to look constantly at what can be decommissioned, and have development teams write new applications. These not only help to move the business forward but, at the same time, enable IT to decommission something else.
Davis believes that the new techniques available to developers, such as reusable microservices, containers, functional programming and continuous delivery, offer enterprises an ideal opportunity to reduce risk and improve reliability.
“It is all about safe, small-scale scalability,” he says. The tools and techniques discussed enable IT departments to avoid the risk of modifying mission-critical applications by augmenting them in a highly controlled way, adds Davis.
Why software development matters
Technology is creating jobs, not just in programming but also in sales, marketing, operations, finance and human resources in tech businesses and traditional industries.
Software engineers power technology, and they are increasingly able to come up with innovative solutions that enable UK businesses to compete in a global market.
More than 600,000 startups were created in 2015, creating jobs and driving the UK economy. Many of these were either technology businesses or used technology to remove significant barriers to market.
Technology has become integral to any business. Marianne Lake, CFO of investment bank JPMorgan, was recently quoted as saying: “We are a technology company.”
Retailers, publishers, manufacturers and other industries that might not traditionally be branded as technology companies, increasingly realise that their survival and future success depends on gaining a competitive advantage through technology.
They often seek help to drive innovation in their engineering teams through engagement with developer communities such as Skills Matter.
Someone put it to me that the UK’s recent decision to leave Europe was as much driven by a discomfort with the speed of innovation and globalisation as it was by other reasons, such as immigration or concerns about sovereignty.
He pointed out that my industry, the tech industry, is growing too quickly and perhaps needs to slow down a little. He illustrated this with an example. A truck driver working for a transport company for the past 20 years, probably earning £30k-£35k per year, still a decade or two away from retirement and two kids to raise, is going to be unhappy with innovation such as driverless cars, and so innovation is creating an unfair world.
This may of course be true, unless we give his daughter the skills to design that driverless car, his son the ability to come up with new business models that will launch a whole new industry, and his grandchildren the opportunity to gain the work experience they need to take a new concept to a global market.
Running a business for just over a decade has taught me you need to pick your fights. Trying to slow innovation is not one I would pick. Coming up with new ideas and approaches to challenges is integral to humankind. Someone invented a car, a delivery truck. Someone invented the business model for a successful transport company.
Even if we were able to slow or stop innovation in the UK, we would never be able to do so in the wider world. And if we try, we will deprive our kids of the opportunity to gain the skills they need to be participants in the future economy, to find jobs in this country instead of elsewhere.
Technology and software development is powering growth in every industry. It is giving many British businesses a competitive advantage and is creating new industries where our kids can find a successful future.
Wendy Devolder is CEO of Skills Matter, a community of 70,000 software engineers.