Innovation, transformation, and the importance of embracing “ugly ideas”

This is a guest blogpost by Madalina Tanasie, Chief Technology Officer, Collibra

Given the problems businesses face today, from supply chain issues to fears of an economic downturn, it is more important than ever to embrace and drive innovation. How companies can foster a culture of innovation, where their engineers and product managers feel confident to push the limits of their technology and services, continues to be a highly discussed and debated topic. And as companies grow larger and they become more process oriented and risk averse, the challenges they face in getting their workers as well as their customers comfortable with change and open to innovative ideas become more complex.

These are challenges and tradeoffs that I have repeatedly faced throughout my career. As an engineering leader in the highly-regulated life sciences industry and then as CTO at Collibra, I have led engineering teams through transformation journeys and have consistently tried to promote the importance of innovation in both what we produce but also in how we do it.

In regulated industries, risk aversion is the name of the game. In times of economic downturn, the search for high impact, low-risk and low-cost solutions becomes the focus of almost everyone, customers and company leaders alike. To continue to innovate in these circumstances, an organisation must be comfortable with some level of uncertainty and ambiguity and be willing to experiment with the hope – not the expectation – of a marketable outcome.

One approach that I found to work well in these circumstances is what a great mentor of mine (Isaac Wong of Cockroach Labs) called “giving ugly ideas a chance to live”. These are high-potential ideas that are initially unpopular and often dismissed because they are either too radical or they come from unexpected sources. In my experience, it is common for great innovations to come from ugly ideas; if you pursue and develop them, they may reveal a gem underneath.

To put this idea into practice in an environment where innovation is desired, employees are encouraged to develop their ideas, create a prototype or business case, test it, analyse the results and share their findings. This can be achieved in a variety of ways, from Hackathons or dedicated capacity for innovation like Google’s famous “20% rule”, to incorporating exploration time into the day-to-day routine. This process, often referred to as “prototype and test”, requires an organisation to have a strong evaluation criteria that is used consistently to measure the results and decide whether or not the idea has potential.

No matter how this is implemented, there are two important components to this approach. First, you need a culture where people feel safe to bring forward unconventional ideas and where the decision makers can hear these ideas with an open mind. Second, you require the tools and means to fail fast.

To address the first component, companies should consider developing a culture where respectful dissent is encouraged. At Collibra, our approach to addressing differences of opinion focuses on being open, direct and kind: everyone is encouraged to speak up when they disagree or if they think that a different approach would be better.

For the second component, we believe it is important to drink your own champagne, so we leverage our data governance products and data intelligence platform to decide what technical and business ideas to implement. We reserve 10% of our Product & Engineering capacity for innovation, and we also have “prototype and test” embedded in various parts of our Software Development Lifecycle methodology.

One pitfall to be aware of with the “prototype and test” approach, especially as the size of an organisation grows, is its start-and-stop nature which can become unproductive if your engineers are pursuing too many ideas. Creative engineering teams will have lots of competing ideas, but creativity alone is never enough. True innovation needs honesty and discipline in the selection process and it must be followed by exceptional execution.

From Pirates to Navy

Implementing the processes required to support and foster a culture of high-value innovation where creativity is harnessed and turned into long-lasting outcomes requires hard work and discipline. This transformative work, whether just within the engineering team or across the organisation, will often take place behind the scenes and may not necessarily be very visible, but it can have a huge impact on an organisation’s ability to innovate.

An analogy for this transformation is the change a crew of pirates would need to undertake to become a crew of the navy. A pirate crew is composed of highly-skilled, versatile, adventurous individuals who are habitually taking high risks for the often unfulfilled promise of high rewards. But it lacks discipline and cohesion. Ideas are pursued and explored haphazardly. On the other hand, a navy crew is as entrepreneurial but is more disciplined, streamlined, and has established objectives and ways of operating. This team is working together towards a common goal, and there is clear accountability and responsibilities.

One of the possible ways to approach this “Pirate to Navy” transformation is to implement a New Product Introduction (NPI) process. This process serves as a framework for taking a product or new feature from the initial idea stage to a prototype and then into a final product. It is a formalised, systematic and structured way to implement the prototype and test methodology.

Prior to implementing our own NPI process, Collibra had an informal approach. Product managers would do market analysis and speak to a few customers, then they would come up with a specific business case that was usually focused on a subset of customers, and then start the development process. In some cases, the products built and released by the engineering team were edge cases that were only useful to a handful of customers and did not meet broad demand. In other instances, the product was highly successful, but we had missed the opportunity to prepare a marketing campaign, or the product was in such high demand that it would overwhelm the engineering team and the release would lack support.

The NPI process recognises that every innovation must have the full support of the company. We need to understand why we are building it, who it is for, and how we should price it. Does it enhance an existing service or is it a new product entirely? How will we release this feature? Does it require a marketing campaign or content creation to support enablement? The product is not released until all aspects of a release have been addressed.

Today, we track all our initiatives through the NPI process, monitoring the progress of an idea through various stages from business plan to pricing to marketing campaigns and so on. The process allows us to whittle away ideas that may prove to be distractions, allowing us to test prototypes and fail faster. Most startups and scale-ups do not adopt an NPI process until they are quite large, but I would highly recommend doing it sooner than later.

Crossing the scaleup threshold

Adopting and refining processes like NPI is not just about fostering innovation. It is part of a company’s broader journey from being a startup to a scaleup to a mature organisation. Along the way, they will discover other areas that necessitate transformation and becoming more data-driven. Perhaps their product has grown organically but now is cumbersome, or their processes that worked in the past are no longer appropriate.

Some companies will be tempted to take a shortcut on this journey and try to rebuild their product and their culture from scratch. If a company is 10 years old, new technologies and best practices will have emerged in that time and the people in the company are likely familiar with these advancements. It can be tempting to try and start over with newer technology and to think that you can quickly rebuild better versions of the products developed over the last decade that have become difficult to maintain, scale or support. Many companies have attempted to reinvent themselves with a re-build from scratch approach, including Collibra, but few are likely to succeed this way.

In Collibra’s case, growing from a startup to a scaleup was challenging and required us to pursue a few “ugly ideas” – the most unpopular but ultimately the right one being to evolve, not rebuild. We transitioned from being an on-premise software vendor to a software-as-a-service (SaaS) model, which necessitated a significant architectural overhaul but also big cultural changes and improved discipline. There was plenty of reluctance to change our sales model – even some customers were sceptical and felt uncomfortable with their data being in the cloud. But making this transition was the right move both for the company and for our customers: it helped the business scale, it was cheaper than trying to manage hundreds of different product versions across our customer base and it allowed us free up capacity for innovation and to provide both better service and new capabilities faster. Also, evolving instead of rebuilding allowed us to take our customers on this journey gradually, without the need for major migration efforts or requirements to let go of old features to leverage the new ones.

Today, most of our customers are cloud-based, and we learned a lot in the process on how to release innovative products and support our customers as they transform and adopt this innovation. Completing this multi-dimensional transformation required a few iterations, and took trust, discipline, strong partnership, resilience and grit. In the end, what seemed like an “ugly idea” got us to where we needed to be and was fundamental in our success. It might seem crazy to change the wheels of the car while the car is in motion, but sometimes this is precisely what you need to do.