Platform engineering - Percona: Driving data in the new normal
This is a guest post for the Computer Weekly Developer Network written by Michal Nosek, senior enterprise architect at Percona.
Nosek writes in full as follows…
Platform engineering is about making developers’ lives easier.
Automating the steps needed to create and maintain infrastructure keeps your developers focused on building software, not keeping the lights on. But building applications or delivering new services leads to a more and more diverse database ecosystem that also has to be managed.
Developers don’t want the job of managing databases and data, even though their applications created it and that data can be essential to the business. So, how can you make data management part of your platform engineering plan, so you can approach it all in the same way?
Getting it right on day one
One useful idea is to look at your issues as day one or day two problems. Day one problems are those that face you when you first deploy an application – so, what will you use to run the application, how will it be hosted and so on. From a data perspective, that will involve choosing how to deploy the databases you want to use, what is the optimal topology for each of them and what integrations with other components are needed.
Each of these elements involves a choice. Should your developers have to make that choice consciously, or should the platform handle this on their behalf automatically?
Day two issues are found around areas like maintenance, management and scaling – just how should that infrastructure run and are your assumptions around data growth or performance accurate? As applications evolve over time, they will get new features that will add to the complexity side. Your application’s platform – and by extension, your platform engineering approach – will have to evolve too.
To make things easier, your platform engineering design should make it easier to get started. For the majority of projects, you can standardise your process around provisioning and running databases to limit the choice and concentrate on getting started. If your developers really need something specific – say a vector database for generative AI, or a time-series database for IoT data – then your developers can check for extensions for the datastores that are already running and available. Solutions like PostgreSQL are easily extensible and can support different use cases while still running the same underlying technology, simplifying the whole platform.
Platform engineering goes hand-in-hand with a cloud-native approach to infrastructure management.
You will probably use Kubernetes to orchestrate and manage your workloads around demand. You can use Kubernetes Operators to carry out tasks around complex components such as databases automatically.
Day two problems
Platform engineering also has to encompass all the day two operations – management, maintenance and operational tasks like backup and security – that infrastructure requires.
Kubernetes operators can manage those database tasks that come up and build up over time. For example, data should be backed up for disaster recovery. You can use a Kubernetes Operator to trigger that task automatically on a regular basis. On top of this, you can also look at how your Kubernetes Operator can deliver the right level of availability for your database components. Does that application need to be available all the time? In this case, you might ask for it to run in a clustered environment that can deal with potential failures in infrastructure and the Kubernetes Operator should then deploy a cluster rather than a single database instance.
To achieve this, you will have to look at how you integrate your management tasks into your Infrastructure as Code templates and plan ahead on what tasks your infrastructure can deliver for you. For more specialist areas like security, Security-as-Code can define what has to be in place around requirements like authentication, authorisation and specific security requirements – this template can then be applied to any workload that creates or handles sensitive data.
Platform engineering is designed to make it easier for developers to run applications and for organisations to save resources on managing them. By automating management tasks and hiding that complexity, the goal for platform engineering is to deliver more reliable and responsive infrastructure without requiring as much in-depth specialist insight into areas like databases.