GreenOps – Astronomer: Workflow orchestration is the hygiene layer a data team needs
This is a guest post for the Computer Weekly Developer Network written by Carter Page in his capacity as EVP of R&D at Astronomer.
Astronomer is the company behind Astro, the modern data orchestration platform built on Apache Airflow. Its work sits at the intersection of workflow automation, data engineering and operational efficiency across cloud environments.
Page writes in full as follows…
Cloud cost conversations tend to focus on infrastructure pricing.
You know the level we’re talking about i.e. this is the zone where we find reserved instances, spot pricing, autoscaling and so on.
These elements, techniques and approaches matter, but they miss something quite major: poor data hygiene is a cloud waste source that is commonly overlooked.
Editor’s note: TechTarget defines data cleansing (also referred to as data cleaning or data scrubbing) as the process of fixing incorrect, incomplete, duplicate or otherwise erroneous data in a data set. It involves identifying data errors and then changing, updating or removing data to correct them.
Data cleansing improves data quality and helps provide more accurate, consistent and reliable information for decision-making in an organisation. Page’s context extends the notion of data hygiene potentially even further to the very source of data (and the way it is subsequently allocated resources) to ensure that teams do not waste compute (or indeed analytics or storage) capacity.
The Astro Observe Data Quality approach reminds us that standard data quality checks (monitoring row volumes, tracking null percentages, detecting schema drift) catch common problems and work well on fixed schedules. But every organisation has unique business rules that determine whether data is actually trustworthy,and those rules often need validation the moment data lands, not hours later on some predetermined schedule.
Sustainable software, not squeezed
Except in a few niche cases where highly specialised software is deployed, nobody is squeezing cycles out of an algorithm by coding portions in assembly. What matters now is whether that code should be running at all… and whether it’s using the right resources when it does.
After a point, correct resource allocation matters more than code efficiency.
Manually tuning the correct resourcing for individual DAGs is rarely worth it. But when you have hundreds of DAGs, collective tuning adds up. Having an orchestrator that can help with that is a big deal.
As we ship more event-driven capabilities, there’s a cost component to this as well. Running workloads only when data actually arrives, rather than on fixed schedules, eliminates waste at the source.
TECHNICAL NOTE: A Directed Acyclic Graph (DAG) is a mathematical structure consisting of nodes connected by directed edges. It flows in one specific direction, ensuring that no path ever loops back to itself.
Lagging indicators, deeper problems
Cloud cost and carbon metrics are lagging indicators. By the time you see the bill, the waste has already occurred. Sometimes these metrics point to something more basic: using the wrong tool for the job.
Using a data warehouse like a graph database is costly and slow. Same for running massive OLAP workloads on row-based relational databases designed for transactions. No amount of infrastructure optimisation fixes an architectural mismatch. These decisions happen upstream, in how data flows are designed and orchestrated.
Orchestration as the hygiene layer
Workflow orchestration becomes the practical hygiene layer for data teams. This is where optimisation happens. It needs to be done at scale to matter, and it requires a view of the end-to-end picture across systems.
An orchestrator sees which pipelines run, when they run, what resources they consume and whether their outputs are actually used. That’s the starting point for both FinOps and GreenOps.
Manual auditing is expensive, time-intensive, and immediately out of date. One-off cleanups don’t treat the underlying problems. The intelligence needs to live in the orchestration layer itself. When your orchestrator understands resource patterns across hundreds of workflows, it can make recommendations that no human team could manage.
That’s where GreenOps and FinOps stop being spreadsheet exercises and become operational.
Editorial analysis
It certainly seems like Astronomer regards views GreenOps as a tangible and defined practice that should work as a natural extension to any FinOps initiative that an organisation has in place. The company has made comments such as “wasted cloud spend is wasted electricity”… and that’s a reality that (certainly in the previous decade, if not also today) we can see many IT teams not directly understanding and working to address.
Astronomer Astro Observe is described as a technology that provides “deep visibility into pipeline health and runtimes” and one that is able to identify “zombie” software application (or other data engine) tasks that represent processes that do run, but that have no value to provide) or long-running DAGs. Astronomer tells us that it allows teams to deploy in over 50+ regions around the world to allow IT functions to intentionally schedule heavy workloads in “greener” regions when and where possible.
Image: Google Gemini