Astronomer Otto: A data engineering agent built for Apache Airflow
Data engineering and data science are the modern software engineering disciplines that run enterprises today.
These are the processes needed to keep dashboards current, AI models trustworthy and customer-facing applications running.
According to Taylor A. Murphy, principal product manager at Astronomer, data teams are still largely the same size they were two years ago, but the demand on them has grown considerably.
This causes pressure on “upgrade” debt that keeps accumulating, 2am production failures that eat the next morning and the need to try and retain institutional knowledge that often leaves.
Otto is a new data engineering agent built to address these issues.
“Data engineers spend too much of their time on work that scales poorly: manually tracing a failure through logs, planning an Airflow upgrade [Apache Airflow is an open source platform used to programmatically author, schedule and monitor complex data pipelines and workflows] across hundreds of DAGS [see below], onboarding a new engineer who doesn’t yet know which Snowflake connection to use for prod. Otto is built to take on that work alongside you and accelerate it,” said Murphy.
NOTE: An Airflow DAG (Directed Acyclic Graph) is a collection of tasks organised to define a workflow, defining the sequence of execution and dependencies.
Otto is purpose-built for Airflow and is the first to bring Astronomer’s operational Airflow knowledge directly to the engineers doing the work.
It knows the environment and learns from your team’s conventions. When asked to build a Dag, investigate a failure, or plan the next upgrade, Otto acts using actual Dag patterns, run history and proven upgrade paths.
Conventions accumulate
The longer a team uses it, the more it knows. Your conventions accumulate, corrections stick and new engineers inherit the context that would otherwise take months to absorb.
“We’ve spent years helping enterprise data teams run Airflow at scale. Through that work, we’ve accumulated knowledge about what breaks, what works and what it actually takes to keep a production data platform healthy. That knowledge has lived in our support teams, our documentation and the expertise of the engineers we’ve worked alongside,” wrote Murphy, in a company blog.
Building it on Atronomer’s Astro platform means Otto has access to full operational history of an organisation’s data platform i.e. every pipeline execution, failure and correction is a record of how data actually moves… and that context is what lets Otto reason about the environment with precision.
Apache Airflow
“Data engineering work doesn’t happen in isolated tasks. It spans the full lifecycle: understanding what data exists and how it moves, building pipelines that process it reliably, diagnosing when things break, keeping the platform current and making sure institutional knowledge outlasts the people who hold it. Otto is built to accelerate every part of that work, not just the authoring step,” explained Murphy.
Before writing a single line of code, a data engineer needs to understand what they’re working with. Otto connects to data warehouses across 25+ connectors, including Snowflake, Databricks, BigQuery and Postgres and lets users query, profile and trace lineage directly in the agent session.
Otto reads task logs, traces dependency chains and checks run history against patterns it has seen before. Ask what happened to a Dag that failed overnight and Otto comes back with a root cause, the history of similar failures and a proposed fix.
CTO deep dive
Astronomer CTO LaNeve: The current state of Otto Memory is knowledge that is customer-specific and driven by actual usage.
The Computer Weekly Developer Network (CWDN) spoke to Julian LaNeve, CTO at Astronomer for a deeper dive in these agentic functions, automations and accelerations.
CWDN: Otto Memory stores team conventions and corrections – how do you prevent accumulated context from drifting into outdated or conflicting states over time?
LaNeve: Otto Memories are stored locally in Git. That’s a deliberate choice. Git-based storage means the memory layer is transparent and versionable. Teams can see exactly what Otto knows about their environment and correct it. It’s not a black box accumulating silently in the background.
The current state of Otto Memory is knowledge that is customer-specific and driven by actual usage, utilizing the recency of when the memory was committed. Coming soon, we’ll be adding review capabilities in Otto that will include the ability to regularly review memories for conflicts and updates. This will work within our upcoming memory API which will automatically curate memories for customers.
The design intent throughout is an agent that doesn’t just use your context, but helps produce it. The longer your team uses Otto, the more it knows about how your environment works and that’s a compounding asset, not a liability.
CWDN: With Otto accessing 25-plus warehouse connectors for lineage tracing, how is query-level data governance enforced to prevent unauthorised cross-environment exposure?
LaNeve: Otto’s exploration and lineage capabilities operate through the same connection credentials and access scopes your team has already established in Astro. There’s no separate data plane, no new attack surface.
When Otto queries your warehouse, traces lineage, or profiles tables, it uses the connections that already exist in your deployment with the same authorisation boundaries already in place. The Astro credential model is the enforcement layer and Otto inherits your permissions. This makes it both easier to get going (Otto has access to what you have access to) and more secure (Otto can _only_ see what you have access to).
At the organisational level, Astro admins control whether AI features are enabled at all via a single org-wide toggle that applies to Otto across every interface: CLI, IDE and API. Access to Otto’s proprietary context through MCP, coming in the near future, will also require authentication to Astro. The governance model is consistent across surfaces by design.
CWDN: Otto’s proprietary compatibility knowledge base covers version matrices and provider interactions – what’s the update cadence and validation methodology when Airflow 3.x providers change rapidly?
LaNeve: The knowledge base is built from three sources that compound on each other and it’s worth being specific about the methodology because it’s genuinely different from what you’d find in a changelog.
The first source is automated extraction across Apache Airflow upstream records, provider changelogs and runtime release notes, parsed programmatically so nothing slips through manual review. The second is Astronomer’s own migration history. We ran our internal platforms through the Airflow 2-to-3 migration first and every issue we hit became a knowledge base entry before it shipped outward. Third is fleet-wide signals: upgrade patterns and failure signatures observed across hundreds of enterprise Airflow deployments. When something breaks for one team, it becomes a detection for every team.
So for subsequent Airflow version updates, that first channel means we’re tracking provider releases as they ship, not on a quarterly review cycle. The third channel means we’re seeing what actually breaks in production environments before it surfaces in a public issue tracker. Those two signals together are what let us say Otto’s upgrade recommendations are grounded in what actually breaks, not just what the documentation says should work.
Otto is also model-agnostic by design. Inference routes through Astronomer’s LLM Gateway, where we whitelist particular models via the gateway and customers can select which model they want to use from that list. We test each model before publishing it to ensure it works well with the type of data engineering workflows that Otto is used for. We also use multiple providers for each model, so if one provider happens to go down, there’s a fallback, making it more durable.