Matillion reduces data pipeline creation from 40 hours to 1 with Claude-powered AI assistant
Industry:
Software
Company size:
Large
Product:
Claude Code
Location:
Europe
41% faster pull request completion with Claude Code
80% of Matillion's developers use Claude Code, reporting an average 41% reduction in time spent on pull requests
40x faster pipeline creation
Sophisticated data transformations that took a customer 40 hours now take 1 hour with Maia
The problem
The solution
Matillion is a data integration platform that helps data teams build and manage pipelines for AI and analytics. Thousands of enterprises including Cisco, London Stock Exchange Group, and Slack use Matillion for insights, operational analytics, and machine learning applications.
With Claude, Matillion:
- Creates data pipelines through natural language: users describe what they need in plain English, and Maia builds production-ready pipelines
- Translates legacy codebases automatically: migrates existing ETL processes without manual recreation
- Performs autonomous data operations: handles everything from sampling data to optimizing transformations at machine speed
- Enables self-service for non-engineers: business analysts can now build data infrastructure that previously required specialized expertise
- Maintains and optimizes existing pipelines: continuously improves performance and adapts to changing requirements
Selecting the right foundation model
Matillion chose Claude after 18 months of testing across multiple model providers. Their evaluation framework measured three critical capabilities: understanding proprietary domain-specific languages, multi-step reasoning for complex data operations, and reliable tool-calling decisions.Claude consistently outperformed other models across all metrics. Specifically:
• Strong reasoning and tool-calling: Claude demonstrated sophisticated ability to select the right tools and sequence complex operations—critical for autonomous data pipeline management
• Reliability: Stable performance across model versions meant Matillion could upgrade to newer Claude releases without re-engineering their entire prompt architecture
• No training required: Claude understood Matillion's proprietary YAML-based Data Pipeline Language (DPL) immediately, despite having no public training data
• Consistent behavior: The team found they never had to "fight" the model to achieve desired outcomes
"The decision to choose Claude was entirely data-driven," the Matillion team explains. "We tested multiple models side by side, and Claude consistently delivered the best results for both accuracy and reliability."
The challenge: Data teams drowning in backlogs
Data engineering teams struggle to keep up with demand. Executives requesting dashboards and analysts needing new data sources often wait weeks or months for simple requests. This problem is about to get worse—generative AI adoption is significantly increasing the load on already stretched data teams who must build pipelines feeding AI processes while handling the flood of data these systems generate.
Data engineering had become a critical bottleneck. Even simple requests needed to climb months-long backlogs, creating friction across entire organizations. Matillion recognized that traditional approaches couldn't scale to meet the exploding demand for data pipeline creation and maintenance.
Building Maia: From concept to production
“Maia handles everything from legacy ETL migrations to building production-ready pipelines at machine speed, with logic quality we can trust,” said Ammad Baig, Director of Enterprise Data & AI Services at Precision Medicine Group. “It's fundamentally accelerating our workflow while reducing manual overhead.”
The transformation began when Matillion found that their proprietary Data Pipeline Language, designed to be human-readable, was also machine-readable. While they initially assumed LLMs couldn't understand a language with no public training data, Claude proved them wrong. The model demonstrated sophisticated understanding of their YAML-based format, respecting instructions and generating valid pipeline definitions consistently.
Matillion built a sophisticated testing framework using three LLMs in concert—one running Maia, another playing the role of tester, and a third acting as judge to verify outcomes. This framework let them measure performance tangibly: tracking time taken, tool selection accuracy, and token consumption across scenario sets.
The evolution from simple auto-documentation to full agentic capabilities took two years, but the results justify the investment. Maia can now access almost the complete functionality of the Data Productivity Cloud through tool calls, performing operations that challenge even expert data engineers.
The outcome: Redefining data team productivity
Since Maia's general availability in June, customers are seeing significant changes in how they operate. Teams are delivering more with their existing resources, with analysts now able to self-serve data infrastructure that previously required specialized engineering expertise.
The impact extends beyond external customers. Eighty percent of Matillion's developers adopted Claude Code, reporting an average 41% reduction in time spent on pull requests based on internal surveys. This creates a multiplier effect—junior developers handle tasks that would previously go to seniors, while senior developers tackle work that only veteran technical leads could manage before.
Looking ahead, Matillion plans to expand Maia's capabilities to capture semantic information from data catalogs, existing pipelines, and documentation. The goal is to give Maia domain expertise about each customer's unique data landscape and business terminology—understanding not just that there are revenue fields in a database, but which ones represent the true source of truth for specific business contexts.
"Capturing these types of semantic detail and having Maia use them efficiently is our next challenge," the Matillion team explains. "Anthropic's models will be key to that."
Video caption
Video caption
Video caption