Porch accelerates data and ML operations by migrating from Airflow to Union.ai

Porch is a technology-driven insurance and home services company supporting 11,000+ businesses across moving, utilities, inspection, and real estate. Their data and ML pipelines fuel insights that reach two-thirds of U.S. homebuyers every month—powering customer recommendations, market intelligence, job matching, and real-time service performance.

Porch initially relied on Airflow to orchestrate Spark jobs on Google Dataproc. But the integration created significant operational overhead:

• Complex cluster setup and job configuration
  
• Frequent engineering support needed for data scientists
  
• A steep learning curve and brittle maintenance
  
• A looming Airflow version deprecation requiring a costly upgrade
  

Without a dedicated DevOps team, managing and upgrading the Airflow cluster became unsustainable. Instead of sinking engineering time into infrastructure, Porch wanted:

• A fully managed orchestration platform
  
• Native Kubernetes support
  
• Simplified Spark execution
  
• Better developer experience and easier troubleshooting
  
• Support for existing workflows without major refactoring
  

After evaluating Google Composer, managed Kubeflow offerings, and Union, they chose Union.ai for its capabilities, usability, and strong open-source Flyte foundation.

Today, Porch runs critical workloads—including data processing, feature ingestion, and model (re)training—on Union.ai’s managed orchestration platform.

A key breakthrough came from using Flyte Agents, specifically the Airflow agent, to migrate off their self-hosted Airflow cluster without rewriting legacy code. This allowed Porch to “lift and shift” existing DAGs into Union workflows and eliminate their on-prem Airflow cluster entirely.

This delivered two major advantages:

1. No costly Airflow upgrade or maintenance
   
2. Immediate migration to a managed platform while buying time to move toward native Spark on Union
   

Porch now plans to adopt Union’s Kubernetes-native Spark execution to further reduce operational complexity and consolidate their stack.

“We want one platform for orchestrating all these jobs. Writing everything in a Union workflow means less to manage—it's awesome.” —Thomas Busath, Machine Learning Engineer

Union also improved developer productivity through:

• End-to-end versioning for rapid experimentation and reproducibility
  
• Clear error propagation and UI insights enabling self-sufficient troubleshooting
  
• GitOps-native integration for full traceability of workflow lineage
  
• Containerized task execution for clean dependency isolation
  

Porch praises Union’s documentation, engineering support, and the strength of the Flyte open-source community.

“We’re amazed at the level of support. Union engineers respond within minutes and dive directly into the issue—no other vendor comes close.” —Thomas Busath

Porch also leverages Union via the GCP Marketplace, simplifying procurement and aligning spend with cloud commitments.

By decommissioning their self-hosted Airflow cluster and using Union’s managed infrastructure, Porch eliminated the overhead of maintaining, upgrading, and troubleshooting their orchestration stack.

With Union’s versioning, teams now iterate rapidly with multiple parallel workflow versions—accelerating experimentation by 200% while retaining reproducibility and governance.

Flyte Agents allowed Porch to fully migrate legacy pipelines without touching model code, freeing ML engineers to focus on improving models instead of managing infrastructure. Containerized execution and native Spark support further simplified pipeline development and debugging.

Union’s strong developer experience and responsive support team give Porch confidence in scaling their ML operations. As the platform continues to evolve—with features like Artifacts and Triggers—additional Porch teams plan to adopt Union for their own data and ML workflows.