How MethaneSAT is Using Flyte™ to Power a Global Methane-Reduction Mission from Space

When MethaneSAT launches in October 2023, the satellite — a joint American-New Zealand mission — will measure methane emissions across the globe. The goal? Reduce the harmful greenhouse gas by 45% by 2025. 

Cutting the methane produced by the oil and gas supply chain — which emits 80 million tons of methane every year around the world, making it the second-largest methane emitter behind agriculture — is the single fastest way to slow global warming today. 

While other satellites measure either very large or very small areas, MethaneSAT, part of the Environmental Defense Fund, will do both. MethaneSAT will track oil and gas emissions using an imaging spectrometer to detect methane concentrations as low as three parts per billion. 

Crunching all of this data is no small task, said Nicholas LoFaso, a senior platform software engineer at MethaneSAT. And Flyte™ — the orchestration engine behind Union Cloud — will be critical to MethaneSAT’s mission. 

“Our goal is to use Flyte™ as our production task orchestrator,” LoFaso said. “We’re going to have 10,000-plus CPUs that we plan to use every day processing the raw data. There'll be 30 different targets that we're collecting data on every day, and it's that's about 200 gigs of raw data, and then our estimate is for final data, we're probably like two terabytes or so on the output. So a lot of data to process, and we're leaning heavily on Flyte™ to make that happen.”

MethaneSAT’s data-processing platform depends heavily on dynamic tasks, launch plans and nested workflows. The team chose Flyte™ for:

1. A Kubernetes-native platform that’s highly scalable.
2. Large batch processing involving thousands of CPUs that work for hours on dozens of tasks per day.
3. Custom containers for tasks. This is especially useful because the many science teams involved in the project are using algorithms written in a variety of languages (including Fortran, which first came into use nearly seven decades ago).
4. Caching, which avoids reprocessing data for long-running processes if there’s no need.
5. Abstraction — or reducing the complexity of certain processes — to pass files between tasks, provided through FlyteFile.
6. A track record of success at leading technology companies including Lyft, Spotify and Freenome.

Sensors similar to MethaneSAT have already been used on an airplane that has scanned the Permian Basin, a large area of oil and gas production in Texas and New Mexico. What the flight was able to scan in two hours, MethaneSAT will be able to do in 30 seconds.“A fraction of the time, larger area,” LoFaso said. “We're really excited about the capabilities that this project will enable us to do, and Flyte™’s going to get us there.”