Consumer applications, E-Commerce, Food Delivery, Logistics, Payments, Transportation

Adopting Flyte™ at Gojek

Gojek is an on-demand multi-service platform and digital payment technology group operating in Southeast Asia. It has 20+ services ranging from food, transportation to payments and entertainment. In 2021, the Gojek ecosystem contributed 1.7% of the Indonesian GDP, indicating that the user base is quite large. Hence, scalability and reliability are of critical importance.

Machine Learning forms a big part of the services offered by Gojek. It’s used for several use cases, such as generating pickup points, voucher allocation, optimizing prices, etc. Back in 2017-18, ML pipelines were built using an abstraction on top of Clockwork and Airflow, which today has 1000+ DAGs. The abstraction was necessary to run pipelines in separate containers in order to handle the dependency management. However, the abstraction wasn’t a smooth ride due to the less than ideal development experience, unscalability of scheduler, missing DAGs, and infrastructure overhead.

After some reflection, the Gojek team understood that they wanted a platform that is scalable, Kubernetes-native, a data-aware orchestrator, and provides a first-class development experience, and Flyte™ fit their criteria.

Pradithya Aria Pura from the Data Science Platform team at Gojek presents a detailed walkthrough of the adoption of Flyte™ at Gojek and what they love about Flyte™:

Some noteworthy excerpts from the video:

“… we started evaluations of several tools in the market that’ll satisfy our requirements, and eventually, we decided to use Flyte™. We’re happy with our experience so far with Flyte™.”

“Workflow versioning is quite important. When productionizing a pipeline, there are only a few platforms that provide this kind of versioning, and to us, it’s critical where we want to roll back to a specific workflow version in case there’s a bug introduced in the pipeline.”

Flytekit (Python SDK) provides a powerful and expressive way of building pipelines. So far, we love the dynamic workflow and flow control.”

Flyte™ is fast and scalable. During our evaluation stage, we did stress testing to understand whether Flyte™ can satisfy our requirements, and it provided us with good results.