Large Scale Processing with Flyte™: Blackshark.ai’s Semantic 3D Planet
Blackshark.ai is an Austrian startup that generates real-time, accurate, searchable, and photorealistic digital 3D maps of the infrastructure of the entire planet. For instance, it enabled Microsoft’s Flight Simulator team to construct over 1.5 billion photorealistic 3D buildings, giving users an unprecedented immersive 3D flight experience.
Blackshark does this through AI models and satellite imagery. Satellite imagery for the whole planet could be as big as ~2.5 petabytes. Processing such enormous data takes a whole lot of computation power. Blackshark runs the computation on 100s of machines in parallel, which results in ~1.5 billion building footprints!
In the past, the Blackshark team was solely responsible for handling the infrastructure, code, resources, among other things. However, with the growth of the company and models, it wasn’t scalable anymore. New models were being trained, people were doing explorations with novel techniques, and all of these needed to be done in parallel, which henceforth, posed challenges to Blackshark. That was when the Blackshark team decided to deploy Flyte™.
Today, Flyte™ runs the AI detection and content generation pieces of the Blackshark pipeline, effectively handling huge amounts of data.
Maarten De Jong from the Blackshark team explains they chose Flyte™ and how they use it in the following video:
Some noteworthy excerpts from the video:
“… the reason we’re mainly using Flyte™ is that it has cloud-native capabilities. We do everything in the cloud, and we don’t want to be limited to a single cloud provider, so having the ability to run everything through Kubernetes is amazing for us.”
“... it’s pretty robust. We haven’t really been able to break it too much, …”