Case Study: Integrating Azure IPv6 PrivateLink with Kubernetes

Meixing Le, Michael Wiederhold at KubeCon + CloudNativeCon North America 2020

Databricks offers a multi-cloud SaaS platform to enable data teams to solve the world’s toughest problems, using best in class technology and providing an open ecosystem. We run an infrastructure footprint consisting of 2M+ VMs across 40+ Cloud environments/regions, processing exabytes of data per day and generating billions of events. This talk shares Databricks’ journey of supporting IPv6 traffic with our existing Kubernetes infrastructures. At Databricks, we run 2000+ Kubernetes clusters (self-managed) across multiple cloud providers and regions with IPv4. To support the populate PrivateLink feature on Azure with IPv6, we explored various architectural options to integrate IPv6 with our existing Kubernetes clusters. After back and forth discussions and experiments, we ended up with a solution which is lightweight and fits our deployment and monitoring systems naturally.

https://sched.co/ekCW