FlagThis AI

OpenAI experienced a significant outage on Wednesday, impacting its ChatGPT, Sora, and developer API services for approximately three hours. The disruption was attributed to a newly deployed telemetry service designed to gather Kubernetes metrics. This service, unfortunately, triggered resource-intensive API operations across all nodes in their Kubernetes clusters. The overwhelming load on the Kubernetes API servers resulted in the control plane going down in many of the large clusters.

The cascading effect of this initial issue led to DNS resolution failures. The service disruption was further prolonged due to the reliance on DNS caching. As cached records expired, services began to fail, as they could not perform the real-time DNS lookups to contact each other. OpenAI acknowledges the incident was not due to a security breach or a new product release, but rather a confluence of system failures interacting in unexpected ways.

References:

• Techmeme - OpenAI says the three hour ChatGPT outage on Wednesday was caused by a telemetry service it deployed to collect Kubernetes metrics (Kyle Wiggers/TechCrunch) - 8d
• Simon Willison's Weblog - OpenAI's postmortem for API, ChatGPT & Sora Facing Issues - 8d
• TechCrunch - OpenAI blames its massive ChatGPT outage on a ‘new telemetry service’ - 8d
• @Techmeme - OpenAI says the three hour ChatGPT outage on Wednesday was caused by a telemetry service it deployed to collect Kubernetes metrics (Kyle Wiggers/TechCrunch) - 8d

Classification:

• HashTags: OpenAIOutage ChatGPTDown Kubernetes
• Company: OpenAI
• Target: ChatGPT Users
• Product: ChatGPT
• Feature: Telemetry Service
• Type: Bug
• Severity: Medium