{"id":10020510,"date":"2024-12-19T19:16:43","date_gmt":"2024-12-19T11:16:43","guid":{"rendered":"https:\/\/itmax.cn\/?p=10020510"},"modified":"2024-12-19T19:16:43","modified_gmt":"2024-12-19T11:16:43","slug":"%e6%88%91%e5%af%b9-openai-kubernetes-%e9%9b%86%e7%be%a4%e6%95%85%e9%9a%9c%e7%9a%84%e8%bf%bd%e9%97%ae%e4%b8%8e%e7%96%91%e6%83%91","status":"publish","type":"post","link":"https:\/\/itmax.cn\/?p=10020510","title":{"rendered":"\u6211\u5bf9 OpenAI Kubernetes \u96c6\u7fa4\u6545\u969c\u7684\u8ffd\u95ee\u4e0e\u7591\u60d1"},"content":{"rendered":"<p>2024 \u5e74 12 \u6708 11 \u53f7\uff0c<code>OpenAI<\/code> \u7684 <code>Kubernetes<\/code> \u96c6\u7fa4\u53d1\u751f\u6545\u969c\uff0c<code>API<\/code>\u3001<code>ChatGPT<\/code>\u3001<code>Sora<\/code> \u7b49\u670d\u52a1\u90fd\u53d7\u5230\u4e86\u5f71\u54cd\uff0c\u4e14\u65f6\u95f4\u957f\u8fbe 4 \u5c0f\u65f6 22 \u5206\u949f\u3002<\/p>\n<p>\u5173\u4e8e\u8fd9\u6b21\u6545\u969c\uff0c\u5b98\u65b9\u6709\u516c\u5f00\u590d\u76d8\uff0c\u4e5f\u6709\u5f88\u591a\u5a92\u4f53\u535a\u4e3b\u8ffd\u8e2a\u62a5\u5bfc\u3002\u7136\u800c\uff0c\u6211\u5bf9\u6b64\u5e76\u4e0d\u6ee1\u610f\uff0c<strong>\u672c\u6587\u6211\u5c06\u4f1a\u63d0\u51fa\u81ea\u5df1\u7684\u7591\u60d1\u4e0e\u8ffd\u95ee<\/strong>\u3002<\/p>\n<h2>\u6545\u969c\u590d\u76d8<\/h2>\n<blockquote>\n<p>At 3:12 PM PST, we deployed a new telemetry service to collect detailed Kubernetes control plane metrics. <\/p>\n<p>Telemetry services have a very wide footprint, so this new service\u2019s configuration unintentionally caused every node in each cluster to execute resource-intensive Kubernetes API operations whose cost scaled with the size of the cluster. With thousands of nodes performing these operations simultaneously, the Kubernetes API servers became overwhelmed, taking down the Kubernetes control plane in most of our large clusters. This issue was most pronounced in our largest clusters, so our testing didn\u2019t catch it \u2013 and DNS caching made the issue far less visible until the rollouts had begun fleet-wide.<\/p>\n<p>The Kubernetes data plane can operate largely independently of the control plane, but DNS relies on the control plane \u2013 services don\u2019t know how to contact one another without the Kubernetes control plane. <\/p>\n<p>In short, the root cause was a new telemetry service configuration that unexpectedly generated massive Kubernetes API load across large clusters, overwhelming the control plane and breaking DNS-based service discovery.<\/p>\n<\/blockquote>\n<p>\u6839\u636e\u4ee5\u4e0a\u5b98\u65b9\u4fe1\u606f\u6240\u8ff0\uff0c\u6545\u969c\u7684\u6839\u672c\u539f\u56e0\u5728\u4e8e\u65b0\u90e8\u7f72\u4e86\u4e00\u4e2a telemetry \u670d\u52a1\uff0c\u8fd9\u4e2a\u670d\u52a1\u7684\u672c\u610f\u662f\u6536\u96c6\u8be6\u7ec6\u7684\u63a7\u5236\u5e73\u9762\u7684\u6307\u6807\uff0c\u5374\u610f\u5916\u5730\u5bfc\u81f4\u96c6\u7fa4\u4e2d\u7684\u6bcf\u4e2a <code>node<\/code> \u8282\u70b9\u90fd\u6267\u884c\u4e86\u8d44\u6e90\u5bc6\u96c6\u578b\u7684 <code>Kubernetes API<\/code> \u64cd\u4f5c\uff0c\u96c6\u7fa4\u4e2d\u7684 <code>node<\/code> \u8282\u70b9\u6570\u91cf\u8d8a\u591a\uff0c\u5bf9 <code>kube-apiserver<\/code> \u7684\u51b2\u51fb\u8d8a\u5927\uff0c\u6700\u7ec8\u5bfc\u81f4 <code>kube-apiserver<\/code> \u762b\u75ea\u3002<\/p>\n<p>\u5bf9\u6b64\uff0c\u6211\u7684\u7591\u95ee\u662f\uff1a<\/p>\n<ul>\n<li><strong>telemetry \u670d\u52a1\u5230\u5e95\u505a\u4e86\u4ec0\u4e48\u4f1a\u5bfc\u81f4 node \u8282\u70b9\u66f4\u9891\u7e41\u7684\u8c03\u7528\u4e86 kube-apiserver \uff1f<\/strong><\/li>\n<li><strong>telemetry \u670d\u52a1\u7684\u672c\u610f\u662f\u6536\u96c6 control plane \u63a7\u5236\u5e73\u9762\u7684\u6307\u6807\uff0c\u4e3a\u4ec0\u4e48\u4f1a\u7275\u626f\u5230 node \u8282\u70b9\uff1f<\/strong><\/li>\n<\/ul>\n<p>node \u8282\u70b9\u53ea\u662f\u4e00\u4e2a\u8303\u56f4\uff0c\u771f\u6b63\u6267\u884c\u64cd\u4f5c\u7684\u5e94\u8be5\u662f\u8282\u70b9\u4e0a\u7684\u67d0\u4e2a\u7ec4\u4ef6\uff0c\u6bd4\u5982 <code>kubelet<\/code>\u3001<code>kube-proxy<\/code>\u3001<code>CNI<\/code>\u3001<code>Pod\/Container<\/code>\u3002<\/p>\n<ul>\n<li><strong>\u5230\u5e95\u662f\u54ea\u4e2a\/\u54ea\u4e9b\u5177\u4f53\u7684\u7ec4\u4ef6\u6267\u884c\u4e86\u66f4\u52a0\u9891\u7e41\u7684\u5bf9 kube-apiserver \u7684\u64cd\u4f5c\uff1f<\/strong><\/li>\n<\/ul>\n<p>\u96c6\u7fa4\u5185\u7684\u670d\u52a1\u4f7f\u7528\u4e86 <code>DNS-based service discovery<\/code>\uff0cDNS \u5728\u7f13\u5b58\u5931\u6548\u540e\u9700\u8981\u4e0e <code>kube-apiserver<\/code> \u4ea4\u4e92\uff0c\u800c\u540e\u8005\u5df2\u7ecf\u762b\u75ea\uff0c\u56e0\u6b64\u96c6\u7fa4\u5185\u7684\u670d\u52a1\u901a\u4fe1\u5f02\u5e38\uff0c\u800c\u8fd9\u79cd\u5f02\u5e38\u4e5f\u5f88\u5bb9\u6613\u8fa8\u522b\uff08\u96c6\u7fa4\u5185\u90e8\u57df\u540d\u65e0\u6cd5\u89e3\u6790\uff09\uff0c\u6240\u4ee5\u4e5f\u80fd\u5f88\u5feb\u5730\u5b9a\u4f4d\u5230\u6545\u969c\u3002<\/p>\n<blockquote><p>DNS caching mitigated the impact temporarily by providing stale but functional DNS records. However, as cached records expired over the following 20 minutes, services began failing due to their reliance on real-time DNS resolution. This timing was critical because it delayed the visibility of the issue, allowing the rollout to continue before the full scope of the problem was understood. Once the DNS caches were empty, the load on the DNS servers was multiplied, adding further load to the control plane and further complicating immediate mitigation.<\/p><\/blockquote>\n<p>\u518d\u770b\u6545\u969c\u5206\u6790\u91cc\u5bf9 DNS \u7684\u63cf\u8ff0\uff0c\u7531\u4e8e DNS \u5b58\u5728\u7f13\u5b58\uff0c\u5b83\u5ef6\u8fdf\u4e86\u95ee\u9898\u7684\u663e\u73b0\uff0c\u6240\u4ee5\u5de5\u7a0b\u5e08\u521a\u5f00\u59cb\u6ca1\u53d1\u73b0\u95ee\u9898\uff0c\u7ee7\u7eed\u63a8\u8fdb\u4e86\u90e8\u7f72\u5de5\u4f5c\u3002\u800c\u5728 DNS \u7f13\u5b58\u5931\u6548\u540e\uff0c\u63a7\u5236\u5e73\u9762 <code>kube-apiserver<\/code> \u7684\u8d1f\u8f7d\u53c8\u88ab\u52a0\u5927\u3002<\/p>\n<p>DNS \u5b58\u5728\u7f13\u5b58\u673a\u5236\u8fd9\u70b9\u5e94\u8be5\u662f\u4e00\u79cd\u5e38\u8bc6\uff0c\u90e8\u7f72\u7684\u65f6\u5019\u5ffd\u7565\u4e86\u4e5f\u53ef\u4ee5\u7406\u89e3\uff0c\u4f46\u662f\uff1a<\/p>\n<ul>\n<li><strong>DNS \u4e0d\u5e94\u8be5\u80cc\u9505\uff0c\u673a\u5236\u5c31\u662f\u8fd9\u6837\uff0c\u600e\u4e48\u4f7f\u7528\u548c\u914d\u7f6e\u5b8c\u5168\u662f\u5de5\u7a0b\u5e08\u7684\u804c\u8d23\u3002<\/strong><\/li>\n<li><strong>\u7b49\u5230 DNS \u6302\u4e86\u624d\u53d1\u73b0\u95ee\u9898\u5c31\u6709\u70b9\u665a\u4e86\uff0c\u65e2\u7136\u77e5\u9053\u90e8\u7f72\u7684\u65b0\u670d\u52a1\u4e0e\u63a7\u5236\u5e73\u9762\u4ea4\u4e92\uff0c\u90a3\u4e48\u5bf9\u63a7\u5236\u5e73\u9762\u7684\u76d1\u63a7\u5462\uff1f<\/strong><\/li>\n<\/ul>\n<blockquote><p>In order to make that fix, we needed to access the Kubernetes control plane \u2013 which we could not do due to the increased load to the Kubernetes API servers.<\/p><\/blockquote>\n<p>\u4e3a\u4e86\u4fee\u590d\u95ee\u9898\uff0c\u9700\u8981\u8bbf\u95ee kube-apiserver\uff0c\u4f46\u662f\u540e\u8005\u6302\u6389\u4e86\u56e0\u6b64\u65e0\u6cd5\u505a\u5230\u3002<\/p>\n<p>\u542c\u4e0a\u53bb\u6709\u4e9b\u9053\u7406\uff0c\u4f46\u662f <code>kube-apiserver<\/code> \u4e13\u95e8\u63d0\u4f9b\u4e86 <code>--max-requests-inflight<\/code> \u548c <code>--max-mutation-requests-inflight<\/code> \u4e24\u4e2a\u53c2\u6570\u7528\u4e8e\u8bbe\u7f6e\u6700\u5927\u5e76\u53d1\u8bf7\u6c42\u6570\u3002<strong>\u5de5\u7a0b\u5e08\u5bf9\u6b64\u4e00\u65e0\u6240\u77e5\u5417\uff1f<\/strong><\/p>\n<p>\u53e6\u5916\uff0c<code>kube-apiserver<\/code> \u8fd8\u63d0\u4f9b\u4e86 <code>APF<\/code>\uff08<code>API Priority and Fairness<\/code>\uff09\u673a\u5236\uff0c\u7528\u6765\u5212\u5206\u8bf7\u6c42\u7684\u4f18\u5148\u7ea7\uff0c\u4fdd\u969c\u96c6\u7fa4\u7ba1\u7406\u5458\u5728\u8fc7\u8f7d\u7684\u60c5\u51b5\u4e0b\u4ecd\u7136\u80fd\u591f\u63a7\u5236 <code>kube-apiserver<\/code>\u3002<\/p>\n<blockquote><p>We do not yet have a mechanism to ensure access to the API server when the data plane is putting too much pressure on the control plane. We\u2019ll implement break-glass mechanisms to ensure that engineers can access Kubernetes API servers in any circumstances.<\/p><\/blockquote>\n<p>\u7136\u800c\uff0c\u4ece\u62ab\u9732\u7684\u4fe1\u606f\u770b\uff0c<strong>OpenAI \u7684\u5de5\u7a0b\u5e08\u4f3c\u4e4e\u4e4b\u524d\u5bf9\u8fd9\u4e9b\u7f3a\u4e4f\u6df1\u5165\u4e86\u89e3\uff0c\u662f\u4f7f\u7528\u7684 kubernetes \u7248\u672c\u8fc7\u4e8e\u8001\u65e7\uff0c\u8fd8\u662f\u57fa\u4e8e kubernetes \u9b54\u6539\u4e86\u5f88\u591a\u4e1c\u897f\uff1f<\/strong><\/p>\n<p><strong>\u4ed6\u4eec\u8bf4\u7684 <code>break-glass<\/code> \u673a\u5236\u53c8\u662f\u4ec0\u4e48\uff1f\u8fd9\u4e0d\u5c31\u662f <code>APF<\/code> \u5e72\u7684\u4e8b\u60c5\u5417\uff1f<\/strong><\/p>\n<p>\u6211\u5bf9 OpenAI Kubernetes \u96c6\u7fa4\u6545\u969c\u7684\u7591\u60d1\u5f88\u591a\uff0c\u4f46\u662f\u8fd9\u4e9b\u7ec6\u8282\u6050\u6015\u53ea\u6709\u4ed6\u4eec\u5185\u90e8\u5de5\u7a0b\u5e08\u77e5\u9053\u4e86\u3002<\/p>\n<h2>\u603b\u7ed3<\/h2>\n<p><code>kube-apiserver<\/code> \u662f\u4e00\u4e2a\u975e\u5e38\u91cd\u8981\u7684\u7ec4\u4ef6\uff0c\u4e0d\u53ea\u662f\u6587\u4e2d\u63d0\u5230\u7684 DNS \u670d\u52a1\u4f1a\u4e0e\u5b83\u4ea4\u4e92\uff0c\u56e0\u6b64\u5bf9\u96c6\u7fa4\u7684\u76d1\u63a7\u4e00\u5b9a\u8981\u989d\u5916\u6ce8\u610f\u5b83\u3002\u53e6\u5916\u5f88\u591a\u7ec4\u4ef6\u7684\u6570\u636e\u5b58\u50a8\u9664\u4e86\u652f\u6301 <code>kubernetes API<\/code> \u4e4b\u5916\uff0c\u8fd8\u53ef\u4ee5\u76f4\u8fde <code>etcd<\/code>\uff0c\u56e0\u6b64\u505a\u9ad8\u53ef\u7528\u7684\u65f6\u5019\uff0c\u4e5f\u53ef\u4ee5\u8003\u8651\u8ba9\u7ec4\u4ef6\u4f7f\u7528\u53e6\u5916\u7684 <code>etcd<\/code>\uff0c\u4ece\u800c\u51cf\u5c0f\u5bf9\u63a7\u5236\u5e73\u9762\u7684\u4f9d\u8d56\u3002<\/p>\n<hr>\n<p>(\u6211\u662f\u51cc\u865a\uff0c\u5173\u6ce8\u6211\uff0c\u65e0\u5e7f\u544a\uff0c\u4e13\u6ce8\u6280\u672f\uff0c\u6b22\u8fce\u4ea4\u6d41\u6216\u4e3a\u6211\u63a8\u8350\u5de5\u4f5c)<\/p>\n<hr>\n<p>\u53c2\u8003\u8d44\u6599\uff1a<\/p>\n<ul>\n<li><em><a href=\"https:\/\/link.segmentfault.com\/?enc=NpyIXoWK8vk2cZTRQAg3DQ%3D%3D.%2F53bWeYPO5Ndcmqy3bpYPrgqi0FAM2FiU4B1L6u38rQWrOwt13q1q70uFT6Zr1ApHN%2BZVsCiFNtvPjWD186J5A%3D%3D\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">https:\/\/status.openai.com\/incidents\/ctrsv3lwd797<\/a><\/em><\/li>\n<li><em><a href=\"https:\/\/link.segmentfault.com\/?enc=vXp4xBLhv%2BT3eXUkNNoPyQ%3D%3D.6khDCnnru5A7z1nuC3yvP16p57uLaMwP%2FnZAjtha0t5cGkuV6Va60K871UMmrBL1NT5GuuwMF4j36JUPSRGGi2spY4op3MeEQYtUgNe52Os%3D\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">https:\/\/kubernetes.io\/docs\/concepts\/cluster-administration\/fl&#8230;<\/a><\/em><\/li>\n<\/ul>\n<p><a href=\"https:\/\/segmentfault.com\/a\/1190000045659219\">https:\/\/segmentfault.com\/a\/1190000045659219<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>2024 \u5e74 12 \u6708 11 \u53f7\uff0cOpenAI \u7684 Kubernetes \u96c6\u7fa4\u53d1\u751f\u6545\u969c\uff0cAPI\u3001ChatGPT [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4],"tags":[],"class_list":["post-10020510","post","type-post","status-publish","format-standard","hentry","category-share"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/itmax.cn\/index.php?rest_route=\/wp\/v2\/posts\/10020510","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itmax.cn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itmax.cn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itmax.cn\/index.php?rest_route=\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/itmax.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10020510"}],"version-history":[{"count":0,"href":"https:\/\/itmax.cn\/index.php?rest_route=\/wp\/v2\/posts\/10020510\/revisions"}],"wp:attachment":[{"href":"https:\/\/itmax.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10020510"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itmax.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10020510"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itmax.cn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10020510"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}