kube-apiserver的listwatche机制

Table of Contents

1. 背景

client-go实际只是一个客户端，list-watch我们经常听到。但实际上是apisever的实现。在apisever注册资源对象的create, update, delete等等hanlder时，也注册了List-watch的实现。

所以在研究client-go是如果处理list watch之前，先了解一个apiserver的list watch机制

2. list watch机制

List-watch是K8S统一的异步消息处理机制，保证了消息的实时性，可靠性，顺序性，性能等等，为声明式风格的API奠定了良好的基础，它是优雅的通信方式，是K8S 架构的精髓。

2.1 如何实现实时性

一般客户端和服务器端的同步，无非就是两种大类：一种是客户端轮训服务器端。第二种就是服务器端主动发起通知。

k8s采用的是第二种，主动发起通知。这里具体就是使用了watch机制。

list, watch其实都是特殊的get接口。

get default命名空间所有的pod url如下： curl http://xxx:port/api/v1/namespaces/default/pods

watch default命名空间所有的pod url如下: curl http://XXX/api/v1/namespaces/default/pods?watch=true 就是多了一个watch=true的参数

root:/# curl http://XXX/api/v1/namespaces/default/pods?watch=true
{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"zx-vpa-786d4b8bb5-xv5zw","generateName":"zx-vpa-786d4b8bb5-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/zx-vpa-786d4b8bb5-xv5zw","uid":"639944b7-3495-4fbb-a21d-cbc7f4d6f7a5","resourceVersion":"157197390","creationTimestamp":"2021-11-12T10:59:39Z","labels":{"app":"zx-vpa-test","pod-template-hash":"786d4b8bb5"},"annotations":{"v2-fixed-ip":"","v2-subnet":"faf7c8b0-55c3-42c7-ba27-ad90290a9cd9","v2-tenant":"","v2-vpc":"6af350be-c456-44bc-909d-4b92c48b3b54","vpaObservedContainers":"zx-vpa, zx-vpa2","vpaUpdates":"Pod resources updated by hamster-vpa: container 0: memory request, cpu request, memory limit, cpu limit; container 1: cpu request, memory request, cpu limit, memory limit"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zx-vpa-786d4b8bb5","uid":"8199639c-40fc-4dc5-81c3-d3faff7f6b4c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-dbxf8","secret":{"secretName":"default-token-dbxf8","defaultMode":420}}],"containers":[{"name":"zx-vpa","image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"zx-vpa2","image":"ncr.nie.netease.com/zouxiang/testcpu:v1","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":5,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"7.34.19.14","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"}],"hostIP":"7.34.19.14","podIP":"7.34.19.14","podIPs":[{"ip":"7.34.19.14"}],"startTime":"2021-11-12T10:59:39Z","containerStatuses":[{"name":"zx-vpa","state":{"running":{"startedAt":"2021-11-14T02:59:46Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:45Z","finishedAt":"2021-11-14T02:59:45Z","containerID":"docker://87a70d2061b7fb37b0f97be3a4f9d44b345fbd54be3dcc4d8a61879dd5c6a127"}},"ready":true,"restartCount":4,"image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","imageID":"docker-pullable://dockerhub.nie.netease.com/fanqihong/ubuntu@sha256:ac49d16f9686c2acd351d436ed7154311e4dba50ed8c18b6abaa578dde696440","containerID":"docker://bc586f53f363e9afb08c7a214eef06c8c1202f72439fc972d4c7d6177cfb8e63","started":true},{"name":"zx-vpa2","state":{"running":{"startedAt":"2021-11-14T02:59:47Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:46Z","finishedAt":"2021-11-14T02:59:46Z","containerID":"docker://37d8dd54be6d27ed9f055049e700f12fa4aa30ec29f2fd16fd5176218b2acce9"}},"ready":true,"restartCount":4,"image":"ncr.nie.netease.com/zouxiang/testcpu:v1","imageID":"docker-pullable://ncr.nie.netease.com/zouxiang/testcpu@sha256:4560824247d61f92c0d4b62224fdb3efc47560339ff05c92f73d6c731eba2717","containerID":"docker://9ccb7968bee2c155c472e03d56a5987c9cf7e6833a4cb125084ceb19158474ed","started":true}],"qosClass":"Guaranteed"}}}
{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"zx-vpa-786d4b8bb5-mw6mr","generateName":"zx-vpa-786d4b8bb5-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/zx-vpa-786d4b8bb5-mw6mr","uid":"4e7f3a44-7483-434d-a917-52b37c0eae33","resourceVersion":"157192079","creationTimestamp":"2021-11-12T10:52:37Z","labels":{"app":"zx-vpa-test","pod-template-hash":"786d4b8bb5"},"annotations":{"v2-fixed-ip":"","v2-subnet":"faf7c8b0-55c3-42c7-ba27-ad90290a9cd9","v2-tenant":"","v2-vpc":"6af350be-c456-44bc-909d-4b92c48b3b54","vpaObservedContainers":"zx-vpa, zx-vpa2","vpaUpdates":"Pod resources updated by hamster-vpa: container 0: cpu request, memory request, cpu limit, memory limit; container 1: memory request, cpu request, cpu limit, memory limit"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zx-vpa-786d4b8bb5","uid":"8199639c-40fc-4dc5-81c3-d3faff7f6b4c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-dbxf8","secret":{"secretName":"default-token-dbxf8","defaultMode":420}}],"containers":[{"name":"zx-vpa","image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"zx-vpa2","image":"ncr.nie.netease.com/zouxiang/testcpu:v1","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":5,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"10.90.67.175","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:52:37Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:52:54Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:52:54Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:52:37Z"}],"hostIP":"10.90.67.175","podIP":"10.90.67.175","podIPs":[{"ip":"10.90.67.175"}],"startTime":"2021-11-12T10:52:37Z","containerStatuses":[{"name":"zx-vpa","state":{"running":{"startedAt":"2021-11-14T02:52:50Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:52:47Z","finishedAt":"2021-11-14T02:52:47Z","containerID":"docker://ddf625ee9c90ba70ba5f1d27caa4d61ded938143a724dccbfada898271ac7fd0"}},"ready":true,"restartCount":4,"image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","imageID":"docker-pullable://dockerhub.nie.netease.com/fanqihong/ubuntu@sha256:ac49d16f9686c2acd351d436ed7154311e4dba50ed8c18b6abaa578dde696440","containerID":"docker://854091543fc0ba88d3dc4a839f8014d21ecbaecf4e40221d2ce9d6a343ddbe29","started":true},{"name":"zx-vpa2","state":{"running":{"startedAt":"2021-11-14T02:52:53Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:52:50Z","finishedAt":"2021-11-14T02:52:50Z","containerID":"docker://0241d05357e1f6b8eec73810341bddf14479a168aaa7958ee580855eb2f4300f"}},"ready":true,"restartCount":4,"image":"ncr.nie.netease.com/zouxiang/testcpu:v1","imageID":"docker-pullable://ncr.nie.netease.com/zouxiang/testcpu@sha256:4560824247d61f92c0d4b62224fdb3efc47560339ff05c92f73d6c731eba2717","containerID":"docker://ad0d09158c0db8b10d2c282f2749c1c1e97fdb707e10392b9033aa619b162450","started":true}],"qosClass":"Guaranteed"}}}





// 一旦有对象改变就会收到事件。type有三种，modifyed, added ,deleted
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"zx-vpa-786d4b8bb5-xv5zw","generateName":"zx-vpa-786d4b8bb5-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/zx-vpa-786d4b8bb5-xv5zw","uid":"639944b7-3495-4fbb-a21d-cbc7f4d6f7a5","resourceVersion":"157401529","creationTimestamp":"2021-11-12T10:59:39Z","labels":{"app":"zx-vpa-test","pod-template-hash":"786d4b8bb5"},"annotations":{"v2-fixed-ip":"","v2-subnet":"faf7c8b0-55c3-42c7-ba27-ad90290a9cd9","v2-tenant":"","v2-vpc":"6af350be-c456-44bc-909d-4b92c48b3b54","vpaObservedContainers":"zx-vpa, zx-vpa2","vpaUpdates":"11111111111Pod resources updated by hamster-vpa: container 0: memory request, cpu request, memory limit, cpu limit; container 1: cpu request, memory request, cpu limit, memory limit"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zx-vpa-786d4b8bb5","uid":"8199639c-40fc-4dc5-81c3-d3faff7f6b4c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-dbxf8","secret":{"secretName":"default-token-dbxf8","defaultMode":420}}],"containers":[{"name":"zx-vpa","image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"zx-vpa2","image":"ncr.nie.netease.com/zouxiang/testcpu:v1","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":5,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"7.34.19.14","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"}],"hostIP":"7.34.19.14","podIP":"7.34.19.14","podIPs":[{"ip":"7.34.19.14"}],"startTime":"2021-11-12T10:59:39Z","containerStatuses":[{"name":"zx-vpa","state":{"running":{"startedAt":"2021-11-14T02:59:46Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:45Z","finishedAt":"2021-11-14T02:59:45Z","containerID":"docker://87a70d2061b7fb37b0f97be3a4f9d44b345fbd54be3dcc4d8a61879dd5c6a127"}},"ready":true,"restartCount":4,"image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","imageID":"docker-pullable://dockerhub.nie.netease.com/fanqihong/ubuntu@sha256:ac49d16f9686c2acd351d436ed7154311e4dba50ed8c18b6abaa578dde696440","containerID":"docker://bc586f53f363e9afb08c7a214eef06c8c1202f72439fc972d4c7d6177cfb8e63","started":true},{"name":"zx-vpa2","state":{"running":{"startedAt":"2021-11-14T02:59:47Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:46Z","finishedAt":"2021-11-14T02:59:46Z","containerID":"docker://37d8dd54be6d27ed9f055049e700f12fa4aa30ec29f2fd16fd5176218b2acce9"}},"ready":true,"restartCount":4,"image":"ncr.nie.netease.com/zouxiang/testcpu:v1","imageID":"docker-pullable://ncr.nie.netease.com/zouxiang/testcpu@sha256:4560824247d61f92c0d4b62224fdb3efc47560339ff05c92f73d6c731eba2717","containerID":"docker://9ccb7968bee2c155c472e03d56a5987c9cf7e6833a4cb125084ceb19158474ed","started":true}],"qosClass":"Guaranteed"}}}



//删除一个pod，发现会进入
MODIFIED （设置deletiontimestamp）-> 
ADDED  "status":{"phase":"Pending","qosClass":"Guaranteed"}}} 新pod pending
MODIFIED podScheduled
MODIFIED ContainerCreating
MODIFIED.. 到pod running
DELETED  删除旧Pod


 curl http://7.34.19.44:58201/api/v1/watch/namespaces/default/pods
 看起来也是一样的效果

通过上面的实践可以发现：

（1）watch其实就是一种特殊的get

（2）可以看到删除操作后，对象的整个变化过程

（3）watch每次都会返回type，和完整的对象信息

2.2 如何实现顺序性

K8S在每个资源的事件中都带一个resourceVersion的标签，这个标签是递增的数字，所以当客户端并发处理同一个资源的事件时，它就可以对比resourceVersion来保证最终的状态和最新的事件所期望的状态保持一致。

2.3 如何实现消息可靠性

list和watch一起保证了消息的可靠性，避免因消息丢失而造成状态不一致场景。具体而言，list API可以查询当前的资源及其对应的状态(即期望的状态)，客户端通过拿期望的状态和实际的状态进行对比，纠正状态不一致的资源。Watch API和apiserver保持一个长链接，接收资源的状态变更事件并做相应处理。如果仅调用watch API，若某个时间点连接中断，就有可能导致消息丢失，所以需要通过list API解决消息丢失的问题。从另一个角度出发，我们可以认为list API获取全量数据，watch API获取增量数据。虽然仅仅通过轮询list API，也能达到同步资源状态的效果，但是存在开销大，实时性不足的问题。

2.4 如何解决性能问题

（1）list-watch机制的结合就已经在apiserver做了性能优化。（是不是可以watch的时候，只传递更新了的字段，而不是全量数据）

（2）client-go的 tool.cache做了客户端的性能优化问题

3.总结

本节主要从apiserver端探究了以下list-watch。接下来从client-go端源码看看具体是如何实现的

©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 196,165评论 5赞 462
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 82,503评论 2赞 373
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 143,295评论 0赞 325
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 52,589评论 1赞 267
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 61,439评论 5赞 358
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 46,342评论 1赞 273
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 36,749评论 3赞 387
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 35,397评论 0赞 255
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 39,700评论 1赞 295
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 34,740评论 2赞 313
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 36,523评论 1赞 326
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 32,364评论 3赞 314
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 37,755评论 3赞 300
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 29,024评论 0赞 19
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 30,297评论 1赞 251
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 41,721评论 2赞 342
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 40,918评论 2赞 336

推荐阅读更多精彩内容