使用eci-profile調(diào)度Pod到VNode
eci-profile可以提供集群維度的ECI資源視圖,支持根據(jù)Selector實現(xiàn)Pod編排。本文介紹如何部署eci-profile組件,以及如何使用eci-profile。
eci-profile功能說明
eci-profile可以通過Pod、Namespace的Label篩選符合條件的Pod,實現(xiàn)以下功能:
添加Annotation和Label
執(zhí)行調(diào)度策略
支持的調(diào)度策略如下:
策略
說明
fair
公平調(diào)度,由kube-scheduler決定調(diào)度到標準Node或VNode。
normalNodePrefer
優(yōu)先調(diào)度到標準Node,標準Node資源不足時允許調(diào)度到VNode。
virtualNodeOnly
Pod只會調(diào)度到VNode。
本文部署的eci-profile組件通過Selector CRD實現(xiàn)Pod自動調(diào)度。如果您之前已經(jīng)部署了舊版eci-profile組件(通過ConfigMap實現(xiàn)),仍可以繼續(xù)使用。建議您升級到Selector CRD模式,后續(xù)ConfigMap模式不再支持新增功能。更多信息,請參見升級說明。
部署eci-profile組件
通過VNodectl工具
如果您已安裝配置VNodectl工具,可以執(zhí)行以下命令一鍵部署eci-profile。
部署eci-profile。
vnode addon enable eci-profile --kubeconfig /path/to/kubeconfig
查看eci-profile部署狀態(tài)。
vnode addon list
預(yù)期返回如下,其中eci-profile的STATUS為enabled。
|----------------|------------|------------|-------------------------------------------------| | ADDON NAME | STATUS | MAINTAINER | REPOSITORY | |----------------|------------|------------|-------------------------------------------------| | eci-profile | enabled ? | ECI Group | https://github.com/aliyuneci/eci-profile.git | | vnode-approver | enabled ? | ECI Group | https://github.com/aliyuneci/vnode-approver.git | |----------------|------------|------------|-------------------------------------------------|
手動部署
將以下內(nèi)容保存為eci-profile.yaml。
請根據(jù)您的集群版本復(fù)制對應(yīng)YAML。
集群版本為1.16及以上
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: eci-profile rules: - apiGroups: - "" resources: - nodes - namespaces - resourcequotas verbs: - get - list - watch - apiGroups: - "" resources: - pods verbs: - get - list - watch - create - patch - apiGroups: - "admissionregistration.k8s.io" resources: - mutatingwebhookconfigurations verbs: - get - patch - create - delete - apiGroups: - "eci.aliyun.com" resources: - selectors verbs: - get - watch - list --- apiVersion: v1 kind: ServiceAccount metadata: name: eci-profile namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: eci-profile roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: eci-profile subjects: - kind: ServiceAccount name: eci-profile namespace: kube-system --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: (devel) creationTimestamp: null name: selectors.eci.aliyun.com spec: group: eci.aliyun.com names: kind: Selector listKind: SelectorList plural: selectors singular: selector scope: Namespaced versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: properties: effect: properties: annotations: additionalProperties: type: string type: object labels: additionalProperties: type: string type: object type: object namespaceLabels: description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array required: - key - operator type: object type: array matchLabels: additionalProperties: type: string description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic objectLabels: description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects. properties: matchExpressions: description: matchExpressions is a list of label selector requirements. The requirements are ANDed. items: description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. properties: key: description: key is the label key that the selector applies to. type: string operator: description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. type: string values: description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. items: type: string type: array required: - key - operator type: object type: array matchLabels: additionalProperties: type: string description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. type: object type: object x-kubernetes-map-type: atomic policy: properties: fair: type: object namespaceResourceLimit: properties: limits: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: ResourceList is a set of (resource name, quantity) pairs. type: object namespace: type: string required: - limits - namespace type: object normalNodeOnly: type: object normalNodePrefer: properties: cpuRatio: type: integer memoryRatio: type: integer type: object virtualNodeOnly: type: object type: object priority: format: int32 type: integer type: object required: - spec type: object served: true storage: true --- apiVersion: v1 kind: Service metadata: labels: role: eci-profile name: eci-profile namespace: kube-system spec: ports: - port: 443 targetPort: 443 selector: app: eci-profile --- apiVersion: apps/v1 kind: Deployment metadata: name: eci-profile namespace: kube-system labels: app: eci-profile spec: replicas: 1 selector: matchLabels: app: eci-profile template: metadata: labels: app: eci-profile spec: serviceAccount: eci-profile containers: - name: eci-profile image: registry.cn-beijing.aliyuncs.com/eci-release/eci-profile:0.0.3 imagePullPolicy: Always resources: requests: cpu: 2 memory: 4Gi limits: cpu: 4 memory: 8Gi env: - name: KUBERNETES_MASTER value: https://kubernetes:443
集群版本為1.16以下
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: eci-profile rules: - apiGroups: - "" resources: - nodes - namespaces - resourcequotas verbs: - get - list - watch - apiGroups: - "" resources: - pods verbs: - get - list - watch - create - patch - apiGroups: - "admissionregistration.k8s.io" resources: - mutatingwebhookconfigurations verbs: - get - patch - create - delete - apiGroups: - "eci.aliyun.com" resources: - selectors verbs: - get - watch - list --- apiVersion: apiextensions.k8s.io/v1beta1 #apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: selectors.eci.aliyun.com spec: group: eci.aliyun.com version: v1beta1 names: kind: Selector plural: selectors shortNames: - selectors categories: - all scope: Cluster validation: openAPIV3Schema: type: object required: - metadata - spec properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object --- apiVersion: v1 kind: ServiceAccount metadata: name: eci-profile namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: eci-profile roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: eci-profile subjects: - kind: ServiceAccount name: eci-profile namespace: kube-system --- apiVersion: v1 kind: Service metadata: labels: role: eci-profile name: eci-profile namespace: kube-system spec: ports: - port: 443 targetPort: 443 selector: app: eci-profile --- apiVersion: apps/v1 kind: Deployment metadata: name: eci-profile namespace: kube-system labels: app: eci-profile spec: replicas: 1 selector: matchLabels: app: eci-profile template: metadata: labels: app: eci-profile spec: serviceAccount: eci-profile containers: - name: eci-profile image: registry.cn-beijing.aliyuncs.com/eci-release/eci-profile:2.0.0-477875b-aliyun imagePullPolicy: Always resources: requests: cpu: 2 memory: 4Gi limits: cpu: 4 memory: 8Gi env: - name: KUBERNETES_MASTER value: https://kubernetes:443
部署eci-profile組件。
kubectl create -f eci-profile.yaml
查看部署結(jié)果。
kubectl -n kube-system get pods
預(yù)期返回如下,eci-profile對應(yīng)Pod的狀態(tài)為Running。
NAME READY STATUS RESTARTS AGE eci-profile-6454756cb8-8xlz8 1/1 Running 0 76s
配置說明及示例
部署eci-profile組件后,您可以創(chuàng)建Selector來配置Pod的調(diào)度策略,以及需要添加的Annotation和Label。Selector的YAML配置如下:
apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
name: test-fair
spec:
objectLabels:
matchLabels:
app: nginx
namespaceLabels:
matchLabels:
app: test
effect:
annotations:
k8s.aliyun.com/eci-auto-imc: "true"
labels:
eci-schedulable: "true"
policy:
fair: {}
priority: 3
spec中的相關(guān)參數(shù)說明如下:
名稱 | 描述 |
objectLabels.matchLabels | 要匹配的Pod Label。 |
namespaceLabels.matchLabels | 要匹配的Namespace Label。 |
effect.annotations | 要添加的Annotation。 |
effect.labels | 要添加的Label。 |
policy | 調(diào)度策略。支持以下幾種:
|
priority | 優(yōu)先級。存在多個Selector且發(fā)生沖突時,優(yōu)先級高的Selector會生效。數(shù)值越大,表示該Selector優(yōu)先級越高。 |
objectLabels和namespaceLabels中,至少要選擇一個配置。如果同時配置了兩者,則Pod需要同時匹配兩者。
配置示例一:fair調(diào)度
創(chuàng)建以下Selector,eci-profile會為帶有app: nginx
Label的Pod添加VNode Toleration,由kube-scheduler決定調(diào)度,同時添加effect中定義的Annotation和Label到Pod。
apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
name: test-fair
spec:
objectLabels:
matchLabels:
app: nginx
effect:
annotations:
k8s.aliyun.com/eci-auto-imc: "true"
labels:
eci-schedulable: "true"
policy:
fair: {}
配置示例二:normalNodePrefer調(diào)度
創(chuàng)建以下Selector,eci-profile會在標準節(jié)點資源不足時將帶有app: nginx
Label的Pod調(diào)度到VNode,同時添加effect中定義的Annotation和Label到Pod。
apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
name: test-normal-node-prefer
spec:
objectLabels:
matchLabels:
app: nginx
effect:
annotations:
k8s.aliyun.com/eci-image-cache: "true"
labels:
eci-schedulable: "true"
policy:
normalNodePrefer: {}
配置示例三:virtualNodeOnly調(diào)度
創(chuàng)建以下Selector,eci-profile會為帶有app: nginx
Label的Pod添加VNode Toleration和VNode nodeSelector,同時添加effect中定義的Annotation和Label到Pod。
apiVersion: eci.aliyun.com/v1beta1
kind: Selector
metadata:
name: test-virtual-node-only
spec:
objectLabels:
matchLabels:
app: nginx
effect:
annotations:
k8s.aliyun.com/eci-auto-imc: "true"
labels:
eci-schedulable: "true"
policy:
virtualNodeOnly: {}
升級說明
如果您之前已經(jīng)部署了舊版eci-profile組件(ConfigMap模式),建議您升級到新版eci-profile組件(Selector CRD模式)。升級方式如下:
記錄kube-system命名空間下,ConfigMap eci-profile中的selectors內(nèi)容。
刪除舊的eci-profie組件。
部署新的eci-profile組件。
根據(jù)原有的selectors內(nèi)容,創(chuàng)建新的Selector資源。
如果您在使用eci-profile的過程中遇到任何需求或問題時,歡迎使用釘釘搜索群號44666389,加入VNode客戶支持群。