赞
踩
k8s集群中搭建有elasticsearch服务一般都会用到pvc,但是考虑到有些自建k8s环境下,搭建的共享存储可能会存在稳定性及性能问题,所以这次是通过采用节点亲和性和hostpath来实现,目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将elasticsearch集群的搭建采用hostpath+亲和性的权重+多个副本分区的方式来实现数据持久化和高可用。
k8s版本k8s-1.29.4,环境搭建在电信机房,六个worker节点,每个节点有一块非系统盘的ssd盘挂载到/data/路径,不采用ceph或nas之类的共享存储,也未采用operator。
节点名称 | 角色 | 标签 |
---|---|---|
k8s-worker-120-81 | 主 | storage-selector=node-a-master |
k8s-worker-120-82 | 主 | storage-selector=node-b-master |
k8s-worker-120-83 | 主 | storage-selector=node-c-master |
k8s-worker-120-84 | 备 | storage-selector=node-a-slve |
k8s-worker-120-85 | 备 | storage-selector=node-a-slve |
k8s-worker-120-86 | 备 | storage-selector=node-a-slve |
注:这里的角色划分是指:redis由于是一主两从,每个redis的pod分别落在三个主的节点上,当主节点发生故障是,就切换到备节点,例如:名称为autopocket-es-a-0的pod,默认落在标签为storage-selector=node-a-master的节点上,当该节点发生故障是,就会根据权重匹配罗在storage-selector=node-a-slve的节点上。
1.elasticsearch_a.yaml 节点a的statuflset配置和无头服务配置
apiVersion: apps/v1 kind: StatefulSet metadata: name: autopocket-es-a namespace: zx-app labels: app: autopocket-es-a appname: pcauto-pocket appgroup: autopocket-es spec: serviceName: "autopocket-es-a" replicas: 1 selector: matchLabels: app: autopocket-es-a appgroup: autopocket-es template: metadata: labels: app: autopocket-es-a appgroup: autopocket-es spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: - key: storage-selector operator: In values: - node-a-master - weight: 20 preference: matchExpressions: - key: storage-selector operator: In values: - node-a-slave containers: - name: elasticsearch image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22" command: ["/bin/sh","-c"] args: - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch" env: - name: cluster.name value: autopocket-es - name: K8S_SERVICE_NAME value: autopocket-es-a - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: discovery.seed_hosts value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local" - name: cluster.initial_master_nodes value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local" - name: node.name value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local - name: path.data value: /home/elastic/$(POD_NAME)/data - name: path.logs value: /home/elastic/$(POD_NAME)/logs - name: xpack.security.enabled value: "false" ports: - name: rest containerPort: 9200 - name: inter-node containerPort: 9300 volumeMounts: - name: elasticsearch-data mountPath: /home/elastic restartPolicy: Always terminationGracePeriodSeconds: 30 volumes: - name: elasticsearch-data hostPath: path: /data/es_data type: "" tolerations: - key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 3600 - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 3600 --- apiVersion: v1 kind: Service metadata: name: autopocket-es-a namespace: zx-app labels: appname: pcauto-pocket app: autopocket-es-a spec: ports: - port: 9200 name: rest - port: 9300 name: inter-node clusterIP: None selector: app: autopocket-es-a
2.elasticsearch_b.yaml 节点b的statuflset配置和无头服务配置
apiVersion: apps/v1 kind: StatefulSet metadata: name: autopocket-es-b namespace: zx-app labels: app: autopocket-es-b appname: pcauto-pocket appgroup: autopocket-es spec: serviceName: "autopocket-es-b" replicas: 1 selector: matchLabels: app: autopocket-es-b appgroup: autopocket-es template: metadata: labels: app: autopocket-es-b appgroup: autopocket-es spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: - key: storage-selector operator: In values: - node-b-master - weight: 20 preference: matchExpressions: - key: storage-selector operator: In values: - node-b-slave containers: - name: elasticsearch image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22" command: ["/bin/sh","-c"] args: - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch" env: - name: cluster.name value: autopocket-es - name: K8S_SERVICE_NAME value: autopocket-es-b - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: discovery.seed_hosts value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local" - name: cluster.initial_master_nodes value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local" - name: node.name value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local - name: path.data value: /home/elastic/$(POD_NAME)/data - name: path.logs value: /home/elastic/$(POD_NAME)/logs - name: xpack.security.enabled value: "false" ports: - name: rest containerPort: 9200 - name: inter-node containerPort: 9300 volumeMounts: - name: elasticsearch-data mountPath: /home/elastic restartPolicy: Always terminationGracePeriodSeconds: 30 volumes: - name: elasticsearch-data hostPath: path: /data/es_data type: "" tolerations: - key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 3600 - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 3600 --- apiVersion: v1 kind: Service metadata: name: autopocket-es-b namespace: zx-app labels: appname: pcauto-pocket app: autopocket-es-b spec: ports: - port: 9200 name: rest - port: 9300 name: inter-node clusterIP: None selector: app: autopocket-es-b
3.elasticsearch_c.yaml 节点c的statuflset配置和无头服务配置
apiVersion: apps/v1 kind: StatefulSet metadata: name: autopocket-es-c namespace: zx-app labels: app: autopocket-es-c appname: pcauto-pocket appgroup: autopocket-es spec: serviceName: "autopocket-es-c" replicas: 1 selector: matchLabels: app: autopocket-es-c appgroup: autopocket-es template: metadata: labels: app: autopocket-es-c appgroup: autopocket-es spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: - key: storage-selector operator: In values: - node-c-master - weight: 20 preference: matchExpressions: - key: storage-selector operator: In values: - node-c-slave containers: - name: elasticsearch image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22" command: ["/bin/sh","-c"] args: - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch" env: - name: cluster.name value: autopocket-es - name: K8S_SERVICE_NAME value: autopocket-es-c - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: discovery.seed_hosts value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local" - name: cluster.initial_master_nodes value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local" - name: node.name value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local - name: path.data value: /home/elastic/$(POD_NAME)/data - name: path.logs value: /home/elastic/$(POD_NAME)/logs - name: xpack.security.enabled value: "false" ports: - name: rest containerPort: 9200 - name: inter-node containerPort: 9300 volumeMounts: - name: elasticsearch-data mountPath: /home/elastic restartPolicy: Always terminationGracePeriodSeconds: 30 volumes: - name: elasticsearch-data hostPath: path: /data/es_data type: "" tolerations: - key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 3600 - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 3600 --- apiVersion: v1 kind: Service metadata: name: autopocket-es-c namespace: zx-app labels: appname: pcauto-pocket app: autopocket-es-c spec: ports: - port: 9200 name: rest - port: 9300 name: inter-node clusterIP: None selector: app: autopocket-es-c
给9200 9300配置一个service的代理服务,注意其中的selector是指向到appgroup,这是将三个statefulset定义了一个appgroup的标签。
apiVersion: v1 kind: Service metadata: name: autopocket-es namespace: zx-app labels: app: autopocket-es appname: pcauto-zx spec: ports: - port: 9200 name: rest - port: 9300 name: inter-node selector: appgroup: autopocket-es type: LoadBalancer
apiVersion: apps/v1 kind: Deployment metadata: name: autopocket-es-kibana namespace: zx-app labels: app: autopocket-es-kibana appname: pcauto-pocket spec: replicas: 1 selector: matchLabels: app: autopocket-es-kibana template: metadata: labels: app: autopocket-es-kibana spec: containers: - name: kibana image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/kibana:7.17.22" ports: - containerPort: 5601 env: - name: ELASTICSEARCH_HOSTS value: '["http://autopocket-es-a-0.autopocket-es-a:9200","http://autopocket-es-b-0.autopocket-es-b:9200","http://autopocket-es-c-0.autopocket-es-c:9200"]' --- apiVersion: v1 kind: Service metadata: name: autopocket-es-kibana namespace: zx-app labels: app: autopocket-es-kibana appname: pcauto-pocket spec: selector: app: autopocket-es-kibana ports: - port: 5601 targetPort: 5601 type: LoadBalancer
查看集群健康情况:
http://10.16.xx.xx:9200/_cat/health?v
elasticsearch集群状态查看:
http://10.16.1xx.xx:9200/_cat/nodes?v
索引状态查看:
http://10.16.1xx.xx:9200/_cat/indices?v
kibana地址登录:
http://10.16.1xx.xx:5601/app/home#/
调整已存在索引的副本分区:
curl -XPUT -H "Content-Type:application/json" -d '{"settings":{"number_of_replicas":2}}' 'http://10.16.xx.xx:9200/testindex/_settings'
调整集群最大分片数:
curl -XPUT -H "Content-Type:application/json" http://10.16.xx.xx:9200/_cluster/settings -d '{"transient":{"cluster":{"max_shards_per_node":10000}}}'
调整默认主分片和副本分片数:
curl -X PUT -H "Content-Type:application/json" -d '{"index_patterns":["*"],"settings":{"number_of_shards":3,"number_of_replicas":2}}' http://10.16.xx.xx:9200/_template/template_http_request_record
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。