Ubuntu系统Elastic Stack 8.12.0 集群平台搭建_elk8.12安装和配置

Ubuntu系统Elastic Stack 8.12.0 集群平台搭建

一、环境准备

elasticsearch: 8.12.0

kibana

1.1.软件介绍

elasticsearch: 8.12.0

1.2.硬件环境：

二、系统环境基础设置（所有机器）

2.1.设置系统名称

sudo hostnamectl set-hostname elk51.forgcat.com
1

2.2.关闭防火墙

sudo systemctl disable ufw
sudo systemctl stop ufw  
1
2

2.3.配置hosts

sudo cat > /etc/hosts << EOF
127.0.0.1 localhost
127.0.1.1       elk-51.forgcat.com elk-51

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

192.168.20.51 elk-51.forgcat.com elk-51
192.168.20.52 elk-52.forgcat.com elk-52
192.168.20.53 elk-53.forgcat.com elk-53
192.168.20.54 elk-54.forgcat.com elk-54
192.168.20.55 elk-55.forgcat.com elk-55
192.168.20.56 elk-56.forgcat.com elk-56
192.168.20.54 kibana.forgcat.com 
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

2.3. 关闭缓存

sudo sed -ri 's/.*swap.*/#&/' /etc/fstab
sudo swapoff -a
1
2

三、es集群部署

3.1.1安装节点介绍

3.2 服务器优化(三台操作一样)

$ sudo cat > /etc/sysctl.d/10-elasticsearch.conf <<EOF
vm.swappiness=1
fs.file-max=655360  #系统最大打开文件描述符数
net.ipv4.tcp_retries2=5
vm.max_map_count = 262144   #用于限制一个进程可以拥有的VMA(虚拟内存区域)的大小，系统默认是65530，建议修改成262144或者更高
EOF
$ sudo cat >> /etc/security/limits.conf <<EOF
#添加以下这行
* soft  nproc   20480
* hard   nproc  20480
* soft  nofile  65535
* hard  nofile  65535
* soft  memlock  unlimited
* hard  memlock  unlimited
elasticsearch  -  nofile  65535
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

3.3.安装elasticsearch(三台操作一样)

3.3.1.下载公钥文件和elastic-8.x存储文件

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
1
2

3.3.2.安装elasticsearch

sudo apt-get update && sudo apt-get install elasticsearch
#安装后会出现安全信息请注意保存
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : 164IeiYBPr7RHMYeHD38

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with 
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with 
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with 
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

3.3.3.elasticsearch配置

备份初始文件

sudo  cp /etc/elasticsearch/elasticsearch.yml  /etc/elasticsearch/elasticsearch.yml_bak
1

修改elasticsearch.yml

sudo vim /etc/elasticsearch/elasticsearch.yml
cluster.name: es-elk #集群名称 三台节点一样
node.name: elk-51 #节点名称 三台节点根据情况修改
#数据和日志的存储目录
path.data: /var/lib/elasticsearch 
path.logs: /var/log/elasticsearch
network.host: elk-51.forgcat.com
http.port: 9200
discovery.seed_hosts: ["elk-51"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["elk-51"]
transport.host: 0.0.0.0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

elk-51启动elasticsearch

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start  elasticsearch
sudo systemctl status   elasticsearch
1
2
3
4

elk-51上查看加入节点的token

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node 
eyJ2ZXIiOiI4LjEyLjAiLCJhZHIiOlsiMTkyLjE2OC4yMC41MTo5MjAwIl0sImZnciI6ImRlNmMzNjFkMzI0MTBlMWYyMmUwNGNkZmQzNzY2MWFmOTJkZTBkZTcyM2FkZDNhYmJiZWQ3ZmM4MjJhNjFmMTUiLCJrZXkiOiJhMk1LUkkwQngzSkpDeS1MOEt4NzpjTVRtMWJsQVFXS29aYlVWQ25VWHJRIn0=
1
2

elk-52和elk-53输入下面命令加入现有集群

sudo /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjEyLjAiLCJhZHIiOlsiMTkyLjE2OC4yMC41MTo5MjAwIl0sImZnciI6ImRlNmMzNjFkMzI0MTBlMWYyMmUwNGNkZmQzNzY2MWFmOTJkZTBkZTcyM2FkZDNhYmJiZWQ3ZmM4MjJhNjFmMTUiLCJrZXkiOiJhMk1LUkkwQngzSkpDeS1MOEt4NzpjTVRtMWJsQVFXS29aYlVWQ25VWHJRIn0=

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically: 
  - Security auto configuration will be removed from elasticsearch.yml
  - The [certs] config directory will be removed
  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore
Do you want to continue with the reconfiguration process [y/N]y
#设置elasticsearch开机自启
sudo systemctl enable elasticsearch
sudo vim /etc/elasticsearch/elasticsearch.yml
cluster.name: es-elk
node.name: elk-52
network.host: elk-52.forgcat.com
http.port: 9200
#启动elasticsearch
sudo systemctl start  elasticsearch
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

检查 Elasticsearch 是否正在运行

export ELASTIC_PASSWORD="164IeiYBPr7RHMYeHD38"
sudo  curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD https://localhost:9200
{
  "name" : "elk-51",
  "cluster_name" : "es-elk",
  "cluster_uuid" : "OzYO44lFTRaVnwVxUb7uDQ",
  "version" : {
    "number" : "8.12.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "1665f706fd9354802c02146c1e6b5c0fbcddfbc9",
    "build_date" : "2024-01-11T10:05:27.953830042Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

查看ES集群的健康状态

sudo curl -XGET  --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD 'https://localhost:9200/_cat/health?v'
epoch      timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1706157797 04:43:17  es-elk  green           3         3      2   1    0    0        0             0                  -                100.0%
1
2
3

查看master节点信息

sudo curl -XGET  --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD 'https://localhost:9200/_cat/master?v'
id                     host      ip        node
Pe2XkCV9TnyTyB_xW5ZImA 127.0.0.1 127.0.0.1 elk-51
1
2
3

查看ES节点信息

 sudo curl -XGET  --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD 'https://localhost:9200/_cat/nodes?v'
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
192.168.20.53           16          97  16    1.06    0.46     0.25 cdfhilmrstw -      elk-53
192.168.20.51           24          95   4    0.23    0.32     0.21 cdfhilmrstw *      elk-51
192.168.20.52           19          97   7    0.67    0.34     0.20 cdfhilmrstw -      elk-52
1
2
3
4
5

重新配置/etc/elasticsearch/elasticsearch.yml

sudo vim /etc/elasticsearch/elasticsearch.yml
discovery.seed_hosts: ["elk51", "elk-52", "elk-53"]
cluster.initial_master_nodes: ["elk-51", "elk-52", "elk-53"]
#重新启动服务
systemctl restart elasticsearch
1
2
3
4
5

3.4.安装kibana(elk-54节点上安装)

3.4.1.下载公钥文件和elastic-8.x存储文件

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
1
2

3.4.2.安装kibana

sudo apt-get update && sudo apt-get install kibana  
1

3.4.3.生成加密key

/usr/share/kibana/bin/kibana-encryption-keys generate
## Kibana Encryption Key Generation Utility

The 'generate' command guides you through the process of setting encryption keys for:

xpack.encryptedSavedObjects.encryptionKey
    Used to encrypt stored objects such as dashboards and visualizations
    https://www.elastic.co/guide/en/kibana/current/xpack-security-secure-saved-objects.html#xpack-security-secure-saved-objects

xpack.reporting.encryptionKey
    Used to encrypt saved reports
    https://www.elastic.co/guide/en/kibana/current/reporting-settings-kb.html#general-reporting-settings

xpack.security.encryptionKey
    Used to encrypt session information
    https://www.elastic.co/guide/en/kibana/current/security-settings-kb.html#security-session-and-cookie-settings


Already defined settings are ignored and can be regenerated using the --force flag.  Check the documentation links for instructions on how to rotate encryption keys.
Definitions should be set in the kibana.yml used configure Kibana.
# kibana.conf 配置
Settings:
xpack.encryptedSavedObjects.encryptionKey: 38e9b31f7ae530defbef1ff699bf4e4f
xpack.reporting.encryptionKey: 53d6cbcaaee336f8e794fe474f580ee9
xpack.security.encryptionKey: a68ecf030cfc4e166a6cd1e5d4097467

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

3.4.4.生成证书

mkdir /etc/kibana/certs
cd /etc/kibana/certs
openssl req -x509 -newkey rsa:4096 -keyout kibana-server.key -out kibana-server.crt -days 3650 -subj "/C=CN/ST=Shanghai/L=Huangpu/O=sunshineit/OU=IT/CN=*forgcat.com"
....+....+..+.......+..+.+...+..+.........+...+.+............+.....+...+.+...............+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..+...+...+..........+..+......+.+........+............................+........+.......+...+.....+.......+..+.+...........+...+.......+...+......+.....+.+...+..+..........+.....+.+.........+.....+......+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+..+....+...+..................+.....+...+.........+.......+.....+.+..+.+.....+......+........................+..........+......+...+..+.........+.........+...+.......+...+.........+.....+.+..+......+....+..+......+.......+......+.........+...+..............+.........+..........+......+...+...+.................+.+..+...................+...+..+.........+....+.................+......+...+..........+......+...................................+......+....+.....+....+...........+..........+...+......+......+.........+......+........+...............+.+......+.....+.+..+..........+........+...+......+...+............................+...+......+......+.................+...+...+............+....+...+.......................+....+...+......+..+..........+...+...............+...........+.+........+.......+.........+......+........+.......+.........+.....+....+..+...............+....+........+.+...........+......+....+...+...+........+...................+...+.....+.........................+..................+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
....+..+...+.+.....+....+...........+.+...+..................+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+.+..+................+..+...+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.........+..+.......+.....+......+....+...+........+....+........+.......+......+.........+.....+............+.........+............+...+................+..............+...................+..+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Enter PEM pass phrase:  #输入密码
Verifying - Enter PEM pass phrase:
-----
1
2
3
4
5
6
7
8

3.4.5.配置kibana.yml

cd /etc/kibana/
cp kibana.yml  kibana.yml_bak
vim kibana.yml
server.port: 5601 # 修改服务端口
server.host: "elk-54" # 修改服务IP
server.publicBaseUrl: "https://kibana.forgcat.com:5601" # 修改服务公共URL
server.name: "elk-54" # 修改服务名称
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana-server.crt #https证书
server.ssl.key: /etc/kibana/certs/kibana-server.key #https证书密钥
server.ssl.keyPassphrase: "P@ssw0rd135" # https证书密钥的密码
logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid
i18n.locale: "zh-CN" #设置页面中文
# 加密密钥信息
xpack.encryptedSavedObjects.encryptionKey: 38e9b31f7ae530defbef1ff699bf4e4f
xpack.reporting.encryptionKey: 53d6cbcaaee336f8e794fe474f580ee9
xpack.security.encryptionKey: a68ecf030cfc4e166a6cd1e5d4097467`
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

3.4.6.启动服务

systemctl enable kibana
systemctl start kibana
1
2

3.4.7.Web页面配置

web打开kibana https://192.168.20.54:5601
打开的页面后需要输入elasticsearch注册令牌

#在elasticsearch服务器上执行查看kibana连接elasticsearch的注册令牌
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana 
eyJ2ZXIiOiI4LjEyLjAiLCJhZHIiOlsiMTkyLjE2OC4yMC41MTo5MjAwIl0sImZnciI6ImRlNmMzNjFkMzI0MTBlMWYyMmUwNGNkZmQzNzY2MWFmOTJkZTBkZTcyM2FkZDNhYmJiZWQ3ZmM4MjJhNjFmMTUiLCJrZXkiOiJ0dDZrUkkwQk9qa0dLZTZMakVzbzpxbFgweEltdlIxaWk0cXJKVGFQRjN3In0=
# 产生的信息复制到页面即可
1
2
3
4

页面输入令牌信息后提示“从 Kibana 服务器复制该代码，或运行 bin/kibana-verification-code 进行检索。”

#在kibana服务器执行
/usr/share/kibana/bin/kibana-verification-code
Your verification code is:  927 097
1
2
3

WEB页面用户名就是elasticsearch的账户和密码
user：elastic 密码：164IeiYBPr7RHMYeHD38

其他的暂未上传