基于Security默认开启:搭建本地3个节点集群
首先在当前节点执行
将xpack.security.enrollment.enabled设置true
执行以下命令
elasticsearch-create-enrollment-token -s node
elasticsearch --enrollment-token <token> //以生成的注册令牌进行启动
查看加入节点:https://127.0.0.1:9200/_cat/nodes/v
ES搭建集群
centos7环境
centos7安装略
配置网络略
es1 192.168.1.20
es2 192.168.1.21
es3 192.168.1.22
es4 192.168.1.23
es4 192.168.1.24
规划:
es1 : master 主节点
es2 : master 主节点
es3 :voting_only 仅投票
es3 、es4、es5 : data、ingest 数据节点和预处理节点
下载elasticsarch8.15.1和kibana8.15.1
上传centos7的相关目录下
解压elasticsearch8.15.1到相关目录下
5台服务器做相同操作
mkdir /opt/es && tar -zxvf elasticsearch-8.15.1-linux-x86_64.tar.gz -C /opt/es
elasticsearch运行环境
1、创建centos7用户
sudo useradd -m elasticsearch
sudo passwd elasticsearch
2、创建elasticsearch数据目录
注:生产环境要求elasticsearch数据目录不能默认
mkdir /opt/esdata
3、目录赋权
sudo chown -R elasticsearch:elasticsearch /opt/es
sudo chown -R elasticsearch:elasticsearch /opt/esdata
4、修改系统资源限制
vi /etc/security/limits.conf
elasticsearch soft nofile 65536
elasticsearch hard nofile 65536
elasticsearch soft nproc 4096
elasticsearch hard nproc 4096
5、编辑/etc/sysctl.conf
增加虚拟内存映射区域的限制
vi /etc/sysctl.conf
vm.max_map_count=262144
sudo sysctl -w vm.max_map_count=262144
6、防火墙设置
sudo firewall-cmd --permanent --zone=public --add-port=9200/tcp
sudo firewall-cmd --permanent --zone=public --add-port=9300/tcp
sudo firewall-cmd --reload
9200是HTTP REST API 端口:这是Elasticsearch的默认HTTP端口,用于客户端与Elasticsearch集群通信
9300端口是用于Elasticsearch节点之间的内部通信。它是一个TCP端口,用于集群内节点之间的通信
7、开启认证:
在 Elasticsearch 中,默认情况下是允许任何用户访问的。为了安全起见,可以为 Elasticsearch 开启认证,并设置账号密码。
生成节点证书和私钥,并将移动到配置证书目录下
openssl req -new -x509 -days 3650 -nodes -out ca.crt -keyout ca.key
openssl req -new -nodes -out node.csr -keyout node.key
openssl x509 -req -in node.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out node.crt -days 365
将
移动到elasticsearch-8.15.1/config目录下
各节点都要拷贝和执行
chmod -R 777 ca.crt
chmod -R 777 ca.key
chmod -R 777 node.crt
chmod -R 777 node.crs
chmod -R 777 node.key
chmod -R 777 ca.srl
在 Elasticsearch 配置文件中开启 x-pack 验证并重启。例如,在 elasticsearch.yml 文件中添加以下配置:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verificationmode: certificate
xpack.security.transport.ssl.keystore.path: /opt/es/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /opt/es/certs/elastic-certificates.p12
在每台节点上修改 elasticsearch.yml
文件,开启 SSL 验证
8、配置elasticsearch8.15.1的配置文件
elasticsearch8.15.1的config目录下的文件
9、配置elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: es2
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
path.data: /opt/esdata/data
#
# Path to log files:
#
#path.logs: /path/to/logs
path.logs: /opt/esdata/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: 192.168.1.21
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
discovery.seed_hosts: ["es1", "es2","es3","es4","es5"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["es1", "es2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#是否允许使用通配符(wildcard)来删除一个或多个索引
action.destructive_requires_name: false
#用于定义Elasticsearch节点应该扮演的角色
node.master: true
node.roles: ["master"]
#当设置为 true 时,Elasticsearch在启动时会自动配置安全设置
#用于控制是否启用X-Pack安全自动配置功能,
#如果集群还没有设置密码,则会自动生成密码。
#为内置用户设置随机密码。
#为Elasticsearch集群生成加密密钥,用于加密传输和存储的数据。
#可能还会执行其他安全相关的初始化步骤。
#当设置为 false 时,Elasticsearch不会自动执行上述安全配置步骤。这意味着你需要手动进行所有安全相关的设置,包括为内置用户设置密码、生成加密密钥等。
#xpack.security.autoconfiguration.enabled: true
#用于控制是否启用X-Pack安全功能,Elasticsearch集群将要求进行身份验证和授权才能访问
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /opt/es/elasticsearch-8.15.1/config/node.key
xpack.security.transport.ssl.certificate: /opt/es/elasticsearch-8.15.1/config/node.crt
xpack.security.transport.ssl.certificate_authorities: [ "/opt/es/elasticsearch-8.15.1/config/ca.crt" ]
其中:
node.name: es3
node.master: true
node.roles: [master]
cluster.initial_master_nodes: ["es1", "es2"]
network.host: 192.168.1.21
等几项配置根据各节点修改配置
以elasticsearch用户启动Elasticsearch
sudo -u elasticsearch /opt/es/elasticsearch-8.15.1/bin/elasticsearch
注:如果创建或编辑systemd管理服务,编辑/etc/systemd/system/elasticsearch.service
[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target
[Service]
User=elasticsearch
Group=elasticsearch
ExecStart=/opt/es/elasticsearch-8.15.1/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
# Replace `-Des.config=/opt/es/config/elasticsearch.yml` with the actual path to your elasticsearch.yml
ExecStart/opt/es/elasticsearch-8.15.1/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Des.config=/opt/es/config/elasticsearch.yml
# These are the defaults, you can change them as required
LimitNOFILE=65536
LimitNPROC=4096
LimitAS=infinity
LimitFSIZE=infinity
TimeoutStartSec=0
# StandardOutput=journal
# StandardError=inherit
[Install]
WantedBy=multi-user.target
#然后启动并使服务在启动时运行:
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
为内置用户设置强密码
./bin/elasticsearch-setup-passwords interactive
elastic
:Elasticsearch 的超级用户。kibana_system
:用于 Kibana 与 Elasticsearch 通信的用户。logstash_system
:用于 Logstash 与 Elasticsearch 通信的用户。beats_system
:用于 Filebeat 和其他 Beats 与 Elasticsearch 通信的用户。apm_system
:用于 APM 服务器与 Elasticsearch 通信的用户。remote_monitoring_user
:用于 Elastic Stack 监控的用户
安装kibana
安装
上传kibana-8.15.1-linux-x86_64.tar.gz并解压
tar -zxvf kibana-8.15.1-linux-x86_64.tar.gz
#并将elasticsearch的config下拷贝到kibana的目录下
配置
# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html
# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601
server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"
server.host: "192.168.1.22"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false
# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""
# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://192.168.1.20:9200","http://192.168.1.21:9200"]
elasticsearch.hosts: ["http://192.168.1.20:9200","http://192.168.1.21:9200"]
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"
elasticsearch.username: "kibana_system"
elasticsearch.password: "elastic"
# Kibana can also authenticate to Elasticsearch via "service account tokens".
# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
# Use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# The maximum number of sockets that can be used for communications with elasticsearch.
# Defaults to `Infinity`.
#elasticsearch.maxSockets: 1024
# Specifies whether Kibana should use compression for communications with elasticsearch
# Defaults to `false`.
#elasticsearch.compression: false
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# =================== System: Elasticsearch (Optional) ===================
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
elasticsearch.ssl.certificate: /opt/kibana-8.15.1/config/node.crt
elasticsearch.ssl.key: /opt/kibana-8.15.1/config/node.key
# Enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
elasticsearch.ssl.verificationMode: certificate
# =================== System: Logging ===================
# Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'info'
#logging.root.level: debug
# Enables you to specify a file where Kibana stores log output.
#logging.appenders.default:
# type: file
# fileName: /var/logs/kibana.log
# layout:
# type: json
# Example with size based log rotation
#logging.appenders.default:
# type: rolling-file
# fileName: /var/logs/kibana.log
# policy:
# type: size-limit
# size: 256mb
# strategy:
# type: numeric
# max: 10
# layout:
# type: json
# Logs queries sent to Elasticsearch.
#logging.loggers:
# - name: elasticsearch.query
# level: debug
# Logs http responses.
#logging.loggers:
# - name: http.server.response
# level: debug
# Logs system usage information.
#logging.loggers:
# - name: metrics.ops
# level: debug
# Enables debug logging on the browser (dev console)
#logging.browser.root:
# level: debug
# =================== System: Other ===================
# The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data
#path.data: data
# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000ms.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English (default) "en", Chinese "zh-CN", Japanese "ja-JP", French "fr-FR".
#i18n.locale: "en"
# =================== Frequently used (Optional)===================
# =================== Saved Objects: Migrations ===================
# Saved object migrations run at startup. If you run into migration-related issues, you might need to adjust these settings.
# The number of documents migrated at a time.
# If Kibana can't start up or upgrade due to an Elasticsearch `circuit_breaking_exception`,
# use a smaller batchSize value to reduce the memory pressure. Defaults to 1000 objects per batch.
#migrations.batchSize: 1000
# The maximum payload size for indexing batches of upgraded saved objects.
# To avoid migrations failing due to a 413 Request Entity Too Large response from Elasticsearch.
# This value should be lower than or equal to your Elasticsearch cluster’s `http.max_content_length`
# configuration option. Default: 100mb
#migrations.maxBatchSizeBytes: 100mb
# The number of times to retry temporary migration failures. Increase the setting
# if migrations fail frequently with a message such as `Unable to complete the [...] step after
# 15 attempts, terminating`. Defaults to 15
#migrations.retryAttempts: 15
# =================== Search Autocomplete ===================
# Time in milliseconds to wait for autocomplete suggestions from Elasticsearch.
# This value must be a whole number greater than zero. Defaults to 1000ms
#unifiedSearch.autocomplete.valueSuggestions.timeout: 1000
# Maximum number of documents loaded by each shard to generate autocomplete suggestions.
# This value must be a whole number greater than zero. Defaults to 100_000
#unifiedSearch.autocomplete.valueSuggestions.terminateAfter: 100000
开放防火墙
sudo firewall-cmd --permanent --zone=public --add-port=5601/tcp #kibana端口
sudo firewall-cmd --reload
启动
sudo -u elasticsearch /opt/kibana-8.15.1/bin/kibana
访问http://192.168.1.22:5601
输入之前设置elasticsearch用户及密码登录