当前位置:   article > 正文

Cassandra Tracing, Hints, Read Repair, Metrics expained & Scylladb mark_jolokia scylladb

jolokia scylladb

Mysql 使用 B+ tree 这种数据结构保存数据,Cassandra 使用 LSM tree。其结果就是 Cassandra 的删除操作也变成了一种写入操作,只不过写入的是一个 tomb, 而真正的删除,在 compaction 时才发生。

Cassandra TRACING:

如果cassandra开启了 TRACING ON, 然后查询报错:
在这里插入图片描述
多半是 system_traces KEYSPACE 的配置有问题,导致跨 DC 写数据超时(因为开启tracing需要向 system_traces 中的 table 写入数据),需要:

alter keyspace system_traces xxx
  • 1

Cassandra Read Repair 参数:

关于 nodetool repair:

nodetool repair # 默认会repair 该node自己的token range 和 保存的 replicas 的 token range, 所以会牵扯到好几个node的相互之间数据对比

nodetool repair -pr  # 只会 repair 该node 自己的token range, 所以,如果需要将某个keyspace的数据全部repair,需要在所有 hold 这个 keyspace的 node 上运行 nodetool repair -pr
  • 1
  • 2
  • 3

除了手动执行 repair, cassandra 还会在query该数据的时候触发 repair, 即 read repair. 关于 Read Repair:
Two types of read repair: foreground and background.

Foreground here means blocking – we complete all operations before returning to the client. Background means non-blocking – we begin the background repair operation and then return to the client before it has completed.

关于配置 background Read Repair 的参数:
dclocal_read_repair_chance = 0.1  # 同一个dc中随机read_repair的概率是 10%
read_repair_chance = 0.0  # cluster 中随机 read_repaire的概率是 0
speculative_retry = xxx
• ALWAYS: The coordinator node always sends extra read requests to all replicas after every read of this table.
• NONE: The coordinator node never sends extra read requests after a read of this table.
• Xpercentile: The coordinator node sends extra read requests if the table’s latency is higher than normal. That is, extra reads are triggered after a percent of the normal read latency of a table elapses. For example, the coordinator sends out extra read requests after waiting for 48ms (80 percent of 60ms) if a table’s latency is 60 seconds on average.
• Nms: The coordinator node sends out extra read requests if it doesn’t receive any results from the target node in N milliseconds.
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8

注意, 参数 dclocal_read_repair_chance 和 speculative_retry 同时使用好像有个bug,或导致跨DC 的read repair, check this link, 需要关闭其一,例如:

alter table test with dclocal_read_repair_chance=0.0;
  • 1

关于 Cassandra metrics:

关于如何查看 replication latency:
replication 发生各个node之间,通过查看CrossNodeLatency 指标可以看到 cassandra的复制延时:

注意, cassandra 的Latency metrics 是统计node争个存活期间的数据,其中的Count是单调递增的,只有在node重启时才会reset.

Count=117 表示从该node start开始,接收到177个Message,其中平均延时是 520712.29534117645 微秒(0.52 秒);
OneMinuteRate = 1.356... 表示每分钟平均收到 1.35 个Message;
99thPercentile : 5960319.812 表示 99%的消息在 5.96 秒内收到;
  • 1
  • 2
  • 3

curl http://127.0.0.1:8778/org.apache.cassandra.metrics:type=Messaging,name=* | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2311    0  2311    0     0   2274      0 --:--:--  0:00:01 --:--:--  2274
{
  "request": {
    "mbean": "org.apache.cassandra.metrics:name=*,type=Messaging",
    "type": "read"
  },
  "value": {
    "org.apache.cassandra.metrics:name=CrossNodeLatency,type=Messaging": {
      "Mean": 520712.29534117645,
      "StdDev": 1200436.6643582033,
      "75thPercentile": 268650.95,
      "98thPercentile": 5960319.812,
      "RateUnit": "events/second",
      "95thPercentile": 4139110.981,
      "99thPercentile": 5960319.812,
      "Max": 5960319.812,
      "Count": 117,
      "FiveMinuteRate": 0.33695134016087785,
      "50thPercentile": 186563.16,
      "MeanRate": 2.661284598969726,
      "Min": 2816.16,
      "OneMinuteRate": 1.3564407341962146,
      "DurationUnit": "microseconds",
      "999thPercentile": 5960319.812,
      "FifteenMinuteRate": 0.11664276131878955
    }
  },
  "timestamp": 1586250903,
  "status": 200
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33

关于 Cassandra Internode 加密传输:
Cassandra 默认使用 7000 端口传输数据,但是配置了ssl加密的话使用的是 7001 端口。
使用 ssl 加密的话,证书不正确的node不能加入cluster,会报证书验证失败,同时开启了 ssl 加密 7000 端口就不能用于 node 间的数据传输了。但是还是可以连接 client(同样非ssl).
推荐使用该脚本生成keystore 和 truststore:

#!/bin/bash

KEY_STORE_PATH="$PWD/resources/opt/cassandra/conf/certs"
mkdir -p "$KEY_STORE_PATH"
KEY_STORE="$KEY_STORE_PATH/cassandra.keystore"
PKS_KEY_STORE="$KEY_STORE_PATH/cassandra.pks12.keystore"
TRUST_STORE="$KEY_STORE_PATH/cassandra.truststore"
PASSWORD=xxxx
CLUSTER_NAME=mycluster
CLUSTER_PUBLIC_CERT="$KEY_STORE_PATH/CLUSTER_${CLUSTER_NAME}_PUBLIC.cer"
CLIENT_PUBLIC_CERT="$KEY_STORE_PATH/CLIENT_${CLUSTER_NAME}_PUBLIC.cer"

### Cluster key setup.
# Create the cluster key for cluster communication.
keytool -genkey -keyalg RSA -alias "${CLUSTER_NAME}_CLUSTER" -keystore "$KEY_STORE" -storepass "$PASSWORD" -keypass "$PASSWORD" \
-dname "CN=MyCompany Image $CLUSTER_NAME cluster, OU=A, O=MyCompany, L=Shanghai, ST=CA, C=CN, DC=MyCompany, DC=com" \
-validity 36500

# Create the public key for the cluster which is used to identify nodes.
keytool -export -alias "${CLUSTER_NAME}_CLUSTER" -file "$CLUSTER_PUBLIC_CERT" -keystore "$KEY_STORE" \
-storepass "$PASSWORD" -keypass "$PASSWORD" -noprompt

# Import the identity of the cluster public cluster key into the trust store so that nodes can identify each other.
keytool -import -v -trustcacerts -alias "${CLUSTER_NAME}_CLUSTER" -file "$CLUSTER_PUBLIC_CERT" -keystore "$TRUST_STORE" \
-storepass "$PASSWORD" -keypass "$PASSWORD" -noprompt


### Client key setup.
# Create the client key for CQL.
keytool -genkey -keyalg RSA -alias "${CLUSTER_NAME}_CLIENT" -keystore "$KEY_STORE" -storepass "$PASSWORD" -keypass "$PASSWORD" \
-dname "CN=MyCompany Image $CLUSTER_NAME client, OU=MyCompany, O=MyCompany, L=Shanghai, ST=CA, C=CN, DC=MyCompany, DC=com" \
-validity 36500

# Create the public key for the client to identify itself.
keytool -export -alias "${CLUSTER_NAME}_CLIENT" -file "$CLIENT_PUBLIC_CERT" -keystore "$KEY_STORE" \
-storepass "$PASSWORD" -keypass "$PASSWORD" -noprompt

# Import the identity of the client pub  key into the trust store so nodes can identify this client.
keytool -importcert -v -trustcacerts -alias "${CLUSTER_NAME}_CLIENT" -file "$CLIENT_PUBLIC_CERT" -keystore "$TRUST_STORE" \
-storepass "$PASSWORD" -keypass "$PASSWORD" -noprompt

keytool -importkeystore -srckeystore "$KEY_STORE" -destkeystore "$PKS_KEY_STORE" -deststoretype PKCS12 \
-srcstorepass "$PASSWORD" -deststorepass "$PASSWORD"

openssl pkcs12 -in "$PKS_KEY_STORE" -nokeys -out "${CLUSTER_NAME}_CLIENT.cer.pem" -passin pass:xxxx
openssl pkcs12 -in "$PKS_KEY_STORE" -nodes -nocerts -out "${CLUSTER_NAME}_CLIENT.key.pem" -passin pass:xxxx


  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48

如果只需要使用node之间通讯加密,可以忽略client证书;client 证书可用于双向ssl认证,即server也会认证client的合法性。 详见连接:http://cloudurable.com/blog/cassandra-ssl-cluster-setup/index.html

Cassandra使用总结:
1. cassandra 在跨区传输时,如果网络丢包严重,当写入量很大时,会导致产生很多的hints, 影响数据同步。可在 /var/lib/cassandar/hints 目录下看到大量文件。适当提高 hinted_handoff_throttle_in_kb,max_hints_delivery_threads 能改善同步性能,但是不能解决根本问题。最好是设置好replicates 的本地存储。避免跨网络不好的的replicates传输。查看日志能看到coordinater又大量的日志:
INFO [HintsDispatcher:73] 2020-05-25 03:26:03,615 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 6766cc74-59a5-455f-90cf-2ef8b91df6ea-1590376417285-1.hints to endpoint /1.2.3.4: 6766cc74-59a5-455f-90cf-2ef8b91df6ea, partially
查看 metric 也能看到很多HintsTimeOut:

curl http://localhost:8778/jolokia/list
curl http://localhost:8778/jolokia/read/org.apache.cassandra.metrics:type=HintsService,name=HintsTimedOut | python -m json.tool
  • 1
  • 2

这时使用 nodetool repair 也无济于事,只能等很长时间后(根据数据量和网络情况,数分钟至数小时。。。),coordinater 将所有的hints传到该node,数据才能同步上。

2 . 在写入时序数据时,可设置compatction为 TimeWindowCompactionStrategy,但是注意,如果 gc_grace_seconds 时间设置为0,会导致不能生成 hints(或者生成 hints 立即 expire了). check this link.
三个参数:

  • max_hint_window_ms controlling how long to collect hints for.
  • gc_grace_seconds indicating hint expiration time. (同时也是 Tombstones 被删除的时间。)
  • Data TTL determining duration of data validity.

推荐设置:

  • When not using data TTL, gc_grace_seconds(默认十天) should be (far) longer than max_hint_window_ms (默认3小时).
  • When using data TTL, gc_grace_seconds should be (reasonably) larger than the smaller of max_hint_window_ms and data TTL. 对于使用 TTL 的数据来说,gc_grace_seconds 一般可以根据TTL改小点,以便快速删除Tombstones, 但是又不应该设置小于 max_hint_window_ms (默认3小时), that’s all…

如果是少量数据,但又想使用cassandra的p2p特性,可以考虑 Hazelcast.


  1. https://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
  2. https://thelastpickle.com/blog/2016/12/08/TWCS-part1.html
  3. http://www.benstopford.com/2015/02/14/log-structured-merge-trees/
  4. https://priyankvex.wordpress.com/2019/04/28/introduction-to-lsm-trees-may-the-logs-be-with-you/
  5. https://thelastpickle.com/blog/
声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/盐析白兔/article/detail/712159
推荐阅读
相关标签
  

闽ICP备14008679号