当前位置:   article > 正文

MySQL · myrocks · data dictionary 分析

rocksdb myrocks索引

data dictionary

rocksdb作为mysql的一个新的存储引擎,在存储引擎层,会维护自已的元数据信息。在innodb存储引擎中,我们通过information_schema下的INNODB_SYS_DATAFILES,INNODB_SYS_TABLES,INNODB_SYS_INDEXES等表,
可以窥视innodb的元数据信息。同样,rocksdb通过information_schema下的ROCKSDB_INDEX_FILE_MAP,ROCKSDB_DDL,ROCKSDB_GLOBAL_INFO等表可以查看原数据信息。

  1. show create table ROCKSDB_INDEX_FILE_MAP\G
  2. *************************** 1. row ***************************
  3. Table: ROCKSDB_INDEX_FILE_MAP
  4. Create Table: CREATE TEMPORARY TABLE `ROCKSDB_INDEX_FILE_MAP` (
  5. `COLUMN_FAMILY` int(4) NOT NULL DEFAULT '0',
  6. `INDEX_NUMBER` int(4) NOT NULL DEFAULT '0',
  7. `SST_NAME` varchar(193) NOT NULL DEFAULT '',
  8. `NUM_ROWS` bigint(8) NOT NULL DEFAULT '0',
  9. `DATA_SIZE` bigint(8) NOT NULL DEFAULT '0',
  10. `ENTRY_DELETES` bigint(8) NOT NULL DEFAULT '0',
  11. `ENTRY_SINGLEDELETES` bigint(8) NOT NULL DEFAULT '0',
  12. `ENTRY_MERGES` bigint(8) NOT NULL DEFAULT '0',
  13. `ENTRY_OTHERS` bigint(8) NOT NULL DEFAULT '0'
  14. ) ENGINE=MEMORY DEFAULT CHARSET=utf8
  15. show create table ROCKSDB_DDL\G
  16. *************************** 1. row ***************************
  17. Table: ROCKSDB_DDL
  18. Create Table: CREATE TEMPORARY TABLE `ROCKSDB_DDL` (
  19. `TABLE_SCHEMA` varchar(193) NOT NULL DEFAULT '',
  20. `TABLE_NAME` varchar(193) NOT NULL DEFAULT '',
  21. `PARTITION_NAME` varchar(193) DEFAULT NULL,
  22. `INDEX_NAME` varchar(193) NOT NULL DEFAULT '',
  23. `COLUMN_FAMILY` int(4) NOT NULL DEFAULT '0',
  24. `INDEX_NUMBER` int(4) NOT NULL DEFAULT '0',
  25. `INDEX_TYPE` smallint(2) NOT NULL DEFAULT '0',
  26. `KV_FORMAT_VERSION` smallint(2) NOT NULL DEFAULT '0',
  27. `CF` varchar(193) NOT NULL DEFAULT ''
  28. ) ENGINE=MEMORY DEFAULT CHARSET=utf8
  29. show create table ROCKSDB_GLOBAL_INFO\G
  30. *************************** 1. row ***************************
  31. Table: ROCKSDB_GLOBAL_INFO
  32. Create Table: CREATE TEMPORARY TABLE `ROCKSDB_GLOBAL_INFO` (
  33. `TYPE` varchar(513) NOT NULL DEFAULT '',
  34. `NAME` varchar(513) NOT NULL DEFAULT '',
  35. `VALUE` varchar(513) NOT NULL DEFAULT ''
  36. ) ENGINE=MEMORY DEFAULT CHARSET=utf8

元数据详情

下面我们来具体看看rocksdb维护了哪些元数据信息,从源码中看定义了以下类型,这些数据都以KV的形式存储在名叫__system__的系统column family中。

  1. // Data dictionary types
  2. enum DATA_DICT_TYPE {
  3. DDL_ENTRY_INDEX_START_NUMBER= 1,
  4. INDEX_INFO= 2,
  5. CF_DEFINITION= 3,
  6. BINLOG_INFO_INDEX_NUMBER= 4,
  7. DDL_DROP_INDEX_ONGOING= 5,
  8. INDEX_STATISTICS= 6,
  9. MAX_INDEX_ID= 7,
  10. DDL_CREATE_INDEX_ONGOING= 8,
  11. END_DICT_INDEX_ID= 255
  12. };
  • DDL_ENTRY_INDEX_START_NUMBER
    表和索引之间的映射关系
    key: Rdb_key_def::DDL_ENTRY_INDEX_START_NUMBER(0x1) + dbname.tablename
    value: version + {global_index_id}*n_indexes_of_the_table

  • INDEX_INFO
    索引id和索引属性的关系
    key: Rdb_key_def::INDEX_INFO(0x2) + global_index_id
    value: version, index_type, key_value_format_version
    index_type:主键/二级索引/隐式主键
    key_value_format_version: 记录存储格式的版本

  • CF_DEFINITION
    column family属性
    key: Rdb_key_def::CF_DEFINITION(0x3) + cf_id
    value: version, {is_reverse_cf, is_auto_cf}
    is_reverse_cf: 是否是reverse column family
    is_auto_cf: column family名字是否是$per_index_cf,名字自动由table.indexname组成

  • BINLOG_INFO_INDEX_NUMBER
    binlog位点及gtid信息,binlog_commit更新此信息
    key: Rdb_key_def::BINLOG_INFO_INDEX_NUMBER (0x4)
    value: version, {binlog_name,binlog_pos,binlog_gtid}

  • DDL_DROP_INDEX_ONGOING
    等待删除的索引信息
    key: Rdb_key_def::DDL_DROP_INDEX_ONGOING(0x5) + global_index_id
    value: version

  • INDEX_STATISTICS
    索引统计信息
    key: Rdb_key_def::INDEX_STATISTICS(0x6) + global_index_id
    value: version, {materialized PropertiesCollector::IndexStats}

  • MAX_INDEX_ID
    当前的index id,每次创建索引index id都从这个获取和更新
    key: Rdb_key_def::CURRENT_MAX_INDEX_ID(0x7)
    value: version, current max index id

  • DDL_CREATE_INDEX_ONGOING
    等待创建的索引信息
    key: Rdb_key_def::DDL_CREATE_INDEX_ONGOING(0x8) + global_index_id
    value: version

rocksdb DDL 实现

这里以建表和删表来举例

  • create table
CREATE TABLE t1 (a INT, b CHAR(8), pk INT AUTO_INCREMENT ,PRIMARY KEY(pk), key idx1(b) comment 'cf_1') ENGINE=rocksdb;

通过以下步骤建表

  1. 创建column family (get_or_create_cf)
    primary key 存在default column family中,idx1存在cf_1中,需增加一条cf_1的,CF_DEFINITION的记录
    {CF_DEFINITION(4)+cf_id(4)} —> {CF_DEFINITION_VERSION(2)+cf_flags(4)}

  2. 创建索引
    两条索引
    {INDEX_INFO(4)+cf_id(0)+index_id(260)—> { INDEX_INFO_VERSION_VERIFY_KV_FORMAT(1)+index_type(1)+kv_version(11)
    {INDEX_INFO(4)+cf_id(2)+index_id(261)—> { INDEX_INFO_VERSION_VERIFY_KV_FORMAT(2)+index_type(2)+kv_version(11)

  3. 建立表和索引的映射
    {DDL_ENTRY_INDEX_START_NUMBER(4)+dbname(test)+tablename(t1) } –> { DDL_ENTRY_INDEX_VERSION+cf_id(0)+index_id(260)+cf_id(2)+index_id(261}

以上信息通过同一batch一起存入rocksdb中。

另外,建索引时,会更新MAX_INDEX_ID信息,使用单独的batch写入,参考(Rdb_seq_generator::get_and_update_next_number)

  1. select * from INFORMATION_SCHEMA.ROCKSDB_DDL where table_name='t1';
  2. +--------------+------------+----------------+------------+---------------+--------------+------------+-------------------+---------+
  3. | TABLE_SCHEMA | TABLE_NAME | PARTITION_NAME | INDEX_NAME | COLUMN_FAMILY | INDEX_NUMBER | INDEX_TYPE | KV_FORMAT_VERSION | CF |
  4. +--------------+------------+----------------+------------+---------------+--------------+------------+-------------------+---------+
  5. | test | t1 | NULL | PRIMARY | 0 | 260 | 1 | 11 | default |
  6. | test | t1 | NULL | idx1 | 2 | 261 | 2 | 11 | cf_1 |
  7. +--------------+------------+----------------+------------+---------------+--------------+------------+-------------------+---------+
  8. select d.*,i.* from INFORMATION_SCHEMA.ROCKSDB_INDEX_FILE_MAP i,INFORMATION_SCHEMA.ROCKSDB_DDL d where i.INDEX_NUMBER=d.INDEX_NUMBER;
  9. +--------------+------------+----------------+------------+---------------+--------------+------------+-------------------+---------+---------------+--------------+------------+----------+-----------+---------------+---------------------+--------------+--------------+
  10. | TABLE_SCHEMA | TABLE_NAME | PARTITION_NAME | INDEX_NAME | COLUMN_FAMILY | INDEX_NUMBER | INDEX_TYPE | KV_FORMAT_VERSION | CF | COLUMN_FAMILY | INDEX_NUMBER | SST_NAME | NUM_ROWS | DATA_SIZE | ENTRY_DELETES | ENTRY_SINGLEDELETES | ENTRY_MERGES | ENTRY_OTHERS |
  11. +--------------+------------+----------------+------------+---------------+--------------+------------+-------------------+---------+---------------+--------------+------------+----------+-----------+---------------+---------------------+--------------+--------------+
  12. | test | t1 | NULL | PRIMARY | 0 | 260 | 1 | 11 | default | 0 | 260 | 000025.sst | 2 | 42 | 0 | 0 | 0 | 0 |
  13. | test | t1 | NULL | idx1 | 2 | 261 | 2 | 11 | cf_1 | 2 | 261 | 000027.sst | 2 | 42 | 0 | 0 | 0 | 0 |
  14. +--------------+------------+----------------+------------+---------------+--------------+------------+-------------------+---------+---------------+--------------+------------+----------+-----------+---------------+---------------------+--------------+--------------+
  15. 2 rows in set (0.00 sec)

实际数据分布如下图:
levedb_columfamily__

元数据分布在系统column family __system__中
primary key 分布在column family default中
idx1 分布在column family cf_1中
黄线之间代表数据分布的范围

  • drop table
drop table t1;

batch->Put 将索引加入到待删的kv队列中
{DDL_DROP_INDEX_ONGOING(4)+cf_id(0)+index_id(260)} –> {DDL_DROP_INDEX_ONGOING_VERSION(2)}
{DDL_DROP_INDEX_ONGOING(4)+cf_id(2)+index_id(261)} –> {DDL_DROP_INDEX_ONGOING_VERSION(2)}
batch->Delete 删除表的映射关系
表和索引的映射关系

后台线程再从待删的kv队列取出待删的索引,通过 DeleteFilesInRange, CompactRange 删除索引数据。
后台线程确定索引数据删除完成后,batch删除相应的DDL_DROP_INDEX_ONGOING和INDEX_INFO的索引信息。

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/小丑西瓜9/article/detail/628220
推荐阅读
相关标签
  

闽ICP备14008679号