赞
踩
实际工作中,按时间分片的比较多,但是也有些特殊的分片,不只是用到一个字段,可能会多个字段,这个时候就要用到复合分片算法来实现了 ComplexKeysShardingAlgorithm 。
我也是偷懒,不想设计其他的表了,就用以前的表结构来完成,我们就以用户id和金额分表。本文示例大概架构如下图:
例子没很大的实用性,你可以扩展为一个平台很多商户,同一个商户的交易按月分表,也是没问题的,但是我们学习的是思路。
pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.6.0</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>com.alian</groupId> <artifactId>sharding-jdbc</artifactId> <version>0.0.1-SNAPSHOT</version> <name>sharding-jdbc</name> <description>sharding-jdbc</description> <properties> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-jpa</artifactId> </dependency> <dependency> <groupId>org.apache.shardingsphere</groupId> <artifactId>sharding-jdbc-spring-boot-starter</artifactId> <version>4.1.1</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>druid</artifactId> <version>1.2.15</version> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>8.0.26</version> <scope>runtime</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.20</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>
有些小伙伴的 druid 可能用的是 druid-spring-boot-starter
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid-spring-boot-starter</artifactId>
<version>1.2.6</version>
</dependency>
然后出现可能使用不了的各种问题,这个时候你只需要在主类上添加 @SpringBootApplication(exclude = {DruidDataSourceAutoConfigure.class}) 即可
package com.alian.shardingjdbc;
import com.alibaba.druid.spring.boot.autoconfigure.DruidDataSourceAutoConfigure;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication(exclude = {DruidDataSourceAutoConfigure.class})
@SpringBootApplication
public class ShardingJdbcApplication {
public static void main(String[] args) {
SpringApplication.run(ShardingJdbcApplication.class, args);
}
}
CREATE DATABASE `sharding_12` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
在数据库sharding_12创建表:tb_order_00、tb_order_01、tb_order_02,三者的结构是一样的:
CREATE TABLE `tb_order_00` ( `order_id` bigint(20) NOT NULL COMMENT '主键', `user_id` int unsigned NOT NULL DEFAULT '0' COMMENT '用户id', `price` int unsigned NOT NULL DEFAULT '0' COMMENT '价格(单位:分)', `order_status` tinyint unsigned NOT NULL DEFAULT '1' COMMENT '订单状态(1:待付款,2:已付款,3:已取消)', `order_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', `title` varchar(100) NOT NULL DEFAULT '' COMMENT '订单标题', PRIMARY KEY (`order_id`), KEY `idx_user_id` (`user_id`), KEY `idx_order_time` (`order_time`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='订单表'; CREATE TABLE `tb_order_01` ( `order_id` bigint(20) NOT NULL COMMENT '主键', `user_id` int unsigned NOT NULL DEFAULT '0' COMMENT '用户id', `price` int unsigned NOT NULL DEFAULT '0' COMMENT '价格(单位:分)', `order_status` tinyint unsigned NOT NULL DEFAULT '1' COMMENT '订单状态(1:待付款,2:已付款,3:已取消)', `order_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', `title` varchar(100) NOT NULL DEFAULT '' COMMENT '订单标题', PRIMARY KEY (`order_id`), KEY `idx_user_id` (`user_id`), KEY `idx_order_time` (`order_time`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='订单表'; CREATE TABLE `tb_order_02` ( `order_id` bigint(20) NOT NULL COMMENT '主键', `user_id` int unsigned NOT NULL DEFAULT '0' COMMENT '用户id', `price` int unsigned NOT NULL DEFAULT '0' COMMENT '价格(单位:分)', `order_status` tinyint unsigned NOT NULL DEFAULT '1' COMMENT '订单状态(1:待付款,2:已付款,3:已取消)', `order_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', `title` varchar(100) NOT NULL DEFAULT '' COMMENT '订单标题', PRIMARY KEY (`order_id`), KEY `idx_user_id` (`user_id`), KEY `idx_order_time` (`order_time`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='订单表';
为了说明 ComplexKeysShardingAlgorithm 算法,我假设有个这样的需求(实际中应该不会有,这里只是演示),生成的订单规则如下:
很明显,这样分表的字段就是多个了,userId和price,那我们看怎么实现的
application.properties
server.port=8899 server.servlet.context-path=/sharding-jdbc # 允许定义相同的bean对象去覆盖原有的 spring.main.allow-bean-definition-overriding=true # 数据源名称,多数据源以逗号分隔 spring.shardingsphere.datasource.names=ds1 # sharding_1数据库连接池类名称 spring.shardingsphere.datasource.ds1.type=com.alibaba.druid.pool.DruidDataSource # sharding_1数据库驱动类名 spring.shardingsphere.datasource.ds1.driver-class-name=com.mysql.cj.jdbc.Driver # sharding_1数据库url连接 spring.shardingsphere.datasource.ds1.url=jdbc:mysql://192.168.0.129:3306/sharding_12?serverTimezone=GMT%2B8&characterEncoding=utf8&useUnicode=true&useSSL=false&zeroDateTimeBehavior=CONVERT_TO_NULL&autoReconnect=true&allowMultiQueries=true&failOverReadOnly=false&connectTimeout=6000&maxReconnects=5 # sharding_1数据库用户名 spring.shardingsphere.datasource.ds1.username=alian # sharding_1数据库密码 spring.shardingsphere.datasource.ds1.password=123456 # 指定库分片策略 spring.shardingsphere.sharding.default-data-source-name=ds1 # 指定tb_order表的数据分布情况,配置数据节点,使用Groovy的表达式 spring.shardingsphere.sharding.tables.tb_order.actual-data-nodes=ds1.tb_order_0$->{0..2} # 采用标准分片策略:ComplexKeysShardingAlgorithm # 指定tb_order表的分片策略中的分片键 spring.shardingsphere.sharding.tables.tb_order.table-strategy.complex.sharding-columns=user_id,price # 指定tb_order表的分片策略中的分片算法全类路径的名称 spring.shardingsphere.sharding.tables.tb_order.table-strategy.complex.algorithm-class-name=com.alian.shardingjdbc.algorithm.OrderComplexShardingAlgorithm # 指定tb_order表的主键为order_id spring.shardingsphere.sharding.tables.tb_order.key-generator.column=order_id # 指定tb_order表的主键生成策略为SNOWFLAKE spring.shardingsphere.sharding.tables.tb_order.key-generator.type=SNOWFLAKE # 指定雪花算法的worker.id spring.shardingsphere.sharding.tables.tb_order.key-generator.props.worker.id=100 # 指定雪花算法的max.tolerate.time.difference.milliseconds spring.shardingsphere.sharding.tables.tb_order.key-generator.props.max.tolerate.time.difference.milliseconds=20 # 打开sql输出日志 spring.shardingsphere.props.sql.show=true
application.yml
server: port: 8899 servlet: context-path: /sharding-jdbc spring: main: # 允许定义相同的bean对象去覆盖原有的 allow-bean-definition-overriding: true shardingsphere: props: sql: # 打开sql输出日志 show: true datasource: # 数据源名称,多数据源以逗号分隔 names: ds1 ds1: # 数据库连接池类名称 type: com.alibaba.druid.pool.DruidDataSource # 数据库驱动类名 driver-class-name: com.mysql.cj.jdbc.Driver # 数据库url连接 url: jdbc:mysql://192.168.0.129:3306/sharding_12?serverTimezone=GMT%2B8&characterEncoding=utf8&useUnicode=true&useSSL=false&zeroDateTimeBehavior=CONVERT_TO_NULL&autoReconnect=true&allowMultiQueries=true&failOverReadOnly=false&connectTimeout=6000&maxReconnects=5 # 数据库用户名 username: alian # 数据库密码 password: 123456 sharding: # 未配置分片规则的表将通过默认数据源定位 default-data-source-name: ds1 tables: tb_order: # 由数据源名 + 表名组成,以小数点分隔。多个表以逗号分隔,支持inline表达式 actual-data-nodes: ds1.tb_order_0$->{0..2} # 分表策略 table-strategy: complex: # 分片键 sharding-columns: user_id,price # 复合分片算法 algorithm-class-name: com.alian.shardingjdbc.algorithm.OrderComplexShardingAlgorithm # key生成器 key-generator: # 自增列名称,缺省表示不使用自增主键生成器 column: order_id # 自增列值生成器类型,缺省表示使用默认自增列值生成器(SNOWFLAKE/UUID) type: SNOWFLAKE # SnowflakeShardingKeyGenerator props: # SNOWFLAKE算法的worker.id worker: id: 100 # SNOWFLAKE算法的max.tolerate.time.difference.milliseconds max: tolerate: time: difference: milliseconds: 20
具体的算法实现如下:
OrderComplexShardingAlgorithm.java
package com.alian.shardingjdbc.algorithm; import lombok.extern.slf4j.Slf4j; import org.apache.shardingsphere.api.sharding.complex.ComplexKeysShardingAlgorithm; import org.apache.shardingsphere.api.sharding.complex.ComplexKeysShardingValue; import java.time.format.DateTimeFormatter; import java.util.ArrayList; import java.util.Collection; import java.util.Map; @Slf4j public class OrderComplexShardingAlgorithm implements ComplexKeysShardingAlgorithm<Integer> { private static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("yyyyMM"); @Override public Collection<String> doSharding(Collection<String> availableTargetNames, ComplexKeysShardingValue<Integer> shardingValue) { Collection<String> result = new ArrayList<>(); Map<String, Collection<Integer>> shardingValuesMap = shardingValue.getColumnNameAndShardingValuesMap(); Collection<Integer> userIds = shardingValuesMap.get("user_id"); Collection<Integer> prices = shardingValuesMap.get("price"); for (Integer userId : userIds) { for (Integer price : prices) { String suffix; if (userId % 2 == 1 && price % 2 == 1) { // 奇数 suffix = "01"; } else if (userId % 2 == 0 && price % 2 == 0) { // 偶数 suffix = "02"; } else { // 1奇数1偶数 suffix = "00"; } for (String targetName : availableTargetNames) { if (targetName.endsWith("_" + suffix)) { // 按照分片值选择目标表 result.add(targetName); } } } } return result; } }
我们是按照订单时间进行分表的,实际使用也很简单,插入数据时实现接口 OrderComplexShardingAlgorithm <Integer> 。然后重写方法 doSharding ,这个方法会有两个参数,第一个就是物理表的集合,第二个是分片对象。但是有人就会说,你这里正好两个字段都是同一个类型,而我们实际中,可能是多种类型,比如String、Long、Integer、Date等,按照这个写法,不是这个类型的就取不到值了,如果你是你说的这种情况,你就把上面的代码改造下,把类型改成 <Comparable<?>> ,然后就自己转类型,别说你自己都不知道类型,具体见下面代码。
package com.alian.shardingjdbc.algorithm; import lombok.extern.slf4j.Slf4j; import org.apache.shardingsphere.api.sharding.complex.ComplexKeysShardingAlgorithm; import org.apache.shardingsphere.api.sharding.complex.ComplexKeysShardingValue; import java.util.ArrayList; import java.util.Collection; import java.util.Map; @Slf4j public class ComparableOrderComplexShardingAlgorithm implements ComplexKeysShardingAlgorithm<Comparable<?>> { @Override public Collection<String> doSharding(Collection<String> availableTargetNames, ComplexKeysShardingValue<Comparable<?>> shardingValue) { Collection<String> result = new ArrayList<>(); Map<String, Collection<Comparable<?>>> shardingValuesMap = shardingValue.getColumnNameAndShardingValuesMap(); Collection<?> userIds = shardingValuesMap.get("user_id"); Collection<?> prices = shardingValuesMap.get("price"); // Collection<?> orderTimes = shardingValuesMap.get("order_time"); for (Object userIdObj : userIds) { for (Object priceObj : prices) { String suffix; int userId = Integer.parseInt(userIdObj + ""); int price = Integer.parseInt(priceObj + ""); if (userId % 2 == 1 && price % 2 == 1) { // 奇数 suffix = "01"; } else if (userId % 2 == 0 && price % 2 == 0) { // 偶数 suffix = "02"; } else { // 1奇数1偶数 suffix = "00"; } for (String targetName : availableTargetNames) { if (targetName.endsWith("_" + suffix)) { // 按照分片值选择目标表 result.add(targetName); } } } } return result; } }
Order.java
@Data @Entity @Table(name = "tb_order") public class Order implements Serializable { @Id @GeneratedValue(strategy = GenerationType.IDENTITY) @Column(name = "order_id") private Long orderId; @Column(name = "user_id") private Integer userId; @Column(name = "price") private Integer price; @Column(name = "order_status") private Integer orderStatus; @Column(name = "title") private String title; @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss") @Column(name = "order_time") private LocalDateTime orderTime; }
OrderRepository.java
public interface OrderRepository extends PagingAndSortingRepository<Order, Long> {
/**
* 根据userId和金额查询订单
*
* @param userId
* @param price
* @return
*/
List<Order> findByUserIdAndPrice(int userId, int price);
}
OrderService.java
@Slf4j @Service public class OrderService { @Autowired private OrderRepository orderRepository; public void saveOrder(Order order) { orderRepository.save(order); } public List<Order> findByUserIdAndPrice(int userId, int price) { return orderRepository.findByUserIdAndPrice(userId, price); } }
OrderTests.java
@Slf4j @RunWith(SpringJUnit4ClassRunner.class) @SpringBootTest public class OrderTests { @Autowired private OrderService orderService; @Test public void saveOrder() { // 创建20条订单记录 for (int i = 0; i < 20; i++) { Order order = new Order(); // order.setOrderId(System.currentTimeMillis()); // 随机生成1000到1009的用户id int userId = (int) Math.round(Math.random() * (1009 - 1000) + 1000); order.setUserId(userId); // 随机生成50到100的金额 int price = (int) Math.round(Math.random() * (10000 - 5000) + 5000); order.setPrice(price); order.setOrderStatus(2); order.setOrderTime(LocalDateTime.now()); order.setTitle(""); orderService.saveOrder(order); } } @Test public void queryOrder() { List<Order> order = orderService.findByUserIdAndPrice(1002,9860); log.info("查询的结果:{}", order); } }
我们插入数据时,采用随机时间插入,具体时间生成见测试类。
效果图:
从上面的数据来看,满足我们分库分表的要求的,实现都是基于我们自定义的算法实现。
13:09:39 403 INFO [main]:Actual SQL: ds1 ::: select order0_.order_id as order_id1_0_, order0_.order_status as order_st2_0_, order0_.order_time as order_ti3_0_, order0_.price as price4_0_, order0_.title as title5_0_, order0_.user_id as user_id6_0_ from tb_order_02 order0_ where order0_.user_id=? and order0_.price=? ::: [1002, 9860]
13:09:39 463 INFO [main]:查询的结果:[Order(orderId=940934509405552641, userId=1002, price=9860, orderStatus=2, title=, orderTime=2023-12-11T11:37:49)]
通过语句查询:
(SELECT *,'tb_order_00' FROM sharding_12.tb_order_00
where user_id=1002 and price=9860)
union all
(SELECT *,'tb_order_01' FROM sharding_12.tb_order_01
where user_id=1002 and price=9860)
union all
(SELECT *,'tb_order_02' FROM sharding_12.tb_order_02
where user_id=1002 and price=9860)
数据库校验:
赞
踩
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。