【黑马头条】-day04自媒体文章审核-阿里云接口-敏感词分析DFA-图像识别OCR-异步调用MQ

---

---

day4学习内容

自媒体文章自动审核

今日内容

1 自媒体文章自动审核

1.1 审核流程

1.2 内容安全第三方接口

1.3 引入阿里云内容安全接口

1.3.1 添加依赖

在heima-leadnews-common包下引入依赖

<dependency>
    <groupId>com.aliyun</groupId>
    <artifactId>aliyun-java-sdk-core</artifactId>
    <version>4.1.1</version>
</dependency>
<dependency>
    <groupId>com.aliyun</groupId>
    <artifactId>aliyun-java-sdk-green</artifactId>
    <version>3.6.6</version>
</dependency>
<dependency>
    <groupId>com.alibaba.fastjson2</groupId>
    <artifactId>fastjson2</artifactId>
    <version>2.0.9</version>
</dependency>
<dependency>
    <groupId>com.aliyun.oss</groupId>
    <artifactId>aliyun-sdk-oss</artifactId>
    <version>2.8.3</version>
</dependency>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

1.3.2 导入aliyun模块

放入heima-leadnews-common模块下的com.heima.common

哪个微服务使用，就在哪个微服务的nacos中配置

在heima-leadnews-wemedia中的nacos配置中心添加以下配置：

aliyun:
 accessKeyId: LTAI5tCWHCcfvqQzu8k2oKmX
 secret: auoKUFsghimbfVQHpy7gtRyBkoR4vc
#aliyun.scenes=porn,terrorism,ad,qrcode,live,logo
 scenes: terrorism
1
2
3
4
5

1.3.3 注入Bean测试

在resource中META-INF的spring-factories中自动配置

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
  com.heima.common.exception.ExceptionCatch,\
  com.heima.common.aliyun.GreenTextScan,\
    com.heima.common.aliyun.GreenImageScan
1
2
3
4

在测试类中进行测试

@SpringBootTest(classes = WemediaApplication.class)
@RunWith(SpringRunner.class)
public class AliyunTest {

    @Autowired
    private GreenTextScan greenTextScan;

    @Autowired
    private GreenImageScan greenImageScan;

    @Autowired
    private FileStorageService fileStorageService;

    @Test
    public void testScanText() throws Exception {
        Map map = greenTextScan.greeTextScan("我是一个好人,冰毒");
        System.out.println(map);
    }

    @Test
    public void testScanImage() throws Exception {
        byte[] bytes = fileStorageService.downLoadFile("http://192.168.200.130:9000/leadnews/2021/04/26/ef3cbe458db249f7bd6fb4339e593e55.jpg");
        Map map = greenImageScan.imageScan(Arrays.asList(bytes));
        System.out.println(map);
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

2 app端文章保存接口

2.1 表结构说明

2.2 分布式id

2.2.1 分布式id-技术选型

2.2.2 雪花算法

2.2.3 配置雪花算法

第一：在实体类中的id上加入如下配置，指定类型为id_worker

@TableId(value = "id",type = IdType.ID_WORKER)
private Long id;
1
2

第二：在application.yml文件中配置数据中心id和机器id

在文章的微服务的nacos配置中leadnews-article中添加

spring:
  datasource:
    driver-class-name: com.mysql.jdbc.Driver
    url: jdbc:mysql://localhost:3306/leadnews_article?useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC&useSSL=false
    username: root
    password: 123sjbsjb

# 设置Mapper接口所对应的XML文件位置，如果你在Mapper接口中有自定义方法，需要进行该配置
mybatis-plus:
  mapper-locations: classpath*:mapper/*.xml
  # 设置别名包扫描路径，通过该属性可以给包中的类注册别名
  type-aliases-package: com.heima.model.article.pojos
  #雪花算法
  global-config:
    datacenter-id: 1
    workerId: 1
minio:
  accessKey: minio
  secretKey: minio123
  bucket: leadnews
  endpoint: http://192.168.204.129:9000
  readPath: http://192.168.204.129:9000
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

2.3 保存app端文章-思路分析

2.4 实现接口

2.4.1 实现步骤

2.4.2 定义feign接口

2.4.2.1 导入feign远程调用依赖

在heima-leadnews-feign-api的pom.xml中导入依赖

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
1
2
3
4

2.4.2.2 定义文章端远程接口

heima-leadnews-feign-api定义com.heima.apis.article.IArticleClient接口

@FeignClient(value = "leadnews-article")
1

@FeignClient指定文章远程调用接口名称

@FeignClient(value = "leadnews-article")
public interface IArticleClient {

    @PostMapping("/api/v1/article/save")
    public ResponseResult saveArticle(@RequestBody ArticleDto dto) ;
}
1
2
3
4
5
6

2.4.2.3 导入ArticleDto

在heima-leadnews-model模块下com.heima.model.article.dto中导入ArticleDto类

@Data
public class ArticleDto  extends ApArticle {
    /**
     * 文章内容
     */
    private String content;
}
1
2
3
4
5
6
7

2.4.3 实现feign接口

在heima-leadnews-service模块下的heima-leadnews-article模块下创建com.heima.article.feign.ArticleClient类

@RestController
public class ArticleClient implements IArticleClient {
    @Autowired
    private ApArticleService apArticleService;

    @PostMapping("/api/v1/article/save")
    @Override
    public ResponseResult saveArticle(@RequestBody ArticleDto dto) {
        return apArticleService.saveArticle(dto);
    }
}
1
2
3
4
5
6
7
8
9
10
11

2.4.4 创建mapper

在heima-leadnews-service模块下的heima-leadnews-article模块下创建com.heima.article.mapper.ApArticleConfigMapper接口

@Mapper
public interface ApArticleConfigMapper extends BaseMapper<ApArticleConfig> {
}
1
2
3

2.4.5 为AparticleConfig设置默认参数

添加@NoArgsConstructor

public ApArticleConfig(Long articleId) {
    this.articleId = articleId;
    this.isDelete = false;
    this.isDown = false;
    this.isForward = true;
    this.isComment = true;
}
1
2
3
4
5
6
7

添加有参构造

2.4.6 在ApArticleService的实现类ApArticleServiceImpl中实现方法

ApArticleService接口

public interface ApArticleService extends IService<ApArticle>{
    /**
     * 加载文章列表
     * @param dto
     * @param type 1 加载更多 2 加载最新
     * @return
     */
    public ResponseResult load(ArticleHomeDto dto, Short type);

    /**
     * 保存文章
     * @param dto
     * @return
     */
    public ResponseResult saveArticle(ArticleHomeDto dto);
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

实现类，实现saveArticle方法

@Autowired
private ApArticleConfigMapper apArticleConfigMapper;
@Autowired
private ApArticleContentMapper apArticleContentMapper;
/**
 * 保存文章
 * @param dto
 * @return
 */
@Override
public ResponseResult saveArticle(ArticleDto dto) {
    //1.参数检查
    if(dto == null){
        return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID);
    }
    ApArticle apArticle = new ApArticle();
    //org.springframework.beans
    BeanUtils.copyProperties(dto, apArticle);
    //2.判断是否存在id
    if(dto.getId() == null) {
        //2.1 不存在id ，新增 文章、内容、配置
        save(apArticle);
        //2.1.2 保存文章配置
        ApArticleConfig apArticleConfig = new ApArticleConfig(apArticle.getId());
        apArticleConfigMapper.insert(apArticleConfig);
        //2.1.3 保存文章内容
        ApArticleContent apArticleContent = new ApArticleContent();
        apArticleContent.setArticleId(apArticle.getId());
        apArticleContent.setContent(dto.getContent());
        apArticleContentMapper.insert(apArticleContent);
    }
    else {
        //2.2 存在id，更新 文章、内容
        //2.2.1 更新文章
        updateById(apArticle);
        //2.2.2 更新文章内容
        ApArticleContent apArticleContent = apArticleContentMapper.selectOne(Wrappers
                .<ApArticleContent>lambdaQuery()
                .eq(ApArticleContent::getArticleId, dto.getId()));
        apArticleContent.setContent(dto.getContent());
        apArticleContentMapper.updateById(apArticleContent);
    }
    //3.返回结果 文章的id
    return ResponseResult.okResult(apArticle.getId());
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

2.4.7 启动ArticleApplication

刚刚是新增，如果是修改。

就会在json中传入id

成功修改

3 自媒体文章审核实现

3.1 创建审核接口

在heima-leadnews-service中heima-leadnews-wemedia中的service新增WmNewAutoScanService接口

public interface WmNewAutoScanService {
    /**
     * 自动审核媒体文章
     */
    public void  autoScanMediaNews(Integer id);
}
1
2
3
4
5
6

3.2 实现审核接口

@Service
@Slf4j
@Transactional
public class WmNewAutoScanServiceImpl implements WmNewAutoScanService {
    @Autowired
    private WmNewsMapper wmNewsMapper;
    @Qualifier("com.heima.apis.article.IArticleClient")
    @Autowired
    private IArticleClient iArticleClient;
    @Autowired
    private WmChannelMapper wmChannelMapper;
    @Autowired
    private WmUserMapper wmUserMapper;

    @Override
    public void autoScanMediaNews(Integer id) {
        //1.查询自媒体文章
        WmNews wmNews = wmNewsMapper.selectById(id);
        if (wmNews == null) {
            throw new RuntimeException("自媒体文章不存在");
        }
        if(wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())){
            Map<String,List<String>> scanMaterialsList = extractImageAndContent(wmNews);
            //2.调用阿里云接口审核文本内容
            List<String> contentTexts =scanMaterialsList.get("contentTexts");
            boolean isTextScan =true;
            if(!isTextScan)return;
            //3.调用阿里云接口审核图片内容
            List<String> imagesUrls =scanMaterialsList.get("imagesUrls");
            boolean isImageScan =true;
            if(!isImageScan)return;
            if(isTextScan && isImageScan) {
                //审核通过
                wmNews.setStatus((short) 9);
                wmNews.setReason("审核通过");
            }
        }
        //4.审核成功保存app端的相关文章数据
        ArticleDto dto=new ArticleDto();
        BeanUtils.copyProperties(wmNews,dto);
        //布局
        dto.setLayout(wmNews.getType());
        //频道
        dto.setChannelId(wmNews.getChannelId());
        //频道名称
        WmChannel wmChannel = wmChannelMapper.selectById(wmNews.getChannelId());
        if(wmChannel!=null){
            dto.setChannelName(wmChannel.getName());
        }
        //作者
        dto.setAuthorId(Long.valueOf(wmNews.getUserId()));
        //作者名称
        WmUser wmUser= wmUserMapper.selectById(wmNews.getUserId());
        if(wmUser!=null){
            dto.setAuthorName(wmUser.getName());
        }
        //设置文章id
        if(wmNews.getArticleId()!=null){
            dto.setId(wmNews.getArticleId());
        }
        dto.setCreatedTime(new Date());
        ResponseResult responseResult = iArticleClient.saveArticle(dto);
        if(responseResult.getCode().equals(200)){
            //保存成功
            wmNews.setArticleId((Long)responseResult.getData());
            wmNewsMapper.updateById(wmNews);
        }
        else{
            //保存失败
            throw new RuntimeException("保存app端文章失败");
        }
    }

    private Map<String,List<String>> extractImageAndContent(WmNews wmNews) {
        //提取文章内容
        String content = wmNews.getContent();
        List<String> imagesUrls =new ArrayList<>();
        List<String> contentTexts =new ArrayList<>();
        Map<String,List<String>> scanMaterialsList =new HashMap<>();
        List<Map> maps = JSON.parseArray(content, Map.class);
        //提取文章图片
        for (Map map : maps) {
            if(map.get("type").equals("image")){
                String imgUrl = (String) map.get("value");
                imagesUrls.add(imgUrl);
            }
            if(map.get("type").equals("text")){
                String text = (String) map.get("value");
                contentTexts.add(text);
            }
        }
        scanMaterialsList.put("imagesUrls",imagesUrls);
        scanMaterialsList.put("contentTexts",contentTexts);
        return scanMaterialsList;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

3.3 启动类扫描feign

调用Feign远程接口时，要在启动类中加入@EnableFeignClients(basePackages = “com.heima.apis”)来对feign的api进行扫描，同时也要引入feign-api模块的依赖

<dependency>
    <groupId>com.heima</groupId>
    <artifactId>heima-leadnews-feign-api</artifactId>
</dependency>
1
2
3
4

@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.wemedia.mapper")
@EnableFeignClients(basePackages = "com.heima.apis")
public class WemediaApplication {

    public static void main(String[] args) {
        SpringApplication.run(WemediaApplication.class,args);
    }

    @Bean
    public MybatisPlusInterceptor mybatisPlusInterceptor() {
        MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
        interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
        return interceptor;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

3.4 测试

转到WmNewAutoScanService接口中，ctrl+shift+T创建测试类

@SpringBootTest(classes = WemediaApplication.class)
@RunWith(SpringRunner.class)
class WmNewAutoScanServiceTest {
    @Autowired
    private WmNewAutoScanService wmNewAutoScanService;
    @Test
    void autoScanMediaNews() {
        wmNewAutoScanService.autoScanMediaNews(6236);
    }
}
1
2
3
4
5
6
7
8
9
10

4 自媒体调用文章微服务feign远程调用服务降级

4.1 feign远程调用服务降级处理的逻辑

4.2 编写降级逻辑

在heima-leadnews-feign-api模块下编写降级逻辑com.heima.apis.article.fallback.IArticleClientFallback类，实现IArticleClient接口

@Component
public class IArticleClientFallback implements IArticleClient {
    @Override
    public ResponseResult saveArticle(ArticleDto dto) {
        return ResponseResult.errorResult(AppHttpCodeEnum.SERVER_ERROR,"获取数据失败");
    }
}
1
2
3
4
5
6
7

4.3 指定IArticleClient接口指向Feign降级逻辑

@FeignClient(value = "leadnews-article",fallback = IArticleClientFallback.class)

@FeignClient(value = "leadnews-article",fallback = IArticleClientFallback.class)
public interface IArticleClient {

    @PostMapping("/api/v1/article/save")
    public ResponseResult saveArticle(@RequestBody ArticleDto dto) ;
}
1
2
3
4
5
6

4.4 加载feign降级逻辑

因为IArticleClientFallback是在com.heima.apis.article.fallback包下，并不能被spring通过@Component直接加载

因此需要在使用的微服务中加载feign

如使用的微服务是heima-leadnews-wemedia，所以要在com.heima.wemedia.config下创建InitConfig类加载feign降级策略

@Configuration
@ComponentScan("com.heima.apis.article.fallback")
public class InitConfig {
}
1
2
3
4

4.5 配置降级策略

要么在bootstrap中开启，要么在nacos中实现热更新

这里采用nacos热更新

feign:
  # 开启feign对hystrix熔断降级的支持
  hystrix:
    enabled: true
  # 修改调用超时时间
  client:
    config:
      default:
        connectTimeout: 2000
        readTimeout: 2000
1
2
3
4
5
6
7
8
9
10

4.6 测试

当前设置超时2s进行降级，测试一下

在com.heima.article.service.impl.ApArticleServiceImpl类中的saveArticle方法添加睡眠3秒进行测试

@Override
public ResponseResult saveArticle(ArticleDto dto) {
    try {
        Thread.sleep(3000);
    } catch (InterruptedException e) {
        throw new RuntimeException(e);
    }
    //1.参数检查
    if(dto == null){
1
2
3
4
5
6
7
8
9

这次审核6239

@SpringBootTest(classes = WemediaApplication.class)
@RunWith(SpringRunner.class)
class WmNewAutoScanServiceTest {
    @Autowired
    private WmNewAutoScanService wmNewAutoScanService;
    @Test
    void autoScanMediaNews() {
        wmNewAutoScanService.autoScanMediaNews(6239);
    }
}
1
2
3
4
5
6
7
8
9
10

5 文章审核异步调用

5.1 在自动审核的方法加上@Async注解

Springboot集成异步线程调用

@Override
@Async//表明这是一个异步方法
public void autoScanMediaNews(Integer id) {
    try {
        Thread.sleep(1000);
    } catch (InterruptedException e) {
        throw new RuntimeException(e);
    }
1
2
3
4
5
6
7
8

5.2 在文章发布后调用自动审核方法

//5.审核文章
wmNewAutoScanService.autoScanMediaNews(wmNews.getId());
1
2

@Autowired
private WmNewAutoScanService wmNewAutoScanService;
@Override
public ResponseResult submitNews(WmNewsDto wmNewsDto) {
    // 0.参数检查
    if(wmNewsDto == null||wmNewsDto.getContent()==null){
        return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID);
    }
    //1. 保存或修改文章
    WmNews wmNews = new WmNews();
    BeanUtils.copyProperties(wmNewsDto,wmNews);
    //1.1 封面
    if(wmNewsDto.getImages()!=null&& wmNewsDto.getImages().size()>0){
        String imageStr = StringUtils.join(wmNewsDto.getImages(), ",");
        wmNews.setImages(imageStr);
    }
    //1.2 如果封面为自动-1，则需要手动设置封面规则
    if(wmNewsDto.getType().equals(WemediaConstants.WM_NEWS_TYPE_AUTO)){
        wmNews.setType(null);
    }
    saveOrUpdateWmNews(wmNews);
    //2.判断是否为草稿，如果为草稿结束当前方法
    if(wmNews.getStatus().equals(WmNews.Status.NORMAL.getCode())){
        return ResponseResult.okResult(AppHttpCodeEnum.SUCCESS);
    }
    //3.不是草稿，保存文章内容与图片素材的关系
    //3.1 获取文章内容的图片素材
    List<String> imageList=extractUrlInfo(wmNewsDto.getContent());
    saveRelativeInfoForContent(imageList,wmNews.getId());

    //4.不是草稿，保存文章封面图片与图片素材的关系
    saveRelativeInfoForCover(wmNewsDto,wmNews,imageList);

    //5.审核文章
    wmNewAutoScanService.autoScanMediaNews(wmNews.getId());

    return ResponseResult.okResult(AppHttpCodeEnum.SUCCESS);
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

5.3 在启动类中添加注解开启异步调用

在自媒体引导类中使用@EnableAsync注解开启异步调用

@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.wemedia.mapper")
@EnableFeignClients(basePackages = "com.heima.apis")
@EnableAsync//开启异步
public class WemediaApplication {

    public static void main(String[] args) {
        SpringApplication.run(WemediaApplication.class,args);
    }

    @Bean
    public MybatisPlusInterceptor mybatisPlusInterceptor() {
        MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
        interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
        return interceptor;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

5.4 综合测试

5.5 使用rabbit MQ来完成异步调用

我的异步调用只要在启动类中加入@EnableAsync就报错，迫不得已采用rabbitMQ

5.5.1 引入依赖

在heima-leadnews-service中引入依赖

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
</dependency>
1
2
3
4
5
6
7
8

5.5.2 为微服务配置MQ

在heima-leadnews-article和wemedia的配置文件中添加配置

spring:
  rabbitmq:
    host: 192.168.204.129
    port: 5672
    virtual-host: /
    username: itcast
    password: 123321
1
2
3
4
5
6
7

5.5.3 改造方法，创建监听队列

修改heima-leadnews-wemedia下的com.heima.wemedia.service.impl.WmNewAutoScanServiceImpl类中的autoScanMediaNews方法

@Autowired
private RabbitTemplate rabbitTemplate;
@Override
public void autoScanMediaNews(Integer id) {
    //1.查询自媒体文章
    WmNews wmNews = wmNewsMapper.selectById(id);
    if (wmNews == null) {
        throw new RuntimeException("自媒体文章不存在");
    }
    if(wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())){
        Map<String,List<String>> scanMaterialsList = extractImageAndContent(wmNews);
        //2.调用阿里云接口审核文本内容
        List<String> contentTexts =scanMaterialsList.get("contentTexts");
        boolean isTextScan =true;
        if(!isTextScan)return;
        //3.调用阿里云接口审核图片内容
        List<String> imagesUrls =scanMaterialsList.get("imagesUrls");
        boolean isImageScan =true;
        if(!isImageScan)return;
        if(isTextScan && isImageScan) {
            //审核通过
            wmNews.setStatus((short) 9);
            wmNews.setReason("审核通过");
        }
    }
    //4.审核成功保存app端的相关文章数据
    ArticleDto dto=new ArticleDto();
    BeanUtils.copyProperties(wmNews,dto);
    //布局
    dto.setLayout(wmNews.getType());
    //频道
    dto.setChannelId(wmNews.getChannelId());
    //频道名称
    WmChannel wmChannel = wmChannelMapper.selectById(wmNews.getChannelId());
    if(wmChannel!=null){
        dto.setChannelName(wmChannel.getName());
    }
    //作者
    dto.setAuthorId(Long.valueOf(wmNews.getUserId()));
    //作者名称
    WmUser wmUser= wmUserMapper.selectById(wmNews.getUserId());
    if(wmUser!=null){
        dto.setAuthorName(wmUser.getName());
    }
    //设置文章id
    if(wmNews.getArticleId()!=null){
        dto.setId(wmNews.getArticleId());
    }
    dto.setCreatedTime(new Date());

    //2.rabbitmq异步处理
    Map<String,Object> map=new HashMap<>();
    map.put("dto",dto);
    map.put("wmNewsId",id);
    rabbitTemplate.convertAndSend("article.queue", map);
    /*ResponseResult responseResult = iArticleClient.saveArticle(dto);
    if(responseResult.getCode().equals(200)){
        //保存成功
        wmNews.setArticleId((Long)responseResult.getData());
        wmNewsMapper.updateById(wmNews);
    }
    else{
        //保存失败
        log.error("保存app端文章失败，responseResult: {}", responseResult);
        throw new RuntimeException("保存app端文章失败");
    }*/
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

rabbitTemplate.convertAndSend("article.queue", map);
1

发送到article.queue队列

在heima-leadnews-article模块下创建com.heima.article.mq.ArticleMessageConsumer消费者监听类监听article.queue

@Slf4j
@Component
public class ArticleMessageConsumer {
    @Autowired
    private IArticleClient iArticleClient;
    @Autowired
    private RabbitTemplate rabbitTemplate;

    @RabbitListener(bindings =@QueueBinding(
            value=@Queue(name="article.queue"),
            exchange=@Exchange(name="article.direct",type= ExchangeTypes.FANOUT)))
    public void processMessage(Map<String,Object> map) {
        ObjectMapper objectMapper = new ObjectMapper();
        Object dto = map.get("dto");
        Integer id= (Integer) map.get("wmNewsId");
        LinkedHashMap<String, Object> linkedHashMap = (LinkedHashMap<String, Object>) dto;
        ArticleDto articleDto = objectMapper.convertValue(linkedHashMap, ArticleDto.class);
        // 异步处理文章数据
        ResponseResult responseResult = iArticleClient.saveArticle(articleDto);
        if(responseResult.getCode().equals(200)){
            WmNews wmNews = new WmNews();
            BeanUtils.copyProperties(dto, wmNews);
            wmNews.setArticleId((Long)responseResult.getData());
            Map<String,Object> params = new HashMap<>();
            params.put("id", id);
            params.put("wmNews", wmNews);
            params.put("articleId",(Long)responseResult.getData());
            rabbitTemplate.convertAndSend("wmNews.queue", params);
            log.info("发送params成功，param: {}", params);
        }
        else{
            //保存失败
            log.error("保存app端文章失败，responseResult: {}", responseResult);
            throw new RuntimeException("保存app端文章失败");
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

将ResponseResult responseResult = iArticleClient.saveArticle(articleDto);回填的id发到wmNews.queue

在heima-leadnews-wemedia模块下创建com.heima.wemedia.mq.ReceiveWmNewsId消费者监听类监听wmNews.queue

@Component
@Slf4j
public class ReceiveWmNewsId {
    @Autowired
    private WmNewsMapper wmNewsMapper;

    @RabbitListener(bindings =@QueueBinding(
            value=@Queue(name="wmNews.queue"),
            exchange=@Exchange(name="wmNews.direct",type= ExchangeTypes.FANOUT)))
    public void processMessage(Map<String,Object> map) {
        ObjectMapper objectMapper = new ObjectMapper();
        Integer id= (Integer)map.get("id");
        Object wmNews= map.get("wmNews");
        Long articleId= (Long)map.get("articleId");
        LinkedHashMap<String, Object> linkedHashMap = (LinkedHashMap<String, Object>) wmNews;
        WmNews articleDto = objectMapper.convertValue(linkedHashMap, WmNews.class);
        WmNews oldwmNews = wmNewsMapper.selectById(id);
        BeanUtils.copyProperties(wmNews,oldwmNews);
        oldwmNews.setStatus((short) 9);
        oldwmNews.setReason("审核通过");
        oldwmNews.setArticleId(articleId);
        int i = wmNewsMapper.updateById(oldwmNews);
        if(i == 0){
            log.error("更新自媒体文章失败，wmNews: {}", oldwmNews);
            throw new RuntimeException("更新自媒体文章失败");
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

5.5.4 序列化MQ消息

在heima-leadnews-article和wemedia的启动类中添加序列化器

@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.wemedia.mapper")
@EnableFeignClients(basePackages = "com.heima.apis")
public class WemediaApplication {

    public static void main(String[] args) {
        SpringApplication.run(WemediaApplication.class,args);
    }

    @Bean
    public MybatisPlusInterceptor mybatisPlusInterceptor() {
        MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
        interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
        return interceptor;
    }
    @Bean
    public MessageConverter messageConverter(){
        return new Jackson2JsonMessageConverter();
    }
    @Bean
    public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
        RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMessageConverter(messageConverter);
        return rabbitTemplate;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.article.mapper")
public class ArticleApplication {

    public static void main(String[] args) {
        SpringApplication.run(ArticleApplication.class,args);
    }

    @Bean
    public MybatisPlusInterceptor mybatisPlusInterceptor() {
        MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
        interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
        return interceptor;
    }
    @Bean
    public MessageConverter messageConverter(){
        return new Jackson2JsonMessageConverter();
    }
    @Bean
    public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
        RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMessageConverter(messageConverter);
        return rabbitTemplate;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

5.5.5 加上mq后的综合测试

测试通过在MQ上也检测到消息

6 自管理敏感词过滤

6.1 DFA实现原理

6.2 DFA检索过程

6.3 实现步骤

6.3.1 创建敏感词表

在leadnews-wemedia数据库中到入wm_sensitive.sql

6.3.2 将wm_sensitive对应的实体类和mapper导入

@Data
@TableName("wm_sensitive")
public class WmSensitive implements Serializable {
    private static final long serialVersionUID = 1L;

    /**
     * 主键
     */
    @TableId(value = "id", type = IdType.AUTO)
    private Integer id;

    /**
     * 敏感词
     */
    @TableField("sensitives")
    private String sensitives;

    /**
     * 创建时间
     */
    @TableField("created_time")
    private Date createdTime;
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

@Mapper
public interface WmSensitiveMapper extends BaseMapper<WmSensitive> {
}
1
2
3

6.3.3 在阿里云接口前自行进行审查

boolean isSensitive= handleSensitiveWords(contentTexts,wmNews);

if(wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())){
    Map<String,List<String>> scanMaterialsList = extractImageAndContent(wmNews);
    //2.调用阿里云接口审核文本内容
    List<String> contentTexts =scanMaterialsList.get("contentTexts");
    //2.1 敏感词过滤
    boolean isSensitive= handleSensitiveWords(contentTexts,wmNews);
    boolean isTextScan =true;
    if(!isTextScan)return;

    //3.调用阿里云接口审核图片内容
    List<String> imagesUrls =scanMaterialsList.get("imagesUrls");
    boolean isImageScan =true;
1
2
3
4
5
6
7
8
9
10
11
12

@Autowired
private WmSensitiveMapper wmSensitiveMapper;
private boolean handleSensitiveWords(List<String> contentTexts, WmNews wmNews) {
    boolean isSensitive = true;
    //1.获取所有敏感词
    List<WmSensitive> wmSensitiveList = wmSensitiveMapper.selectList(Wrappers.<WmSensitive>lambdaQuery().select(WmSensitive::getSensitives));
    List<String> collect = wmSensitiveList.stream().map(WmSensitive::getSensitives).collect(Collectors.toList());
    //2.初始化敏感词库
    SensitiveWordUtil.initMap(collect);
    //3.遍历文章内容查看是否包含敏感词
    for(String contentText:contentTexts){
        Map<String, Integer> map = SensitiveWordUtil.matchWords(contentText);
        if(map.size()>0){
            //4.如果包含敏感词，修改文章状态
            wmNews.setStatus((short) 2);
            wmNews.setReason("文章内容包含敏感词");
            wmNewsMapper.updateById(wmNews);
            isSensitive = false;
            break;
        }
    }
    return isSensitive;
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

6.3.4 测试

7 图片文字敏感词过滤

7.1 文字图片识别

7.2 Tesseract-OCR

7.3 Tess4j案例

7.3.1 导入依赖

在heima-leadnews-test模块下的tess4j-demo的模块下导入依赖

<dependency>
    <groupId>net.sourceforge.tess4j</groupId>
    <artifactId>tess4j</artifactId>
    <version>4.1.1</version>
</dependency>
1
2
3
4
5

7.3.2 将训练好的分类器放入资源中

7.3.3 demo

在tess4j-demo的Applcation中

public class Application {
    /**
     * 识别图片中的文字
     * @param args
     */
    public static void main(String[] args) {
        // 1.创建Tesseract对象
        Tesseract tesseract = new Tesseract();
        // 2.设置训练库的位置
        tesseract.setDatapath("D:\\Code\\JavaCode\\HeimaToutiao\\heima-leadnews\\heima-leadnews-test\\tess4j-demo\\src\\main\\resources\\tessdata");
        // 3.设置识别语言
        tesseract.setLanguage("chi_sim");
        // 4.设置识别图片
        File file = new File("D:\\Code\\JavaCode\\HeimaToutiao\\heima-leadnews\\heima-leadnews-test\\tess4j-demo\\src\\main\\resources\\testdata\\testImage.png");
        // 5.识别图片
        try {
            String result = tesseract.doOCR(file);
            System.out.println(result.replace("\\n|\\r", ""));
        } catch (TesseractException e) {
            e.printStackTrace();
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

7.3.4 结果

7.4 图片文字敏感词过滤实现

7.4.1 创建工具类

在heima-leadnews-common中创建com.heima.common.tess4j.Tess4jClient工具类，封装tess4j

@Getter
@Setter
@Component
@ConfigurationProperties(prefix = "tess4j")
public class Tess4jClient {

    private String dataPath;
    private String language;

    public String doOCR(BufferedImage image) throws TesseractException {
        //创建Tesseract对象
        ITesseract tesseract = new Tesseract();
        //设置字体库路径
        tesseract.setDatapath(dataPath);
        //中文识别
        tesseract.setLanguage(language);
        //执行ocr识别
        String result = tesseract.doOCR(image);
        //替换回车和tal键  使结果为一行
        result = result.replaceAll("\\r|\\n", "-").replaceAll(" ", "");
        return result;
    }

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

7.4.2 工具类被其他微服务使用

想让工具类被其他微服务使用就要拷贝全路径，在当前的resource中的META-INF的spring.factories中添加配置

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
  com.heima.common.exception.ExceptionCatch,\
  com.heima.common.aliyun.GreenTextScan,\
    com.heima.common.aliyun.GreenImageScan,\
    com.heima.common.tess4j.Tess4jClient
1
2
3
4
5

7.4.3 在微服务中配置

在heima-leadnews-wemedia中的resource的boostrap.yml中进行配置

tess4j:
  data-path: D:\Code\JavaCode\HeimaToutiao\heima-leadnews\heima-leadnews-test\tess4j-demo\src\main\resources\tessdata
  language: chi_sim
1
2
3

7.4.4 添加实现

在WmNewsAutoScanServiceImpl中的handleImageScan方法上添加如下代码

try {
    for (String image : images) {
        byte[] bytes = fileStorageService.downLoadFile(image);

        //图片识别文字审核---begin-----

        //从byte[]转换为butteredImage
        ByteArrayInputStream in = new ByteArrayInputStream(bytes);
        BufferedImage imageFile = ImageIO.read(in);
        //识别图片的文字
        String result = tess4jClient.doOCR(imageFile);

        //审核是否包含自管理的敏感词
        boolean isSensitive = handleSensitiveScan(result, wmNews);
        if(!isSensitive){
            return isSensitive;
        }
        //图片识别文字审核---end-----
        imageList.add(bytes);

    } 
}catch (Exception e){
    e.printStackTrace();
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

8 静态文件生成

8.1 实现思路

我们在保存/修改文章时就应该同时异步的的生成静态文件，生成静态文件上传到minio中

8.1.1 生成minio接口和实现，并且异步调用

在com.heima.article.service.ArticleFreemarkerService接口

生成静态文件，上传到minio中

public interface ArticleFreemarkerService {
    /**
     * 生成静态化页面
     * @param apArticle
     * @param content
     */
    public void buildArticleToMinio(ApArticle apArticle,String content);
}
1
2
3
4
5
6
7
8

@Service
@Slf4j
@Transactional
public class ArticleFreemarkerServiceImpl implements ArticleFreemarkerService {
    @Autowired
    private ApArticleContentMapper apArticleContentMapper;
    @Autowired
    private Configuration configuration;
    @Autowired
    private FileStorageService fileStorageService;
    @Autowired
    private ApArticleService apArticleService;
    /**
     * 生成静态化页面
     * @param apArticle
     * @param content
     */
    @Async
    @Override
    public void buildArticleToMinio(ApArticle apArticle, String content) {
        if(StringUtils.isNotBlank(content)){
            //1.文章内容通过freemarker生成静态html页面
            Template template = null;
            //2 输出流
            StringWriter writer = new StringWriter();
            try {
                template = configuration.getTemplate("article.ftl");
                //2.1 创建模型
                Map<String,Object> contentDataModel=new HashMap();
                //content是固定的，因为article.ftl中有<#if content??>${content}</#if>
                //因为apArticleContent.getContent()获取的是字符串，所以需要转换成对象
                contentDataModel.put("content", JSONArray.parseArray(content));
                //2.2 合成方法
                template.process(contentDataModel,writer);
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
            //3.把静态页面上传到minio
            //3.1 文件流
            InputStream inputStream = new ByteArrayInputStream(writer.toString().getBytes());
            String path = fileStorageService.uploadHtmlFile("",apArticle.getId()+".html",inputStream);
            //4.把静态页面的路径保存到数据库
            apArticleService.update(Wrappers
                    .<ApArticle>lambdaUpdate()
                    .eq(ApArticle::getId,apArticle.getId())
                    .set(ApArticle::getStaticUrl,path));
        }

    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

8.1.2 修改saveArticle逻辑

修改com.heima.article.service.impl.ApArticleServiceImpl的saveArticle方法，添加buildArticleToMinio

articleFreemarkerService.buildArticleToMinio(apArticle, dto.getContent());
1

    @Autowired
    private ApArticleConfigMapper apArticleConfigMapper;
    @Autowired
    private ApArticleContentMapper apArticleContentMapper;
    @Autowired
    private ArticleFreemarkerService articleFreemarkerService;
    /**
     * 保存文章
     * @param dto
     * @return
     */
    @Override
    public ResponseResult saveArticle(ArticleDto dto) {
        //1.参数检查
        if(dto == null){
            return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID);
        }
        ApArticle apArticle = new ApArticle();
        //org.springframework.beans
        BeanUtils.copyProperties(dto, apArticle);
        //2.判断是否存在id
        if(dto.getId() == null) {
            //2.1 不存在id ，新增 文章、内容、配置
            save(apArticle);
            //2.1.2 保存文章配置
            ApArticleConfig apArticleConfig = new ApArticleConfig(apArticle.getId());
            apArticleConfigMapper.insert(apArticleConfig);
            //2.1.3 保存文章内容
            ApArticleContent apArticleContent = new ApArticleContent();
            apArticleContent.setArticleId(apArticle.getId());
            apArticleContent.setContent(dto.getContent());
            apArticleContentMapper.insert(apArticleContent);
        }
        else {
            //2.2 存在id，更新 文章、内容
            //2.2.1 更新文章
            updateById(apArticle);
            //2.2.2 更新文章内容
            ApArticleContent apArticleContent = apArticleContentMapper.selectOne(Wrappers
                    .<ApArticleContent>lambdaQuery()
                    .eq(ApArticleContent::getArticleId, dto.getId()));
            apArticleContent.setContent(dto.getContent());
            apArticleContentMapper.updateById(apArticleContent);
        }
        //异步调用 生成静态文件上传到minio中
        articleFreemarkerService.buildArticleToMinio(apArticle, dto.getContent());

        //3.返回结果 文章的id
        return ResponseResult.okResult(apArticle.getId());
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

8.1.3 开启异步调用

引导类加上@EnableAsyn

@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.article.mapper")
@EnableAsync
public class ArticleApplication {

    public static void main(String[] args) {
        SpringApplication.run(ArticleApplication.class,args);
    }

    @Bean
    public MybatisPlusInterceptor mybatisPlusInterceptor() {
        MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
        interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
        return interceptor;
    }
    @Bean
    public MessageConverter messageConverter(){
        return new Jackson2JsonMessageConverter();
    }
    @Bean
    public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
        RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMessageConverter(messageConverter);
        return rabbitTemplate;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

8.1.4 测试

查看minio有没有生成

生成成功，查看数据库，有html生成，说明功能成功