Flink链接kafka并解析Json文件(一)_flink处理kafka中的json数据

背景:某小型互联网公司每日的疫情填报

Flink消费Kafka中的Json数据

遇到的问题:Flink的jar包上传Flink服务器报错

web页面报错:
2021-01-19 10:26:42,437 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka version : 0.11.0.2
2021-01-19 10:26:42,438 INFO  org.apache.kafka.common.utils.AppInfoParser                   - Kafka commitId : 73be1e1168f91ee2
2021-01-19 10:26:42,495 INFO  org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase  - Consumer subtask 0 will start reading the following 3 partitions from the committed group offsets in Kafka: [KafkaTopicPartition{topic='epidemic_db_1', partition=1}, KafkaTopicPartition{topic='epidemic_db_1', partition=2}, KafkaTopicPartition{topic='epidemic_db_1', partition=0}]
2021-01-19 10:26:42,496 INFO  org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase  - Consumer subtask 0 creating fetcher with offsets {KafkaTopicPartition{topic='epidemic_db_1', partition=1}=-915623761773, KafkaTopicPartition{topic='epidemic_db_1', partition=2}=-915623761773, KafkaTopicPartition{topic='epidemic_db_1', partition=0}=-915623761773}.
2021-01-19 10:26:42,502 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source (1/1) (1ea73af0e0f412a51db96404f436fe5b) switched from RUNNING to FAILED.
java.lang.NoSuchMethodError: org.apache.flink.streaming.connectors.kafka.internal.KafkaConsumerThread.<init>(Lorg/slf4j/Logger;Lorg/apache/flink/streaming/connectors/kafka/internal/Handover;Ljava/util/Properties;Lorg/apache/flink/streaming/connectors/kafka/internals/ClosableBlockingQueue;Lorg/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge09;Ljava/lang/String;JZLorg/apache/flink/metrics/MetricGroup;Lorg/apache/flink/metrics/MetricGroup;Lorg/apache/flink/api/common/io/ratelimiting/FlinkConnectorRateLimiter;)V
at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.<init>(Kafka09Fetcher.java:113)
at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.<init>(Kafka010Fetcher.java:66)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010.createFetcher(FlinkKafkaConsumer010.java:212)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:698)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:200)
2021-01-19 10:26:42,503 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Source: Custom Source (1/1) (1ea73af0e0f412a51db96404f436fe5b).
2021-01-19 10:26:42,503 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Source: Custom Source (1/1) (1ea73af0e0f412a51db96404f436fe5b) [FAILED]
2021-01-19 10:26:42,506 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state FAILED to JobManager for task Source: Custom Source (1/1) 1ea73af0e0f412a51db96404f436fe5b.
2021-01-19 10:26:42,515 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Flat Map (1/3) (f688c3d09c871ebf8605e275370df94f).
2021-01-19 10:26:42,515 INFO  org.apache.flink.runtime.taskmanager.Task                     - Flat Map (1/3) (f688c3d09c871ebf8605e275370df94f) switched from RUNNING to CANCELING.
2021-01-19 10:26:42,515 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Flat Map (1/3) (f688c3d09c871ebf8605e275370df94f).
2021-01-19 10:26:42,516 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Flat Map (2/3) (a494ad6ec7afd30eb113dec544f9e721).
2021-01-19 10:26:42,516 INFO  org.apache.flink.runtime.taskmanager.Task                     - Flat Map (2/3) (a494ad6ec7afd30eb113dec544f9e721) switched from RUNNING to CANCELING.
2021-01-19 10:26:42,516 INFO  org.apache.flink.runtime.taskmanager.Task                     - Flat Map (1/3) (f688c3d09c871ebf8605e275370df94f) switched from CANCELING to CANCELED.
2021-01-19 10:26:42,516 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Flat Map (2/3) (a494ad6ec7afd30eb113dec544f9e721).
2021-01-19 10:26:42,516 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Flat Map (1/3) (f688c3d09c871ebf8605e275370df94f).
2021-01-19 10:26:42,516 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Flat Map (1/3) (f688c3d09c871ebf8605e275370df94f) [CANCELED]
2021-01-19 10:26:42,516 INFO  org.apache.flink.runtime.taskmanager.Task                     - Flat Map (2/3) (a494ad6ec7afd30eb113dec544f9e721) switched from CANCELING to CANCELED.
2021-01-19 10:26:42,516 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Flat Map (3/3) (c1f052d75b237210db701fc5c427eb2f).
2021-01-19 10:26:42,516 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Flat Map (2/3) (a494ad6ec7afd30eb113dec544f9e721).
2021-01-19 10:26:42,517 INFO  org.apache.flink.runtime.taskmanager.Task                     - Flat Map (3/3) (c1f052d75b237210db701fc5c427eb2f) switched from RUNNING to CANCELING.
2021-01-19 10:26:42,517 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Flat Map (2/3) (a494ad6ec7afd30eb113dec544f9e721) [CANCELED]
2021-01-19 10:26:42,517 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Flat Map (3/3) (c1f052d75b237210db701fc5c427eb2f)
2021-01-19 10:26:42,518 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Sink: Print to Std. Out (1/2) (ee813d356c5820637da7fe6674732292).
2021-01-19 10:26:42,518 INFO  org.apache.flink.runtime.taskmanager.Task                     - Flat Map (3/3) (c1f052d75b237210db701fc5c427eb2f) switched from CANCELING to CANCELED.
2021-01-19 10:26:42,518 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Flat Map (3/3) (c1f052d75b237210db701fc5c427eb2f).
2021-01-19 10:26:42,518 INFO  org.apache.flink.runtime.taskmanager.Task                     - Sink: Print to Std. Out (1/2) (ee813d356c5820637da7fe6674732292) switched from RUNNING to CANCELING.
2021-01-19 10:26:42,518 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Sink: Print to Std. Out (1/2) (ee813d356c5820637da7fe6674732292).
2021-01-19 10:26:42,518 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Flat Map (3/3) (c1f052d75b237210db701fc5c427eb2f) [CANCELED]
2021-01-19 10:26:42,519 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to cancel task Sink: Print to Std. Out (2/2) (5173f956fdee153342b02a2a6c48daeb).
2021-01-19 10:26:42,519 INFO  org.apache.flink.runtime.taskmanager.Task                     - Sink: Print to Std. Out (2/2) (5173f956fdee153342b02a2a6c48daeb) switched from RUNNING to CANCELING.
2021-01-19 10:26:42,519 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Sink: Print to Std. Out (2/2) (5173f956fdee153342b02a2a6c48daeb).
2021-01-19 10:26:42,519 INFO  org.apache.flink.runtime.taskmanager.Task                     - Sink: Print to Std. Out (1/2) (ee813d356c5820637da7fe6674732292) switched from CANCELING to CANCELED.
2021-01-19 10:26:42,519 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Sink: Print to Std. Out (1/2) (ee813d356c5820637da7fe6674732292).
2021-01-19 10:26:42,519 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Sink: Print to Std. Out (1/2) (ee813d356c5820637da7fe6674732292) [CANCELED]
2021-01-19 10:26:42,520 INFO  org.apache.flink.runtime.taskmanager.Task                     - Sink: Print to Std. Out (2/2) (5173f956fdee153342b02a2a6c48daeb) switched from CANCELING to CANCELED.
2021-01-19 10:26:42,520 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Sink: Print to Std. Out (2/2) (5173f956fdee153342b02a2a6c48daeb).
2021-01-19 10:26:42,520 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Flat Map (1/3) f688c3d09c871ebf8605e275370df94f.
2021-01-19 10:26:42,521 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Sink: Print to Std. Out (2/2) (5173f956fdee153342b02a2a6c48daeb) [CANCELED]
2021-01-19 10:26:42,521 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Flat Map (2/3) a494ad6ec7afd30eb113dec544f9e721.
2021-01-19 10:26:42,522 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Flat Map (3/3) c1f052d75b237210db701fc5c427eb2f.
2021-01-19 10:26:42,522 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Sink: Print to Std. Out (1/2) ee813d356c5820637da7fe6674732292.
2021-01-19 10:26:42,522 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Un-registering task and sending final execution state CANCELED to JobManager for task Sink: Print to Std. Out (2/2) 5173f956fdee153342b02a2a6c48daeb.
2021-01-19 10:26:42,528 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl  - Free slot TaskSlot(index:2, state:ACTIVE, resource profile: ResourceProfile{cpuCores=1.0000000000000000, taskHeapMemory=48.000mb (50331646 bytes), taskOffHeapMemory=0 bytes, managedMemory=64.000mb (67108865 bytes), networkMemory=16.000mb (16777216 bytes)}, allocationId: 0016ed5e06671d091b60afa45c19c3cb, jobId: 29feb7f1fab3b620a9da4f4019f0815f).
2021-01-19 10:26:42,529 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl  - Free slot TaskSlot(index:4, state:ACTIVE, resource profile: ResourceProfile{cpuCores=1.0000000000000000, taskHeapMemory=48.000mb (50331646 bytes), taskOffHeapMemory=0 bytes, managedMemory=64.000mb (67108865 bytes), networkMemory=16.000mb (16777216 bytes)}, allocationId: 7c1d226151ed5def3972885457fdd08e, jobId: 29feb7f1fab3b620a9da4f4019f0815f).
2021-01-19 10:26:42,531 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl  - Free slot TaskSlot(index:1, state:ACTIVE, resource profile: ResourceProfile{cpuCores=1.0000000000000000, taskHeapMemory=48.000mb (50331646 bytes), taskOffHeapMemory=0 bytes, managedMemory=64.000mb (67108865 bytes), networkMemory=16.000mb (16777216 bytes)}, allocationId: b963fc2721bc119b4893a95657a78b4a, jobId: 29feb7f1fab3b620a9da4f4019f0815f).
2021-01-19 10:26:42,531 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Remove job 29feb7f1fab3b620a9da4f4019f0815f from job leader monitoring.
2021-01-19 10:26:42,531 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Close JobManager connection for job 29feb7f1fab3b620a9da4f4019f0815f.
2021-01-19 10:26:42,532 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Close JobManager connection for job 29feb7f1fab3b620a9da4f4019f0815f.
2021-01-19 10:26:42,532 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Cannot reconnect to job 29feb7f1fab3b620a9da4f4019f0815f because it is not registered.

服务器上执行命令报错:
Caused by: java.io.InvalidClassException: scala.Tuple4; local class incompatible: stream classdesc serialVersionUID = -3999741010374250998, local class serialVersionUID = -3284039410903489814

解决方案:重新建立一个项目，将原版本的scala版本(2.12.11)全部换成scala-2.11.12