当前位置:   article > 正文

UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies._unsupported for this filesystem class org.apache.h

unsupported for this filesystem class org.apache.hadoop.fs.ceph.cephfilesyst

flink程序在idea上运行正常,打成jar包上传服务器运行,各种报错。下面这个错误印象最为深刻

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme ‘hdfs’. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:472)
at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:62)
at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:248)
… 22 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Cannot support file system for ‘hdfs’ via Hadoop, because Hadoop is not in the classpath, or some classes are missing from the classpath.
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:179)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
… 27 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hdfs.HdfsConfiguration
at org.apache.flink.runtime.util.HadoopUtils.getHadoopConfiguration(HadoopUtils.java:49)
at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:85)
… 28 more

上网查了各种资料,比如说下面的两步很多教程都说了。
解决方法:
1.环境变量加入以下配置(别忘了source /etc/profile 后重启Flink)
HADOOP_HOME=xxx
export HADOOP_HOME
export HADOOP_CLASSPATH=`hadoop classpath`
2.下载一个flink集成hadoop的jar包放在Flink的lib目录下
下载地址:https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/

hadoop的版本是2.6.0,flink的版本是1.10.0
在这里插入图片描述

在这里插入图片描述
但是上面两步操作后,重启flink后还是报相同的错误。

最后发现重启linux系统就好了。。。。。。

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/知新_RL/article/detail/580380
推荐阅读
相关标签
  

闽ICP备14008679号