hadoop UnsupportedFileSystemException 问题_org.apache.hadoop.fs.unsupportedfilesystemexceptio

使用hdfs操作文件的时候遇到以下两个异常：

1.java.io.IOException: No FileSystem for scheme: hdfs

2.org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme “file”

类似No FileSystem for scheme: hdfs （file）原因是jar包或打包问题。

依赖的jar包是hadoop-commons 和 hadoop-hdfs

Different JARs (hadoop-commonsforLocalFileSystem,hadoop-hdfsforDistributedFileSystem) each contain a different file calledorg.apache.hadoop.fs.FileSystemin theirMETA-INFO/servicesdirectory. This file lists the canonical classnames of the filesystem implementations they want to declare (This is called a Service Provider Interface implemented viajava.util.ServiceLoader, seeorg.apache.hadoop.FileSystemline 2622).

When we usemaven-assembly-plugin, it merges all our JARs into one, and allMETA-INFO/services/org.apache.hadoop.fs.FileSystemoverwrite each-other. Only one of these files remains (the last one that was added). In this case, theFileSystemlist fromhadoop-commonsoverwrites the list fromhadoop-hdfs, soDistributedFileSystemwas no longer declared.

解决方法如下：

1.通过配置方式：

		// 这个解决hdfs问题
		hadoopConf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
		// 这个解决本地file问题
		hadoopConf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
1
2
3
4

2.通过打包插件方式

org.apache.maven.plugins maven-shade-plugin 2.3 package shade