devbox
小蓝xlanll
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

hadoop：Win10本地环境配置hadoop（附wordcount实例） org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/_win10下使用maven连接hadoop2.6.0集群并运行wordcount例程

作者：小蓝xlanll | 2024-04-24 09:52:41

赞

踩

win10下使用maven连接hadoop2.6.0集群并运行wordcount例程

下载 hadoop包后先安装本地环境，配置环境变量：

解压hadoop-2.6.0.tar.gz

变量名 HADOOP_HOME   路径：E:\spark\anzhuangsoft\hadoop-2.6.0

path中：%HADOOP_HOME%\lib和 %HADOOP_HOME%\bin

 

 

验证：

 

 

IDEA中跑wordcount程序：

建一个 maven项目，目录结构：

代码：

RunJob 执行类：
  1. package main.java;
  2.  
  3.  
  4. import org.apache.hadoop.conf.Configuration;
  5. import org.apache.hadoop.fs.FileSystem;
  6. import org.apache.hadoop.fs.Path;
  7. import org.apache.hadoop.io.IntWritable;
  8. import org.apache.hadoop.io.Text;
  9. import org.apache.hadoop.mapreduce.Job;
  10. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  11. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  12.  
  13. public class RunJob {
  14. //    static {
  15. //        try {
  16. //            // 设置 HADOOP_HOME 目录
  17. //            System.setProperty("hadoop.home.dir", "E:\\spark\\anzhuangsoft\\hadoop-2.7.5");
  18. //            // 加载库文件
  19. //            System.load("E:\\spark\\anzhuangsoft\\hadoop-2.7.5\\bin\\hadoop.dll");
  20. //        } catch (UnsatisfiedLinkError e) {
  21. //            System.err.println("Native code library failed to load.\n" + e);
  22. //            System.exit(1);
  23. //        }
  24. //    }
  25.  
  26.     public static void main(String[] args) throws Exception {
  27.  
  28.         Configuration configuration = new Configuration();
  29.  
  30.         FileSystem fs = FileSystem.get(configuration);
  31.  
  32.         Job job = Job.getInstance(configuration);
  33.         job.setJarByClass(RunJob.class);
  34.         job.setJobName("wordCount");
  35.  
  36.         job.setMapperClass(WordCount.WordCountMapper.class);
  37.         job.setReducerClass(WordCount.WordCountReducer.class);
  38.  
  39.         job.setMapOutputKeyClass(Text.class);
  40.         job.setMapOutputValueClass(IntWritable.class);
  41.  
  42.         FileInputFormat.addInputPath(job, new Path("input"));
  43.         Path outPath = new Path("output");
  44.         if (fs.exists(outPath)) {
  45.             fs.delete(outPath, true);
  46.         }
  47.         FileOutputFormat.setOutputPath(job, outPath);
  48.  
  49.         boolean completion = job.waitForCompletion(true);
  50.         if (completion) {
  51.             System.out.println("执行完成");
  52.         }
  53.  
  54.  
  55.     }
  56. }

 

WordCount（含mapper和reducer）
  1. package main.java;
  2.  
  3. import java.io.IOException;
  4. import org.apache.hadoop.conf.Configuration;
  5. import org.apache.hadoop.fs.Path;
  6. import org.apache.hadoop.io.IntWritable;
  7. import org.apache.hadoop.io.LongWritable;
  8. import org.apache.hadoop.io.Text;
  9. import org.apache.hadoop.mapreduce.Job;
  10. import org.apache.hadoop.mapreduce.Mapper;
  11. import org.apache.hadoop.mapreduce.Reducer;
  12. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  13. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  14.  
  15. public class WordCount {
  16.     public static class WordCountMapper extends Mapper<LongWritable, Text, Text,IntWritable>{
  17.         @Override
  18.         protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context)
  19.                 throws IOException, InterruptedException {
  20.             String [] words = value.toString().split(" ");
  21.             for (String word : words) {
  22.                 context.write(new Text(word), new IntWritable(1));
  23.             }
  24.         }
  25.     }
  26.     public static class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
  27.         @Override
  28.         protected void reduce(Text key, Iterable<IntWritable> value,Context context) throws IOException, InterruptedException {
  29.             int sum = 0;
  30.             for (IntWritable intWritable : value) {
  31.                 sum += intWritable.get();
  32.             }
  33.             context.write(key, new IntWritable(sum));
  34.         }
  35.     }
  36.  
  37. //        static {
  38. //        try {
  39. //            // 设置 HADOOP_HOME 目录
  40. //            System.setProperty("hadoop.home.dir", "E:\\spark\\anzhuangsoft\\hadoop-2.7.5");
  41. //            // 加载库文件
  42. //            System.load("E:\\spark\\anzhuangsoft\\hadoop-2.7.5\\bin\\hadoop.dll");
  43. //        } catch (UnsatisfiedLinkError e) {
  44. //            System.err.println("Native code library failed to load.\n" + e);
  45. //            System.exit(1);
  46. //        }
  47. //    }
  48.  
  49.     public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
  50.         Configuration conf = new Configuration();
  51.         conf.set("mapreduce.framework.name", "local");
  52.         Job job = Job.getInstance(conf);
  53.  
  54.         job.setJarByClass(WordCount.class);
  55.  
  56.         //设置Mapper类和Reducer类
  57.         job.setMapperClass(WordCountMapper.class);
  58.         job.setReducerClass(WordCountReducer.class);
  59.         //设置Map端输出key类和输出value类
  60.         job.setMapOutputKeyClass(Text.class);
  61.         job.setMapOutputValueClass(IntWritable.class);
  62.         //设置Reduce端输出key类和输出value类
  63.         job.setOutputKeyClass(Text.class);
  64.         job.setOutputValueClass(IntWritable.class);
  65.  
  66.         FileInputFormat.addInputPath(job,new Path("input\\words.txt"));
  67.         FileOutputFormat.setOutputPath(job,new Path("output"));
  68.         //执行任务
  69.         boolean status = job.waitForCompletion(true);
  70.         System.out.println(status);
  71.     }
  72.  
  73. }

pom： 

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <project xmlns="http://maven.apache.org/POM/4.0.0"
  3.          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  4.          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  5.     <modelVersion>4.0.0</modelVersion>
  6.  
  7.     <groupId>com.tzb</groupId>
  8.     <artifactId>hadoop</artifactId>
  9.     <version>1.0-SNAPSHOT</version>
  10.  
  11.     <properties>
  12.         <!--<hadoop.version>2.7.5</hadoop.version>-->
  13.         <hadoop.version>2.6.0</hadoop.version>
  14.     </properties>
  15.     <dependencies>
  16.         <dependency>
  17.             <groupId>org.apache.hadoop</groupId>
  18.             <artifactId>hadoop-client</artifactId>
  19.             <version>${hadoop.version}</version>
  20.         </dependency>
  21.         <dependency>
  22.             <groupId>org.apache.hadoop</groupId>
  23.             <artifactId>hadoop-common</artifactId>
  24.             <version>${hadoop.version}</version>
  25.         </dependency>
  26.         <dependency>
  27.             <groupId>org.apache.hadoop</groupId>
  28.             <artifactId>hadoop-hdfs</artifactId>
  29.             <version>${hadoop.version}</version>
  30.         </dependency>
  31.         <!--<dependency>-->
  32.             <!--<groupId>org.apache.hadoop</groupId>-->
  33.             <!--<artifactId>hadoop-client</artifactId>-->
  34.             <!--<version>${hadoop.version}</version>-->
  35.         <!--</dependency>-->
  36.         <dependency>
  37.             <groupId>org.apache.hadoop</groupId>
  38.             <artifactId>hadoop-mapreduce-client-core</artifactId>
  39.             <version>${hadoop.version}</version>
  40.         </dependency>
  41.         <!--<dependency>-->
  42.         <!--<groupId>jdk.tools</groupId>-->
  43.         <!--<artifactId>jdk.tools</artifactId>-->
  44.         <!--<version>1.8</version>-->
  45.         <!--<scope>system</scope>-->
  46.         <!--<systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>-->
  47.         <!--</dependency>-->
  48.     </dependencies>
  49.     <build>
  50.         <plugins>
  51.             <plugin>
  52.                 <groupId>org.apache.maven.plugins</groupId>
  53.                 <artifactId>maven-jar-plugin</artifactId>
  54.                 <version>2.4</version>
  55.                 <configuration>
  56.                     <archive>
  57.                         <manifest>
  58.                             <addClasspath>true</addClasspath>
  59.                             <classpathPrefix>lib/</classpathPrefix>
  60.                             <mainClass>cn.itcast.hadoop.mr.WordCountDriver</mainClass>
  61.                         </manifest>
  62.                     </archive>
  63.                 </configuration>
  64.             </plugin>
  65.             <plugin>
  66.                 <groupId>org.apache.maven.plugins</groupId>
  67.                 <artifactId>maven-compiler-plugin</artifactId>
  68.                 <!--<version>3.0</version>-->
  69.                 <configuration>
  70.                     <source>1.8</source>
  71.                     <target>1.8</target>
  72.                     <encoding>UTF-8</encoding>
  73.                 </configuration>
  74.             </plugin>
  75.         </plugins>
  76.     </build>
  77. </project>

执行效果：

将Reducer的代码注释掉，只执行mapper的代码（可以看出未作聚合的效果）：

 

 

 

执行代码

报错1：

  1. F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\bin\java "-javaagent:H:\Program Files\JetBrains\IntelliJ IDEA 2017.2.5\lib\idea_rt.jar=52764:H:\Program Files\JetBrains\IntelliJ IDEA 2017.2.5\bin" -Dfile.encoding=GBK -classpath F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\charsets.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\deploy.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\access-bridge-32.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\cldrdata.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\dnsns.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\jaccess.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\jfxrt.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\localedata.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\nashorn.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\sunec.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\sunjce_provider.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\sunmscapi.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\sunpkcs11.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\zipfs.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\javaws.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\jce.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\jfr.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\jfxswt.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\jsse.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\management-agent.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\plugin.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\resources.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\rt.jar;D:\Workspaces\hadoop\target\classes;D:\repository\org\apache\hadoop\hadoop-client\2.6.0\hadoop-client-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-app\2.6.0\hadoop-mapreduce-client-app-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-common\2.6.0\hadoop-mapreduce-client-common-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-yarn-client\2.6.0\hadoop-yarn-client-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-yarn-server-common\2.6.0\hadoop-yarn-server-common-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-shuffle\2.6.0\hadoop-mapreduce-client-shuffle-2.6.0.jar;D:\repository\org\fusesource\leveldbjni\leveldbjni-all\1.8\leveldbjni-all-1.8.jar;D:\repository\org\apache\hadoop\hadoop-yarn-api\2.6.0\hadoop-yarn-api-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-jobclient\2.6.0\hadoop-mapreduce-client-jobclient-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-annotations\2.6.0\hadoop-annotations-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-common\2.6.0\hadoop-common-2.6.0.jar;D:\repository\com\google\guava\guava\11.0.2\guava-11.0.2.jar;D:\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;D:\repository\org\apache\commons\commons-math3\3.1.1\commons-math3-3.1.1.jar;D:\repository\xmlenc\xmlenc\0.52\xmlenc-0.52.jar;D:\repository\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;D:\repository\commons-codec\commons-codec\1.4\commons-codec-1.4.jar;D:\repository\commons-io\commons-io\2.4\commons-io-2.4.jar;D:\repository\commons-net\commons-net\3.1\commons-net-3.1.jar;D:\repository\commons-collections\commons-collections\3.2.1\commons-collections-3.2.1.jar;D:\repository\javax\servlet\servlet-api\2.5\servlet-api-2.5.jar;D:\repository\org\mortbay\jetty\jetty\6.1.26\jetty-6.1.26.jar;D:\repository\org\mortbay\jetty\jetty-util\6.1.26\jetty-util-6.1.26.jar;D:\repository\com\sun\jersey\jersey-core\1.9\jersey-core-1.9.jar;D:\repository\com\sun\jersey\jersey-json\1.9\jersey-json-1.9.jar;D:\repository\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar;D:\repository\com\sun\xml\bind\jaxb-impl\2.2.3-1\jaxb-impl-2.2.3-1.jar;D:\repository\org\codehaus\jackson\jackson-jaxrs\1.8.3\jackson-jaxrs-1.8.3.jar;D:\repository\org\codehaus\jackson\jackson-xc\1.8.3\jackson-xc-1.8.3.jar;D:\repository\com\sun\jersey\jersey-server\1.9\jersey-server-1.9.jar;D:\repository\asm\asm\3.1\asm-3.1.jar;D:\repository\tomcat\jasper-compiler\5.5.23\jasper-compiler-5.5.23.jar;D:\repository\tomcat\jasper-runtime\5.5.23\jasper-runtime-5.5.23.jar;D:\repository\javax\servlet\jsp\jsp-api\2.1\jsp-api-2.1.jar;D:\repository\commons-el\commons-el\1.0\commons-el-1.0.jar;D:\repository\commons-logging\commons-logging\1.1.3\commons-logging-1.1.3.jar;D:\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar;D:\repository\net\java\dev\jets3t\jets3t\0.9.0\jets3t-0.9.0.jar;D:\repository\org\apache\httpcomponents\httpclient\4.1.2\httpclient-4.1.2.jar;D:\repository\org\apache\httpcomponents\httpcore\4.1.2\httpcore-4.1.2.jar;D:\repository\com\jamesmurty\utils\java-xmlbuilder\0.4\java-xmlbuilder-0.4.jar;D:\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;D:\repository\commons-configuration\commons-configuration\1.6\commons-configuration-1.6.jar;D:\repository\commons-digester\commons-digester\1.8\commons-digester-1.8.jar;D:\repository\commons-beanutils\commons-beanutils\1.7.0\commons-beanutils-1.7.0.jar;D:\repository\commons-beanutils\commons-beanutils-core\1.8.0\commons-beanutils-core-1.8.0.jar;D:\repository\org\slf4j\slf4j-api\1.7.5\slf4j-api-1.7.5.jar;D:\repository\org\slf4j\slf4j-log4j12\1.7.5\slf4j-log4j12-1.7.5.jar;D:\repository\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar;D:\repository\org\codehaus\jackson\jackson-mapper-asl\1.9.13\jackson-mapper-asl-1.9.13.jar;D:\repository\org\apache\avro\avro\1.7.4\avro-1.7.4.jar;D:\repository\com\thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;D:\repository\org\xerial\snappy\snappy-java\1.0.4.1\snappy-java-1.0.4.1.jar;D:\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;D:\repository\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;D:\repository\org\apache\hadoop\hadoop-auth\2.6.0\hadoop-auth-2.6.0.jar;D:\repository\org\apache\directory\server\apacheds-kerberos-codec\2.0.0-M15\apacheds-kerberos-codec-2.0.0-M15.jar;D:\repository\org\apache\directory\server\apacheds-i18n\2.0.0-M15\apacheds-i18n-2.0.0-M15.jar;D:\repository\org\apache\directory\api\api-asn1-api\1.0.0-M20\api-asn1-api-1.0.0-M20.jar;D:\repository\org\apache\directory\api\api-util\1.0.0-M20\api-util-1.0.0-M20.jar;D:\repository\org\apache\curator\curator-framework\2.6.0\curator-framework-2.6.0.jar;D:\repository\com\jcraft\jsch\0.1.42\jsch-0.1.42.jar;D:\repository\org\apache\curator\curator-client\2.6.0\curator-client-2.6.0.jar;D:\repository\org\apache\curator\curator-recipes\2.6.0\curator-recipes-2.6.0.jar;D:\repository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;D:\repository\org\htrace\htrace-core\3.0.4\htrace-core-3.0.4.jar;D:\repository\org\apache\zookeeper\zookeeper\3.4.6\zookeeper-3.4.6.jar;D:\repository\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar;D:\repository\org\tukaani\xz\1.0\xz-1.0.jar;D:\repository\org\apache\hadoop\hadoop-hdfs\2.6.0\hadoop-hdfs-2.6.0.jar;D:\repository\commons-daemon\commons-daemon\1.0.13\commons-daemon-1.0.13.jar;D:\repository\io\netty\netty\3.6.2.Final\netty-3.6.2.Final.jar;D:\repository\xerces\xercesImpl\2.9.1\xercesImpl-2.9.1.jar;D:\repository\xml-apis\xml-apis\1.3.04\xml-apis-1.3.04.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-core\2.6.0\hadoop-mapreduce-client-core-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-yarn-common\2.6.0\hadoop-yarn-common-2.6.0.jar;D:\repository\javax\xml\bind\jaxb-api\2.2.2\jaxb-api-2.2.2.jar;D:\repository\javax\xml\stream\stax-api\1.0-2\stax-api-1.0-2.jar;D:\repository\javax\activation\activation\1.1\activation-1.1.jar;D:\repository\com\sun\jersey\jersey-client\1.9\jersey-client-1.9.jar;D:\repository\com\google\inject\guice\3.0\guice-3.0.jar;D:\repository\javax\inject\javax.inject\1\javax.inject-1.jar;D:\repository\aopalliance\aopalliance\1.0\aopalliance-1.0.jar;D:\repository\com\sun\jersey\contribs\jersey-guice\1.9\jersey-guice-1.9.jar;D:\repository\com\google\inject\extensions\guice-servlet\3.0\guice-servlet-3.0.jar main.java.RunJob
  2. log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
  3. log4j:WARN Please initialize the log4j system properly.
  4. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  5. Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  6. 	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
  7. 	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:557)
  8. 	at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)
  9. 	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:187)
  10. 	at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:174)
  11. 	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:108)
  12. 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:285)
  13. 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
  14. 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
  15. 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
  16. 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
  17. 	at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:131)
  18. 	at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
  19. 	at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
  20. 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:536)
  21. 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
  22. 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
  23. 	at java.security.AccessController.doPrivileged(Native Method)
  24. 	at javax.security.auth.Subject.doAs(Subject.java:422)
  25. 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
  26. 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
  27. 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
  28. 	at main.java.RunJob.main(RunJob.java:49)
  29.  
  30. Process finished with exit code 1

以上报错原因，我经过各种尝试，最终锁定目录：C:\Windows\SysWOW64

只有当这个目录中有 hadoop.dll文件时，wordcount程序才可以执行成功。

但是最好把hadoop.dll和winutils.exe两个文件都放到下面两个目录中去

C:\Windows\System32 和 C:\Windows\SysWOW64

如果还是报这个错试试将如下代码放到runjob中去，main方法的外面 ，然后再执行试试，里边目录指向你的hadoop目录。

  1. static {
  2.         try {
  3.             // 设置 HADOOP_HOME 目录
  4.             System.setProperty("hadoop.home.dir", "E:\\spark\\anzhuangsoft\\hadoop-2.7.5");
  5.             // 加载库文件
  6.             System.load("E:\\spark\\anzhuangsoft\\hadoop-2.7.5\\bin\\hadoop.dll");
  7.         } catch (UnsatisfiedLinkError e) {
  8.             System.err.println("Native code library failed to load.\n" + e);
  9.             System.exit(1);
  10.         }
  11.     }

报错2：

  1. F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\bin\java "-javaagent:H:\Program Files\JetBrains\IntelliJ IDEA 2017.2.5\lib\idea_rt.jar=53113:H:\Program Files\JetBrains\IntelliJ IDEA 2017.2.5\bin" -Dfile.encoding=GBK -classpath F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\charsets.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\deploy.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\access-bridge-32.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\cldrdata.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\dnsns.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\jaccess.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\jfxrt.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\localedata.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\nashorn.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\sunec.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\sunjce_provider.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\sunmscapi.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\sunpkcs11.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\ext\zipfs.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\javaws.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\jce.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\jfr.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\jfxswt.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\jsse.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\management-agent.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\plugin.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\resources.jar;F:\MyDevelopKit\1.software\jdk\1.8\jdk1.8.0_131\jre\lib\rt.jar;D:\Workspaces\hadoop\target\classes;D:\repository\org\apache\hadoop\hadoop-client\2.6.0\hadoop-client-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-app\2.6.0\hadoop-mapreduce-client-app-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-common\2.6.0\hadoop-mapreduce-client-common-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-yarn-client\2.6.0\hadoop-yarn-client-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-yarn-server-common\2.6.0\hadoop-yarn-server-common-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-shuffle\2.6.0\hadoop-mapreduce-client-shuffle-2.6.0.jar;D:\repository\org\fusesource\leveldbjni\leveldbjni-all\1.8\leveldbjni-all-1.8.jar;D:\repository\org\apache\hadoop\hadoop-yarn-api\2.6.0\hadoop-yarn-api-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-jobclient\2.6.0\hadoop-mapreduce-client-jobclient-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-annotations\2.6.0\hadoop-annotations-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-common\2.6.0\hadoop-common-2.6.0.jar;D:\repository\com\google\guava\guava\11.0.2\guava-11.0.2.jar;D:\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;D:\repository\org\apache\commons\commons-math3\3.1.1\commons-math3-3.1.1.jar;D:\repository\xmlenc\xmlenc\0.52\xmlenc-0.52.jar;D:\repository\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;D:\repository\commons-codec\commons-codec\1.4\commons-codec-1.4.jar;D:\repository\commons-io\commons-io\2.4\commons-io-2.4.jar;D:\repository\commons-net\commons-net\3.1\commons-net-3.1.jar;D:\repository\commons-collections\commons-collections\3.2.1\commons-collections-3.2.1.jar;D:\repository\javax\servlet\servlet-api\2.5\servlet-api-2.5.jar;D:\repository\org\mortbay\jetty\jetty\6.1.26\jetty-6.1.26.jar;D:\repository\org\mortbay\jetty\jetty-util\6.1.26\jetty-util-6.1.26.jar;D:\repository\com\sun\jersey\jersey-core\1.9\jersey-core-1.9.jar;D:\repository\com\sun\jersey\jersey-json\1.9\jersey-json-1.9.jar;D:\repository\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar;D:\repository\com\sun\xml\bind\jaxb-impl\2.2.3-1\jaxb-impl-2.2.3-1.jar;D:\repository\org\codehaus\jackson\jackson-jaxrs\1.8.3\jackson-jaxrs-1.8.3.jar;D:\repository\org\codehaus\jackson\jackson-xc\1.8.3\jackson-xc-1.8.3.jar;D:\repository\com\sun\jersey\jersey-server\1.9\jersey-server-1.9.jar;D:\repository\asm\asm\3.1\asm-3.1.jar;D:\repository\tomcat\jasper-compiler\5.5.23\jasper-compiler-5.5.23.jar;D:\repository\tomcat\jasper-runtime\5.5.23\jasper-runtime-5.5.23.jar;D:\repository\javax\servlet\jsp\jsp-api\2.1\jsp-api-2.1.jar;D:\repository\commons-el\commons-el\1.0\commons-el-1.0.jar;D:\repository\commons-logging\commons-logging\1.1.3\commons-logging-1.1.3.jar;D:\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar;D:\repository\net\java\dev\jets3t\jets3t\0.9.0\jets3t-0.9.0.jar;D:\repository\org\apache\httpcomponents\httpclient\4.1.2\httpclient-4.1.2.jar;D:\repository\org\apache\httpcomponents\httpcore\4.1.2\httpcore-4.1.2.jar;D:\repository\com\jamesmurty\utils\java-xmlbuilder\0.4\java-xmlbuilder-0.4.jar;D:\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;D:\repository\commons-configuration\commons-configuration\1.6\commons-configuration-1.6.jar;D:\repository\commons-digester\commons-digester\1.8\commons-digester-1.8.jar;D:\repository\commons-beanutils\commons-beanutils\1.7.0\commons-beanutils-1.7.0.jar;D:\repository\commons-beanutils\commons-beanutils-core\1.8.0\commons-beanutils-core-1.8.0.jar;D:\repository\org\slf4j\slf4j-api\1.7.5\slf4j-api-1.7.5.jar;D:\repository\org\slf4j\slf4j-log4j12\1.7.5\slf4j-log4j12-1.7.5.jar;D:\repository\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar;D:\repository\org\codehaus\jackson\jackson-mapper-asl\1.9.13\jackson-mapper-asl-1.9.13.jar;D:\repository\org\apache\avro\avro\1.7.4\avro-1.7.4.jar;D:\repository\com\thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;D:\repository\org\xerial\snappy\snappy-java\1.0.4.1\snappy-java-1.0.4.1.jar;D:\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;D:\repository\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;D:\repository\org\apache\hadoop\hadoop-auth\2.6.0\hadoop-auth-2.6.0.jar;D:\repository\org\apache\directory\server\apacheds-kerberos-codec\2.0.0-M15\apacheds-kerberos-codec-2.0.0-M15.jar;D:\repository\org\apache\directory\server\apacheds-i18n\2.0.0-M15\apacheds-i18n-2.0.0-M15.jar;D:\repository\org\apache\directory\api\api-asn1-api\1.0.0-M20\api-asn1-api-1.0.0-M20.jar;D:\repository\org\apache\directory\api\api-util\1.0.0-M20\api-util-1.0.0-M20.jar;D:\repository\org\apache\curator\curator-framework\2.6.0\curator-framework-2.6.0.jar;D:\repository\com\jcraft\jsch\0.1.42\jsch-0.1.42.jar;D:\repository\org\apache\curator\curator-client\2.6.0\curator-client-2.6.0.jar;D:\repository\org\apache\curator\curator-recipes\2.6.0\curator-recipes-2.6.0.jar;D:\repository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;D:\repository\org\htrace\htrace-core\3.0.4\htrace-core-3.0.4.jar;D:\repository\org\apache\zookeeper\zookeeper\3.4.6\zookeeper-3.4.6.jar;D:\repository\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar;D:\repository\org\tukaani\xz\1.0\xz-1.0.jar;D:\repository\org\apache\hadoop\hadoop-hdfs\2.6.0\hadoop-hdfs-2.6.0.jar;D:\repository\commons-daemon\commons-daemon\1.0.13\commons-daemon-1.0.13.jar;D:\repository\io\netty\netty\3.6.2.Final\netty-3.6.2.Final.jar;D:\repository\xerces\xercesImpl\2.9.1\xercesImpl-2.9.1.jar;D:\repository\xml-apis\xml-apis\1.3.04\xml-apis-1.3.04.jar;D:\repository\org\apache\hadoop\hadoop-mapreduce-client-core\2.6.0\hadoop-mapreduce-client-core-2.6.0.jar;D:\repository\org\apache\hadoop\hadoop-yarn-common\2.6.0\hadoop-yarn-common-2.6.0.jar;D:\repository\javax\xml\bind\jaxb-api\2.2.2\jaxb-api-2.2.2.jar;D:\repository\javax\xml\stream\stax-api\1.0-2\stax-api-1.0-2.jar;D:\repository\javax\activation\activation\1.1\activation-1.1.jar;D:\repository\com\sun\jersey\jersey-client\1.9\jersey-client-1.9.jar;D:\repository\com\google\inject\guice\3.0\guice-3.0.jar;D:\repository\javax\inject\javax.inject\1\javax.inject-1.jar;D:\repository\aopalliance\aopalliance\1.0\aopalliance-1.0.jar;D:\repository\com\sun\jersey\contribs\jersey-guice\1.9\jersey-guice-1.9.jar;D:\repository\com\google\inject\extensions\guice-servlet\3.0\guice-servlet-3.0.jar main.java.RunJob
  2. log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
  3. log4j:WARN Please initialize the log4j system properly.
  4. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  5. Exception in thread "main" java.lang.NullPointerException
  6. 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
  7. 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
  8. 	at org.apache.hadoop.util.Shell.run(Shell.java:455)
  9. 	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
  10. 	at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)
  11. 	at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
  12. 	at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
  13. 	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:582)
  14. 	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:557)
  15. 	at org.apache.hadoop.fs.LocatedFileStatus.<init>(LocatedFileStatus.java:42)
  16. 	at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1699)
  17. 	at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1681)
  18. 	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:303)
  19. 	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:264)
  20. 	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:385)
  21. 	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:597)
  22. 	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614)
  23. 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
  24. 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
  25. 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
  26. 	at java.security.AccessController.doPrivileged(Native Method)
  27. 	at javax.security.auth.Subject.doAs(Subject.java:422)
  28. 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
  29. 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
  30. 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
  31. 	at main.java.RunJob.main(RunJob.java:49)
  32.  
  33. Process finished with exit code 1

这个报错原因是由于hadoop的bin目录下没有winutils.exe文件。复制进去。

 

转载请注明出处，感谢！

参考：

https://www.cnblogs.com/kevin-lee123/p/11020570.html

https://blog.csdn.net/a2099948768/article/details/79577246

https://www.cnblogs.com/jhxxb/p/10723369.html

winutil.exe和hadoop.dll下载：

https://github.com/steveloughran/winutils

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/小蓝xlanll/article/detail/478794
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号