devbox
很楠不爱3
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Hadoop + Spark 大数据巨量分析与机器学习整合开发实战-学习笔记_hadoop+spark大数据巨量分析

作者：很楠不爱3 | 2024-06-07 15:21:56

赞

踩

hadoop+spark大数据巨量分析

 

  1. spark_Document:			http://spark.apache.org/docs/latest/index.html
  2. spark_API:			http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package
  3. 测试数据集movielens		https://grouplens.org/datasets/movielens/
  4. hadoop集群搭建			http://hadoop.apache.org/docs/r1.0.4/cn/cluster_setup.html
  5. JARs				http://www.java2s.com
  6. ubuntu更新源			https://www.linuxidc.com/Linux/2017-11/148627.html
  7. unknown host:	--->	sudo gedit /etc/resolv.conf	--->	nameserver 8.8.8.8
  8. --------------------------------------------------D1 linux_enviromrnt-----------------------------------------
  9. Linux查看进程		ps -ef / ps -aux -> ps -ef|grep java ->	kill -9 1827
  10. 查看第3-10行		cat filename | tail -n +3 | head -n +10
  11.  
  12. #其他操作
  13. 	sudo apt-get update  				更新源
  14. 	sudo apt-get install package 			安装包
  15. 	sudo apt-get remove package 			删除包
  16. 	sudo apt-cache search package 			搜索软件包
  17. 	sudo apt-cache show package  			获取包的相关信息，如说明、大小、版本等
  18. 	sudo apt-get install package --reinstall  	重新安装包
  19. 	sudo apt-get -f install  			修复安装
  20. 	sudo apt-get remove package --purge 		删除包，包括配置文件等
  21. 	sudo apt-get build-dep package 			安装相关的编译环境
  22. 	sudo apt-get upgrade 				更新已安装的包
  23. 	sudo apt-get dist-upgrade 			升级系统
  24. 	sudo apt-cache depends package 			了解使用该包依赖那些包
  25. 	sudo apt-cache rdepends package 		查看该包被哪些包依赖
  26. 	sudo apt-get source package  			下载该包的源代码
  27. 	sudo apt-get clean && sudo apt-get autoclean 	清理无用的包
  28. 	sudo apt-get check 				检查是否有损坏的依赖
  29. --------------------------------------------------------------------------------------------------------------
  30.  
  31.  
  32. -----------------------------------------THE BOOK OF <HADOOP WITH SPARK>-----------------------------------------
  33. 书中范例下载：
  34. http://pan.baidu.com.cn/s/1qYMtjNQ
  35. http://pan.baidu.com.cn/hadoopsparkbook
  36. ----------------------------------------------------------CP1 Info of big data&ML p1-8
  37. ----------------------------------------------------------CP2 VirtualBox p11-18
  38. 1.virtualbox5.2.34下载			https://www.virtualbox.org/wiki/Download_Old_Builds_5_2
  39. 	---> Next ~> ~> ~> Yes ~> Install ~> Finish
  40. 	---> Set languages (File ~> preferences ~> language
  41. 	---> Set Restore_File ( 管理 ~> 全局设定 ~> VRDP认证库 <其他> ~> 默认虚拟电脑位置：)
  42. 	---> Build a Vm
  43. 	~> next ~> 4096M ~> ~> VDI ~> D ~> 80G ~>
  44. 2.ubuntu18.04				https://ubuntu.com/download/desktop
  45. 	ubuntu18.04很卡解决方案 	sudo apt install gnome-session-flashback
  46. 					https://jingyan.baidu.com/article/37bce2bea3c07f1002f3a22a.html
  47. #更新linux				sudo apt-get update
  48. 3.安装：mysql				http://www.cnblogs.com/jpfss/p/7944622.html
  49. 	--安装mysql: sudo apt-get install mysql-server
  50. 	->获取mysql用户名密码文件:	 sudo gedit /etc/mysql/debian.cnf
  51. 	->登录mysql:	 mysql -u用户名 -p密码
  52. 	->修改mysql密码:
  53. 	->配置快捷命令
  54. 		sudo gedit ~/.bashrc
  55. 		alias mysql='mysql -u debian-sys-maint -pAQeZFkTb0y5EECNU'
  56. 		source ~/.bashrc
  57. 	->修改mysql在不支持中文		https://www.cnblogs.com/guodao/p/9702465.html
  58. 	#启动、关闭服务和查看运行状态
  59. 		sudo service mysql start
  60. 		sudo service mysql stop
  61. 		sudo service mysql status
  62. 删除 mysql		https://blog.csdn.net/iehadoop/article/details/82961264
  63. 	查看MySQL的依赖项：	dpkg --list|grep mysql
  64. 	sudo apt-get remove mysql-common
  65. 	sudo apt-get autoremove --purge mysql-server-5.7
  66. 	清除残留数据：	dpkg -l|grep ^rc|awk '{print$2}'|sudo xargs dpkg -P
  67. 	pkg --list|grep mysql
  68. 	继续删除剩余依赖项，如：sudo apt-get autoremove --purge mysql-apt-config
  69. 4.挂载文件夹
  70. 	vbox设置 -> 共享文件夹+ -> OK
  71. 	sudo mount -t vboxsf BZ /home/zieox/Desktop/BZ
  72.  
  73. 5.安装anaconda				https://blog.csdn.net/ksws0292756/article/details/79143460
  74. 	启动anaconda.SH:		sudo bash Anaconda3-2018.12-Linux-x86_64.sh
  75. 	启动anaconda.navigator:		anaconda-navigator
  76. 6.安装java8/step12INSTEAD		https://blog.csdn.net/claire017/article/details/80953632
  77. 7.安装IDEA				https://www.cnblogs.com/gzu-link-pyu/p/8263312.html
  78. 	解压：				sudo tar -zxvf ideaIC-2018.3.5-no-jdk.tar.gz -C /opt
  79. 8.安装PyCharm
  80.  
  81.  
  82.  
  83. 9.安装scala-plugs			https://www.cnblogs.com/starwater/p/6766831.html
  84. 	scala_SDK:
  85. 		1. 右键项目名称找到Open Module Settings
  86. 		2. 左侧Project Settings目录中点击Libraries
  87. 		3. 点击+new Project Library选择Scala SDK
  88. 		4. 添加下载好的jar文件夹
  89.  
  90. 10.安装scala_shell			https://blog.csdn.net/wangkai_123456/article/details/53669094
  91. 	升级高版本scala：		sudo apt-get remove scala
  92. 	解压：				tar zxvf scala-2.12.8.tgz 
  93. 	移动：				mv scala /usr/local/scala
  94. 	Source /etc/profile		https://blog.csdn.net/qq_35571554/article/details/82850563
  95. 	卸载：				sudo apt-get --purge remove polipo
  96. 11.maven					https://baijiahao.baidu.com/s?id=1612907927393262341&wfr=spider&for=pc
  97. 					tar -xvf apache-maven-3.6.0-bin.tar.gz
  98. 					sudo mv apache-maven-3.6.0 /usr/local
  99. 	下载地址：			http://maven.apache.org/download.cgi
  100. 12.关于maven.pom.xml库			https://mvnrepository.com/
  101. 13.linux下安装gradle			https://blog.csdn.net/yzpbright/article/details/53359855
  102. 14.卸载open-jdk				sudo apt-get remove openjdk*
  103. 15.安装sunJDK12				https://blog.csdn.net/smile_from_2015/article/details/80056297
  104. 	tar -zxvf jdk-12_linux-x64_bin.tar.gz
  105. 	cd  /usr/lib
  106. 	sudo mkdir jdk
  107. 	sudo mv jdk-12  /usr/lib/jdk
  108. 	sudo gedit /etc/profile
  109. ----------------------------------------------------
  110. export JAVA_HOME=/usr/lib/jdk/jdk-12
  111. export JRE_HOME=${JAVA_HOME}/jre
  112. export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
  113. export PATH=${JAVA_HOME}/bin:$PATH 
  114. ----------------------------------------------------
  115. 	source /etc/profile 
  116. sudo update-alternatives --install /usr/bin/java  java  /usr/lib/jdk/jdk-8/bin/java 300
  117. sudo update-alternatives --install /usr/bin/javac javac  /usr/lib/jdk/jdk-8/bin/javac 300
  118.  
  119. 16.下载maven	wget http://mirror.bit.edu.cn/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
  120. 	tar -zxvf apache-maven-3.6.3-bin.tar.gz
  121. 	rm apache-maven-3.6.3-bin.tar.gz
  122. 	sudo gedit /etc/profile
  123. 	------------------------	
  124. 	export M2_HOME=/usr/local/apache-maven-3.6.1
  125. 	export PATH=${M2_HOME}/bin:$PATH
  126. 	------------------------
  127. 	source /etc/profile 
  128. 17.创建scala 项目
  129. new project -> maven (create from more archetype) ->scala-archetype-simple
  130. 配置maven镜像:		https://blog.csdn.net/qq_39929929/article/details/103753905
  131.  
  132. ----------------------------------------------------------CP3 Ubuntu Linux P23-43
  133. //Set Virtual_File			setting ~> restore ~> Controler:IDE (no cd) ~> choose (open) ~>
  134. //Install Ubuntu			-> launch ~~~> (clear CD & install Ubuntu) Install --->Launch Ubuntu
  135. //Install Plug-in P34
  136. //Set Default Typewriting		system setting -> "text imput" -> "+" ->
  137. //Set terminal : (win)->ter-> drag->
  138. //Set terminal color : terminal (Configuration file preferences)->
  139. //share clipboard (equipment) ~> share_clipboard
  140.  
  141.  
  142. ----------------------------------------------------------CP4 The Installing Of Hadoop Single Node Cluster
  143. //4.1 install JDK
  144. 1.ctrl+alt+t
  145. 2.java-version
  146. 3.sudo apt-get update
  147. 4.sudo apt-get install default-jdk
  148. 5.update-alternatives --display java
  149.  
  150. //4.2 Set SSH logging without code
  151. 1. install SSH :					sudo apt-get install ssh
  152. 2. install rsync					sudo apt-get install rsync
  153. 3.//产生秘匙 produce SSH Key:				ssh-keygen -t dsa -P '' -f ~/.ssh/id.dsa
  154. 4.//查看秘匙 see the file of SSH Key:			ll ~/.ssh
  155. 5.//将产生的key放入许可证文件 :				cat ~/.ssh/id.dsa.pub >> ~/.ssh/authorized_keys
  156.  
  157. //4.3 Download Hadoop 下载hadoop
  158. 		---> https://archive.apache.org/dist/hadoop/common/
  159. 		---> Index of /dist/hadoop/common/hadoop-2.6.0
  160.  
  161. --->Copy Link Location
  162. ===> wget "~~~coped~~~" 
  163. //使用weget下载hadoop2.6		wget https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
  164. //解压					sudo tar -zxvf hadoop-2.6.0.tar.gz
  165. //移动文件				sudo mv hadoop-2.6.0 /usr/local/hadoop
  166. --->see & comfirm			ll /usr/local/hadoop
  167.  
  168. ============================================================================================================
  169. bin/各项运行文件
  170. sbin/各项shell运行文件
  171. etc/etc/hadoop 子目录包含Hadoop配置文件，例 hadoop -env.sh,core-site.xml,yarn-site.xml,mapred-site.xml,hdfs.site.xml
  172. lib/Hadoop函数库
  173. logs/系统日志，可以查看运行状况，运行有问题可以线哦嗯日志找出错误的原因
  174. ============================================================================================================
  175. //4.4 Set Hadoop variance 设置hadoop环境变量
  176. --->Edit bash				sudo gedit ~/.bashrc
  177. ---path of jdk				which java -> ls -l /usr/bin/java -> ls -l /etc/alternatives/java ->
  178. --->/usr/lib/jvm/java-8-oracle	
  179. --->Set JDK
  180. --------------------------------------------------------------------
  181. export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  182. export HADOOP_HOME=/usr/local/hadoop
  183. export PATH=$PATH:$HADOOP_HOME/bin
  184. export PATH=$PATH:$HADOOP_HOME/sbin
  185. export HADOOP_MAPRED_HOME=$HADOOP_HOME
  186. export HADOOP_COMMON_HOME=$HADOOP_HOME
  187. export HADOOP_HDFS_HOME=$HADOOP_HOME
  188. export YARN_HOME=$HADOOP_HOME
  189. export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
  190. export HADOOP_OPTS="-DJava.library.path=$HADOOP_HOME/lib"
  191. export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
  192. --------------------------------------------------------------------
  193. --->Set HADOOP_HOME hadoop usr/local/hadoop			export HADOOP_HOME=/usr/local/hadoop
  194. --->Set PATH
  195. export PATH=$PATH:$HADOOP_HOME/bin
  196. export PATH=$PATH:$HADOOP_HOME/sbin
  197. --->set Hadoop-env
  198. export HADOOP_MAPRED_HOME=$HADOOP_HOME
  199. export HADOOP_COMMON_HOME=$HADOOP_HOME
  200. export HADOOP_HDFS_HOME=$HADOOP_HOME
  201. export YARN_HOME=$HADOOP_HOME
  202. --->lib_link
  203. export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
  204. export HADOOP_OPTS="-DJava.library.path=$HADOOP_HOME/lib"
  205. export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
  206. --->Let bash work.						source ~/.bashrc
  207.  
  208. //4.5 modify Hadoop Setting_files
  209. --->修改hadoop-env.sh配置文件：				sudo gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh
  210. //change 'export JAVA_HOME=${JAVA_HOME}' to		export JAVA_HOME=/usr/lib/jvm/java-8-oracle
  211. --->修改core-site.xml配置文件				sudo gedit /usr/local/hadoop/etc/hadoop/core-site.xml
  212. #set Default-name of HDFS
  213. -------------------------------------
  214. <property>
  215. <name>fs.default.name</name>
  216. <value>hdfs://localhost:9000</value>
  217. <property>
  218. -------------------------------------
  219. --->修改 yarn-site.xml配置文件				sudo gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml
  220. -------------------------------------------------------------------
  221. <property>
  222. <name>yarn.nodemanager.aux-services</name>
  223. <value>mapreduce_shuffle</value>
  224. </property>
  225. <property>
  226. <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  227. <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  228. </property>
  229. -------------------------------------------------------------------
  230. --->修改 mapred-site.xml配置文件
  231. #copy template-file (复制模板文件)
  232. 	sudo cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
  233. --->编辑 mapred-site.xml配置文件			sudo gedit /usr/local/hadoop/etc/hadoop/mapred-site.xml
  234. --------------------------------------
  235. <property>
  236. <name>mapreduce.framework.name</name>
  237. <value>yarn</value>
  238. </property>
  239. --------------------------------------
  240. --->编辑 hdfs-site.xml配置文件				sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  241. 10.0.2.15
  242. ----------------------------------------------------------------------------
  243. <property>
  244. <name>dfs.replication</name>
  245. <value>3</value>
  246. </property>
  247. <property>
  248. <name>dfs.namenode.name.dir</name>
  249. <value> file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
  250. </property>
  251. <property>
  252. <name>dfs.datanode.name.dir</name>
  253. <value> file:/usr/local/hadoop/hadoop_data/hdfs/datanode</value>
  254. </property>
  255. ----------------------------------------------------------------------------
  256. //launch without namenode				https://www.cnblogs.com/lishpei/p/6136043.html
  257. //4.6 Creating & Formatting HDFS_dir 创建并格式化hdfs目录
  258. ---> 创建NameNode数据存储目录			sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/namenode
  259. ---> 创建DataNode数据存储目录			sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode
  260. ---> 将Hadoop目录所有者更改为 hduser		sudo chown zieox:zieox -R /usr/local/hadoop
  261. --->格式化hdfs					hadoop namenode -format
  262. ---# 启动 NameNode				sudo ./start-yarn.sh
  263. ---# 启动 NameNode				sudo ./start-dfs-sh
  264. --->显示错误日志				cd /usr/local/hadoop/logs/
  265. --->启动 name_node				./yarn-daemon.sh start namenode
  266. //查看本机ip					ifconfig		cd /usr/local/hadoop/	lsof -i :50070
  267. //打开hadoop resource manager web 界面：	localhost:8808
  268. //打开NameNode HDFS Web界面			http://172.28.30.12:50070
  269.  
  270. -------------------------------r-c-------------------------------
  271. 1469 gedit hadoop-lufax-datanode-lufax.log
  272. 1470 mkdir -r /home/user/hadoop_tmp/dfs/data
  273. 1471 mkdir -p /home/user/hadoop_tmp/dfs/data
  274. 1472 sudo mkdir -p /home/user/hadoop_tmp/dfs/data
  275. 1473 sudo chown lufax:lufax /home/user/hadoop_tmp/dfs/data
  276. 1474 ./hadoop-daemon.sh start datanode
  277. 1475 cd ../
  278. 1476 ./sbin/hadoop-daemon.sh start datanode
  279. 1477 jps
  280. 1478 hadoop fs -ls /
  281. 1479 hadoop fs -put README.txt /
  282. 1480 history 20
  283.  
  284. -----------------------------------------------------------------
  285.  
  286. ----------------------------------------------------------CP5 Installing:Hadoop Multi Node Cluster
  287. //1. 复制虚拟机
  288.  
  289. //设置data1虚拟机
  290. //编辑interfaces网络配置文件：			sudo gedit /etc/network/interfaces
  291. ------------------------------------------
  292. #NAT interface 		//网卡1
  293. auto auth0
  294. iface eth0 inet dhcp
  295.  
  296. #host only interface 	//网卡2
  297. auto eth0
  298. iface eth1 inet static 
  299. address 		192.168.56.101
  300. netmask 		255.255.255.0
  301. network			192.168.56.0
  302. broadcast		192.168.56.255
  303. ------------------------------------------
  304.  
  305. //设置hostname				sudo gedit /etc/hostname   --->    data1
  306. //设置hosts文件				sudo gedit /etc/hosts
  307. //加入主机ip地址
  308. ------------------------------------------
  309. 192.168.56.100		master
  310. 192.168.56.101		data1
  311. 192.168.56.102		data2
  312. 192.168.56.103		data3
  313. ------------------------------------------
  314. //修改localhost为master					sudo gedit /usr/local/hadoop/etc/hadoop/core-site.xml
  315. sudo gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml
  316.  
  317. //设置resourcemanager主机与nodemanager的连接地址
  318. <property>
  319. <name>yarn.resourcemanager.scheduler.address</name>
  320. <value>master:8025</value>
  321. </property>
  322.  
  323. //设置resourcemanager与applicationmaster的连接地址
  324. <property>
  325. <name>yarn.resourcemanager.address</name>
  326. <value>master:8030</value>
  327. </property>
  328.  
  329. //设置resourcemanager与客户端的连接地址
  330. <property>
  331. <name>yarn.resourcemanager.address</name>
  332. <value>master:8050</value>
  333. </property>
  334.  
  335. //查看YARN架构图		http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
  336.  
  337. //设置mapred-site.xml		sudo gedit /usr/local/hadoop/etc/hadoop/mapred-site.xml
  338. ------------------------------------------
  339. <property>
  340. <name>mapred.job.tracker</name>
  341. <value>master:54311</value>
  342. </property>
  343. ------------------------------------------
  344.  
  345. //设置hdfs-site.xml		sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  346. ------------------------------------------
  347. <property>
  348. <name>dfs.datanode.data.dir</name>
  349. <value> file:/usr/local/hadoop/hadoop_data/hdfs/datanode</value>
  350. </property>
  351. ------------------------------------------
  352. restart
  353. //确认网络设置			ifconfig
  354.  
  355. //设置master节点
  356. sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml
  357. -----------------------------------------------------------------
  358. <property>
  359. <name>dfs.namenode.name.dir</name>
  360. <value> file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
  361. </property>
  362. -----------------------------------------------------------------
  363.  
  364. //编辑masters文件			sudo gedit /usr/local/hadoop/etc/hadoop/masters
  365. //编辑slaves文件			sudo gedit /usr/local/hadoop/etc/hadoop/slaves
  366.  
  367. //master 连接到data1虚拟机		ssh data1
  368. //连接到data1创建HDFS相关目录
  369. --->删除hdfs所有目录			sudo rm -rf /usr/local/hadoop/hadoop_data/hdfs
  370. --->创建DataNode存储目录		mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode
  371. --->将目录所有者改成zieox		sudo chown -R zieox:zieox /usr/local/hadoop
  372. //中断data1连接回到master		exit
  373.  
  374. //master 连接到data2虚拟机		ssh data2
  375. --->删除hdfs所有目录			sudo rm -rf /usr/local/hadoop/hadoop_data/hdfs
  376. --->创建DataNode存储目录		mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode
  377. --->将目录所有者改成zieox		sudo chown -R zieox:zieox /usr/local/hadoop
  378. --->中断data2连接回到master		exit
  379.  
  380. //master 连接到data2虚拟机		ssh data3
  381. --->删除hdfs所有目录			sudo rm -rf /usr/local/hadoop/hadoop_data/hdfs
  382. --->创建DataNode存储目录		mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode
  383. --->将目录所有者改成zieox		sudo chown -R zieox:zieox /usr/local/hadoop
  384. --->中断data3连接回到master		exit
  385.  
  386. //创建并格式化NameNode HDFS目录
  387. --->删除之前的hdfs目录			sudo rm -rf /usr/local/hadoop/hadoop_data/hdfs
  388. --->创建DataNode存储目录		mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode
  389. --->将目录所有者改成zieox		sudo chown -R zieox:zieox /usr/local/hadoop
  390. --->格式化NameNode HDFS目录		hadoop namenode -format
  391.  
  392. //启动Hadoop Multi Node Cluster 
  393. --->分别启动HDFS & YARN 		start-dfs.sh 	start-yarn.sh	start-all.sh
  394.  
  395. ssh免密操作
  396. https://www.cnblogs.com/robert-blue/p/4133467.html
  397. //编辑固定IP地址
  398. 	scp id_rsa.pub zieox@master:/home/zieox
  399. 	cat id.dsa.pub >>~/.ssh/authorized_keys 
  400. 	service sshd restart
  401. 	cd .ssh
  402. 	scp authorized_keys zieox@data1:/home/zieox
  403. 	scp authorized_keys zieox@data2:/home/zieox
  404. 	scp authorized_keys zieox@data3:/home/zieox
  405. 	http://master:8088/cluster
  406. 	sudo chown zieox:zieox *
  407. //Datanode启动不了 datanode的clusterID 和 namenode的clusterID 不匹配(master中把namenode的cluterID改的和datanodeID一致)
  408. 	https://www.cnblogs.com/artistdata/p/8410429.html
  409. 	sudo gedit /usr/local/hadoop/hadoop_data/hdfs/datanode/current/VERSION
  410. 	clusterID=CID-834686bf-02bf-4933-b2fb-0a2288e97cc9
  411. 	sudo gedit /usr/local/hadoop/hadoop_data/hdfs/namenode/current/VERSION
  412.  
  413.  
  414. ----------------------------------------------------------CP6 Hadoop HDFS Order
  415. //hdfs 常用命令
  416. http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_shell.html
  417. -----------------------------------------HDFS基本命令-----------------------------------------
  418. hadoop fs-mkdir				创建hdfs目录
  419. hadoop fs-ls				列出hdfs目录
  420. hadoop fs-copyFromLocal			使用(copyFromLocal)复制本地文件到hdfs
  421. hadoop fs-put				使用(put)复制本地文件到hdfs
  422. hadoop fs-cat				列出hdfs目录下的文件内容
  423. hadoop fs-copyToLocal			使用(-copyToLocal)将hdfs上的文件复制到本地
  424. hadoop fs-get				使用(-get)将hdfs上的文件复制到本地
  425. hadoop fs-cp				复制hdfs文件
  426. hadoop fs-rm				删除hdfs文件
  427. ----------------------------------------------------------------------------------------------
  428. //创建hadoop目录
  429. hadoop fs -mkdir /user
  430. hadoop fs -mkdir /user/hduser
  431. hadoop fs -mkdir /user/hduser/test
  432. //hadoop: ls (list)
  433. hadoop fs -ls
  434. hadoop fs -ls /
  435. hadoop fs -ls /user
  436. hadoop fs -ls /user/hduser
  437. hadoop fs -ls -R /
  438. //创建多级目录				hadoop fs -mkdir -p /dir1/dir2/dir3
  439. //查看全部文件				hadoop fs -ls -R /
  440.  
  441. ---------------------------------------------Output---------------------------------------------
  442. -rw-r--r-- 3 lufax supergroup 1366 2018-09-12 23:57 /README.txt
  443. drwxr-xr-x - lufax supergroup 0 2018-09-20 03:31 /dir1
  444. drwxr-xr-x - lufax supergroup 0 2018-09-20 03:31 /dir1/dir2
  445. drwxr-xr-x - lufax supergroup 0 2018-09-20 03:31 /dir1/dir2/dir3
  446. drwxr-xr-x - lufax supergroup 0 2018-09-20 03:23 /user
  447. drwxr-xr-x - lufax supergroup 0 2018-09-20 03:23 /user/hduser
  448. drwxr-xr-x - lufax supergroup 0 2018-09-20 03:23 /user/hduser/test
  449. -----------------------------------------------------------------------------------------------
  450. //上传文件至HDFS (copyFromLocal)
  451. hadoop fs -copyFromLocal /home/zieox/桌面/test1.txt /user/hduser/test
  452. hadoop fs -copyFromLocal /usr/local/hadoop/README.txt /user/hduser/test
  453. hadoop fs -copyFromLocal /user/local/hadoop/README.txt /user/hduser/test/test1.txt
  454.  
  455. //打开查看文件				hadoop fs -cat /user/hduser/test/test1.txt |more
  456. //强制复制				hadoop fs -copyFromLocal -f /usr/local/hadoop/README.txt /user/hduser/test
  457. //查看目录				hadoop fs -ls /user/hduser/test
  458. // put:使用-put会直接覆盖文件		hadoop fs -put /usr/local/hadoop/README.txt /user/hduser/test/test1.txt
  459. //将屏幕上内容存储到HDFS文件		echo abc| hadoop fs -put - /user/hduser/test/echoin.txt
  460. //查看文件				hadoop fs -cat /user/hduser/test/echoin.txt
  461.  
  462. //将本地目录的列表，存储到HDFS文件	ls /usr/local/hadoop | hadoop fs -put - /user/hduser/test/hadooplist.txt
  463.  
  464. //将hdfs文件拷贝至本地 (copyToLocal)
  465. 	mkdir /home/zieox/桌面/test  ~>  cd /home/zieox/桌面/test  ~>  hadoop fs -copyToLocal /user/hduser/test/hadooplist.txt
  466.  
  467. //将整个HDFS上的目录复制到本地计算机：	hadoop fs -copyToLocal /user/hduser/test/ect
  468. //将HDFS上的文件复制到本地计算机 (get)：	hadoop fs -get /user/hduser/test/README.txt localREADME.txt
  469. ///复制与删除HDFS文件
  470. //在HDFS上创建测试目录			hadoop fs -mkdir /user/hduser/test/tmp
  471. //复制HDFS文件到HDFS测试目录		hadoop fs -cp /user/hduser/test/README.txt /user/hduser/test/tmp
  472. //查看测试目录				hadoop fs -ls /user/hduser/test/tmp
  473. //删除文件				hadoop fs -rm /user/hduser/test/test2.txt
  474. //删除文件目录				hadoop fs -rm -R /user/hduser/test
  475. ###///在Hadoop HDFS Web - Browse Directory			http://master:50070
  476.  
  477.  
  478. ----------------------------------------------------------CP7 Hadoop MapReduce
  479. http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html
  480. //创建wordcount目录				mkdir -p ~/wordcount/input
  481. //进入wordcount文件夹				cd ~/wordcount
  482. //编辑java脚本					sudo gedit WordCount.Java
  483. //设置路径					sudo gedit ~/.bashrc
  484. --------------------------------------------------
  485. export PATH=${JAVA_HOME}/bin:${path}
  486. export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
  487. --------------------------------------------------
  488. //生效bashrc					source ~/.bashrc
  489. ***
  490. hadoop com.sun.tools.Javac.Main WordCount.Java
  491. jar wc.jar WordCount*.class
  492.  
  493.  
  494. //创建测试文本文件			cp /usr/local/hadoop/LICENSE.txt ~/wordcount/input
  495. ll ~/wordcount/input
  496. //hdfs操作
  497. 		hadoop fs -mkdir -p /user/hduser/wordcount/input
  498. 		hadoop fs -ls /user/hduser/wordcount/input
  499. 		cd ~/wordcount
  500. //运行WordCount程序
  501. 		hadoop jar wc.jar WordCount /user/hduser/wordcount/input/LICENSE.txt /user/hduser/wordcount/output
  502.  
  503. //查看运行结果
  504. 		hadoop fs -ls /user/hduser/wordcount/output
  505. 		hadoop fs -cat /user/hduser/wordcount/output/part-r-00000|more
  506.  
  507.  
  508. ----------------------------------------------------------CP8 Installing Spark
  509.  
  510. //启动hadoop服务
  511. 	启动服务的脚本全部在 ./sbin 目录下
  512. 	start-all.sh 可以启动全部的服务，不建议！这样做！
  513. 	start-dfs.sh 启动DFS中的namenode 和 datanode 全部启动，不建议！这样做！
  514. //启动namenode：/usr/local/hadoop/sbin/hadoop-daemon.sh start namenode
  515. //启动datenode：/usr/local/hadoop/sbin/hadoop-daemon.sh start datanode
  516. //查看是否有datanode 的进程jps		然后就可以通过Web浏览器查看dfs了！http://localhost:50070
  517. //启动Yarn /usr/local/hadoop/sbin/start-yarn.sh
  518. -------------------------------------------------------------------------------------------------------------------
  519. 测试yarn
  520. /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 2 100
  521. //启动hdfs/usr/local/hadoop/sbin/start-dfs.sh
  522. bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.2.jar pi 2 100
  523. ssh-copy-id -i /root/.ssh/id_rsa.pub root@<lufax>
  524. -------------------------------------------------------------------------------------------------------------------
  525. //Install spark			wget https://archive.apache.org/dist/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.6.tgz
  526. 	cd /home/lufax/下载
  527. 	tar -zxvf spark-2.3.1-bin-hadoop2.6.tgz -C /opt/spark/
  528. 	tar xzvf spark-2.3.1-bin-hadoop2.6.tgz
  529. 	sudo mv spark-2.3.1-bin-hadoop2.6 /usr/local/spark/
  530. 	gedit ~/.bashrc
  531. -----------------------------------
  532. export SPARK_HOME=/usr/local/spark
  533. export PATH=$PATH:$SPARK_HOME/bin
  534. -----------------------------------
  535. 	source ~/.bashrc
  536. 	spark-shell
  537. 	https://master:50070
  538.  
  539. //设置spark-shell
  540. 	cd /usr/local/spark/conf
  541. 	cp log4j.properties.template log4j.properties
  542. 	sudo gedit log4j.properties
  543. -----------------set-----------------
  544. log4j.rootCategory=WARN
  545. -------------------------------------
  546. //本地运行spark
  547. //本地启动(3线程)			spark-shell --master local[3]
  548. //读取本地文件				val tf = sctextFile("hdfs//master:9000/user/hduser/test/tes2.txt")
  549. //Spark读取与写入文件			https://blog.csdn.net/a294233897/article/details/80904305
  550. //在yarn上运行spark-shell
  551. SPARK_JAR=` /usr/local/spark/lib/jars/*.jar `
  552. HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
  553. MASTER=yarn-client
  554. /usr/local/spark/bin/spark-shell
  556. //用spark读取HDFS文件
  557. 	val tf = sc.textFile("hdfs://master:9000/user/hduser/test/README.txt")
  558. 	scala> tf.count				--->res0: Long = 31
  559. 	http://172.22.230.18/
  560. //8.9 构建spark standalone cluster环境
  561. //复制模板文件来创建spark-env.sh
  562. 	cp /usr/local/spark/conf/spark-env.sh.template spark-env.sh
  563. 	sudo gedit spark-env.sh
  564. ---------------------------------------------------------------------------------------------
  565. export SPARK_MASTER_IP=master			设置master的IP或服务器名称
  566. export SPARK_WORKER_CORES=1			设置每个Worker使用的CPU核心
  567. export SPARK_WORKER_MEMORY=800m			设置每个Worker使用内存
  568. export SPARK_WORKER_INSTANCES=2			设置每个Worker实例
  569. ---------------------------------------------------------------------------------------------
  570. ssh data1
  571. sudo mkdir /usr/local/spark
  572. sudo chown -R zieox:zieox /usr/local/spark
  573. sudo scp -r /usr/local/spark zieox@data:/usr/local
  574. ssh data3
  575. ...
  577. sudo gedit /usr/local/spark/conf/slaves
  578. ------------
  579. data1
  580. data3
  581. ------------
  582. //启动：spark standalone cluster(启动works)			/usr/local/spark/sbin/start-all.sh
  583. datanode=3
  584. SPARK_WORKER_INSTANCES=2
  585. worker=6
  586. //分别启动master&slaves
  587. 	//启动spark_master			/usr/local/spark/sbin/start-master.sh
  588. 	//启动spark_slaves			/usr/local/spark/sbin/start-slaves.sh
  589. //在Spark Standalone运行spark-shell程序				spark-shell --master spark://master:7077
  590. //查看Spark Standalone 的 Web_UI界面				http://master:8080/
  591. //读取本地文件
  592. 	val tf = spark.read.textFile("file:/home/zieox/桌面/tes2.txt")
  593. 	val tf = sc.textFile("file:/user/hduser/test/README.txt")
  594. //读取hdfs文件(确保works完全启动)			val tf = spark.read.textFile("hdfs://master:9000/user/hduser/test/README.txt")
  595. --->scala> tf.count						res0: Long = 31
  596. //停止Spark Stand alone cluster					/usr/local/spark/sbin/stop-all.sh
  599. ----------------------------------------------------------CP9 Spark RDD
  600. spark-shell
  601. scala> val intRDD = sc.parallelize(List(1,2,3,1,3,2,3,4))
  602. 	--->intRDD: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:24
  603. scala> intRDD.collect()				res0: Array[Int] = Array(1, 2, 3, 1, 3, 2, 3, 4)
  604. scala> val stringRDD=sc.parallelize(List("zieox","luccy","fucker"))
  605. 	--->stringRDD: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[1] at parallelize at <console>:24
  606. scala> stringRDD.collect()			res1: Array[String] = Array(zieox, luccy, fucker)
  607. //map
  608. def addone(x:Int):Int = {return (x+1)}
  609. intRDD.map(addone).collect()
  610. intRDD.map(x=>x+1).collect()
  611. intRDD.map(_+1).collect()
  612. scala> stringRDD.map(x=>"name: "+x).collect			--->res9: Array[String] = Array(name: zieox, name: luccy, name: fucker)
  613. //filter
  614. intRDD.filter(_<3).collect()
  615. scala> stringRDD.filter(x=>x.contains("eo")).collect()		--->res10: Array[String] = Array(zieox)
  616. //distinct 
  617. scala> intRDD.distinct().collect				--->res11: Array[Int] = Array(4, 1, 2, 3)
  618. //randomSplit
  619. //以随机数的方式按照4:6的比例分割为两个RDD			val sRDD=intRDD.randomSplit(Array(0.4,0.6))
  620. //groupBy
  621. val gDD = intRDD.groupBy(x=>{if (x%2==0) "even" else "odd"}).collect
  623. //9.3多个RDD“转换”运算
  624. val r1=sc.parallelize(List(1,2))
  625. val r2=sc.parallelize(List(3,4,5))
  626. val r3=sc.parallelize(List(1,6,7))
  628. //union (并集运算)				r1.union(r2).union(r3).collect()	&	(r1++r2++r3).collect()
  629. //intersection	(交集运算)			r1.intersection(r3).collect
  630. //subtract (差集运算)				r1.subtract(r2).collect() 	//在r1,不再r2
  631. //cartesian (笛卡尔积运算)			r1.cartesian(r2).collect()
  633. //9.4 基本动作运算
  634. val r=sc.parallelize(List(1,2,3,4,5))
  635. scala> r.first						--->res30: Int = 1
  636. scala> r.take(2)					--->res32: Array[Int] = Array(1, 2)
  637. scala> r.takeOrdered(3)					--->res33: Array[Int] = Array(1, 2, 3)
  638. scala> r.takeOrdered(3)(Ordering[Int].reverse)		--->res34: Array[Int] = Array(5, 4, 3)
  640. --------基本统计--------
  641. r.stats		统计
  642. r.min		最小值
  643. r.max		最大值
  644. r.stdev		标准差
  645. r.count		计数
  646. r.sum		求和
  647. r.mean		平均
  648. ------------------------
  650. //9.5 RDD Key-Value 基本“转换”运算
  651. val kv=sc.parallelize(List((1,2),(1,3),(3,2),(3,4),(4,5)))
  652. kv.keys.collect()
  653. kv.filter{case (key,value)=>key<5}.collect
  654. //mapValues运算					kv.mapValues(x=>x*x).collect()
  655. //sortByKey (按key，从小到大排列)		kv.sortByKey(true).collect()
  656. //reduceByKey					kv.reduceByKey((x,y)=>x+y).collect	//根据key进行reduce并对value执行func
  657. 						kv.reduceByKey(_+_).collect
  659. //9.6 多个RDD Key-Value “转换”运算
  660. val r1 = sc.parallelize(List((1,2),(1,3),(3,2),(3,4),(4,5)))
  661. val r2 = sc.parallelize(List((3,8)))
  662. scala> r1.join(r2).foreach(println)			//join(by key) return(key with  key's value)
  663. 		--->(3,(2,8))
  664. 		    (3,(4,8))
  666. scala> r1.leftOuterJoin(r2).foreach(println)		//join(by key-left_all) return (key's left_value-all)
  667. 		--->(1,(2,None))
  668. 		    (1,(3,None))
  669. 		    (3,(2,Some(8)))
  670. 		    (4,(5,None))
  671. 		    (3,(4,Some(8)))
  673. scala> r1.rightOuterJoin(r2).foreach(println)		//join (by key-right_all) return (key's right_value-all)
  674. 		--->(3,(Some(2),8))
  675. 		    (3,(Some(4),8))
  678. scala> r1.subtractByKey(r2).collect			//join (exclude common_keys) return (keys) (join的逆反)
  679. 		--->res56: Array[(Int, Int)] = Array((4,5), (1,2), (1,3))
  681. //9.7 Key-Value “动作”运算
  682. val r1 = sc.parallelize(List((1,2),(1,3),(3,2),(3,4),(4,5)))
  683. scala> r1.first				--->res0: (Int, Int) = (1,2)
  684. scala> r1.take(2)			--->res2: Array[(Int, Int)] = Array((1,2), (1,3))
  685. scala> r1.first._1			--->res5: Int = 1
  686. //countByKey		根据key进行count
  687. scala> r1.countByKey			--->res8: scala.collection.Map[Int,Long] = Map(4 -> 1, 1 -> 2, 3 -> 2)
  688. //collectAsMap		创建Key-Value对照表
  689. scala> r1.collectAsMap			--->res11: scala.collection.Map[Int,Int] = Map(4 -> 5, 1 -> 3, 3 -> 4)
  690. //lookup运算(by_key)
  691. scala> r1.collect			--->res16: Array[(Int, Int)] = Array((1,2), (1,3), (3,2), (3,4), (4,5))
  693. //9.8 Broadcast 广播变量
  694. //不使用广播变量的映射对照
  695. val kf = sc.parallelize(List((1,"apple"),(2,"banana"),(3,"orange")))
  696. scala> val kfm=kf.collectAsMap			--->kfm: scala.collection.Map[Int,String] = Map(2 -> banana, 1 -> apple, 3 -> orange)
  697. scala> val id = sc.parallelize(List(2,1,3,1))
  698. scala> val fn = id.map(x=>kfm(x)).collect	--->fn: Array[String] = Array(banana, apple, orange, apple)
  700. //使用广播变量
  701. val kf = sc.parallelize(List((1,"apple"),(2,"banana"),(3,"orange")))
  702. scala> val kfm=kf.collectAsMap			--->kfm: scala.collection.Map[Int,String] = Map(2 -> banana, 1 -> apple, 3 -> orange)
  703. val bfm = sc.broadcast(kfm)			//使用广播以节省内存与传送时间
  704. scala> val id = sc.parallelize(List(2,1,3,1))
  705. val fn = id.map(x=>bfm.value(x)).collect
  707. //9.9 accumulator 累加器
  708. val acc = sc.parallelize(List(2,1,3,1,5,1))
  709. val total = sc.accumulator(0.0)
  710. val num = sc.accumulator(0)
  711. acc.foreach(i=>{total+=1 ; num+=1})
  712. println("total=" + total.value + ",num=" + num.value)
  713. val avg = total.value/num.value
  715. //9.10 RDD Persistence持久化
  716. Spark RDD 持久化机制：可以用于将需要重复运算的RDD存储在内存中，以便大幅提升运算效率。
  717. ------------------------------------------------------------------------------------------------------------------------------------
  718. MEMORY_ONLY					spark会将RDD对象以Java对象反串行化（序列化）在JVM的堆空间中，而不经过序列化处理。
  719. 						如果RDD太大无法完全存储在内存中，多余的RDD partitions不会缓存在内存中，而是需要重新计算
  720. MEMORY_AND_DISK					尽量将RDD以Java对象反串行化在JVM的在内存中，如果内存缓存不下了，就将剩余分区缓存在磁盘中
  721. MEMORY_ONLY_SER					将RDD进行序列化处理(每个分区序列化为一个字节数组)然后缓存在内存中。
  722. 						因为需要再进行反序列化，会多使用CPU计算资源，但是比较省内存的存储空间
  723. 						多余的RDD partitions不会缓存在内存中，而是需要重新计算
  724. MEMORY_AND_DISK_SER				和MEMORY_ONLY_SER类似，多余的RDD partitions缓存在磁盘中
  725. DISK_ONLY					仅仅使用磁盘存储RDD的数据(未经序列化)
  726. MEMORY_ONLY_2,MEMORY_AND_DISK_2			
  727. ------------------------------------------------------------------------------------------------------------------------------------
  729. RDD.persist
  730. RDD.unpersist
  732. val a = sc.parallelize(List(2,1,3,1,5,1))
  733. a.presist()
  734. //设置RDD.persist存储等级范例
  735. import org.apache.spark.storage.StorageLevel
  736. val intRDDMemoryAndDisk = sc.parallelize(List(2,1,3,1,5,1))
  737. intRDDMemoryAndDisk.persist(StorageLevel.MEMORY_AND_DISK)
  738. intRDDMemoryAndDisk.unpersist()
  740. //9.11 使用spark创建WordCount
  741. sudo gedit /home/zieox/桌面/tes.txt
  742. val tf = sc.textFile("file:/home/zieox/桌面/tes.txt")		//读取
  743. val string = tf.flatMap(l=>l.split(" "))			//处理  （这里使用flatMap读取并创建stringRDD）
  744. val wordscount = string.map(w => (w,1)).reduceByKey(_+_)	//计算每个单词出现的次数
  745. val wordscount = tf.flatMap(l=>l.split(" ")).map(w => (w,1)).reduceByKey(_+_)
  746. wordscount.saveAsTextFile("file:/home/zieox/桌面/output")	//保存计算结果
  749. //9.12 Spark WordCount 
  750. scala> tf.flatMap(i=>i.split(" ")).map((_,1)).collect
  751. 	--->res59: Array[(String, Int)] = Array((28,1), (march,1), (,,1), (its,1), (a,1), (raining,1), (day,1), (!,1))
  752. //WordsCount
  753. val wordscount=sc.textFile("file:/home/zieox/桌面/tes.txt").flatMap(l=>l.split(" ")).map((_,1)).reduceByKey(_+_).foreach(println)
  756. ----------------------------------------------------------CP10 Spark的集成开发环境
  757. //10.1 下载与安装eclipse Scala IDE
  758. 下载：			http://scala-ide.org/
  759. 解压：			tar -xzvf scala-SDK-4.7.0-vfinal-2.12-linux.gtk.x86_64.tar.gz
  760. 移动：			mv eclipse /home
  761. 创建link：
  763. //10.2 下载项目所需的Library
  764. spark-assembly
  765. joda-Time
  766. jfreeChart
  767. jcommon
  768. //创建lib目录存放连接库					mkdir -p ~/workspace/Lib
  769. //复制spark.jar						sudo cp /usr/local/spark/jars ~/workspace/Lib
  770. //进入jar下载网站					www.Java2s.com
  771. //下载joda-time	
  772. cd ~/workspace/Lib ---> wegt http://www.java2s.com/Code/JarDownload/joda/joda-time-2.1.jar.zip ---> unzip -j joda-time-2.1.jar.zip
  773. //下载jfreechart
  774. wget http://www.java2s.com/Code/JarDownload/jfreechart/jfreechart.jar.zip ---> unzip -j jfreechart.jar.zip
  775. //下载saprk-core
  776. wget http://www.java2s.com/Code/JarDownload/spark/spark-core_2.9.2-0.6.1.jar.zip    unzip -j spark-core_2.9.2-0.6.1.jar.zip
  777. wget http://www.java2s.com/Code/JarDownload/spark/spark-core_2.9.2-0.6.1-sources.jar.zip   unzip -j spark-core_2.9.2-0.6.1-sources.jar.zip
  780. //下载jcommon
  781. wget http://www.java2s.com/Code/JarDownload/jcommon/jcommon-1.0.14.jar.zip
  782. wget http://www.java2s.com/Code/JarDownload/jcommon/jcommon-1.0.14-sources.jar.zip
  783. unzip -j jcommon-1.0.14-sources.jar.zip
  786. //删除zip文件以节省空间					rm *.zip
  787. //launch Eclipse
  789. //10.4 创建Spark项目					file -> new -> scala project -> addexternalJARs -> changeScalaVERSION
  790. //10.5 设置项目链接库（referenced Libraries --> build path --> configure build path...）
  791. //10.6 新建scala程序					new scala_object
  792. //10.7 创建WordCount测试文件
  793. 	mkdir -p ~/workspace/wordcount2/data
  794. 	cd ~/workspace/wordcount2/data
  795. 	cp /usr/local/hadoop/LICENSE.txt LICENSE.txt
  796. //10.8 创建WordCount.scala
  797. import org.apache.log4j.Logger
  798. import org.apache.log4j.Level
  799. import org.apache.spark.{SparkConf,SparkContext}
  800. import org.apache.spark.rdd.RDD
  801. import org.apache.hadoop.io.IOUtils;
  802. object wordcount {
  803.   def main(args: Array[String]): Unit = {
  804.     // 以這兩行設定不顯示 spark 內部的訊息
  805.     Logger.getLogger("org").setLevel(Level.OFF)
  806.     System.setProperty("spark.ui.showConsoleProgress", "false")
  807.     // 清除 output folder
  808.     println("執行RunWordCount")
  809.     // 設定 application 提交到 MASTER 指向的 cluster 或是 local 執行的模式
  810.     // local[4] 代表是在本地以 四核心的 CPU 執行
  811.     val sc = new SparkContext(new SparkConf().setAppName("wordCount").setMaster("local[4]"))
  812.     println("讀取文字檔...")
  813.     val textFile = sc.textFile("hdfs://master:9000/user/hduser/test/README.txt")
  814.     println("開始建立RDD...")		// flapMap 是取出文字檔的每一行資料，並以 " " 進行 split，分成一個一個的 word
  815.     // map 是將每一個 word 轉換成 (word, 1) 的 tuple
  816.     // reduceByKey 會根據 word 這個 key，將後面的 1 加總起來，就會得到 (word, 數量) 的結果
  817.     val countsRDD = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
  818.     println("儲存結果至文字檔...")
  819.     try {countsRDD.saveAsTextFile("data/output") ; println("存檔成功")}
  820.     catch {case e: Exception => println("輸出目錄已經存在,請先刪除原有目錄");}
  821.     println("hello")
  822.   }
  823. }
  825. 使用intellIDEA创建scalaProject				https://www.cnblogs.com/luguoyuanf/p/19c1e4d88a094c07331e912f40ed46c7.html
  827. scalaProject maven配置
  828. ------------------------------------------------------------------------------------------------------------------------------
  829. <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  830.   <modelVersion>4.0.0</modelVersion>
  831.   <groupId>zieox</groupId>
  832.   <artifactId>zieox</artifactId>
  833.   <version>1.0-SNAPSHOT</version>
  834.   <inceptionYear>2008</inceptionYear>
  835.   <properties> <scala.version>2.11.4</scala.version> </properties>
  837.   <repositories>
  838.     <repository>
  839.       <id>scala-tools.org</id>
  840.       <name>Scala-Tools Maven2 Repository</name>
  841.       <url>http://central.maven.org/maven2/</url>
  842.     </repository>
  843.   </repositories>
  845.   <pluginRepositories>
  846.     <pluginRepository>
  847.       <id>scala-tools.org</id>
  848.       <name>Scala-Tools Maven2 Repository</name>
  849.       <url>https://mvnrepository.com/artifact</url>
  850.     </pluginRepository>
  851.   </pluginRepositories>
  853.   <dependencies>
  854.     <dependency>
  855.       <groupId>org.scala-lang</groupId>
  856.       <artifactId>scala-library</artifactId>
  857.       <version>${scala.version}</version>
  858.     </dependency>
  859.     <dependency>
  860.       <groupId>junit</groupId>
  861.       <artifactId>junit</artifactId>
  862.       <version>4.11</version>
  863.       <scope>test</scope>
  864.     </dependency>
  866.     <dependency>
  867.       <groupId>org.scala-lang</groupId>
  868.       <artifactId>scala-reflect</artifactId>
  869.       <version>${scala.version}</version>
  870.     </dependency>
  871.     <dependency>
  872.       <groupId>org.scala-lang</groupId>
  873.       <artifactId>scala-compiler</artifactId>
  874.       <version>${scala.version}</version>
  875.     </dependency>
  876.     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
  877.     <dependency>
  878.       <groupId>org.apache.spark</groupId>
  879.       <artifactId>spark-core_2.11</artifactId>
  880.       <version>2.4.0</version>
  881.     </dependency>
  882.     <!-- https://mvnrepository.com/artifact/joda-time/joda-time -->
  883.     <dependency>
  884.       <groupId>joda-time</groupId>
  885.       <artifactId>joda-time</artifactId>
  886.       <version>2.10.1</version>
  887.     </dependency>
  888.     <!-- https://mvnrepository.com/artifact/org.jfree/jcommon -->
  889.     <dependency>
  890.       <groupId>org.jfree</groupId>
  891.       <artifactId>jcommon</artifactId>
  892.       <version>1.0.23</version>
  893.     </dependency>
  895.     <!-- https://mvnrepository.com/artifact/org.jfree/jfreechart -->
  896.     <dependency>
  897.       <groupId>org.jfree</groupId>
  898.       <artifactId>jfreechart</artifactId>
  899.       <version>1.0.19</version>
  900.     </dependency>
  901.   </dependencies>
  902.   <build>
  903.     <sourceDirectory>src/main/scala</sourceDirectory>
  904.     <testSourceDirectory>src/test/scala</testSourceDirectory>
  905.     <plugins>
  906.       <plugin>
  907.         <groupId>org.scala-tools</groupId>
  908.         <artifactId>maven-scala-plugin</artifactId>
  909.         <executions>
  910.           <execution>
  911.             <goals>
  912.               <goal>compile</goal>
  913.               <goal>testCompile</goal>
  914.             </goals>
  915.           </execution>
  916.         </executions>
  917.         <configuration>
  918.           <scalaVersion>${scala.version}</scalaVersion>
  919.           <args>
  920.             <arg>-target:jvm-1.5</arg>
  921.           </args>
  922.         </configuration>
  923.       </plugin>
  924.       <plugin>
  925.         <groupId>org.apache.maven.plugins</groupId>
  926.         <artifactId>maven-eclipse-plugin</artifactId>
  927.         <configuration>
  928.           <downloadSources>true</downloadSources>
  929.           <buildcommands>
  930.             <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
  931.           </buildcommands>
  932.           <additionalProjectnatures>
  933.             <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
  934.           </additionalProjectnatures>
  935.           <classpathContainers>
  936.             <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
  937.             <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
  938.           </classpathContainers>
  939.         </configuration>
  940.       </plugin>
  941.     </plugins>
  942.   </build>
  943.   <reporting>
  944.     <plugins>
  945.       <plugin>
  946.         <groupId>org.scala-tools</groupId>
  947.         <artifactId>maven-scala-plugin</artifactId>
  948.         <configuration>
  949.           <scalaVersion>${scala.version}</scalaVersion>
  950.         </configuration>
  951.       </plugin>
  952.     </plugins>
  953.   </reporting>
  954. </project>
  955. ------------------------------------------------------------------------------------------------------------------------------
  956. 用IDEA打jar包				https://www.cnblogs.com/blog5277/p/5920560.html
  959. //10.12 spark-submit 的详细介绍
  960. ------------------------------------------------------常用选项--------------------------------------------------------
  961. --master MASTER_URL				设置spark运行环境
  962. --driver-memory MEM				Driver程序所使用的内存
  963. --executor-memory MEM				executor程序所使用的内存
  964. --jars JARS					要运行的application会引用到的外部链接库
  965. --name NAME					要运行的application名称
  966. --class CLASS_NAME				要运行的application主要类名称
  967. ----------------------------------------------------------------------------------------------------------------------
  968. --master MASTER_URL设置选项
  969. local						在本地运行：只是用一个线程
  970. local[K]					在本地运行：使用K个线程
  971. local[*]					在本地运行：spark会尽量利用计算机上的多核CPU
  972. spark://HOST:PORT				在Spark Standalone Cluster 上运行，spark://master:7077
  973. mesos://HOST:PORT				在mesos cluster上运行（default_port:5050）
  974. yarn-client					在yarn-client上运行，必须设置HADOOP_CONF_DIR or YAEN_CONF_DIR环境变量
  975. yarn-cluster					在yarn-cluster上运行，必须设置HADOOP_CONF_DIR or YAEN_CONF_DIR环境变量
  976. ----------------------------------------------------------------------------------------------------------------------
  978. //10.13 在本地local模式运行WordCount程序
  979. 1.打jar包		https://blog.csdn.net/zrc199021/article/details/53999293
  980. 2.将jar放到usr/local/hadoop/bin 下
  981. 3.spark_submit
  983. spark-submit --driver-memory 2g --master local[4] --class WordCount /usr/local/hadoop/bin/zieoxscala.jar
  984. spark-submit
  985. 	--driver-memory 2g					设置dirver程序使用2G内存
  986. 	--master local[4] 					本地运行使用4个线程
  987. 	--class WordCount					设置main类
  988. /usr/local/hadoop/bin/zieoxscala.jar			jar路径
  991. //10.14 在hadoop yarn-client 运行wordcount程序
  992. //1.上传LICENSE.txt至HDFS
  993. hadoop fs -mkdir /data
  994. hadoop fs -copyFromLocal /home/zieox/桌面/LICENSE.txt data
  995. //2.修改错误标签
  996. sudo gedit ~/.bashrc
  997. 添加：				export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
  998. 生效bashrc：			source ~/.bashrc
  999. zieox@master:~/IdeaProjects/zieoxscala/out/artifacts/zieoxscala_jar$ cp zieoxscala.jar /usr/local/hadoop/bin
  1000. spark-submit --driver-memory 2g --master local[4] --class wordcount --master yarn-client ~/IdeaProjects/zieoxscala/out/artifacts/zieoxscala_jar/zieoxscala.jar
  1001. //查看执行结果							hadoop fs -cat /user/hduser/test/output/part-00000
  1002. //IDEA项目远程调试hadoop入门（maven项目）			https://www.cnblogs.com/zh-yp/p/7884084.html
  1004. //10.15 在Spark Standalone Cluster 上运行wordcount程序
  1005. spark-submit --driver-memory 2g --master local[4] --class wordcount --master spark://master:7077 ~/IdeaProjects/zieoxscala/out/artifacts/zieoxscala_jar/zieoxscala.jar
  1008. ----------------------------------------------------------CP11 创建推荐引擎
  1009. https://www.jianshu.com/p/b909d78f7d72
  1010. mvn dependency:resolve -Dclassifier=sources
  1012. https://www.cnblogs.com/flymercurial/p/7859595.html
  1013. //mvlens数据集下载				https://grouplens.org/datasets/movielens/
  1014. mkdir -p ~/workspace/recommend/data
  1015. cd ~/workspace/recommend/data
  1016. unzip -j ml-100k
  1017. u.data		评分数据
  1018. u.item		电影数据
  1020. //11.5 使用spark-shell导入ml-100k
  1021. //读取数据
  1022. val rawData = sc.textFile("file:/home/zieox/workspace/recommend/data/u.data")
  1023. rawData.first()
  1024. rawData.take(5).foreach(println)				//打印前5行
  1025. rawData.map(_.split('\t')(1).toDouble).stats()			//查看第二列统计信息
  1027. //使用ALS.train进行训练
  1028. import org.apache.spark.mllib.recommendation.ALS
  1029. import org.apache.spark.mllib.recommendation.Rating
  1030. val ratings = rawData.map(_.split('\t').take(3))
  1031. val ratingRDD = ratings.map{case Array(user,movie,rating)=>Rating(user.toInt,movie.toInt,rating.toDouble)}
  1033. //显示评分训练 p249
  1034. ALS.train(ratings:RDD[Rating],rank:Int,iterations:Int,lambda:Double):MatrixFactorizationModel  
  1035. //Rating 数据源RDD，rank 原矩阵m*n 分解为 m*rank和rank*n矩阵，iterations 计算次数，lambda 建议值0.01，返回数据MatrixFactorizationModel
  1036. //隐式训练
  1037. ALS.trainlmplicit(ratings:RDD[Rating],rank:Int,iterations:Int,lambda:Double):MatrixFactorizationModel
  1038. //进行显式训练
  1039. val model = ALS.train(ratingsRDD,10,10,0.01)
  1041. //11.8 使用模型进行推荐
  1042. MatrixFactorizationModel.model.recommendProducts(user:Int,num:Int):Array[Rating]		//使用模型推荐	
  1043. model.recommendProducts(196,5).mkString("\n")				//针对用户推荐
  1044. model.predict(196,464)							//查看针对用户推荐评分 （查看向user:196 推荐464号产的推荐评分）
  1045. MatrixFactorizationModel.model.recommendUsers(product，num)		//针对产品推荐给用户
  1046. model.recommendUsers(464,5).mkString("\n")				//实际执行（针对电影推荐的用户）
  1047. //11.9 显示推荐电影的名称
  1048. val itemRDD = sc.textFile("file:/home/zieox/workspace/recommend/data/u.item")			//导入文件item
  1049. //创建ID_Movie对照表
  1050. val movieTitle = 
  1051. 		itemRDD.map(line=>line.split("\\|")		//处理 按照 ‘\\|’分列
  1052. 		.take(2))					//取出前2个字段
  1053. 		.map(array=>(array(0).toInt,array(1)))		//对取出的array做处理，转化id为数字
  1054. 		.collectAsMap()					//创建ID_Movie对照表
  1055. movieTitle.take(5).foreach(println)				//查看前5条
  1056. movieTitle(146)							//查看某ID电影名称
  1057. //显示推荐电影
  1058. model.recommendProducts(196,5).map(rating => (rating.product,movieTitle(rating.product),rating.rating)).foreach(println))
  1060. //11.10 创建Recommend项目P261
  1061. https://www.cnblogs.com/flymercurial/p/7868606.html
  1062. https://www.jianshu.com/p/61fac2245f1d
  1064. import java.io.File
  1065. import scala.io.Source
  1066. import org.apache.log4j.Logger
  1067. import org.apache.log4j.Level
  1068. import org.apache.spark.SparkConf
  1069. import org.apache.spark.SparkContext
  1070. import org.apache.spark.SparkContext._
  1071. import org.apache.spark.rdd
  1072. import org.apache.spark.mllib.recommendation.{ALS,Rating,MatrixFactorizationModel}
  1073. import scala.collection.immutable.Map
  1074. object recommend {
  1075. 	def PrepareData(): (RDD[Rating], Map[Int, String]) = {
  1076. 		val sc = new SparkContext(new SparkConf().setAppName("Recommend").setMaster("local[4]"))  
  1077. 		print("开始读取用户评分数据中...")
  1078. 		val rawUserData = sc.textFile("file:/home/zieox/workspace/recommend/data/u.data")
  1079. 		val rawRatings = rawUserData.map(_.split("\t").take(3))
  1080. 		val ratingsRDD = rawRatings.map{ case Array(user,movie,rating) => Rating(user.toInt,movie.toInt,rating.toDouble) }
  1081. 		println("共计： "+ratingsRDD.count.toString()+"条 ratings")
  1082. 		print("开始读取电影数据中...")
  1083. 		val itemRDD = sc.textFile("file:/home/zieox/workspace/recommend/data/u.item")
  1084. 		val movieTitle = itemRDD.map(line =>line.split("\\|").take(2)).map(array => (array(0).toInt,array(1))).collect().toMap
  1085. 		val numRatings = ratingsRDD.count()
  1086. 		val numUsers = ratingsRDD.map(_.user).distinct().count()
  1087. 		val numMovies = ratingsRDD.map(_.product).distinct().count()
  1088. 		println("共计： ratings："+numRatings+" User "+numUsers+" Movie "+numMovies)
  1089. 		return (ratingsRDD,movieTitle)
  1090. 	}
  1091. 	def RecommendMovies(model:MatrixFactorizationModel,movieTitle:Map[Int,String],inputUserID:Int) = {
  1092. 		val RecommendMovie = model.recommendProducts(inputUserID,10)
  1093. 		var i = 1
  1094. 		println("针对用户id"+inputUserID+"推荐下列电影：")
  1095. 		RecommendMovie.foreach{
  1096. 			r => println(i.toString() + "." + movieTitle(r.product) + "评分： " + r.rating.toString())
  1097. 			i += 1
  1098. 		}
  1099. 	}
  1100. 	def RecommendUsers(model:MatrixFactorizationModel,movieTitle:Map[Int,String],inputMovieID:Int) = {
  1101. 		val RecommendUser = model.recommendUsers(inputMovieID, 10)
  1102. 		var i = 1
  1103. 		println("针对电影 id" + inputMovieID + "电影名: " + movieTitle(inputMovieID.toInt) + "推荐下列用户id:" )
  1104. 		RecommendUser.foreach{
  1105. 			r => println(i.toString + "用户id：" + r.user + "评分：" + r.rating)
  1106. 			i = i + 1
  1107. 		}
  1108. 	}
  1109. 	def recommend(model:MatrixFactorizationModel,movieTitle:Map[Int,String]) = {
  1110. 		var choose = ""
  1111. 		while (choose != "3") {
  1112. 			print("请选择要推荐的类型 1.针对用户推荐电影 2.针对电影推荐感兴趣的用户 3.离开？")
  1113. 			choose = readLine()
  1114. 			if (choose == "1") {
  1115. 				print("请输入用户id？")
  1116. 				val inputUserID = readLine()
  1117. 				RecommendMovies(model,movieTitle,inputUserID.toInt)
  1118. 			}
  1119. 			else if (choose == "2") {
  1120. 				print("请输入电影的id？")
  1121. 				val inputMovieID = readLine()
  1122. 				RecommendUsers(model,movieTitle,inputMovieID.toInt)
  1123. 			}
  1124. 		}
  1125. 	}
  1126. 	def main(args:Array[String]) {
  1127. 		val (ratings,movieTitle) = PrepareData()
  1128. 		val model = ALS.train(ratings,5,20,0.1)
  1129. 		recommend(model,movieTitle)
  1130. 	}
  1131. }
  1133. //11.15 创建AlsEvaluation.scala 调校推荐引擎参数				http://blog.sina.com.cn/s/blog_1823e4e0f0102x0ov.html
  1134. import java.io.File
  1135. import scala.io.Source
  1136. import org.apache.log4j.{Level, Logger}
  1137. import org.apache.spark.mllib.recommendation.{ALS, MatrixFactorizationModel, Rating}
  1138. import org.apache.spark.rdd.RDD
  1139. import org.apache.spark.{SparkConf, SparkContext}
  1140. import org.joda.time.{DateTime, Duration}
  1141. import org.joda.time._
  1142. import org.joda.time.format._
  1143. import org.jfree.data.category.DefaultCategoryDataset
  1144. import org.apache.spark.mllib.regression.LabeledPoint
  1146. /* Created by Weipengfei on 2017/5/3 0003. ALS过滤算法调校参数 */
  1147. object AlsEvaluation {
  1148.   /*设置日志及乱七八糟的配置*/
  1149.   def SetLogger: Unit ={
  1150.     System.setProperty("hadoop.home.dir", "/usr/local/hadoop")
  1151.     Logger.getLogger("org").setLevel(Level.OFF)
  1152.     Logger.getLogger("com").setLevel(Level.OFF)
  1153.     System.setProperty("spark.ui.showConsoleProgress","false")
  1154.     Logger.getRootLogger.setLevel(Level.OFF)
  1155.   }
  1156.  
  1157.   /*数据准备 @return 返回（训练数据，评估数据，测试数据）*/
  1158.   def PrepareData():(RDD[Rating],RDD[Rating],RDD[Rating])={
  1159.     val sc=new SparkContext(new SparkConf().setAppName("Recommend").setMaster("local[2]").set("spark.testing.memory","21474800000"))
  1160.     //创建用户评分数据
  1161.     print("开始读取用户评分数据中...")
  1162.     val rawUserData=sc.textFile("file:/home/zieox/workspace/recommend/data/u.data")
  1163.     val rawRatings=rawUserData.map(_.split("\t").take(3))
  1164.     val ratingsRDD=rawRatings.map{case Array(user,movie,rating) => Rating( user.toInt ,movie.toInt,rating.toFloat)}
  1165.     println("共计："+ratingsRDD.count().toString+"条评分")
  1166.     //创建电影ID和名称对应表
  1167.     print("开始读取电影数据中...")
  1168.     val itemRDD=sc.textFile("file:/home/zieox/workspace/recommend/data/u.item")
  1169.     val moiveTitle=itemRDD.map(_.split("\\|").take(2)).map(array=>(array(0).toInt,array(1))).collect().toMap
  1170.     //显示数据记录数
  1171.     val numRatings=ratingsRDD.count()
  1172.     val numUser=ratingsRDD.map(_.user).distinct().count()
  1173.     val numMoive=ratingsRDD.map(_.product).distinct().count()
  1174.     println("共计：评分"+numRatings+"条 用户"+numUser+"个 电影"+numMoive+"个")
  1175.     //将数据分为三个部分并且返回
  1176.     print("将数据分为：")
  1177.     val Array(trainData,validationData,testData)=ratingsRDD.randomSplit(Array(0.8,0.1,0.1))
  1178.     println("训练数据："+trainData.count()+"条 评估数据："+validationData.count()+"条 测试数据："+testData.count()+"条")
  1179.     (trainData,validationData,testData)
  1180.   }
  1181.  
  1182.   /*计算RMSE值	*@param model 训练模型	*@param validationData 评估数据	*@return RMSE值 */
  1183.   def computeRmse(model: MatrixFactorizationModel, validationData: RDD[Rating]):(Double) ={
  1184.     val num=validationData.count();
  1185.     val predictedRDD=model.predict(validationData.map(r=>(r.user,r.product)))
  1186.     val predictedAndVali=predictedRDD.map(p=>((p.user,p.product),p.rating)).join(validationData.map(r=>((r.user,r.product),r.rating))).values
  1187.     math.sqrt(predictedAndVali.map(x=>(x._1-x._2)*(x._1-x._2)).reduce(_+_)/num)
  1188.   }
  1189.  
  1190.   /** 训练模型
  1191.     * @param trainData 训练数据
  1192.     * @param validationData 评估数据
  1193.     * @param rank 训练模型参数
  1194.     * @param numIterations 训练模型参数
  1195.     * @param lambda 训练模型参数
  1196.     * @return 模型返回的RMSE（该值越小，误差越小）值，训练模型所需要的时间
  1197.     */
  1198.   def trainModel(trainData: RDD[Rating], validationData: RDD[Rating], rank: Int, numIterations: Int, lambda: Double):(Double,Double)={
  1199.     val startTime=new DateTime()
  1200.     val model=ALS.train(trainData,rank,numIterations,lambda)
  1201.     val endTime=new DateTime()
  1202.     val Rmse=computeRmse(model,validationData)
  1203.     val duration=new Duration(startTime,endTime)
  1204.     println(f"训练参数：rank：$rank= 迭代次数：$numIterations%.2f lambda：$lambda%.2f 结果 Rmse $Rmse%.2f"+" 训练需要时间："+duration.getMillis+"毫秒")
  1205.     (Rmse,duration.getStandardSeconds)
  1206.   }
  1207.  
  1208.   /** 使用jfree.char评估单个参数,这里没有实现
  1209.     * @param trainData 训练数据
  1210.     * @param validationData 评估数据
  1211.     * @param evaluateParameter 评估参数名称
  1212.     * @param rankArray rank参数数组
  1213.     * @param numIterationsArray 迭代次数参数数组
  1214.     * @param lambdaArray lambda参数数组
  1215.     */
  1216.  
  1217.   def evaluateParameter(trainData:RDD[Rating],validationData:RDD[Rating],evaluateParameter:String,rankArray:Array[Int],numIterationsArray:Array[Int],lambdaArray:Array[Double])={
  1218.     val dataBarChart = new DefaultCategoryDataset()
  1219.     val dataLineChart = new DefaultCategoryDataset()
  1220.     for(rank <- rankArray;numIterations <- numIterationsArray;lambda <- lambdaArray){
  1221.       val (rmse,time) = trainModel(trainData,validationData,rank,numIterations,lambda)
  1222.       val parameterData = evaluateParameter match{
  1223.         case "rank" => rank;
  1224.         case "numIterations" => numIterations;
  1225.         case "lambda" => lambda
  1226.       }
  1227.       dataBarChart.addValue(rmse,evaluateParameter,parameterData.toString())
  1228.       dataLineChart.addValue(time,"Time",parameterData.toString())
  1229.     }
  1230.     Chart.plotBarLineChart("ALS evaluations " + evaluateParameter,evaluateParameter,"RMSE",0.58,5,"Time",dataBarChart,dataLineChart)
  1231.   }
  1232.  
  1233.   /*
  1234.     * 三个参数交叉评估，找出最好的参数组合
  1235.     * @param trainData 训练数据
  1236.     * @param validationData 评估数据
  1237.     * @param rankArray rank参数数组
  1238.     * @param numIterationsArray 迭代次数参数数组
  1239.     * @param lambdaArray lambda参数数组
  1240.     * @return 返回由最好参数组合训练出的模型
  1241.     */
  1242.  
  1243.   def evaluateAllParameter(trainData:RDD[Rating],validationData:RDD[Rating],rankArray:Array[Int],numIterationsArray:Array[Int],lambdaArray:Array[Double]): MatrixFactorizationModel = {
  1244.     val evaluations=for(rank <- rankArray;numIterations <- numIterationsArray;lambda <- lambdaArray) yield {
  1245.       val (rmse,time)=trainModel(trainData,validationData,rank,numIterations,lambda)
  1246.       (rank,numIterations,lambda,rmse)
  1247.     }
  1248.     val Eval=(evaluations.sortBy(_._4))
  1249.     val bestEval=Eval(0)
  1250.     println("最佳模型参数：rank："+bestEval._1+" 迭代次数："+bestEval._2+" lambda:"+bestEval._3+" 结果rmse："+bestEval._4)
  1251.     val bestModel=ALS.train(trainData,bestEval._1,bestEval._2,bestEval._3)
  1252.     (bestModel)
  1253.   }
  1254.  
  1255.   /*训练评估	*@param trainData 训练数据	*@param validationData 评估数据		*@return 返回一个最理想的模型 */
  1256.   def trainValidation(trainData:RDD[Rating], validationData:RDD[Rating]):MatrixFactorizationModel={
  1257.     println("------评估rank参数使用------")
  1258.     evaluateParameter(trainData,validationData,"rank",Array(5,10,15,20,50,100),Array(10),Array(0.1))
  1259.     println("------评估numIterations------")
  1260.     evaluateParameter(trainData,validationData,"numIterations",Array(10),Array(5,10,15,2,25),Array(0.1))
  1261.     println("------评估lambda------")
  1262.     evaluateParameter(trainData,validationData,"lambda",Array(10),Array(10),Array(0.05,0.1,1,5,10.0))
  1263.     println("------所有参数交叉评估找出最好的参数组合------")
  1264.     val bestmodel=evaluateAllParameter(trainData,validationData,Array(5,10,15,20,50,100),Array(5,10,15,20,25),Array(0.01,0.05,0.1,1,5,10.0))
  1265.     bestmodel
  1266.   }
  1267.  
  1268.   def main(args: Array[String]) {
  1269.     SetLogger
  1270.     println("========================数据准备阶段========================")
  1271.     val (trainData,validationData,testData)=PrepareData()
  1272.     trainData.persist();validationData.persist();testData.persist()
  1273.     println("========================训练验证阶段========================")
  1274.     val bestModel=trainValidation(trainData,validationData)
  1275.     println("======================测试阶段===========================")
  1276.     val testRmse=computeRmse(bestModel,testData)
  1277.     println("使用bestModel测试testData，结果rmse="+testRmse)
  1278.   }
  1279. }
  1280.  
  1281.  
  1282.  
  1283. import org.jfree.chart._
  1284. import org.jfree.data.xy._
  1285. import org.jfree.data.category.DefaultCategoryDataset
  1286. import org.jfree.chart.axis.NumberAxis
  1287. import org.jfree.chart.axis._
  1288. import java.awt.Color
  1289. import org.jfree.chart.renderer.category.LineAndShapeRenderer;
  1290. import org.jfree.chart.plot.DatasetRenderingOrder;
  1291. import org.jfree.chart.labels.StandardCategoryToolTipGenerator;
  1292. import java.awt.BasicStroke
  1293.  
  1294. object Chart {
  1295.   def plotBarLineChart(Title: String, xLabel: String, yBarLabel: String, yBarMin: Double, yBarMax: Double, yLineLabel: String, dataBarChart : DefaultCategoryDataset, dataLineChart: DefaultCategoryDataset): Unit = {
  1296.  
  1297.     //画出Bar Chart
  1298.     val chart = ChartFactory
  1299.       .createBarChart(
  1300.         "", // Bar Chart 标题
  1301.         xLabel, // X轴标题
  1302.         yBarLabel, // Bar Chart 标题 y轴标题l
  1303.         dataBarChart , // Bar Chart数据
  1304.         org.jfree.chart.plot.PlotOrientation.VERTICAL,//画图方向垂直
  1305.         true, // 包含 legend
  1306.         true, // 显示tooltips
  1307.         false // 不要URL generator
  1308.       );
  1309.     //取得plot
  1310.     val plot = chart.getCategoryPlot();
  1311.     plot.setBackgroundPaint(new Color(0xEE, 0xEE, 0xFF));
  1312.     plot.setDomainAxisLocation(AxisLocation.BOTTOM_OR_RIGHT);
  1313.     plot.setDataset(1, dataLineChart); plot.mapDatasetToRangeAxis(1, 1)
  1314.     //画直方图y轴
  1315.     val vn = plot.getRangeAxis(); vn.setRange(yBarMin, yBarMax);  vn.setAutoTickUnitSelection(true)
  1316.     //画折线图y轴
  1317.     val axis2 = new NumberAxis(yLineLabel); plot.setRangeAxis(1, axis2);
  1318.     val renderer2 = new LineAndShapeRenderer()
  1319.     renderer2.setToolTipGenerator(new StandardCategoryToolTipGenerator());
  1320.     //设置先画直方图,再画折线图以免折线图被盖掉
  1321.     plot.setRenderer(1, renderer2);plot.setDatasetRenderingOrder(DatasetRenderingOrder.FORWARD);
  1322.     //创建画框
  1323.     val frame = new ChartFrame(Title,chart); frame.setSize(500, 500);
  1324.     frame.pack(); frame.setVisible(true)
  1325.   }
  1326. }
  1327. --------------------------------------------------------------------------
  1328. 最佳模型参数：rank：5 迭代次数：15 lambda:0.1 结果rmse：0.9178637704570859
  1329. ======================测试阶段===========================
  1330. 使用bestModel测试testData，结果rmse=0.9177717267204295
  1331. --------------------------------------------------------------------------
  1332.  
  1333.  
  1334. ----------------------------------------------------------CP12 StumbleUpon数据集
  1335. https://www.kaggle.com/c/stumbleupon/data
  1336.  
  1337. ----------------------------------------------------------CP13 决策树二元分类
  1338. object RunDecisionTreeBinary{
  1339. 	def main(args:Array[String]):Unit={
  1340. 		SetLogger()
  1341. 		val sc = new SparkContext(new SparkConf().setAppName("DecisionTreeBinary").setMaster("local[4]"))
  1342. 		println("==============preparing===============")
  1343. 		val (trainData,validationData,testData,categoriesMap)=PrepareData(sc)
  1344. 		trainDate.persist();validationData.persist();testData.persist();
  1345. 		println("==============evaluating==============")
  1346. 		val model=trainEvaluate(trainDate,validationData)
  1347. 		println("===============testing================")
  1348. 		val auc = evaluateModel(model,testData)
  1349. 		println("使用testdata测试的最佳模型，结果 AUC："+auc)
  1350. 		println("============predicted data=============")
  1351. 		PredictData(sc,model,categoriesMap)
  1352. 		trainDate.Unpersist();validationData.Unpersist();testData.Unpersist();
  1353. 	}
  1354. }
  1355.  
  1356. def PrepareData(sc:SparkContext):(RDD[LabeledPoint],RDD[LabeledPoint],RDD[LabeledPoint],Map[String,Int])={
  1357. 	//1.导入并转换数据
  1358. 	print("开始导入数据...")
  1359. 	val rawDataWithHeader=sc.textFile("data/train.tsv")
  1360. 	val rawData=rawDataWithHeader.mapPartitionsWithIndex{(idx,iter)=> if (idx==0) iter.drop(1) else iter} //删除第一行表头
  1361. 	val lines=rawData.map(_.split("\t"))	//读取每一行的数据字段
  1362. 	println("共计："+lines.count.toString()+"条")
  1363. 	//2.创建训练评估所需数据的RDD[LabeledPoint]
  1364. 	val categoriesMap = lines.map(fields => fields(3)).distinct.collect.zipWithIndex.toMap //创建网页分类对照表
  1365. 	val LabeledPointRDD=lines.map{fields => 
  1366. 		val trFields=fields.map(_.replaceAll("\"",""))
  1367. 		val categoryFeaturesArray=Array.ofDim[Double](categoriesMap.size)
  1368. 		val categoryIdx=categoriesMap(fields(3))
  1369. 		categoryFeaturesArray(categoryIdx)=1
  1370. 		val numericalFeatures=trFields.slice(4,fields.size-1).map(d=>if (d=="?") 0.0 else d.toDouble)
  1371. 		val label = trFields(fields.size-1).toInt
  1372. 		LabeledPoint(label,Vectors.dense(categoryFeaturesArray++numericalFeatures))
  1373. 	}
  1374. 	//3.用随机方式将数据分为3个部分并返回
  1375. 	val Array(trainData,validationData,testData) = LabeledPointRDD.randomSplit(Array(0.8,0.1,0.1))
  1376. 	println("将数据分trainData: "+trainData.count() + " validationData:"+validationData.count()+" testData:"+testData.count())
  1377. 	return (trainData,validationData,testData,categoriesMap)
  1378. 	//训练评估阶段
  1379. 	def trainEvaluate(trainData:RDD[LabeledPoint],validationData:RDD[LabeledPoint]):DecisionTreeModel = {
  1380. 		print("开始训练...")
  1381. 		val (model,time) = trainModel(trainData,"entropy",10,10)
  1382. 		println("训练完成，所需时间："+time+" 毫秒")
  1383. 		val AUC = evaluateModel(model,validationData)
  1384. 		println("评估结果 AUC = "+AUC)
  1385. 		return (model)
  1386. 	}
  1387. 	//训练DecisionTree模型Model
  1388. 	def trainModel(trainData:RDD[LabeledPoint],impurity:String,maxDepth:Int,maxBins:Int):(DescisionTreeModel,double)={
  1389. 		val startTime = new DateTime()
  1390. 		val model = DecisionTree.trainClassifier(trainData,2,Map[Int,Int](),impurity,maxDepth,maxBins)
  1391. 		val endTime=new DateTime()
  1392. 		val duration=new Duration(startTime,endTime)
  1393. 		(model,duration.getMillis())
  1394. 	}
  1395. 	//评估模型
  1396. 	def evaluateModel(model:DecisionTreeModel,validationData:RDD[LabeledPoint]):(Double)={
  1397. 		val secoundAndLabels=validationData.map{data=>
  1398. 			var predict=model.predict(data.features)
  1399. 			(predict,data.label)
  1400. 		}
  1401. 		val Metrics = new BinaryClassificationMetrics(scoreAndLabels)
  1402. 		val AUC = Metrics.areaUnderROC
  1403. 		(AUC)
  1404. 	}	
  1405. 	//预测阶段
  1406. 	def PredictData(sc:SparkContext,model:DecisionTreeModel,categoriesMap:Map[String,Int]):Unit = {
  1407. 		//1.导入并转换数据
  1408. 		val rawDataWithHeader=sc.textFile("data/test.tsv")
  1409. 		val rawData = rawDataWithHeader.mapPartitionsWithIndex{ (idx,iter)=>if (idx==0) iter.drop(1) else iter}
  1410. 		val line = rawData.map(_.split("/t"))
  1411. 		println("共计： "+lines.count.toString()+" 条")
  1412. 		//2.创建训练评估所需数据RDD[LabeledPoint]
  1413. 		val dataRDD=lines.take(20).map{fields =>
  1414. 			val trFields = fields.map(_.replaceAll("\"",""))
  1415. 			val categoryFeaturesArray=Array.ofDim[Double](categoriesMap.size)
  1416. 			val categoryIdx=categoriesMap(fields(3))
  1417. 			categoryFeaturesArray(categoryIdx)=1
  1418. 			val numericalFeatures=trFields.slice(4,fields.size-1).map(d=>if (d=="?") 0.0 else d.toDouble)
  1419. 			val label = 0
  1420. 		//3.进行预测
  1421. 			val url = trFields(0)
  1422. 			val Features = Vectors.dense(categoryFeaturesArray++numericalFeatures)
  1423. 			val predictDesc = {predict match{ case 0 =>"暂时性网页"case 1 => "常青网页"}}
  1424. 			println("网址： "+url+"==>预测："+predictDesc)
  1425. 		}
  1426. 	}
  1427. 	//评估模型
  1428. 	def evaluateModel(model:DecisionTreeModel,validationData:RDD[LabeledPoint]):(Double)={
  1429. 		val scoreAndLabels = validationData.map{data=>
  1430. 			var predict = model.predict(data.features)
  1431. 			(predict,data.label)
  1432. 		}
  1433. 		val Metrics = new BinaryClassificationMetrics(scoreAndLabels)
  1434. 		val AUC = Metrics.areaUnderROC
  1435. 		(AUC)
  1436. 	}
  1437. //预测模型
  1438. def PredictData(sc:SparkContext,model:DecisionTreeModel,categoriesMap:Map[String,Int]):Unit={	
  1439. 	//1.导入并转换数据
  1440. 	val rawDataWithHeader = sc.textFile("data/train.tsv")
  1441. 	val rawData = rawDataWithHeader.mapPartitionsWithIndex{(idx,iter)=> if (idx == 0) iter.drop(1) else iter}
  1442. 	val lines = rawData.map(_.split("\t"))
  1443. 	println("共计："+lines.count.toString()+"条")
  1444. 	//2.创建训练评估所需的数据RDD[LabeledPoint]
  1445. 	val dataRDD = lines.take(20).map{ fields =>
  1446. 		val trFields=fields.map(_.replaceAll("\"",""))
  1447. 		val categoryFeaturesArray=Array.ofDim[Double](categoriesMap.size)
  1448. 		val categoryIdx=categoriesMap(fields(3))
  1449. 		categoryFeaturesArray(categoryIdx)=1
  1450. 		val numericalFeatures=trFields.slice(4,fields.size).map(d=>if (d=="?") 0.0 else d.toDouble)
  1451. 		val label = 0
  1452. 		//3.进行预测
  1453. 		val url = trFields(0)
  1454. 		val Features = Vectors.dense(categoryFeaturesArray++numericalFeatures)
  1455. 		val predict = model.predict(Features).toInt
  1456. 		val predictDesc = {predict match{ case 0 =>"暂时性网页"case 1 => "常青网页"}}
  1457. 		println("网址： "+url+"==>预测："+predictDesc)
  1458. 	}
  1459. }
  1460. //parametersTunning 参数调校函数
  1461. def parametersTunning(trainData:RDD[LabeledPoint],validationData:RDD[LabeledPoint]):DecisionTreeModel={
  1462. 	println("-------评估Impurity参数使用gini,entropy-------")
  1463. 	evaluateParameter(trainDta,validationData,"impurity",Array("gini","entropy"),Array(10),Array(10))
  1464. 	println("-------评估MaxDepth参数使用(3,5,10,15,20)-------")
  1465. 	evaluateParameter(trainDta,validationData,"MaxDepth",Array("gini"),Array(3,5,10,15,20),Array(10))
  1466. 	println("-------所有参数交叉评估找出最好的参数组合-------")
  1467. 	evaluateParameter(trainDta,validationData,,Array("gini","entropy"),Array(3,5,10,15,20))
  1468. 	return (bestModel)
  1469. }
  1470. 	
  1471. //evaluateParameter 函数
  1472. def evaluateParameter(trainData:RDD[LabeledPoint],validationData:RDD[LabeledPoint],evaluateParameter:String,impurityArray:Array[String],maxdepthArray:Array[Int],maxBinsArray:Array[Int])={
  1473. 	var dataBarChart = new DefaultCategoryDataset()
  1474. 	var dataLineChart = new DefaultCategoryDataset()
  1475. 	for (impurity <- impurityArray; maxDepth <- maxdepthArray; maxBins <- maxBinsArray){
  1476. 		val (model,time)=trainModel(trainData,impurity,maxDepth,maxBins)
  1477. 		val auc = evaluateModel(model,validationData)
  1478. 		val parameterData = evaluateParameter match{
  1479. 			case "impurity"=>impurity;
  1480. 			case "maxDepth"=>maxDepth;
  1481. 			case "maxBins"=>maxBins
  1482. 		}
  1483. 		dataBarChart.addValue(auc,parameterData.toString())
  1484. 		dataLineChart.addValue(time,"Time",parameterData.toString())
  1485. 	}	
  1486. 	Chart.plotBarLineChart("DecisionTree evaluations"+evaluateParameter,evaluateParameter,"AUC",0.58,0.7,"Time",dataBarChart,dataLineLineChart)
  1487. }	
  1488.  
  1489. def evaluateAllParameter(trainData:RDD[LabeledPoint],validationData:RDD[LabeledPoint],impurityArray:Array[String],maxdepthArray:Array[Int],maxBinsArray:Array[Int]): DecisionTreeModel = { 
  1490. 	val evaluationsArray = 
  1491. 		for (impurity<-impurityArray;maxdepth<-maxdepthArray;maxBins<-maxBinsArray) yield {	
  1492. 			val (model,time)=trainModel(trainData,impurity,maxDepth,maxBins)
  1493. 			val auc = evaluateModel(model,validationData)
  1494. 			(impurity,maxdepth,maxBins,auc)
  1495. 		}
  1496. 	val BestEval = (evaluationsArray.sortBy(_._4).reverse)(0)
  1497. 	println("调校后最佳参数：impurity:"+BestEval._1+" ,maxDepth:"+BestEval._2+" ,maxBins:"+BestEval._3)
  1498. 	return bestModel
  1499. }
  1500.  
  1501. 	
  1502.  
  1503.  
  1504.  
  1505.  
  1506.  
  1507.  
  1508.  
  1509.  
  1510.  
  1511.  
  1512.  
  1513.  
  1514.  
  1515. ----------------------------------------------------------CP18 决策树回归分析
  1516. mkdir -p ~/workspace/Classfication/data
  1517. cd ~/workspace/
  1518. //Bike_Sharing数据
  1519. wget https://archive.ics.uci.edu/ml/machine-learning-databases/00275/Bike-Sharing-Dataset.zip
  1520. unzip -j Bike-Sharing-Dataset.zip
  1521.  
  1522.  
  1523.  
  1524.  
  1525.  
  1526.  
  1527.  
  1528.  
  1529.  
  1530.  
  1531.  
  1532.  
  1533.  
  1534.  
  1535.  
  1536.  
  1537.  
  1538.  
  1539.  
  1540.  
  1541.  
  1542.  
  1543.  
  1544.  
  1545.  
  1546.  
  1547.  
  1548.  
  1549.  
  1550. ----------------------------------------------------------CP19 使用 Apache Zeppelin 数据可视化
  1551. http://zeppelin.incubator.apache.org/
  1552. http://zeppelin.apache.org/
  1553. https://www.cnblogs.com/purstar/p/6294412.html
  1554.  
  1555. cd ~/下载
  1556. tar zxvf zeppelin-0.8.1-bin-all.tgz
  1557. sudo mv zeppelin-0.8.1-bin-all /usr/local/zeppelin
  1558. sudo chown -R zieox:zieox /usr/local/zeppelin
  1559. cd /usr/local/zeppelin/conf/
  1560. cp zeppelin-env.sh.template zeppelin-env.sh
  1561. cp zeppelin-site.xml.template zeppelin-site.xml
  1562. sudo gedit /usr/local/zeppelin/conf/zeppelin-env.sh
  1563. //配置zeppelin
  1564. ----------------------------------------------------------------------------
  1565. export JAVA_HOME=/usr/lib/jdk/jdk-8
  1566. export SPARK_HOME=/usr/local/spark
  1567. export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
  1568. export ZEPPELIN_INTP_JAVA_OPTS="-XX:PermSize=512M -XX:MaxPermSize=1024M"
  1569. ----------------------------------------------------------------------------
  1570. export SPARK_MASTER_IP=192.168.56.100
  1571. export SPARK_LOCAL_IP=192.168.56.100
  1572. ----------------------------------------------------------------------------
  1573. 在maven找到
  1574. cp jackson-annotations-2.4.4.jar /usr/local/zeppelin
  1575. cp jackson-core-2.4.4.jar /usr/local/zeppelin
  1576. cp jackson-databind-2.4.4.jar /usr/local/zeppelin
  1577. 替换
  1578. ./lib/jackson-annotations-2.5.0.jar
  1579. ./lib/jackson-core-2.5.3.jar
  1580. ./lib/jackson-databind-2.5.3.jar
  1581.  
  1582. //将这2个文件移至/usr/local/zeppelin/lib
  1583. mv jackson-annotations-2.4.4.jar jackson-core-2.4.4.jar jackson-databind-2.4.4.jar -t /usr/local/zeppelin/lib
  1584.  
  1585.  
  1586.  
  1587.  
  1588. //启动zeppelin					/usr/local/zeppelin/bin/zeppelin-daemon.sh start
  1589. //登录zeppelin Web_UI界面			http://master:8181/#/
  1590.  
  1591. sudo mkdir -p /workspace/zeppelin/data
  1592. sudo mv ~/workspace/recommend/data/ml-100k.zip ml-100k.zip
  1593. sudo unzip -j ml-100k
  1594.  
  1595. //zeppelin 命令集
  1596. -------------------------------
  1597. %sh
  1598. ls -l /workspace/zeppelin/data
  1599. %sh
  1600. sudo head /workspace/zeppelin/data/u.user
  1601. -------------------------------
  1602. val usertext = sc.textFile("file:/home/zieox/workspace/recommend/data/u.user")
  1603. case class usertable(id:String,age:String,gender:String,occupation:String,zipcode:String)
  1604. val userRDD = usertext.map(s=>s.split("\\|")).map(s=>usertable(s(0),s(1),s(2),s(3),s(4)))
  1605. userRDD.toDF().registerTempTable("usertable")
  1606. println("input: "+userRDD.count+" s")
  1607.  
  1608. //在子目录需找名带jackson的文件				find .|grep jackson
  1609.  
  1610.  
  1611.  
  1612.  
  1613.  
  1614.  
  1615.  
  1616.  
  1617.  
  1618.  
  1619.  
  1620.  
  1621.  
  1622.  
  1623.  
  1624.  
  1625.  
  1626.  
  1627.  
  1628.  
  1629.  
  1630.  
  1631.  
  1632.  
  1633.  
  1634.  
  1635.  
  1636.  
  1637. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++CP10构建Hadoop集群 P280
  1638.  
  1639. //安装hadoop
  1640.  
  1641. 1. install java 安装java
  1642. 2. linux account可通linux帐号
  1643. 3. install Hadoop安装hadoop
  1644. 4. Set SSH设置无密匙SSH
  1645. 5. set Hadoop配置hadoop
  1646. 6. 格式化HDFS:hdfs namenode -format
  1647. 7. 启动和停止守护进程:
  1648. 启动守护进程：start-dfs.sh
  1649. 找到namenode主机名：hdfs getconf -namenodes / hdfs getconf -secondarynamenodes
  1650. 启动YARN守护进程：start-yarn.sh/stop-yarn.sh
  1651. 启动MapReduce守护进程：mr-jobhistory-daemon.sh start historyserver
  1652. 8. 创建用户目录：
  1653. hadoop fs -mkdir /user/username
  1654. hadoop fs -chown username:username /user/username
  1655. 设置容量空间：hdfs dfsadmin -setSpaceQuota 1T /user/us
  1656. hadoop配置：https://www.cnblogs.com/yinghun/p/6230436.html
  1657.  
  1658. //10.3.3 Hadoop守护进程的关键属性
  1659. 1.HDFS
  1660. dfs.datenode.date.dir 设定datenode存储数据块的目录列表
  1661. dfs.datenode.name.dir 编辑日志和文件系统映像 （支持namenode进行冗余备份）
  1662. dfs.datenode.checkpoint.dir 指定辅助文件系统的检查点目录
  1663. p295 HDFS守护进程关键属性
  1664. p297 Yarn守护进程关键属性
  1665. 2. Yarn
  1666. yarn.nodemanager.resource.memory-mb 设置内存分配量
  1667. hadoop -fs -expunge//删除已在回收站中超过最小时限的所有文件
  1668. //10.5 hadoop 基准测评程序 测试hadoop程序
  1669. p312 //使用TeraSort测评HDFS （基准测评）
  1670. //关于其他基准测评程序
  1671. TestSFSIO主要用于测试HDFS的I/O性能
  1672. MRBench检验小型作业能否快速响应
  1673. NNBench测试namenode硬件的加载过程
  1674. Gridmix基准评测程序套装
  1675. SWIM用来为被测系统生成代表性的测试负载
  1676. TPCx-HS基于TeraSort的标准测评程序
  1677.  
  1678. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++CP11 管理hadoop
  1679.  
  1680. //进入和离开安全模式
  1681. hdfs dfsadmin [-safemode enter | leave | get | wait]
  1682. get：查看是否处于安全模式
  1683. wait：执行某条命令之前先退出安全模式
  1684. enter：进入安全模式
  1685. leave：离开安全模式
  1686. //启动审计日志
  1687. gedit hadoop-env.sh
  1688. export HDFS_AUDIT_LOGGER="INFO,RFAAUDIT"
  1689. //tools
  1690. //1.dfsadmin 命令
  1691. //Hadoop常用命令：
  1692. https://blog.csdn.net/suixinlun/article/details/81630902
  1693. https://blog.csdn.net/m0_38003171/article/details/79086780
  1694.  
  1695. 2.文件系统检查fsck工具hadoop fsck /
  1696. 3.datanode块扫描器：各个datanode运行一个块扫描器，定期检测节点上的所有块 dfs.datanode.scan.period.hopurs设置
  1697. 4.均衡器 balancer
  1698. 启动均衡器：start-balancer.sh
  1699.  
  1700. //设置日志级别
  1701. hadoop daemonlog -setlevel resource-manager-host:8088 \ org.apache.hadoop.yarn.server.resourcemanager DEBUG
  1702.  
  1703. //通过修改hadoop-env.sh配置 HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote.port=8804" 来使远程监控得以访问
  1704. //维护hadoop
  1705. 1.元数据备份:hdfs dfsadmin -fetchImage fsimage.backup
  1706. 2.数据备份：distcp
  1707. 3.文件系统检查：fsck
  1708. 4.文件系统均衡器
  1709. p332
  1710. //关于委任和解除节点
  1711. //1.委任新节点
  1712. 配置hdfs-site.xml指向namemode -> 配置 yarn-site.xml 文件，指向资源管理器 -> 启动datanode和资源管理器守护进程
  1713.  
  1714. //添加新节点步骤
  1715. 1. 将新节点IP地址添加到include文件中
  1716. 2. 运行 hdfs dfsadmin -refreshNodes，将审核过的一系列datanode集合更新至namenode信息
  1717. 3. 运行 yarn rmadmin -refreshNodes,将审核过的一系列节点管理器信息更新至资源管理器
  1718. 4. 以新节点更新slaves文件
  1719. 5. 启动新的datanode和节点管理器
  1720. 6. 检查新的datanode和节点管理器是否都出现在网页界面中
  1721.  
  1722. //2. 解除旧节点
  1723. 解除节点由exclude文件控制
  1724.  
  1725. //从集群中移除节点的步骤如下
  1726. 1. 将待解除的网络地址添加岛exclude文件中，不更新include文件
  1727. 2. 执行 hdfs dfsadmin -refreshNodes，使用一组新的审核过的datanode来更新namenode设置
  1728. 3. 执行 yarn rmsadmin -refreshNodes，使用一组新的审核过的节点管理器来更新资源管理器设置
  1729. 4. 转到网页界面，查看待解除datanode的管理状态是否已经变成“正在解除(Decommission In Progress)”
  1730. 5. 当所有datanode的状态变为“解除完毕”(Decommissioned)时，表明所有块都已经复制完毕
  1731. 6. 从include文件中移除这些节点
  1732. hdfs dfsadmin -refreshNodes
  1733. yarn rmsadmin -refreshNodes
  1734.  
  1735. 7. 从slaves文件中移除节点
  1736. //hadoop的升级
  1737. steps P334
  1738. start-dfs.sh -upgrade ...
  1739.  
  1740. //为python安装avro
  1741. easy_install avro
  1742. python3 ch-12-avro/src/main/py/write_pairs.py pairs.avro
  1743.  
  1744. //pig 宏
  1745. DEFINE max_by_group(X,groupby_key,max_field) RETURNS Y{
  1746. A = GROUP $X BY $group_key;
  1747. $Y=FOREACH A GENERATE group ,MAX($X.$max_field);
  1748. }
  1749.  
  1750. record= LOAD "lr/text.txt" AS (year:chararray,temperature:int,quality:int);
  1751. filter_records = FILTER records BY temperature != 9999 and quality IN(1,2,3)
  1752. macro_max_by_group_A = GROUP filtered_records by (year);
  1753. max_tmp = FOREACH macro_max_by_group_A GENERATE group,
  1754. MAX(filter_records.(temperature));
  1755. DUMP max_tmp
  1756. import "./max_tmp.macro"
  1757.  
  1758. ----------------------------------------------------------------------
  1759. linux系统进程 ：gnome-system-monitor
  1760. //Parquet 是什么：
  1761. http://www.zhimengzhe.com/shujuku/other/21763.html
  1762. ---------------------------------------------------------------------- with hive
  1763. 查看linux版本：lsb_release -a
  1764. ----------------------------------------------------------------------hadoop with spark
  1765.  
  1766. sql中提取表
  1767. https://gitlab.com/matteo.redaelli/sqlu/-/blob/master/sqlu/__init__.py
  1768. gsp
  1769. http://www.sqlparser.com/download.php
  1770.  
  1771.  
  1772.  
  1773. //20200924 
  1774. //IDEA git 添加项目
  1775. --ubuntu-
  1776. --ssh
  1777. ssh-keygen -t rsa	=>	cd ~/.ssh	=>	cat ~/.ssh/id_rsa.pub	=> 	<copytoGitLab>
  1778. idea - vcs - clone - >
  1779.  
  1780. //git发布流程
  1781. --dev
  1782. git拉取(pull) 
  1783. =>文件放至git上的指定文件夹 
  1784. =>git再次拉取
  1785. =>git提交(commit)
  1786. =>git同步(merge into current)
  1787. master文件夹同步：git拉取=>git合并=>git同步
  1788.  
  1789. git pull 
  1790. new file
  1791. pull
  1792. commit
  1793. git-push

声明：本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号