hadoop集群spark配置，jupyter notebook，ubuntu_ubuntu jupyter spark 集群

• 下载安装jupyter notebook，并配置，详见另一篇博客，jupyter notebook配置
• 安装python包 findspark、pyspark。
• 用findspark.init函数配置集群中spark-client2路径，python路径。

import findspark

findspark.init(spark_home="/usr/hdp/current/spark2-client/",python_path="/usr/bin/python3")

from pyspark import SparkConf,SparkContext,SQLContext
1
2
3
4
5

• 导入sparkcontext

conf = SparkConf().setMaster("yarn").setAppName("http")

sc = SparkContext(conf = conf)

sqlContext = SQLContext(sc)
1
2
3
4
5

• 读取HDFS中的CSV文件

df2 = sqlContext.read.schema(schema).option("header", "false").option("delimiter","\t").option("mode", "DROPMALFORMED").csv("/s_tmp/100k_rows.csv")
1