spark代码sc统一配置_spark sc

之前写了两篇关于mlib的应用，其实都是在一个project内的，在创建sc的过程中，可以通过继承统一的trait简化sc的创建。

val postgprop = new Properties
  val path = this.getClass.getResourceAsStream("/config.properties")
1
2

读取相关配置文件

key.name=value
jdbcURL=jdbc:mysql://**********:3306/******
recResultTable_fpGrowth=ods_fpgrowth_ds
recResultTable_als=ods_commender_ds
mysqlusername=*******
mysqlpassword=*******
localClusterURL=local[2]
clusterMasterURL=spark://namenode:7077
mode=spark://namenode:7077
minSupport=0.2
minConfidence=0.8
numPartitions=2
1
2
3
4
5
6
7
8
9
10
11
12

通过读取的配置文件的内容，创建sc

/配置sparkconf
  val mode = postgprop.getProperty("mode")
  val conf = new SparkConf().setMaster(mode)
  val sc = new SparkContext(conf)
  val sqlContext = new SQLContext(sc)
1
2
3
4
5

可以在里边直接创建sqlcontext
mysql的相关配置也可以直接在trait里边写

 //jdbc连接
  val jdbcURL = postgprop.getProperty("jdbcURL")
  val recResultTable_fpGrowth = postgprop.getProperty("recResultTable_fpGrowth")
  val recResultTable_als = postgprop.getProperty("recResultTable_als")
  val mysqlusername = postgprop.getProperty("mysqlusername")
  val mysqlpassword = postgprop.getProperty("mysqlpassword")
  val prop = new Properties
  prop.put("driver", "com.mysql.jdbc.Driver")
  prop.put("user", mysqlusername)
  prop.put("password", mysqlpassword)
1
2
3
4
5
6
7
8
9
10