赞
踩
https://yixuan.cos.name/cn/2015/04/spark-beginner-1/
1.读取文件
import scala.io.Source object myfirst { //titanic,LR def main(args: Array[String]) { val data= Source.fromFile("D:\\IDEA\\_01\\train.csv") data.foreach(print) } }
import org.apache.spark.{SparkContext, SparkConf} object myfirst{ //titanic,LR def main(args: Array[String]) { val conf=new SparkConf().setAppName("regression").setMaster("local[1]") val sc=new SparkContext(conf) val lines = { sc.textFile("D:\\IDEA\\_01\\train.csv") } //lines.foreach(print) } }
2.数据预处理
2.1数据结构
spark MLlib之数据类型http://uohzoaix.github.io/studies/2014/10/14/sparkMLlib(1)/
dense和sparse矩阵
LabeldPoint
import org.apache.spark.mllib.linalg.Vectorsimport org.apache.spark.mllib.regression.LabeledPoint
// Create a labeled point with a positive label and a dense feature vector.
val pos = LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0))
// Create a labeled point with a negative label and a sparse feature vector.
val neg = LabeledPoint(0.0, Vectors.sparse(3, Array(0, 2), Array(1.0, 3.0)))
2.2特征提取
tfidf
word2vec
countvectorizer
2.3特征转换
https://my.oschina.net/hblt147/blog/1519465
3.建模
模型选择
4.模型训练
5.模型优化
6.模型优化
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。