devbox
羊村懒王
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

spark XGBoost算法demo_import ml.dmlc.xgboost4j.scala.spark.xgboost demo

作者：羊村懒王 | 2024-02-16 22:50:46

赞

踩

import ml.dmlc.xgboost4j.scala.spark.xgboost demo

1.运行环境配置

     该算法需要运行Linux环境下，运行的版本为：spark2.4.0，scala 2.11

2.maven配置

  1.  
  2.     <properties>
  3.         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  4.         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  5.         <scala.version>2.11</scala.version>
  6.         <spark.version>2.4.0</spark.version>
  7.     </properties>
  8.     <dependencies>
  9.         <dependency>
  10.             <groupId>org.apache.spark</groupId>
  11.             <artifactId>spark-sql_${scala.version}</artifactId>
  12.             <version>${spark.version}</version>
  13.         </dependency>
  14.         <dependency>
  15.             <groupId>org.apache.spark</groupId>
  16.             <artifactId>spark-core_${scala.version}</artifactId>
  17.             <version>${spark.version}</version>
  18.         </dependency>
  19.         <dependency>
  20.             <groupId>org.apache.spark</groupId>
  21.             <artifactId>spark-mllib_${scala.version}</artifactId>
  22.             <version>${spark.version}</version>
  23.         </dependency>
  24.         <dependency>
  25.             <groupId>ml.dmlc</groupId>
  26.             <artifactId>xgboost4j</artifactId>
  27.             <version>0.72</version>
  28.         </dependency>
  29.         <dependency>
  30.             <groupId>ml.dmlc</groupId>
  31.             <artifactId>xgboost4j-spark</artifactId>
  32.             <version>0.72</version>
  33.         </dependency>
  34.     </dependencies>
  35.     <build>
  36.         <plugins>
  37.             <plugin>
  38.                 <groupId>org.scala-tools</groupId>
  39.                 <artifactId>maven-scala-plugin</artifactId>
  40.                 <version>2.15.2</version>
  41.                 <executions>
  42.                     <execution>
  43.                         <goals>
  44.                             <goal>compile</goal>
  45.                             <goal>testCompile</goal>
  46.                         </goals>
  47.                     </execution>
  48.                 </executions>
  49.             </plugin>
  50.  
  51.             <plugin>
  52.                 <artifactId>maven-compiler-plugin</artifactId>
  53.                 <version>3.6.0</version>
  54.                 <configuration>
  55.                     <source>1.8</source>
  56.                     <target>1.8</target>
  57.                 </configuration>
  58.             </plugin>
  59.  
  60.             <plugin>
  61.                 <groupId>org.apache.maven.plugins</groupId>
  62.                 <artifactId>maven-surefire-plugin</artifactId>
  63.                 <version>2.19</version>
  64.                 <configuration>
  65.                     <skip>true</skip>
  66.                 </configuration>
  67.             </plugin>
  68.             <!-- 打出jar包引用关联包 -->
  69.             <plugin>
  70.                 <groupId>org.apache.maven.plugins</groupId>
  71.                 <artifactId>maven-jar-plugin</artifactId>
  72.                 <version>2.4</version>
  73.                 <configuration>
  74.                     <archive>
  75.                         <manifest>
  76.                             <addClasspath>true</addClasspath>
  77.                             <classpathPrefix>lib/</classpathPrefix>
  78.                             <!--<mainClass>com.caxs.artemis.model.schedule.ModelInvoke</mainClass>-->
  79.                         </manifest>
  80.                     </archive>
  81.                 </configuration>
  82.             </plugin>
  83.             <!-- 将依赖包放到lib文件夹中 -->
  84.             <plugin>
  85.                 <groupId>org.apache.maven.plugins</groupId>
  86.                 <artifactId>maven-dependency-plugin</artifactId>
  87.                 <executions>
  88.                     <execution>
  89.                         <id>copy</id>
  90.                         <phase>package</phase>
  91.                         <goals>
  92.                             <goal>copy-dependencies</goal>
  93.                         </goals>
  94.                         <configuration>
  95.                             <outputDirectory>
  96.                                 ${project.build.directory}/lib
  97.                             </outputDirectory>
  98.                         </configuration>
  99.                     </execution>
  100.                 </executions>
  101.             </plugin>
  102.         </plugins>
  103.     </build>

3.运行demo

  1. package spark
  2.  
  3. import ml.dmlc.xgboost4j.scala.spark.XGBoost
  4. import org.apache.spark.ml.feature.VectorAssembler
  5. import org.apache.spark.sql.{DataFrame, SparkSession}
  6.  
  7. /**
  8.   * author     ：test-abc
  9.   * date       ：Created in 2019/9/3 11:04
  10.   * description：xgboost demo
  11.   * modified By：
  12.   */
  13.  
  14. object XgboostDemo {
  15.   def main(args: Array[String]): Unit = {
  16.     val spark: SparkSession = SparkSession.builder()
  17.       .appName("SparkSql")
  18. //      .master("local[2]")
  19.       .getOrCreate()
  20.     //准备示例数据，将数据转为dataframe
  21.     import spark.implicits._
  22.     val dataList: List[(Int, Double, Double, Double, Double, Double, Double)] = List(
  23.       (0, 8.9255, -6.7863, 11.9081, 5.093, 11.4607, -9.2834),
  24.       (1, 11.5006, -4.1473, 13.8588, 5.389, 12.3622, 7.0433),
  25.       (0, 8.6093, -2.7457, 12.0805, 7.8928, 10.5825, -9.0837),
  26.       (1, 11.0604, -2.1518, 8.9522, 7.1957, 12.5846, -1.8361),
  27.       (1, 9.8369, -1.4834, 12.8746, 6.6375, 12.2772, 2.4486),
  28.       (1, 11.4763, -2.3182, 12.608, 8.6264, 10.9621, 3.5609),
  29.       (0, 11.8091, -0.0832, 9.3494, 4.2916, 11.1355, -8.0198),
  30.       (0, 13.558, -7.9881, 13.8776, 7.5985, 8.6543, 0.831),
  31.       (0, 16.1071, 2.4426, 13.9307, 5.6327, 8.8014, 6.163),
  32.       (1, 12.5088, 1.9743, 8.896, 5.4508, 13.6043, -16.2859),
  33.       (0, 5.0702, -0.5447, 9.59, 4.2987, 12.391, -18.8687),
  34.       (0, 12.7188, -7.975, 10.3757, 9.0101, 12.857, -12.0852),
  35.       (0, 8.7671, -4.6154, 9.7242, 7.4242, 9.0254, 1.4247),
  36.       (1, 16.3699, 1.5934, 16.7395, 7.333, 12.145, 5.9004),
  37.       (0, 13.808, 5.0514, 17.2611, 8.512, 12.8517, -9.1622),
  38.       (0, 3.9416, 2.6562, 13.3633, 6.8895, 12.2806, -16.162),
  39.       (0, 5.0615, 0.2689, 15.1325, 3.6587, 13.5276, -6.5477),
  40.       (1, 8.4199, -1.8128, 8.1202, 5.3955, 9.7184, -17.839),
  41.       (0, 4.875, 1.2646, 11.919, 8.465, 10.7203, -0.6707),
  42.       (1, 4.409, -0.7863, 15.1828, 8.0631, 11.2831, -0.7356))
  43.  
  44.     val inputDF: DataFrame = dataList.toDF("label", "feature1", "feature2", "feature3", "feature4", "feature5", "feature6")
  45.     //将需要转换的列合并为向量列
  46.     val transCols: Array[String] = Array("feature1", "feature2", "feature3", "feature4", "feature5", "feature6")
  47.     val assembler: VectorAssembler = new VectorAssembler().setInputCols(transCols).setOutputCol("features")
  48.     val xGBoostTrainInput: DataFrame = assembler.transform(inputDF).select("features","label")
  49.     xGBoostTrainInput.show(10)
  50.  
  51. //    val paramMap = List(
  52. //      "eta" -> 0.01, //学习率
  53. //      "gamma" -> 0.1, //用于控制是否后剪枝的参数,越大越保守，一般0.1、0.2这样子。
  54. //      "lambda" -> 2, //控制模型复杂度的权重值的L2正则化项参数，参数越大，模型越不容易过拟合。
  55. //      "subsample" -> 0.8, //随机采样训练样本
  56. //      "colsample_bytree" -> 0.8, //生成树时进行的列采样
  57. //      "max_depth" -> 5, //构建树的深度，越大越容易过拟合
  58. //      "min_child_weight" -> 5,
  59. //      "objective" -> "multi:softprob",  //定义学习任务及相应的学习目标
  60. //      "eval_metric" -> "merror",
  61. //      "num_class" -> 21
  62. //    ).toMap
  63.  
  64.     val paramMap = List(
  65.       "colsample_bytree" -> 1,
  66.       "eta" -> 0.05f, //就是学习率
  67.       "max_depth" -> 8, //树的最大深度
  68.       "min_child_weight" -> 5, //
  69.       "n_estimators" -> 120,
  70.       "subsample" -> 0.7
  71.     ).toMap
  72.  
  73.  
  74.     //模型训练
  75.     val xgBoostModel = XGBoost.trainWithDataFrame(xGBoostTrainInput, paramMap, round = 10, nWorkers = 3,
  76.       useExternalMemory = true, featureCol = "features", labelCol = "label")
  77.  
  78.     //准备预测数据
  79.     val testList: List[( Double, Double, Double, Double, Double, Double)] = List(
  80.       ( 8.9225, -6.7863, 11.9081, 5.093, 11.4607, -9.2834),
  81.       ( 11.5006, -4.1473, 13.8588, 5.389, 12.3622, 7.0433),
  82.       ( 8.6093, -2.7457, 12.0805, 7.8928, 10.5825, -9.0837),
  83.       ( 11.0604, -2.1518, 8.9522, 7.1957, 12.5846, -1.8361),
  84.       ( 9.8369, -11.4834, 12.8746, 6.6375, 12.2772, 2.4486),
  85.       ( 11.4763, -2.3182, 12.608, 8.6264, 10.9621, 3.5609),
  86.       ( 11.8091, -10.0832, 9.3494, 4.2916, 11.1355, -8.0198),
  87.       ( 13.558, -7.9881, 13.8776, 7.5985, 8.6543, 0.831),
  88.       ( 16.1071, 1.4426, 13.9307, 5.6327, 8.8014, 6.163),
  89.       ( 12.5088, 2.9743, 8.896, 5.4508, 13.6043, -16.2859),
  90.       ( 5.0702, -0.5447, 9.59, 4.2987, 12.391, -18.8687),
  91.       ( 12.7188, -7.975, 10.3757, 9.0101, 12.857, -12.0852),
  92.       ( 8.7671, -4.6154, 8.7242, 7.4242, 9.0254, 1.4247),
  93.       ( 16.3699, 1.5934, 16.7395, 7.333, 12.145, 5.9004),
  94.       ( 13.808, 5.0514, 17.2611, 8.512, 12.8517, -9.1622),
  95.       ( 3.9416, 2.6562, 13.3633, 6.8895, 12.2806, -16.162),
  96.       ( 5.0615, 0.2689, 15.1325, 3.6587, 13.5276, -6.5477),
  97.       ( 8.4199, -1.8128, 9.1202, 5.3955, 9.7184, -17.839),
  98.       ( 5.875, 1.2646, 11.919, 8.465, 10.7203, -0.6707),
  99.       ( 5.409, -0.7863, 15.1828, 8.0631, 11.2831, -0.7356))
  100.  
  101.     val testDf: DataFrame = testList.toDF("feature1", "feature2", "feature3", "feature4", "feature5", "feature6")
  102.     //将测试数据集转为向量
  103.     val xGBoostTestInput: DataFrame = assembler.transform(testDf).select("features")
  104.     xGBoostTestInput.show(10)
  105.     //模型预测
  106.     val output: DataFrame = xgBoostModel.transform(xGBoostTestInput)
  107.     output.show()
  108.     spark.close()
  109.   }
  110. }

运行结果为：

本文内容由网友自发贡献，转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号