《机器学习实战（Scala实现）》（五）——Logistic回归_scala 机器学习 回归

原理

逻辑回归的一种二分类。我们先来看下以下公式：

$\large z = w_0x_0 + w_1x_1 + ... + w_nx_n$

z是Sigmoid函数的输入：

$\large \sigma(z) = \frac{1}{1+e^{-z}}$

我们可以看出上式子，当z = 0时，$\sigma(z) = 0.5$;当z不断的减小时，$\sigma(z)$会不断的趋于0;当z不断的增大时，$\sigma(z)$会不断的趋于1。

所以我们就可以利用Sigmoid函数，当$\sigma(z) < 0.5$，我们就将label判断为0;当$\sigma(z) > 0.5$，我们就将label判断为1。

x是特征的值，而逻辑回归模型训练的就是最优的权值w。

梯度向上

我们可以使用偏导来确定权值优化的最快的方向，例如二维：

x移动方向为$\large \frac{\vartheta f(x,y)}{\vartheta x}$

y移动方向为$\large \frac{\vartheta f(x,y)}{\vartheta y }$

训练算法

Python

def loadDataSet():
    dataMat = []; labelMat = []
    fr = open('testSet.txt')
    for line in fr.readlines():
        lineArr = line.strip().split()
        dataMat.append([1.0, float(lineArr[0]), float(lineArr[1])])
        labelMat.append(int(lineArr[2]))
    return dataMat,labelMat

def sigmoid(inX):
    return 1.0/(1+exp(-inX))
1
2
3
4
5
6
7
8
9
10
11

随机梯度上升

• 所有的权值初始化为1
• 对数据集中的每个一个样本
  
  • 计算该样本的梯度
  • 使用$alpha * gradient$更新权值
• 返回权值

这里我们给出的是优化后的代码：

# numIter 迭代次数 默认150次
def stocGradAscent1(dataMatrix, classLabels, numIter=150):
    m,n = shape(dataMatrix)
    weights = ones(n)   
    for j in range(numIter):
        dataIndex = range(m)
        for i in range(m):
            # 每次迭代都更新alpha
            # 可以发现alpha值越来越小
            # 确保每次移动的步长越来越小
            alpha = 4/(1.0+j+i)+0.0001  
            # 随机得到某行  
            randIndex = int(random.uniform(0,len(dataIndex)))
            # 预测结果
            h = sigmoid(sum(dataMatrix[randIndex]*weights))
            # 得到误差
            error = classLabels[randIndex] - h
            # 更新权值
            weights = weights + alpha * error * dataMatrix[randIndex]
            # 移除该行
            del(dataIndex[randIndex])
    return weights
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

测试算法

def classifyVector(inX, weights):
    prob = sigmoid(sum(inX*weights))
    if prob > 0.5: return 1.0
    else: return 0.0
1
2
3
4

Scala

package Logistic

import scala.io.Source
import scala.collection.mutable.ArrayBuffer
import scala.util.Random

object Logistic {

  def loadDataSet() = {
    val dataMat = new ArrayBuffer[Array[Double]]
    val labelMat = new ArrayBuffer[Double]
    val fr = Source.fromFile("LogisticTestSet.txt")
    for (line <- fr.getLines()) {
      val lineArr = line.trim().split("\t").map(_.toDouble)
      dataMat.append(Array(1.0, lineArr(0), lineArr(1)))
      labelMat.append(lineArr(2))
    }
    (dataMat.toArray, labelMat.toArray)
  }

  def sigmoid(inX: Double) = {
    1.0 / (1 + math.exp(-inX))
  }

  def stocGradAscent1(dataMatrix: Array[Array[Double]], classLabels: Array[Double], numIter: Int = 150) = {
    val m = dataMatrix.length
    val n = dataMatrix(0).length
    var weights = Array.fill(n)(1.0)
    for (j <- 1 to numIter) {
      var dataIndex: ArrayBuffer[Int] = ArrayBuffer.empty
      for (loc <- 0 to m - 1) dataIndex.append(loc)
      for (i <- 0 to m - 1) {
        val alpha = 4 / (1.0 + j + i) + 0.0001
        val randIndex = Random.nextInt(dataIndex.length)
        //go to 0 because of the constant
        val rowZipWeight = dataMatrix(dataIndex(randIndex)).zip(weights)
        val h = sigmoid(rowZipWeight.map(x => x._1 * x._2).sum)
        val error = classLabels(randIndex) - h
        weights = rowZipWeight.map(x => x._1 + alpha * error * x._2)
        dataIndex.remove(randIndex)
      }
    }
    weights
  }

  def classifyVector(inX: Array[Double], weights: Array[Double]) = {
    val prob = sigmoid(inX.zip(weights).map(x => x._1 * x._2).sum)
    if (prob > 0.5) 1.0 else 0.0
  }

  def main(args: Array[String]): Unit = {
    val dataSet = loadDataSet()
    val dataMatrix = dataSet._1
    val classLabels = dataSet._2
    val weights = stocGradAscent1(dataMatrix, classLabels,500)
    val result = dataMatrix.map(x => classifyVector(x, weights))
    println(result.mkString("\n"))
    println(result.zip(classLabels).filter(x => x._1 == x._2).length.toDouble / classLabels.length)
  }
}

/*
运行结果：

0.0
1.0
0.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
1.0
0.0
1.0
1.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
1.0
1.0
0.0
0.0
1.0
1.0
1.0
0.0
0.0
1.0
1.0
0.0
0.0
1.0
1.0
0.0
0.0
1.0
0.0
0.0
0.9973131930172848 -1.6933548385975405 -0.5646293142100435
0.68
*/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167