机器学习 | 向量化_机器学习向量化

前言

下面对线性回归模型、代价函数、梯度下降算法等基础概念进行向量化。在这里我们讨论的都是这些概念的最一般的形式，毕竟，数学家都喜欢这么做。

一、线性回归模型

线性回归模型最一般的形式为：

$$h_{\theta }(x)={\theta }_0{x}_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n$$

令 θ = [ θ 0 θ 1 ⋮ θ n ] \theta=\left[

$$\begin{array}{c}\theta_0 \\ \theta_1\\\vdots\\\theta_n\end{array}$$ \right] θ=⎣⎢⎢⎢⎡​θ0​θ1​⋮θn​​⎦⎥⎥⎥⎤​   ，    $y=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ ⋮\\ y^{(m)}\end{array}\right]$

对于 $x_1,x_2,\cdots ,x_n$ 如何用向量表示，其实就我目前所知，有两种不同的设法，讨论如下：

---

第一种：

令 X = [ x 0 x 1 ⋮ x n ] X=\left[

$$\begin{array}{c}x_0 \\ x_1\\\vdots\\x_n\end{array}$$ \right] X=⎣⎢⎢⎢⎡​x0​x1​⋮xn​​⎦⎥⎥⎥⎤​

则 $h_{\theta }(x)={\theta }^{T}X=X^T\theta$

这种 $X$ 的设法可以让 $h_{\theta }(x)$ 的表达式简化一些。

---

第二种则更实用一些（做编程作业时）：

令 X = [ x 0 x 1 ( 1 ) ⋯ x n ( 1 ) x 0 x 1 ( 2 ) ⋯ x n ( 2 ) ⋮ ⋮ ⋮ x 0 x 1 ( m ) ⋯ x n ( m ) ] X=\left[

$$\begin{array}{cc}x_0 & x_1^{(1)} & \cdots & x_n^{(1)} \\ x_0 & x_1^{(2)} & \cdots & x_n^{(2)} \\ \vdots & \vdots & & \vdots \\ x_0 & x_1^{(m)} & \cdots & x_n^{(m)} \end{array}$$ \right] X=⎣⎢⎢⎢⎢⎡​x0​x0​⋮x0​​x1(1)​x1(2)​⋮x1(m)​​⋯⋯⋯​xn(1)​xn(2)​⋮xn(m)​​⎦⎥⎥⎥⎥⎤​

对于这种设法，可以理解为以特征为列，以数据样本为行。

并且此时的 $h_{\theta }(x)$ 需要改写一下：

$h_{\theta }(x^{(i)})={\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+{\theta }_2{x}_2^{(i)}+\cdots +{\theta }_n{x}_n^{(i)}$

然后令 h θ ( x ) = [ h θ ( x ( 1 ) ) h θ ( x ( 2 ) ) ⋮ h θ ( x ( m ) ) ] h_\theta(x)=\left[

$$\begin{array}{cc}h_\theta(x^{(1)})\\h_\theta(x^{(2)}) \\\vdots \\ h_\theta(x^{(m)}) \end{array}$$ \right] hθ​(x)=⎣⎢⎢⎢⎡​hθ​(x(1))hθ​(x(2))⋮hθ​(x(m))​⎦⎥⎥⎥⎤​

于是有，$h_{\theta }(x)=X\theta$

---

在接下来的向量化推导中我都将采用第二种设法。

二、代价函数

代价函数的一般形式：

$$J\left(\theta \right)=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2$$

其中 θ = [ θ 0 θ 1 ⋮ θ n ] \theta=\left[

$$\begin{array}{c}\theta_0 \\ \theta_1\\\vdots\\\theta_n\end{array}$$ \right] θ=⎣⎢⎢⎢⎡​θ0​θ1​⋮θn​​⎦⎥⎥⎥⎤​

对代价函数进行向量化的结果为：

$$J\left(\theta \right)=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2=\frac{1}{2m}(X\theta -y)\cdot (X\theta -y)$$

注意这里 $(X\theta -y)$ 与 $(X\theta -y)$ 之间使用的是点积。

具体推导过程如下：

X θ − y = h θ ( x ) − y = [ h θ ( x ( 1 ) ) h θ ( x ( 2 ) ) ⋮ h θ ( x ( m ) ) ] − [ y ( 1 ) y ( 2 ) ⋮ y ( m ) ] = [ h θ ( x ( 1 ) ) − y ( 1 ) h θ ( x ( 2 ) ) − y ( 2 ) ⋮ h θ ( x ( m ) ) − y ( m ) ] X\theta-y=h_\theta{(x)}-y=\left[

$$\begin{array}{cc}h_\theta(x^{(1)})\\h_\theta(x^{(2)}) \\\vdots \\ h_\theta(x^{(m)}) \end{array}$$ \right]-\left[ $$\begin{array}{c}y^{(1)}\\y^{(2)}\\\vdots\\y^{(m)}\end{array}$$ \right]= \left[ $$\begin{array}{cc}h_\theta(x^{(1)})-y^{(1)}\\h_\theta(x^{(2)})-y^{(2)} \\\vdots \\ h_\theta(x^{(m)})-y^{(m)} \end{array}$$ \right] Xθ−y=hθ​(x)−y=⎣⎢⎢⎢⎡​hθ​(x(1))hθ​(x(2))⋮hθ​(x(m))​⎦⎥⎥⎥⎤​−⎣⎢⎢⎢⎡​y(1)y(2)⋮y(m)​⎦⎥⎥⎥⎤​=⎣⎢⎢⎢⎡​hθ​(x(1))−y(1)hθ​(x(2))−y(2)⋮hθ​(x(m))−y(m)​⎦⎥⎥⎥⎤​

于是有，

( X θ − y ) ⋅ ( X θ − y ) = [ h θ ( x ( 1 ) ) − y ( 1 ) h θ ( x ( 2 ) ) − y ( 2 ) ⋮ h θ ( x ( m ) ) − y ( m ) ] ⋅ [ h θ ( x ( 1 ) ) − y ( 1 ) h θ ( x ( 2 ) ) − y ( 2 ) ⋮ h θ ( x ( m ) ) − y ( m ) ] (X\theta-y)\cdot(X\theta-y)= \left[

$$\begin{array}{cc}h_\theta(x^{(1)})-y^{(1)}\\h_\theta(x^{(2)})-y^{(2)} \\\vdots \\ h_\theta(x^{(m)})-y^{(m)} \end{array}$$ \right] \cdot \left[ $$\begin{array}{cc}h_\theta(x^{(1)})-y^{(1)}\\h_\theta(x^{(2)})-y^{(2)} \\\vdots \\ h_\theta(x^{(m)})-y^{(m)} \end{array}$$ \right] (Xθ−y)⋅(Xθ−y)=⎣⎢⎢⎢⎡​hθ​(x(1))−y(1)hθ​(x(2))−y(2)⋮hθ​(x(m))−y(m)​⎦⎥⎥⎥⎤​⋅⎣⎢⎢⎢⎡​hθ​(x(1))−y(1)hθ​(x(2))−y(2)⋮hθ​(x(m))−y(m)​⎦⎥⎥⎥⎤​

$={\left(h_{\theta }(x(1))-y^{(1)}\right)}^2+{\left(h_{\theta }(x^{(2)})-y^{(2)}\right)}^2+\cdots +{\left(h_{\theta }(x^{(m)})-y^{(m)}\right)}^2$

$=\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2$

总结：

$$J\left(\theta \right)=\frac{1}{2m}(X\theta -y)\cdot (X\theta -y)$$

…有没有隐约地感觉到数学的美妙之处？

三、梯度下降算法

梯度下降函数的一般形式为：

$${\theta }_j:={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$$其中，$0\le j\le n$

将上式进行向量化后的结果：

$$\theta :=\theta -\alpha \frac{1}{m}{X}^T(X\theta -y)$$

其中 θ = [ θ 0 θ 1 ⋮ θ n ] \theta=\left[

$$\begin{array}{c}\theta_0 \\ \theta_1\\\vdots\\\theta_n\end{array}$$ \right] θ=⎣⎢⎢⎢⎡​θ0​θ1​⋮θn​​⎦⎥⎥⎥⎤​   ，    $X=\left[\begin{array}{cccc}{x}_0^{(1)}& x_1^{(1)}& \cdots & x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& \cdots & x_n^{(2)}\\ ⋮& ⋮& & ⋮\\ x_0^{(m)}& x_1^{(m)}& \cdots & x_n^{(m)}\end{array}\right]$  ，    $y=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ ⋮\\ y^{(m)}\end{array}\right]$

推导过程如下：

令 $\theta :=\theta -\alpha \frac{1}{m}\delta$，显然， δ = [ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x 0 ( i ) ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x 1 ( i ) ⋮ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x n ( i ) ] \delta=\left[

$$\begin{array}{cc}\sum^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_0\\\sum^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_1 \\\vdots \\ \sum^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_n \end{array}$$ \right] δ=⎣⎢⎢⎢⎢⎡​∑i=1m​(hθ​(x(i))−y(i))x0(i)​∑i=1m​(hθ​(x(i))−y(i))x1(i)​⋮∑i=1m​(hθ​(x(i))−y(i))xn(i)​​⎦⎥⎥⎥⎥⎤​

则有 δ = [ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x 0 ( i ) ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x 1 ( i ) ⋮ ∑ i = 1 m ( h θ ( x ( i ) ) − y ( i ) ) x n ( i ) ] \delta= \left[

$$\begin{array}{cc}\sum^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_0\\\sum^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_1 \\\vdots \\ \sum^m_{i=1}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_n \end{array}$$ \right]\quad δ=⎣⎢⎢⎢⎢⎡​∑i=1m​(hθ​(x(i))−y(i))x0(i)​∑i=1m​(hθ​(x(i))−y(i))x1(i)​⋮∑i=1m​(hθ​(x(i))−y(i))xn(i)​​⎦⎥⎥⎥⎥⎤​

= [ ( h θ ( x ( 1 ) ) − y ( 1 ) ) x 0 ( 1 ) + ( h θ ( x ( 2 ) ) − y ( 2 ) ) x 0 ( 2 ) + ⋯ + ( h θ ( x ( m ) ) − y ( m ) ) x 0 ( m ) ( h θ ( x ( 1 ) ) − y ( 1 ) ) x 1 ( 1 ) + ( h θ ( x ( 2 ) ) − y ( 2 ) ) x 1 ( 2 ) + ⋯ + ( h θ ( x ( m ) ) − y ( m ) ) x 1 ( m ) ⋮ ( h θ ( x ( 1 ) ) − y ( 1 ) ) x n ( 1 ) + ( h θ ( x ( 2 ) ) − y ( 2 ) ) x n ( 2 ) + ⋯ + ( h θ ( x ( m ) ) − y ( m ) ) x n ( m ) ] \\= \left[

$$\begin{array}{cc}\left( h_{\theta}(x^{(1)})-y^{(1)} \right)x^{(1)}_0+\left( h_{\theta}(x^{(2)})-y^{(2)} \right)x^{(2)}_0+\cdots+\left( h_{\theta}(x^{(m)})-y^{(m)} \right)x^{(m)}_0 \\ \left( h_{\theta}(x^{(1)})-y^{(1)} \right)x^{(1)}_1+\left( h_{\theta}(x^{(2)})-y^{(2)} \right)x^{(2)}_1+\cdots+\left( h_{\theta}(x^{(m)})-y^{(m)} \right)x^{(m)}_1 \\ \vdots \\ \left( h_{\theta}(x^{(1)})-y^{(1)} \right)x^{(1)}_n+\left( h_{\theta}(x^{(2)})-y^{(2)} \right)x^{(2)}_n+\cdots+\left( h_{\theta}(x^{(m)})-y^{(m)} \right)x^{(m)}_n\\ \end{array}$$ \right] =⎣⎢⎢⎢⎢⎡​(hθ​(x(1))−y(1))x0(1)​+(hθ​(x(2))−y(2))x0(2)​+⋯+(hθ​(x(m))−y(m))x0(m)​(hθ​(x(1))−y(1))x1(1)​+(hθ​(x(2))−y(2))x1(2)​+⋯+(hθ​(x(m))−y(m))x1(m)​⋮(hθ​(x(1))−y(1))xn(1)​+(hθ​(x(2))−y(2))xn(2)​+⋯+(hθ​(x(m))−y(m))xn(m)​​⎦⎥⎥⎥⎥⎤​

= [ x 0 ( 1 ) x 0 ( 2 ) ⋯ x 0 ( m ) x 1 ( 1 ) x 1 ( 2 ) ⋯ x 1 ( m ) ⋮ ⋮ ⋮ x n ( 1 ) x n ( 2 ) ⋯ x n ( m ) ] [ h θ ( x ( 1 ) ) − y ( 1 ) h θ ( x ( 2 ) ) − y ( 2 ) ⋮ h θ ( x ( m ) ) − y ( m ) ] \\=\left[

$$\begin{array}{cc}x_0^{(1)} & x_0^{(2)} & \cdots & x_0^{(m)} \\ x_1^{(1)} & x_1^{(2)} & \cdots & x_1^{(m)} \\ \vdots & \vdots & & \vdots \\ x_n^{(1)} & x_n^{(2)} & \cdots & x_n^{(m)}\end{array}$$ \right] \left[ $$\begin{array}{cc}h_\theta(x^{(1)})-y^{(1)}\\h_\theta(x^{(2)})-y^{(2)} \\\vdots \\ h_\theta(x^{(m)})-y^{(m)} \end{array}$$ \right] =⎣⎢⎢⎢⎢⎡​x0(1)​x1(1)​⋮xn(1)​​x0(2)​x1(2)​⋮xn(2)​​⋯⋯⋯​x0(m)​x1(m)​⋮xn(m)​​⎦⎥⎥⎥⎥⎤​⎣⎢⎢⎢⎡​hθ​(x(1))−y(1)hθ​(x(2))−y(2)⋮hθ​(x(m))−y(m)​⎦⎥⎥⎥⎤​

很明显， [ x 0 ( 1 ) x 0 ( 2 ) ⋯ x 0 ( m ) x 1 ( 1 ) x 1 ( 2 ) ⋯ x 1 ( m ) ⋮ ⋮ ⋮ x n ( 1 ) x n ( 2 ) ⋯ x n ( m ) ] = X T \left[

$$\begin{array}{cc}x_0^{(1)} & x_0^{(2)} & \cdots & x_0^{(m)} \\ x_1^{(1)} & x_1^{(2)} & \cdots & x_1^{(m)} \\ \vdots & \vdots & & \vdots \\ x_n^{(1)} & x_n^{(2)} & \cdots & x_n^{(m)}\end{array}$$ \right]=X^T ⎣⎢⎢⎢⎢⎡​x0(1)​x1(1)​⋮xn(1)​​x0(2)​x1(2)​⋮xn(2)​​⋯⋯⋯​x0(m)​x1(m)​⋮xn(m)​​⎦⎥⎥⎥⎥⎤​=XT ，又因为 $X\theta -y=\left[\begin{array}{c}{h}_{\theta }(x^{(1)})-y^{(1)}\\ h_{\theta }(x^{(2)})-y^{(2)}\\ ⋮\\ h_{\theta }(x^{(m)})-y^{(m)}\end{array}\right]$

故 $\delta =X^T(X\theta -y)$

总结：

$$\theta :=\theta -\alpha \frac{1}{m}{X}^T(X\theta -y)$$