利用python进行数据分析（5）

第十三章Python建模库介绍

13.1 pandas与建模代码的结合

• 特征工程是指从原生数据集中提取可用于模型上下文的有效信息的数据转换过程或分析。

#pandas和其他分析库的结合点通常是NumPy数组。
#要将DataFrame转换为NumPy数组，使用．values属性
import pandas as pd
import numpy as np
1
2
3
4

data = pd.DataFrame({'x0':[1,2,3,4,5],
                    'x1':[0.01,-0.01,0.25,-4.1,0],
                    'y':[-1.5,0.,3.6,1.3,-2]})
data
1
2
3
4

data.columns
1

Index(['x0', 'x1', 'y'], dtype='object')
1

data.values
1

array([[ 1.  ,  0.01, -1.5 ],
       [ 2.  , -0.01,  0.  ],
       [ 3.  ,  0.25,  3.6 ],
       [ 4.  , -4.1 ,  1.3 ],
       [ 5.  ,  0.  , -2.  ]])
1
2
3
4
5

#将数组再转换为DataFrame
df2 = pd.DataFrame(data.values,columns=['one','two','three'])
df2
1
2
3

# .values属性一般在你的数据是同构化的时候使用
df3 = data.copy()
df3['strings'] = ['a','b','c','d','e']
df3
1
2
3
4

df3.values
1

array([[1, 0.01, -1.5, 'a'],
       [2, -0.01, 0.0, 'b'],
       [3, 0.25, 3.6, 'c'],
       [4, -4.1, 1.3, 'd'],
       [5, 0.0, -2.0, 'e']], dtype=object)
1
2
3
4
5

#对于某些模型，你可能只想使用一部分列。我推荐使用loc索引和values
model_cols = ['x0','x1']
data.loc[:,model_cols].values
1
2
3

array([[ 1.  ,  0.01],
       [ 2.  , -0.01],
       [ 3.  ,  0.25],
       [ 4.  , -4.1 ],
       [ 5.  ,  0.  ]])
1
2
3
4
5

data['category'] = pd.Categorical(['a','b','a','a','b'],categories=['a','b'])
data
1
2

#如果我们想使用虚拟变量替代’category’列，我们先创建虚拟变量，
#之后删除’categroy’列，然后连接结果
dummies = pd.get_dummies(data.category,prefix='category')
dummies
1
2
3
4

data_with_dummies = data.drop('category',axis=1).join(dummies)
data_with_dummies
1
2

13.2 使用Patsy创建模型描述

• Patsy（https://patsy.readthedocs.io/）是一个用于描述统计模型（尤其是线性模型）的Python库。
• 它使用一种小型基于字符串的“公式语法”，这种语法受到了R、S统计编程语言中公式语法的启发。

data = pd.DataFrame({'x0':[1,2,3,4,5],
                    'x1':[0.01,-0.01,0.25,-4.1,0],
                    'y':[-1.5,0.,3.6,1.3,-2]})
data
1
2
3
4

import patsy
1

y,X = patsy.dmatrices('y~x0+x1',data)
1

y
1

DesignMatrix with shape (5, 1)
     y
  -1.5
   0.0
   3.6
   1.3
  -2.0
  Terms:
    'y' (column 0)
1
2
3
4
5
6
7
8
9

X
1

DesignMatrix with shape (5, 3)
  Intercept  x0     x1
          1   1   0.01
          1   2  -0.01
          1   3   0.25
          1   4  -4.10
          1   5   0.00
  Terms:
    'Intercept' (column 0)
    'x0' (column 1)
    'x1' (column 2)
1
2
3
4
5
6
7
8
9
10
11

np.asarray(y)
1

array([[-1.5],
       [ 0. ],
       [ 3.6],
       [ 1.3],
       [-2. ]])
1
2
3
4
5

np.asarray(X)
1

array([[ 1.  ,  1.  ,  0.01],
       [ 1.  ,  2.  , -0.01],
       [ 1.  ,  3.  ,  0.25],
       [ 1.  ,  4.  , -4.1 ],
       [ 1.  ,  5.  ,  0.  ]])
1
2
3
4
5

#你可以通过给模型添加名词列+0来加入截距
patsy.dmatrices('y~x0+x1+0',data)[1]
1
2

DesignMatrix with shape (5, 2)
  x0     x1
   1   0.01
   2  -0.01
   3   0.25
   4  -4.10
   5   0.00
  Terms:
    'x0' (column 0)
    'x1' (column 1)
1
2
3
4
5
6
7
8
9
10

coef,resid,_,_ = np.linalg.lstsq(X,y)
1

coef
1

array([[ 0.31290976],
       [-0.07910564],
       [-0.26546384]])
1
2
3

coef = pd.Series(coef.squeeze(),index=X.design_info.column_names)
1

coef
1

Intercept    0.312910
x0          -0.079106
x1          -0.265464
dtype: float64
1
2
3
4

13.2.1 Patsy公式中的数据转换

• 你可以将Python代码混合到你的Patsy公式中，在执行公式时，Patsy库将尝试在封闭作用域中寻找你使用的函数

y,X = patsy.dmatrices('y~x0+np.log(np.abs(x1+1))',data)
1

X
1

DesignMatrix with shape (5, 3)
  Intercept  x0  np.log(np.abs(x1 + 1))
          1   1                 0.00995
          1   2                -0.01005
          1   3                 0.22314
          1   4                 1.13140
          1   5                 0.00000
  Terms:
    'Intercept' (column 0)
    'x0' (column 1)
    'np.log(np.abs(x1 + 1))' (column 2)
1
2
3
4
5
6
7
8
9
10
11

#一些常用的变量转换包括标准化（对均值0和方差1）和居中（减去平均值）
y,X = patsy.dmatrices('y~standardize(x0)+center(x1)',data)
1
2

X
1

DesignMatrix with shape (5, 3)
  Intercept  standardize(x0)  center(x1)
          1         -1.41421        0.78
          1         -0.70711        0.76
          1          0.00000        1.02
          1          0.70711       -3.33
          1          1.41421        0.77
  Terms:
    'Intercept' (column 0)
    'standardize(x0)' (column 1)
    'center(x1)' (column 2)
1
2
3
4
5
6
7
8
9
10
11

#patsy.build_design_matrices函数可以使用原始样本内数据集中保存的信息将变换应用于新的样本外数据上
new_data = pd.DataFrame({'x0':[6,7,8,9],
                        'x1':[3.1,-0.5,0,2.3],
                        'y':[1,2,3,4]})
new_X = patsy.build_design_matrices([X.design_info],new_data)
new_X
1
2
3
4
5
6

[DesignMatrix with shape (4, 3)
   Intercept  standardize(x0)  center(x1)
           1          2.12132        3.87
           1          2.82843        0.27
           1          3.53553        0.77
           1          4.24264        3.07
   Terms:
     'Intercept' (column 0)
     'standardize(x0)' (column 1)
     'center(x1)' (column 2)]
1
2
3
4
5
6
7
8
9
10

y,X = patsy.dmatrices('y~I(x0+x1)',data)
1

X
1

DesignMatrix with shape (5, 2)
  Intercept  I(x0 + x1)
          1        1.01
          1        1.99
          1        3.25
          1       -0.10
          1        5.00
  Terms:
    'Intercept' (column 0)
    'I(x0 + x1)' (column 1)
1
2
3
4
5
6
7
8
9
10

13.2.2 分类数据与Patsy

data = pd.DataFrame({'key1':['a','a','b','b','a','b','a','b'],
                    'key2':[0,1,0,1,0,1,0,0],
                    'v1':[1,2,3,4,5,6,7,8],
                    'v2':[-1,0,2.5,-0.5,4.0,-1.2,0.2,-1.7]})

y,X = patsy.dmatrices('v2~key1',data)

X
1
2
3
4
5
6
7
8

DesignMatrix with shape (8, 2)
  Intercept  key1[T.b]
          1          0
          1          0
          1          1
          1          1
          1          0
          1          1
          1          0
          1          1
  Terms:
    'Intercept' (column 0)
    'key1' (column 1)
1
2
3
4
5
6
7
8
9
10
11
12
13

y,X = patsy.dmatrices('v2~key1+0',data)

X
1
2
3

DesignMatrix with shape (8, 2)
  key1[a]  key1[b]
        1        0
        1        0
        0        1
        0        1
        1        0
        0        1
        1        0
        0        1
  Terms:
    'key1' (columns 0:2)
1
2
3
4
5
6
7
8
9
10
11
12

#数字类型列可以使用C函数解释为分类类型
y,X = patsy.dmatrices('v2~C(key2)',data)

X
1
2
3
4

DesignMatrix with shape (8, 2)
  Intercept  C(key2)[T.1]
          1             0
          1             1
          1             0
          1             1
          1             0
          1             1
          1             0
          1             0
  Terms:
    'Intercept' (column 0)
    'C(key2)' (column 1)
1
2
3
4
5
6
7
8
9
10
11
12
13

data['key2'] = data['key2'].map({0:'zero',1:'one'})
data
1
2

y,X = patsy.dmatrices('v2~key1+key2',data)

X
1
2
3

DesignMatrix with shape (8, 3)
  Intercept  key1[T.b]  key2[T.zero]
          1          0             1
          1          0             0
          1          1             1
          1          1             0
          1          0             1
          1          1             0
          1          0             1
          1          1             1
  Terms:
    'Intercept' (column 0)
    'key1' (column 1)
    'key2' (column 2)
1
2
3
4
5
6
7
8
9
10
11
12
13
14

y,X = patsy.dmatrices('v2~key1+key2+key1:key2',data)

X
1
2
3

DesignMatrix with shape (8, 4)
  Intercept  key1[T.b]  key2[T.zero]  key1[T.b]:key2[T.zero]
          1          0             1                       0
          1          0             0                       0
          1          1             1                       1
          1          1             0                       0
          1          0             1                       0
          1          1             0                       0
          1          0             1                       0
          1          1             1                       1
  Terms:
    'Intercept' (column 0)
    'key1' (column 1)
    'key2' (column 2)
    'key1:key2' (column 3)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

13.3 statsmodels介绍

• statsmodels（http://www.statsmodels.org）是一个Python库，用于拟合多种统计模型，执行统计测试以及数据探索和可视化。statsmodels包含更多的“经典”频率学派统计方法，而贝叶斯方法和机器学习模型可在其他库中找到。
• 线性模型，广义线性模型和鲁棒线性模型
• 线性混合效应模型
• 方差分析（ANOVA）方法
• 时间序列过程和状态空间模型
• 广义的矩量法

13.3.1 评估线性模型

import statsmodels.api as sm
import statsmodels.formula.api as smf
1
2

def dnorm(mean,variance,size=1):
    if isinstance(size,int):
        size = size,
        return mean+np.sqrt(variance)*np.random.randn(*size)
1
2
3
4

np.random.seed(12345)
N = 100
X = np.c_[dnorm(0,0.4,size=N),
         dnorm(0,0.6,size=N),
         dnorm(0,0.2,size=N)]

eps = dnorm(0,0.1,size=N)
beta = [0.1,0.3,0.5]
y = np.dot(X,beta)+eps
1
2
3
4
5
6
7
8
9

X[:5]
1

array([[-0.12946849, -1.21275292,  0.50422488],
       [ 0.30291036, -0.43574176, -0.25417986],
       [-0.32852189, -0.02530153,  0.13835097],
       [-0.35147471, -0.71960511, -0.25821463],
       [ 1.2432688 , -0.37379916, -0.52262905]])
1
2
3
4
5

y[:5]
1

array([ 0.42786349, -0.67348041, -0.09087764, -0.48949442, -0.12894109])
1

#线性模型通常与我们在Patsy中看到的截距项相匹配。sm.add_constant函数可以将截距列添加到现有矩阵
X_model = sm.add_constant(X)
X_model[:5]
1
2
3

array([[ 1.        , -0.12946849, -1.21275292,  0.50422488],
       [ 1.        ,  0.30291036, -0.43574176, -0.25417986],
       [ 1.        , -0.32852189, -0.02530153,  0.13835097],
       [ 1.        , -0.35147471, -0.71960511, -0.25821463],
       [ 1.        ,  1.2432688 , -0.37379916, -0.52262905]])
1
2
3
4
5

#sm.OLS类可以拟合一个最小二乘线性回归
model = sm.OLS(y,X)
1
2

results = model.fit()
results.params
1
2

array([0.17826108, 0.22303962, 0.50095093])
1

print(results.summary())
1

                                 OLS Regression Results                                
=======================================================================================
Dep. Variable:                      y   R-squared (uncentered):                   0.430
Model:                            OLS   Adj. R-squared (uncentered):              0.413
Method:                 Least Squares   F-statistic:                              24.42
Date:                Thu, 30 Dec 2021   Prob (F-statistic):                    7.44e-12
Time:                        11:09:06   Log-Likelihood:                         -34.305
No. Observations:                 100   AIC:                                      74.61
Df Residuals:                      97   BIC:                                      82.42
Df Model:                           3                                                  
Covariance Type:            nonrobust                                                  
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
x1             0.1783      0.053      3.364      0.001       0.073       0.283
x2             0.2230      0.046      4.818      0.000       0.131       0.315
x3             0.5010      0.080      6.237      0.000       0.342       0.660
==============================================================================
Omnibus:                        4.662   Durbin-Watson:                   2.201
Prob(Omnibus):                  0.097   Jarque-Bera (JB):                4.098
Skew:                           0.481   Prob(JB):                        0.129
Kurtosis:                       3.243   Cond. No.                         1.74
==============================================================================

Notes:
[1] R² is computed without centering (uncentered) since the model does not contain a constant.
[2] Standard Errors assume that the covariance matrix of the errors is correctly specified.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

data = pd.DataFrame(X,columns=['col0','col1','col2'])
1

data['y'] = y
1

data[:5]
1

#可以使用statsmodels公式API和Patsy公式字符串
results = smf.ols('y~col0+col1+col2',data=data).fit()
1
2

results.params
1

Intercept    0.033559
col0         0.176149
col1         0.224826
col2         0.514808
dtype: float64
1
2
3
4
5

results.tvalues
1

Intercept    0.952188
col0         3.319754
col1         4.850730
col2         6.303971
dtype: float64
1
2
3
4
5

results.predict(data[:5])
1

0   -0.002327
1   -0.141904
2    0.041226
3   -0.323070
4   -0.100535
dtype: float64
1
2
3
4
5
6

13.3.2 评估时间序列处理

• statsmodels中的另一类模型用于时间序列分析。
• 其中包括自回归过程，卡尔曼滤波和其他状态空间模型，以及多变量自回归模型。

init_x = 4
import random
values = [init_x,init_x]
N = 1000
b0 = 0.8
b1 = -0.4
1
2
3
4
5
6

noise = dnorm(0,0.1,N)
for i in range(N):
    new_x = values[-1]*b0+values[-2]*b1+noise[i]
    values.append(new_x)
1
2
3
4

MAXLAGX = 5
model = sm.tsa.AR(values)
results = model.fit(MAXLAGX)
1
2
3

results.params
1

array([-0.00616093,  0.78446347, -0.40847891, -0.01364148,  0.01496872,
        0.01429462])
1
2

13.4 scikit-learn介绍

• scikit-learn（http://scikit-learn.org）是使用最广泛且最受信任的通用Python机器学习库。
• 它包含广泛的标准监督的和无监督的机器学习方法，包括用于模型选择和评估、数据转换、数据加载和模型持久化的工具。
• 这些模型可用于分类、聚类、预测和其他常见任务。

train = pd.read_csv('datasets/titanic/train.csv')
test = pd.read_csv('datasets/titanic/test.csv')
1
2

train[:4]
1

train.isnull().sum()
1

PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64
1
2
3
4
5
6
7
8
9
10
11
12
13

test.isnull().sum()
1

PassengerId      0
Pclass           0
Name             0
Sex              0
Age             86
SibSp            0
Parch            0
Ticket           0
Fare             1
Cabin          327
Embarked         0
dtype: int64
1
2
3
4
5
6
7
8
9
10
11
12

impute_value = train['Age'].median()
train['Age'] = train['Age'].fillna(impute_value)
test['Age'] = test['Age'].fillna(impute_value)
1
2
3

train['IsFemale'] = (train['Sex']=='female').astype(int)
test['IsFemale'] = (test['Sex']=='female').astype(int)
1
2

train[:5]
1

test[:5]
1

predictors = ['Pclass','IsFemale','Age']
X_train = train[predictors].values
X_train[:5]
1
2
3

array([[ 3.,  0., 22.],
       [ 1.,  1., 38.],
       [ 3.,  1., 26.],
       [ 1.,  1., 35.],
       [ 3.,  0., 35.]])
1
2
3
4
5

X_test = test[predictors].values
X_test[:5]
1
2

array([[ 3. ,  0. , 34.5],
       [ 3. ,  1. , 47. ],
       [ 2. ,  0. , 62. ],
       [ 3. ,  0. , 27. ],
       [ 3. ,  1. , 22. ]])
1
2
3
4
5

y_train = train['Survived'].values
y_train[:5]
1
2

array([0, 1, 1, 1, 0], dtype=int64)
1

#使用scikit-learn的LogisticRegression模型创建一个模型实例
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
1
2
3

model.fit(X_train,y_train)
1

LogisticRegression()
1

y_predict = model.predict(X_test)
y_predict[:10]
1
2

array([0, 0, 0, 0, 1, 0, 1, 0, 1, 0], dtype=int64)
1

from sklearn.linear_model import LogisticRegressionCV
model_cv = LogisticRegressionCV(10)
1
2

model_cv.fit(X_train,y_train)
1

LogisticRegressionCV()
1

from sklearn.model_selection import cross_val_score
model = LogisticRegression(C=10)
scores = cross_val_score(model,X_train,y_train,cv=4)
scores
1
2
3
4

array([0.77578475, 0.79820628, 0.77578475, 0.78828829])
1

13.5 继续你的教育

第十四章数据分析示例

14.1 从Bitly获取1.USA.gov数据

path = 'datasets/bitly_usagov/example.txt'
open(path).readline()
1
2

'{ "a": "Mozilla\\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\\/535.11 (KHTML, like Gecko) Chrome\\/17.0.963.78 Safari\\/535.11", "c": "US", "nk": 1, "tz": "America\\/New_York", "gr": "MA", "g": "A6qOVH", "h": "wfLQtf", "l": "orofrog", "al": "en-US,en;q=0.8", "hh": "1.usa.gov", "r": "http:\\/\\/www.facebook.com\\/l\\/7AQEFzjSi\\/1.usa.gov\\/wfLQtf", "u": "http:\\/\\/www.ncbi.nlm.nih.gov\\/pubmed\\/22415991", "t": 1331923247, "hc": 1331822918, "cy": "Danvers", "ll": [ 42.576698, -70.954903 ] }\n'
1

import json 
path = 'datasets/bitly_usagov/example.txt'
records = [json.loads(line) for line in open(path)]
records[0]
1
2
3
4

{'a': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.78 Safari/535.11',
 'c': 'US',
 'nk': 1,
 'tz': 'America/New_York',
 'gr': 'MA',
 'g': 'A6qOVH',
 'h': 'wfLQtf',
 'l': 'orofrog',
 'al': 'en-US,en;q=0.8',
 'hh': '1.usa.gov',
 'r': 'http://www.facebook.com/l/7AQEFzjSi/1.usa.gov/wfLQtf',
 'u': 'http://www.ncbi.nlm.nih.gov/pubmed/22415991',
 't': 1331923247,
 'hc': 1331822918,
 'cy': 'Danvers',
 'll': [42.576698, -70.954903]}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

14.1.1 纯Python时区计数

time_zones = [rec['tz'] for rec in records]
#结果并不是所有的记录都有时区字段，导致出错
1
2

---------------------------------------------------------------------------

KeyError                                  Traceback (most recent call last)

<ipython-input-6-767e72d0f2fa> in <module>
----> 1 time_zones = [rec['tz'] for rec in records]
      2 #结果并不是所有的记录都有时区字段，导致出错


<ipython-input-6-767e72d0f2fa> in <listcomp>(.0)
----> 1 time_zones = [rec['tz'] for rec in records]
      2 #结果并不是所有的记录都有时区字段，导致出错


KeyError: 'tz'
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

time_zones = [rec['tz'] for rec in records if 'tz' in rec]
1

#计数的一种方法是在遍历时区时使用字典来存储计数
def get_counts(sequence):
    counts = {}
    for x in sequence:
        if x in counts:
            counts[x]+=1
        else :
            counts[x]=1
    return counts
1
2
3
4
5
6
7
8
9

from collections import defaultdict
def get_counts2(sequence):
    counts = defaultdict(int)#值将会初始化为0
    for x in sequence:
        counts[x] +=1
    return counts
1
2
3
4
5
6

counts = get_counts(time_zones)
counts['America/New_York']
1
2

1251
1

len(time_zones)
1

3440
1

#如果我们想要前十的时区和它们的计数
def top_counts(count_dict,n=10):
    value_key_pairs = [(count,tz) for tz,count in count_dict.items()]
    value_key_pairs.sort()
    return value_key_pairs[-n:]
1
2
3
4
5

top_counts(counts)
1

[(33, 'America/Sao_Paulo'),
 (35, 'Europe/Madrid'),
 (36, 'Pacific/Honolulu'),
 (37, 'Asia/Tokyo'),
 (74, 'Europe/London'),
 (191, 'America/Denver'),
 (382, 'America/Los_Angeles'),
 (400, 'America/Chicago'),
 (521, ''),
 (1251, 'America/New_York')]
1
2
3
4
5
6
7
8
9
10

#如果我们搜索Python标准库，你可能会发现collections.Counter类，它可以使任务更加简单
from collections import Counter
counts = Counter(time_zones)
counts.most_common(10)
1
2
3
4

[('America/New_York', 1251),
 ('', 521),
 ('America/Chicago', 400),
 ('America/Los_Angeles', 382),
 ('America/Denver', 191),
 ('Europe/London', 74),
 ('Asia/Tokyo', 37),
 ('Pacific/Honolulu', 36),
 ('Europe/Madrid', 35),
 ('America/Sao_Paulo', 33)]
1
2
3
4
5
6
7
8
9
10

14.1.2 使用pandas进行时区计数

import pandas as pd
frame = pd.DataFrame(records)
frame.info()
1
2
3

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3560 entries, 0 to 3559
Data columns (total 18 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   a            3440 non-null   object 
 1   c            2919 non-null   object 
 2   nk           3440 non-null   float64
 3   tz           3440 non-null   object 
 4   gr           2919 non-null   object 
 5   g            3440 non-null   object 
 6   h            3440 non-null   object 
 7   l            3440 non-null   object 
 8   al           3094 non-null   object 
 9   hh           3440 non-null   object 
 10  r            3440 non-null   object 
 11  u            3440 non-null   object 
 12  t            3440 non-null   float64
 13  hc           3440 non-null   float64
 14  cy           2919 non-null   object 
 15  ll           2919 non-null   object 
 16  _heartbeat_  120 non-null    float64
 17  kw           93 non-null     object 
dtypes: float64(4), object(14)
memory usage: 500.8+ KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

frame['tz'][:10]
1

0     America/New_York
1       America/Denver
2     America/New_York
3    America/Sao_Paulo
4     America/New_York
5     America/New_York
6        Europe/Warsaw
7                     
8                     
9                     
Name: tz, dtype: object
1
2
3
4
5
6
7
8
9
10
11

#对于Series，我们可以使用value_counts方法
tz_counts = frame['tz'].value_counts()
tz_counts[:10]
1
2
3

America/New_York       1251
                        521
America/Chicago         400
America/Los_Angeles     382
America/Denver          191
Europe/London            74
Asia/Tokyo               37
Pacific/Honolulu         36
Europe/Madrid            35
America/Sao_Paulo        33
Name: tz, dtype: int64
1
2
3
4
5
6
7
8
9
10
11

frame['tz'][:10]
1

0     America/New_York
1       America/Denver
2     America/New_York
3    America/Sao_Paulo
4     America/New_York
5     America/New_York
6        Europe/Warsaw
7                     
8                     
9                     
Name: tz, dtype: object
1
2
3
4
5
6
7
8
9
10
11

#用fillna方法替换缺失值，并为空字符串使用布尔数组索引
clean_tz = frame['tz'].fillna('Missing')
1
2

clean_tz[:10]
1

0     America/New_York
1       America/Denver
2     America/New_York
3    America/Sao_Paulo
4     America/New_York
5     America/New_York
6        Europe/Warsaw
7                     
8                     
9                     
Name: tz, dtype: object
1
2
3
4
5
6
7
8
9
10
11

clean_tz[clean_tz ==''] = 'Unknown'
1

clean_tz[:10]
1

0     America/New_York
1       America/Denver
2     America/New_York
3    America/Sao_Paulo
4     America/New_York
5     America/New_York
6        Europe/Warsaw
7              Unknown
8              Unknown
9              Unknown
Name: tz, dtype: object
1
2
3
4
5
6
7
8
9
10
11

tz_counts = clean_tz.value_counts()
tz_counts[:10]
1
2

America/New_York       1251
Unknown                 521
America/Chicago         400
America/Los_Angeles     382
America/Denver          191
Missing                 120
Europe/London            74
Asia/Tokyo               37
Pacific/Honolulu         36
Europe/Madrid            35
Name: tz, dtype: int64
1
2
3
4
5
6
7
8
9
10
11

import seaborn as sns
subset = tz_counts[:10]
sns.barplot(y=subset.index,x=subset.values)
1
2
3

#a列包含了执行网址缩短的浏览器、设备或应用的信息
frame['a'][1]
1
2

'GoogleMaps/RochesterNY'
1

frame['a'][50]
1

'Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2'
1

frame['a'][51]
1

'Mozilla/5.0 (Linux; U; Android 2.2.2; en-us; LG-P925/V10e Build/FRG83G) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1'
1

#默认空格分割
frame['a'][51].split('; ')
1
2

['Mozilla/5.0 (Linux',
 'U',
 'Android 2.2.2',
 'en-us',
 'LG-P925/V10e Build/FRG83G) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1']
1
2
3
4
5

results = pd.Series([x.split()[0] for x in frame.a.dropna()])
results[:5]
1
2

0               Mozilla/5.0
1    GoogleMaps/RochesterNY
2               Mozilla/4.0
3               Mozilla/5.0
4               Mozilla/5.0
dtype: object
1
2
3
4
5
6

results.value_counts()[:8]
1

Mozilla/5.0                 2594
Mozilla/4.0                  601
GoogleMaps/RochesterNY       121
Opera/9.80                    34
TEST_INTERNET_AGENT           24
GoogleProducer                21
Mozilla/6.0                    5
BlackBerry8520/5.0.0.681       4
dtype: int64
1
2
3
4
5
6
7
8
9

frame.a.notnull()[-10:]
1

3550    True
3551    True
3552    True
3553    True
3554    True
3555    True
3556    True
3557    True
3558    True
3559    True
Name: a, dtype: bool
1
2
3
4
5
6
7
8
9
10
11

#由于一些代理字符串的缺失，我们将从数据中排除这些代理字符串
cframe = frame[frame.a.notnull()]
cframe[:10]
1
2
3

#之后我们想要计算一个代表每一行是否是Windows的值
cframe['os'] = np.where(cframe['a'].str.contains('Windows'),'Windows','Not Windows')
1
2

<ipython-input-69-02329ab5f824>:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  cframe['os'] = np.where(cframe['a'].str.contains('Windows'),'Windows','Not Windows')
1
2
3
4
5
6

cframe['os'] [:10]
1

0        Windows
1    Not Windows
2        Windows
3    Not Windows
4        Windows
5        Windows
6        Windows
7        Windows
8    Not Windows
9        Windows
Name: os, dtype: object
1
2
3
4
5
6
7
8
9
10
11

#可以根据时区列以及新生成的操作系统列对数据进行分组
by_tz_os = cframe.groupby(['tz','os'])
1
2

#与value_counts函数类似，分组计数可以使用size计算。然后可以使用unstack对计算结果进行重塑
agg_counts = by_tz_os.size().unstack().fillna(0)
agg_counts[:10]
1
2
3

#最后，让我们选出总体计数最高的时区。
#要实现这个功能，我在agg_counts中根据行的计数构造了一个间接索引数组
#用于升序排列
indexer = agg_counts.sum(1).argsort()
indexer[:10]
1
2
3
4
5

tz
                                  24
Africa/Cairo                      20
Africa/Casablanca                 21
Africa/Ceuta                      92
Africa/Johannesburg               87
Africa/Lusaka                     53
America/Anchorage                 54
America/Argentina/Buenos_Aires    57
America/Argentina/Cordoba         26
America/Argentina/Mendoza         55
dtype: int64
1
2
3
4
5
6
7
8
9
10
11
12

#使用take方法按顺序选出行，之后再对最后10行进行切片（最大的10个值）
count_subset = agg_counts.take(indexer[-10:])
count_subset
1
2
3

#pandas有一个便捷的方法叫作nlargest，可以做同样的事情
agg_counts.sum(1).nlargest(10)
1
2

tz
America/New_York       1251.0
                        521.0
America/Chicago         400.0
America/Los_Angeles     382.0
America/Denver          191.0
Europe/London            74.0
Asia/Tokyo               37.0
Pacific/Honolulu         36.0
Europe/Madrid            35.0
America/Sao_Paulo        33.0
dtype: float64
1
2
3
4
5
6
7
8
9
10
11
12

#对绘图数据重新排列
count_subset = count_subset.stack()
count_subset
1
2
3

tz                   os         
America/Sao_Paulo    Not Windows     13.0
                     Windows         20.0
Europe/Madrid        Not Windows     16.0
                     Windows         19.0
Pacific/Honolulu     Not Windows      0.0
                     Windows         36.0
Asia/Tokyo           Not Windows      2.0
                     Windows         35.0
Europe/London        Not Windows     43.0
                     Windows         31.0
America/Denver       Not Windows    132.0
                     Windows         59.0
America/Los_Angeles  Not Windows    130.0
                     Windows        252.0
America/Chicago      Not Windows    115.0
                     Windows        285.0
                     Not Windows    245.0
                     Windows        276.0
America/New_York     Not Windows    339.0
                     Windows        912.0
dtype: float64
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

count_subset.name = 'total'
count_subset = count_subset.reset_index()
count_subset[:10]
1
2
3

sns.barplot(x='total',y='tz',hue='os',data=count_subset)
1

#上图不容易看到较小组中的Windows用户的相对百分比，因此让我们将组百分比归一化为1
def norm_total(group):
    group['normed_total'] = group.total/group.total.sum()
    return group 

results = count_subset.groupby('tz').apply(norm_total)
1
2
3
4
5
6

results[:10]
1

sns.barplot(x='normed_total',y='tz',hue='os',data=results)
1

#可以通过transform方法和groupby方法更有效地计算归一化之和
g = count_subset.groupby('tz')
results2 = count_subset.total/g.total.transform('sum')
1
2
3

results2[:10]
1

0    0.393939
1    0.606061
2    0.457143
3    0.542857
4    0.000000
5    1.000000
6    0.054054
7    0.945946
8    0.581081
9    0.418919
Name: total, dtype: float64
1
2
3
4
5
6
7
8
9
10
11

14.2 MovieLens 1M数据集

• 数据提供了电影的评分、电影的元数据（流派和年份）以及观众数据（年龄、邮编、性别、职业）
• MovieLens 1M数据集包含6,000个用户对4,000部电影的100万个评分。数据分布在三个表格中：评分，用户信息和电影信息。
• 从ZIP文件中提取数据后，我们可以使用pandas.read_table将每个表加载到一个pandas DataFrame对象中

import pandas as pd 
#让展示内容少一点
pd.options.display.max_rows = 10
1
2
3

unames = ['user_id','gender','age','occupation','zip']
users = pd.read_table('datasets/movielens/users.dat',sep='::',header=None,names=unames)
users[:5]
1
2
3

<ipython-input-101-ffe8596a8cfd>:2: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.
  users = pd.read_table('datasets/movielens/users.dat',sep='::',header=None,names=unames)
1
2

rnames = ['user_id','movie_id','rating','timestamp']
ratings = pd.read_table('datasets/movielens/ratings.dat',sep='::',header=None,names=rnames)
ratings[:5]
1
2
3

<ipython-input-103-bafd8ea1cf17>:2: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.
  ratings = pd.read_table('datasets/movielens/ratings.dat',sep='::',header=None,names=rnames)
1
2

mnames = ['movie_id','title','genres']
movies = pd.read_table('datasets/movielens/movies.dat',sep='::',header=None,names=mnames)
movies[:5]
1
2
3

<ipython-input-118-35e3f9b1d007>:2: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.
  movies = pd.read_table('datasets/movielens/movies.dat',sep='::',header=None,names=mnames)
1
2

#使用pandas的合并功能，我们首先将ratings表与users表合并，然后将该结果与movies表数据合并。
#pandas根据重叠名称推断哪些列用作合并的（或连接）键位
data = pd.merge(pd.merge(ratings,users),movies)
data[:5]
1
2
3
4

data.iloc[0]
1

user_id                                            1
movie_id                                        1193
rating                                             5
timestamp                                  978300760
gender                                             F
age                                                1
occupation                                        10
zip                                            48067
title         One Flew Over the Cuckoo's Nest (1975)
genres                                         Drama
Name: 0, dtype: object
1
2
3
4
5
6
7
8
9
10
11

#为了获得按性别分级的每部电影的平均电影评分，我们可以使用pivot_table方法
mean_ratings = data.pivot_table('rating',index='title',columns='gender',aggfunc='mean')
mean_ratings
1
2
3

3706 rows × 2 columns

mean_ratings[:5]
1

#先过滤掉少于250（完全随意定的数字）个评分的电影；
#为此，我接着按标题对数据进行分组，并使用size()为每个标题获取一个元素是各分组大小的Series
ratings_by_title = data.groupby('title').size()
ratings_by_title[:10]
1
2
3
4

title
$1,000,000 Duck (1971)                37
'Night Mother (1986)                  70
'Til There Was You (1997)             52
'burbs, The (1989)                   303
...And Justice for All (1979)        199
1-900 (1994)                           2
10 Things I Hate About You (1999)    700
101 Dalmatians (1961)                565
101 Dalmatians (1996)                364
12 Angry Men (1957)                  616
dtype: int64
1
2
3
4
5
6
7
8
9
10
11
12

active_titles = ratings_by_title.index[ratings_by_title >= 250]
active_titles
1
2

Index([''burbs, The (1989)', '10 Things I Hate About You (1999)',
       '101 Dalmatians (1961)', '101 Dalmatians (1996)', '12 Angry Men (1957)',
       '13th Warrior, The (1999)', '2 Days in the Valley (1996)',
       '20,000 Leagues Under the Sea (1954)', '2001: A Space Odyssey (1968)',
       '2010 (1984)',
       ...
       'X-Men (2000)', 'Year of Living Dangerously (1982)',
       'Yellow Submarine (1968)', 'You've Got Mail (1998)',
       'Young Frankenstein (1974)', 'Young Guns (1988)',
       'Young Guns II (1990)', 'Young Sherlock Holmes (1985)',
       'Zero Effect (1998)', 'eXistenZ (1999)'],
      dtype='object', name='title', length=1216)
1
2
3
4
5
6
7
8
9
10
11
12

#评分多于250个的电影标题的索引之后可以用于从mean_ratings中选出所需的行
mean_ratings = mean_ratings.loc[active_titles]
mean_ratings
1
2
3

1216 rows × 2 columns

#要看到女性观众的top电影，我们可以按F列降序排序
top_female_ratings = mean_ratings.sort_values(by='F',ascending=False)
top_female_ratings[:10]
1
2
3

14.2.1 测量评价分歧

• 假设你想找到男性和女性观众之间最具分歧性的电影

#一种方法是添加一列到含有均值差的mean_ratings中，然后按以下方式排序
mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']
1
2

mean_ratings[:10]
1

sorted_by_diff = mean_ratings.sort_values(by='diff')
sorted_by_diff[:10]
1
2

#转换行的顺序，并切片出top 10的行，我们就可以获得男性更喜欢但女性评分不高的电影
sorted_by_diff[::-1][:10]
1
2

#假设你想要的是不依赖于性别标识而在观众中引起最大异议的电影。
#异议可以通过评分的方差或标准差来衡量
ratings_std_by_title = data.groupby('title')['rating'].std()
ratings_std_by_title[:10]
1
2
3
4

title
$1,000,000 Duck (1971)               1.092563
'Night Mother (1986)                 1.118636
'Til There Was You (1997)            1.020159
'burbs, The (1989)                   1.107760
...And Justice for All (1979)        0.878110
1-900 (1994)                         0.707107
10 Things I Hate About You (1999)    0.989815
101 Dalmatians (1961)                0.982103
101 Dalmatians (1996)                1.098717
12 Angry Men (1957)                  0.812731
Name: rating, dtype: float64
1
2
3
4
5
6
7
8
9
10
11
12

ratings_std_by_title = ratings_std_by_title.loc[active_titles]
ratings_std_by_title[:10]
1
2

title
'burbs, The (1989)                     1.107760
10 Things I Hate About You (1999)      0.989815
101 Dalmatians (1961)                  0.982103
101 Dalmatians (1996)                  1.098717
12 Angry Men (1957)                    0.812731
13th Warrior, The (1999)               1.140421
2 Days in the Valley (1996)            0.921592
20,000 Leagues Under the Sea (1954)    0.869685
2001: A Space Odyssey (1968)           1.042504
2010 (1984)                            0.946618
Name: rating, dtype: float64
1
2
3
4
5
6
7
8
9
10
11
12

ratings_std_by_title.sort_values(ascending=False)[:10]
1

title
Dumb & Dumber (1994)                     1.321333
Blair Witch Project, The (1999)          1.316368
Natural Born Killers (1994)              1.307198
Tank Girl (1995)                         1.277695
Rocky Horror Picture Show, The (1975)    1.260177
Eyes Wide Shut (1999)                    1.259624
Evita (1996)                             1.253631
Billy Madison (1995)                     1.249970
Fear and Loathing in Las Vegas (1998)    1.246408
Bicentennial Man (1999)                  1.245533
Name: rating, dtype: float64
1
2
3
4
5
6
7
8
9
10
11
12

14.3 美国1880～2010年的婴儿名字

• 美国社会保障局（SSA）提供了从1880年至现在的婴儿姓名频率的数据。

names1880 = pd.read_table('datasets/babynames/yob1880.txt',sep=',',names=['name','sex','births'])
names1880.head()
1
2

#为简单起见，我们可以使用按性别列出的出生总和作为当年的出生总数
names1880.groupby('sex').births.sum()
1
2

sex
F     90993
M    110493
Name: births, dtype: int64
1
2
3
4

#由于数据集按年分为多个文件，首先要做的事情之一是将所有数据集中到一个DataFrame中，然后再添加一个年份字段。
#你可以使用pandas.concat来做到这一点
years = range(1880,2011)
pieces = []
columns = ['name','sex','births']
for year in years:
    path = 'datasets/babynames/yob%d.txt' %year
    frame = pd.read_csv(path,names=columns)
    frame['year'] = year
    pieces.append(frame)
#将所有内容粘贴进一个
names = pd.concat(pieces,ignore_index=True)
1
2
3
4
5
6
7
8
9
10
11
12

names.head()
1

#可以使用groupby或pivot_table开始聚合年份和性别的数据
total_births = names.pivot_table('births',index='year',columns='sex',aggfunc=sum)
total_births.tail()
1
2
3

total_births.plot(title='Total births by sex and year')
1

def add_prop(group):
    group['prop'] = group.births/group.births.sum()
    return group

names = names.groupby(['year','sex']).apply(add_prop)
1
2
3
4
5

names
1

1690784 rows × 6 columns

#在执行此类组操作时，进行完整性检查通常很有价值，例如验证所有组中的prop列总计为1
names.groupby(['year','sex']).prop.sum()
1
2

year  sex
1880  F      1.0
      M      1.0
1881  F      1.0
      M      1.0
1882  F      1.0
            ... 
2008  M      1.0
2009  F      1.0
      M      1.0
2010  F      1.0
      M      1.0
Name: prop, Length: 262, dtype: float64
1
2
3
4
5
6
7
8
9
10
11
12
13

#每个性别/年份组合的前1,000名
def get_top1000(group):
    return group.sort_values(by='births',ascending=False)[:1000]

grouped = names.groupby(['year','sex'])
top1000 = grouped.apply(get_top1000)
#删除组索引，不需要它
top1000.reset_index(inplace=True,drop=True)
1
2
3
4
5
6
7
8

#如果你喜欢DIY方式，可以试试以下代码
pieces = []
for year,group in names.groupby(['year','sex']):
    pieces.append(group.sort_values(by='births',ascending=False)[:1000])
    top1000 = pd.concat(pieces,ignore_index=True)
1
2
3
4
5

top1000
1

261877 rows × 6 columns

14.3.1 分析名字趋势

boys = top1000[top1000.sex=='M']
girls = top1000[top1000.sex=='F']
1
2

total_births = top1000.pivot_table('births',index='year',columns='name',aggfunc=sum)
total_births
1
2

131 rows × 6868 columns

total_births.info()
1

<class 'pandas.core.frame.DataFrame'>
Int64Index: 131 entries, 1880 to 2010
Columns: 6868 entries, Aaden to Zuri
dtypes: float64(6868)
memory usage: 6.9 MB
1
2
3
4
5

subset = total_births[['John','Harry','Mary','Marilyn']]
subset.plot(subplots=True,figsize=(12,10),grid=False,title='Number of births per year')
1
2

14.3.1.1 计量命名多样性的增加

table = top1000.pivot_table('prop',index='year',columns='sex',aggfunc=sum)
table.plot(title='Sum of table1000.prop by year and sex',
          yticks=np.linspace(0,1.2,13),
          xticks=range(1880,2020,10))
1
2
3
4

#另一个有趣的指标是不同名字的数量，按最高到最低的受欢迎程度在出生人数最高的50%的名字中排序
df = boys[boys.year==2010]
df
1
2
3

1000 rows × 6 columns

prop_cumsum = df.sort_values(by='prop',ascending=False).prop.cumsum()
prop_cumsum[:10]
1
2

260877    0.011523
260878    0.020934
260879    0.029959
260880    0.038930
260881    0.047817
260882    0.056579
260883    0.065155
260884    0.073414
260885    0.081528
260886    0.089621
Name: prop, dtype: float64
1
2
3
4
5
6
7
8
9
10
11

prop_cumsum.values.searchsorted(0.5)
1

116
1

df = boys[boys.year==1900]
df
1
2

1000 rows × 6 columns

in1900 = df.sort_values(by='prop',ascending=False).prop.cumsum()
1

in1900.values.searchsorted(0.5)+1
1

25
1

#你现在可以将此操作应用于每个年/性别分组，通过这些字段进行groupby，
#并将返回值是每个分组计数值的函数apply到每个分组上
def get_quantile_count(group,q=0.5):
    group = group.sort_values(by='prop',ascending=False)
    return group.prop.cumsum().values.searchsorted(q)+1

diversity = top1000.groupby(['year','sex']).apply(get_quantile_count)
diversity = diversity.unstack('sex')
1
2
3
4
5
6
7
8

diversity.head()
1

#导入matplotlib相关库
import matplotlib.pyplot as plt

# 中文乱码的处理
plt.rcParams['font.sans-serif'] =['Microsoft YaHei']
plt.rcParams['axes.unicode_minus'] = False
diversity.plot(title='按年份划分的多样性指标图')
1
2
3
4
5
6
7

14.3.1.2 “最后一个字母”革命

#从name列提取最后一个字母
get_last_letter = lambda x:x[-1]
last_letters = names.name.map(get_last_letter)
last_letters.name = 'last_letter'
table = names.pivot_table('births',index=last_letters,columns=['sex','year'],aggfunc=sum)
1
2
3
4
5

table[:5]
1

5 rows × 262 columns

subtable = table.reindex(columns=[1910,1960,2010],level='year')
subtable.head()
1
2

#按照出生总数对表格进行归一化处理，计算一个新表格，其中包含每个性别的每个结束字母占总出生数的比例
subtable.sum()
1
2

sex  year
F    1910     396416.0
     1960    2022062.0
     2010    1759010.0
M    1910     194198.0
     1960    2132588.0
     2010    1898382.0
dtype: float64
1
2
3
4
5
6
7
8

letter_prop = subtable/subtable.sum()
letter_prop[:10]
1
2

#现在根据掌握的字母比例，我们可以绘制出按年划分的每个性别的条形图
import matplotlib.pyplot as plt
fig,axes = plt.subplots(2,1,figsize=(10,8))
letter_prop['M'].plot(kind='bar',rot=0,ax=axes[0],title='Male')
letter_prop['F'].plot(kind='bar',rot=0,ax=axes[1],title='Female')
1
2
3
4
5

letter_prop = table/table.sum()
dny_ts = letter_prop.loc[['d','n','y'],'M'].T
dny_ts.head()
1
2
3

dny_ts.plot(title = '随着时间推移名字以d/n/y结尾的男孩的比例变化趋势')
1

14.3.1.3 男孩名字变成女孩名字（以及反向）

all_names = pd.Series(top1000.name.unique())
all_names[:10]
1
2

0         Mary
1         Anna
2         Emma
3    Elizabeth
4       Minnie
5     Margaret
6          Ida
7        Alice
8       Bertha
9        Sarah
dtype: object
1
2
3
4
5
6
7
8
9
10
11

lesley_like = all_names[all_names.str.lower().str.contains('lesl')]
lesley_like
1
2

632     Leslie
2294    Lesley
4262    Leslee
4728     Lesli
6103     Lesly
dtype: object
1
2
3
4
5
6

filtered = top1000[top1000.name.isin(lesley_like)]
filtered.groupby('name').births.sum()
1
2

name
Leslee      1082
Lesley     35022
Lesli        929
Leslie    370429
Lesly      10067
Name: births, dtype: int64
1
2
3
4
5
6
7

#让我们按性别和年份进行聚合，并在年内进行标准化
table = filtered.pivot_table('births',index='year',columns='sex',aggfunc='sum')
table = table.div(table.sum(1),axis=0)
table.tail()
1
2
3
4

table.plot(style={'M':'k-','F':'k--'})
1

14.4 美国农业部食品数据库

import json
db = json.load(open('datasets/usda_food/database.json'))
len(db)
1
2
3

6636
1

#db中的每个条目都是一个包含单个食物所有数据的词典。
#'nutrients’字段是一个字典的列表，每个营养元素对应一个字典
db[0].keys()
1
2
3

dict_keys(['id', 'description', 'tags', 'manufacturer', 'group', 'portions', 'nutrients'])
1

db[0]['nutrients'][0]
1

{'value': 25.18,
 'units': 'g',
 'description': 'Protein',
 'group': 'Composition'}
1
2
3
4

nutrients = pd.DataFrame(db[0]['nutrients'])
1

nutrients[:7]
1

#将字典的列表转换为DataFrame时，
#我们可以指定一个需要提取的字段列表。我们将提取食物名称、分类、ID和制造商
info_keys = ['description','group','id','manufacturer']
info = pd.DataFrame(db,columns=info_keys)
1
2
3
4

info[-10:]
1

info.info()
1

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6636 entries, 0 to 6635
Data columns (total 4 columns):
 #   Column        Non-Null Count  Dtype 
---  ------        --------------  ----- 
 0   description   6636 non-null   object
 1   group         6636 non-null   object
 2   id            6636 non-null   int64 
 3   manufacturer  5195 non-null   object
dtypes: int64(1), object(3)
memory usage: 207.5+ KB
1
2
3
4
5
6
7
8
9
10
11

#可以通过value_counts查看食物组的分布情况
pd.value_counts(info.group)[:10]
1
2

Vegetables and Vegetable Products    812
Beef Products                        618
Baked Products                       496
Breakfast Cereals                    403
Legumes and Legume Products          365
Fast Foods                           365
Lamb, Veal, and Game Products        345
Sweets                               341
Fruits and Fruit Juices              328
Pork Products                        328
Name: group, dtype: int64
1
2
3
4
5
6
7
8
9
10
11

14.5 2012年联邦选举委员会数据库

• 美国联邦选举委员会公布了有关政治运动贡献的数据。这些数据包括捐赠者姓名、职业和雇主、地址和缴费金额。

fec = pd.read_csv('datasets/fec/P00000001-ALL.csv')
fec.info()
1
2

D:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py:3165: DtypeWarning: Columns (6) have mixed types.Specify dtype option on import or set low_memory=False.
  has_raised = await self.run_ast_nodes(code_ast.body, cell_name,


<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1001731 entries, 0 to 1001730
Data columns (total 16 columns):
 #   Column             Non-Null Count    Dtype  
---  ------             --------------    -----  
 0   cmte_id            1001731 non-null  object 
 1   cand_id            1001731 non-null  object 
 2   cand_nm            1001731 non-null  object 
 3   contbr_nm          1001731 non-null  object 
 4   contbr_city        1001712 non-null  object 
 5   contbr_st          1001727 non-null  object 
 6   contbr_zip         1001620 non-null  object 
 7   contbr_employer    988002 non-null   object 
 8   contbr_occupation  993301 non-null   object 
 9   contb_receipt_amt  1001731 non-null  float64
 10  contb_receipt_dt   1001731 non-null  object 
 11  receipt_desc       14166 non-null    object 
 12  memo_cd            92482 non-null    object 
 13  memo_text          97770 non-null    object 
 14  form_tp            1001731 non-null  object 
 15  file_num           1001731 non-null  int64  
dtypes: float64(1), int64(1), object(14)
memory usage: 122.3+ MB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

fec.iloc[123456]
1

cmte_id                             C00431445
cand_id                             P80003338
cand_nm                         Obama, Barack
contbr_nm                         ELLMAN, IRA
contbr_city                             TEMPE
contbr_st                                  AZ
contbr_zip                          852816719
contbr_employer      ARIZONA STATE UNIVERSITY
contbr_occupation                   PROFESSOR
contb_receipt_amt                        50.0
contb_receipt_dt                    01-DEC-11
receipt_desc                              NaN
memo_cd                                   NaN
memo_text                                 NaN
form_tp                                 SA17A
file_num                               772372
Name: 123456, dtype: object
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#可以使用unique获得所有不同的政治候选人名单
unique_cands = fec.cand_nm.unique()
unique_cands
1
2
3

array(['Bachmann, Michelle', 'Romney, Mitt', 'Obama, Barack',
       "Roemer, Charles E. 'Buddy' III", 'Pawlenty, Timothy',
       'Johnson, Gary Earl', 'Paul, Ron', 'Santorum, Rick',
       'Cain, Herman', 'Gingrich, Newt', 'McCotter, Thaddeus G',
       'Huntsman, Jon', 'Perry, Rick'], dtype=object)
1
2
3
4
5

unique_cands[2]
1

'Obama, Barack'
1

parties = {'Bachmann, Michelle':'Republican',
            'Romney, Mitt':'Republican',
            'Obama, Barack':'Democrat',
            "Roemer, Charles E. 'Buddy' III":'Republican',
            'Pawlenty, Timothy':'Republican',
            'Johnson, Gary Earl':'Republican',
            'Paul, Ron':'Republican',
            'Santorum, Rick':'Republican',
            'Cain, Herman':'Republican',
            'Gingrich, Newt':'Republican',
            'McCotter, Thaddeus G':'Republican',
            'Huntsman, Jon':'Republican',
            'Perry, Rick':'Republican'}

1
2
3
4
5
6
7
8
9
10
11
12
13
14

fec.cand_nm[123456:123461]
1

123456    Obama, Barack
123457    Obama, Barack
123458    Obama, Barack
123459    Obama, Barack
123460    Obama, Barack
Name: cand_nm, dtype: object
1
2
3
4
5
6

fec.cand_nm[123456:123461].map(parties)
1

123456    Democrat
123457    Democrat
123458    Democrat
123459    Democrat
123460    Democrat
Name: cand_nm, dtype: object
1
2
3
4
5
6

fec['party'] = fec.cand_nm.map(parties)
1

fec['party'].value_counts()
1

Democrat      593746
Republican    407985
Name: party, dtype: int64
1
2
3

#首先，这些数据既包括捐款也包括退款（即负贡献金额）
(fec.contb_receipt_amt>0).value_counts()
1
2

True     991475
False     10256
Name: contb_receipt_amt, dtype: int64
1
2
3

fec = fec[fec.contb_receipt_amt>0]
1

fec_mrbo = fec[fec.cand_nm.isin(['Obama, Barack','Romney, Mitt'])] 
1

14.5.1 按职业和雇主的捐献统计

fec.contbr_occupation.value_counts()[:10]
1

RETIRED                                   233990
INFORMATION REQUESTED                      35107
ATTORNEY                                   34286
HOMEMAKER                                  29931
PHYSICIAN                                  23432
INFORMATION REQUESTED PER BEST EFFORTS     21138
ENGINEER                                   14334
TEACHER                                    13990
CONSULTANT                                 13273
PROFESSOR                                  12555
Name: contbr_occupation, dtype: int64
1
2
3
4
5
6
7
8
9
10
11

occ_mapping = {'INFORMATION REQUESTED PER BEST EFFORTS':'NOT PROVIDED',
              'INFORMATION REQUESTED':'NOT PROVIDED',
              'INFORMATION REQUESTED(BEST EFFORTS)':'NOT PROVIDED',
              'C.E.O':'CEO'}

#如果没有映射，则返回x
f = lambda x :occ_mapping.get(x,x)
fec.contbr_occupation = fec.contbr_occupation.map(f)
1
2
3
4
5
6
7
8

occ_mapping = {'INFORMATION REQUESTED PER BEST EFFORTS':'NOT PROVIDED',
              'INFORMATION REQUESTED':'NOT PROVIDED',
               'SELF':'SELF-EMPLOYED',
               'SELF EMPLOYED':'SELF-EMPLOYED'
               }

#如果没有映射，则返回x
f = lambda x :occ_mapping.get(x,x)
fec.contbr_employer = fec.contbr_employer.map(f)
1
2
3
4
5
6
7
8
9

by_occupation = fec.pivot_table('contb_receipt_amt',index='contbr_occupation',
                                columns='party',aggfunc='sum')
1
2

over_2mm = by_occupation[by_occupation.sum(1)>2000000]
over_2mm
1
2

over_2mm.plot(kind='barh')
1

def get_top_amounts(group,key,n=5):
    totals = group.groupby(key)['contb_receipt_amt'].sum()
    return totals.nlargest(n)
1
2
3

grouped = fec_mrbo.groupby('cand_nm')
grouped.apply(get_top_amounts,'contbr_occupation',n=7)
1
2

cand_nm        contbr_occupation                     
Obama, Barack  RETIRED                                   25305116.38
               ATTORNEY                                  11141982.97
               INFORMATION REQUESTED                      4866973.96
               HOMEMAKER                                  4248875.80
               PHYSICIAN                                  3735124.94
               LAWYER                                     3160478.87
               CONSULTANT                                 2459912.71
Romney, Mitt   RETIRED                                   11508473.59
               INFORMATION REQUESTED PER BEST EFFORTS    11396894.84
               HOMEMAKER                                  8147446.22
               ATTORNEY                                   5364718.82
               PRESIDENT                                  2491244.89
               EXECUTIVE                                  2300947.03
               C.E.O.                                     1968386.11
Name: contb_receipt_amt, dtype: float64
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

14.5.2 捐赠金额分桶

bins = np.array([0,1,10,100,1000,10000,100000,1000000,10000000])
1

labels = pd.cut(fec_mrbo.contb_receipt_amt,bins)
labels
1
2

411         (10, 100]
412       (100, 1000]
413       (100, 1000]
414         (10, 100]
415         (10, 100]
             ...     
701381      (10, 100]
701382    (100, 1000]
701383        (1, 10]
701384      (10, 100]
701385    (100, 1000]
Name: contb_receipt_amt, Length: 694282, dtype: category
Categories (8, interval[int64]): [(0, 1] < (1, 10] < (10, 100] < (100, 1000] < (1000, 10000] < (10000, 100000] < (100000, 1000000] < (1000000, 10000000]]
1
2
3
4
5
6
7
8
9
10
11
12
13

grouped = fec_mrbo.groupby(['cand_nm',labels])
grouped.size().unstack(0)
1
2

bucket_sum = grouped.contb_receipt_amt.sum().unstack(0)
normed_sums = bucket_sum.div(bucket_sum.sum(axis=1),axis=0)
normed_sums
1
2
3

normed_sums[:-2].plot(kind='barh')
1

14.5.3 按州进行捐赠统计

#将数据按照候选人和州进行聚合是一项常规分析
grouped = fec_mrbo.groupby(['cand_nm','contbr_st'])
totals = grouped.contb_receipt_amt.sum().unstack(0).fillna(0)
totals[totals.sum(1)>100000]
totals[:10]
1
2
3
4
5

percent = totals.div(totals.sum(1),axis=0)
percent[:10]
1
2

# 附录A高阶NumPy

A.1 ndarray对象内幕

• ndarray内部包含以下内容
• 指向数据的指针——即RAM中或内存映射文件中的数据块
• 数据类型或dtype，描述数组中固定大小的值单元格
• 表示数组形状（shape）的元组
• 步长元组，表示要“步进”的字节数的整数以便沿维度推进一个元素

np.ones((10,5)).shape
1

(10, 5)
1

#一个典型的（C阶）3×4×5 float64值（8字节）的数组具有跨度（160,40,8）
np.ones((3,4,5),dtype=np.float64).strides
1
2

(160, 40, 8)
1

A.1.1 NumPy dtype层次结构

ints = np.ones(10,dtype=np.uint16)
ints
1
2

array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=uint16)
1

floats = np.ones(10,dtype=np.float32)
floats
1
2

array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32)
1

np.issubdtype(ints.dtype,np.integer)
1

True
1

np.issubdtype(floats.dtype,np.floating)
1

True
1

#可以通过调用类型的mro方法来查看特定dtype的所有父类
np.float64.mro()
1
2

[numpy.float64,
 numpy.floating,
 numpy.inexact,
 numpy.number,
 numpy.generic,
 float,
 object]
1
2
3
4
5
6
7

np.issubdtype(ints.dtype,np.number)
1

True
1

A.2 高阶数组操作

A.2.1 重塑数组

arr = np.arange(8)
arr
1
2

array([0, 1, 2, 3, 4, 5, 6, 7])
1

#在很多情况下，你将数组从一个形状转换为另一个形状，并且不复制任何数据
arr.reshape(4,2)
1
2

array([[0, 1],
       [2, 3],
       [4, 5],
       [6, 7]])
1
2
3
4

#多维数组也可以被重塑
arr.reshape(4,2).reshape(2,4)
1
2

array([[0, 1, 2, 3],
       [4, 5, 6, 7]])
1
2

#传递的形状维度可以有一个值是-1，表示维度通过数据进行推断
arr = np.arange(15)
arr.reshape((5,-1))
1
2
3

array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11],
       [12, 13, 14]])
1
2
3
4
5

#由于数组的shape属性是一个元组，它也可以被传递给reshape
other_arr = np.ones((3,5))
other_arr.shape
1
2
3

(3, 5)
1

arr.reshape(other_arr.shape)
1

array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])
1
2
3

#reshape的反操作可以将更高维度的数组转换为一维数组，
#这种操作通常被成为扁平化（flattening）或分散化（raveling）
arr = np.arange(15).reshape((5,3))
arr
1
2
3
4

array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11],
       [12, 13, 14]])
1
2
3
4
5

arr.ravel()
#如果结果中的值在原始数组中是连续的，则ravel不会生成底层数值的副本。
1
2

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14])
1

arr.flatten()
#flatten方法的行为类似于ravel，但它总是返回数据的副本
1
2

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14])
1

A.2.2 C顺序和Fortran顺序

• C顺序/行方向顺序
• 首先遍历更高的维度（例如，在轴0上行进之前先在轴1上行进）
• Fortran顺序/列方向顺序
• 最后遍历更高的维度（例如，在轴1上行进之前先在轴0上行进）

arr = np.arange(12).reshape((3,4))
arr
1
2

array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
1
2
3

arr.ravel()
1

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])
1

arr.ravel('F')
1

array([ 0,  4,  8,  1,  5,  9,  2,  6, 10,  3,  7, 11])
1

A.2.3 连接和分隔数组

• numpy.concatenate可以获取数组的序列（元组、列表等），并沿着输入轴将它们按顺序连接在一起

arr1 = np.array([[1,2,3],[4,5,6]])
arr1
1
2

array([[1, 2, 3],
       [4, 5, 6]])
1
2

arr2 = np.array([[7,8,9],[10,11,12]])
arr2
1
2

array([[ 7,  8,  9],
       [10, 11, 12]])
1
2

np.concatenate([arr1,arr2],axis=0)
1

array([[ 1,  2,  3],
       [ 4,  5,  6],
       [ 7,  8,  9],
       [10, 11, 12]])
1
2
3
4

np.concatenate([arr1,arr2],axis=1)
1

array([[ 1,  2,  3,  7,  8,  9],
       [ 4,  5,  6, 10, 11, 12]])
1
2

np.vstack((arr1,arr2))
1

array([[ 1,  2,  3],
       [ 4,  5,  6],
       [ 7,  8,  9],
       [10, 11, 12]])
1
2
3
4

np.hstack((arr1,arr2))
1

array([[ 1,  2,  3,  7,  8,  9],
       [ 4,  5,  6, 10, 11, 12]])
1
2

arr = np.random.randn(5,2)
arr
1
2

array([[-0.37933271, -1.04852791],
       [-0.3278915 ,  1.11594819],
       [ 0.77077511, -1.19903381],
       [ 0.38477425, -0.35244269],
       [ 1.38135852, -0.10439573]])
1
2
3
4
5

#split可以将一个数组沿轴向切片成多个数组
#传递给np.split的值[1,3]表示将数组拆分时的索引位置
first,second,third = np.split(arr,[2,3])
first
1
2
3
4

array([[-0.37933271, -1.04852791],
       [-0.3278915 ,  1.11594819]])
1
2

second
1

array([[ 0.77077511, -1.19903381]])
1

third
1

array([[ 0.38477425, -0.35244269],
       [ 1.38135852, -0.10439573]])
1
2

• 数组连接函数

A.2.3.1 堆叠助手：r 和c

arr = np.arange(6)
arr
1
2

array([0, 1, 2, 3, 4, 5])
1

arr1= arr.reshape((3,2))
arr1
1
2

array([[0, 1],
       [2, 3],
       [4, 5]])
1
2
3

arr2 = np.random.randn(3,2)
arr2
1
2

array([[-2.17693174,  1.20516725],
       [-0.44083574,  0.84645799],
       [ 0.02369097,  0.63556261]])
1
2
3

np.r_[arr1,arr2]
1

array([[ 0.        ,  1.        ],
       [ 2.        ,  3.        ],
       [ 4.        ,  5.        ],
       [-2.17693174,  1.20516725],
       [-0.44083574,  0.84645799],
       [ 0.02369097,  0.63556261]])
1
2
3
4
5
6

np.c_[arr1,arr2]
1

array([[ 0.        ,  1.        , -2.17693174,  1.20516725],
       [ 2.        ,  3.        , -0.44083574,  0.84645799],
       [ 4.        ,  5.        ,  0.02369097,  0.63556261]])
1
2
3

np.c_[np.r_[arr1,arr2],arr]
1

array([[ 0.        ,  1.        ,  0.        ],
       [ 2.        ,  3.        ,  1.        ],
       [ 4.        ,  5.        ,  2.        ],
       [-2.17693174,  1.20516725,  3.        ],
       [-0.44083574,  0.84645799,  4.        ],
       [ 0.02369097,  0.63556261,  5.        ]])
1
2
3
4
5
6

#可以将切片转换为数组
np.c_[1:6,-10:-5]
1
2

array([[  1, -10],
       [  2,  -9],
       [  3,  -8],
       [  4,  -7],
       [  5,  -6]])
1
2
3
4
5

A.2.4 重复元素：tile和repeat

• repeat和tile函数是用于重复或复制数组的两个有用的工具。
• repeat函数按照给定次数对数组中的每个元素进行复制，生成一个更大的数组

arr = np.arange(3)
arr
1
2

array([0, 1, 2])
1

arr.repeat(3)
1

array([0, 0, 0, 1, 1, 1, 2, 2, 2])
1

arr.repeat([2,3,4])
1

array([0, 0, 1, 1, 1, 2, 2, 2, 2])
1

#多维数组可以在指定的轴向上对它们的元素进行重复
arr = np.random.randn(2,2)
arr
1
2
3

array([[-0.86642515, -0.21137086],
       [ 0.4945539 , -0.02745328]])
1
2

arr.repeat(2,axis=0)
1

array([[-0.86642515, -0.21137086],
       [-0.86642515, -0.21137086],
       [ 0.4945539 , -0.02745328],
       [ 0.4945539 , -0.02745328]])
1
2
3
4

#请注意，如果没有传递轴，数组将首先扁平化，这可能不是你想要的
arr.repeat(2)
1
2

array([-0.86642515, -0.86642515, -0.21137086, -0.21137086,  0.4945539 ,
        0.4945539 , -0.02745328, -0.02745328])
1
2

#需要按照不同次数重复多维数组的切片时，你可以传递一个整数数组
arr.repeat([2,3],axis=0)
1
2

array([[-0.86642515, -0.21137086],
       [-0.86642515, -0.21137086],
       [ 0.4945539 , -0.02745328],
       [ 0.4945539 , -0.02745328],
       [ 0.4945539 , -0.02745328]])
1
2
3
4
5

arr.repeat([2,3],axis=1)
1

array([[-0.86642515, -0.86642515, -0.21137086, -0.21137086, -0.21137086],
       [ 0.4945539 ,  0.4945539 , -0.02745328, -0.02745328, -0.02745328]])
1
2

#tile是一种快捷方法，它可以沿着轴向堆叠副本。在视觉上，你可以把它看作类似于“铺设瓷砖”
arr
1
2

array([[-0.86642515, -0.21137086],
       [ 0.4945539 , -0.02745328]])
1
2

np.tile(arr,2)
1

array([[-0.86642515, -0.21137086, -0.86642515, -0.21137086],
       [ 0.4945539 , -0.02745328,  0.4945539 , -0.02745328]])
1
2

np.tile(arr,(2,1))
1

array([[-0.86642515, -0.21137086],
       [ 0.4945539 , -0.02745328],
       [-0.86642515, -0.21137086],
       [ 0.4945539 , -0.02745328]])
1
2
3
4

#tile的第二个参数可以是表示“铺瓷砖”布局的元组
np.tile(arr,(2,2))
1
2

array([[-0.86642515, -0.21137086, -0.86642515, -0.21137086],
       [ 0.4945539 , -0.02745328,  0.4945539 , -0.02745328],
       [-0.86642515, -0.21137086, -0.86642515, -0.21137086],
       [ 0.4945539 , -0.02745328,  0.4945539 , -0.02745328]])
1
2
3
4

A.2.5 神奇索引的等价方法：take和put

arr = np.arange(10)*100
arr
1
2

array([  0, 100, 200, 300, 400, 500, 600, 700, 800, 900])
1

inds = [7,1,2,6]
arr[inds]
1
2

array([700, 100, 200, 600])
1

arr.take(inds)
1

array([700, 100, 200, 600])
1

arr.put(inds,42)
arr
1
2

array([  0,  42,  42, 300, 400, 500,  42,  42, 800, 900])
1

arr.put(inds,[40,41,42,43])
arr
1
2

array([  0,  41,  42, 300, 400, 500,  43,  40, 800, 900])
1

#如果要在别的轴上使用take，你可以传递axis关键字
inds = [2,0,2,1]
arr = np.random.randn(2,4)
arr
1
2
3
4

array([[ 0.42067458,  1.11465134,  0.80097006, -0.37064359],
       [-0.57974434,  1.24554556,  0.25903436, -0.10895085]])
1
2

arr.take(inds,axis=1)
1

array([[ 0.80097006,  0.42067458,  0.80097006,  1.11465134],
       [ 0.25903436, -0.57974434,  0.25903436,  1.24554556]])
1
2

A.3 广播

• 广播描述了算法如何在不同形状的数组之间进行运算。
• 广播的规则
• 如果对于每个结尾维度（即从尾部开始的），轴长度都匹配或者长度都是1，两个二维数组就是可以兼容广播的。之后，广播会在丢失的或长度为1的轴上进行。

arr = np.arange(5)
arr
1
2

array([0, 1, 2, 3, 4])
1

#这里我们说标量值4已经被广播给乘法运算中的所有其他元素
arr*4
1
2

array([ 0,  4,  8, 12, 16])
1

arr = np.random.randn(4,3)
arr
1
2

array([[-0.26130828,  0.21031853,  0.09806178],
       [-1.89409267, -0.30607457,  1.14174612],
       [-0.04140891, -1.4256403 ,  0.17503634],
       [ 0.94815936, -0.47780023, -0.17362592]])
1
2
3
4

arr.mean(0)
1

array([-0.31216263, -0.49979914,  0.31030458])
1

demeaned = arr - arr.mean(0)
demeaned
1
2

array([[ 0.05085435,  0.71011767, -0.2122428 ],
       [-1.58193004,  0.19372457,  0.83144154],
       [ 0.27075371, -0.92584116, -0.13526824],
       [ 1.26032198,  0.02199892, -0.4839305 ]])
1
2
3
4

demeaned.mean(0)
1

array([5.55111512e-17, 1.38777878e-17, 1.38777878e-17])
1

arr
1

array([[-0.26130828,  0.21031853,  0.09806178],
       [-1.89409267, -0.30607457,  1.14174612],
       [-0.04140891, -1.4256403 ,  0.17503634],
       [ 0.94815936, -0.47780023, -0.17362592]])
1
2
3
4

row_means = arr.mean(1)
row_means
1
2

array([ 0.01569068, -0.35280704, -0.43067096,  0.09891107])
1

row_means.shape
1

(4,)
1

row_means.reshape((4,1))
1

array([[ 0.01569068],
       [-0.35280704],
       [-0.43067096],
       [ 0.09891107]])
1
2
3
4

demeaned = arr - row_means.reshape((4,1))
demeaned.mean(1)
1
2

array([4.62592927e-18, 7.40148683e-17, 7.40148683e-17, 0.00000000e+00])
1

A.3.1 在其他轴上广播

arr - arr.mean(1)
1

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-111-8b8ada26fac0> in <module>
----> 1 arr - arr.mean(1)


ValueError: operands could not be broadcast together with shapes (4,3) (4,) 
1
2
3
4
5
6
7
8
9

arr - arr.mean(1).reshape((4,1))
1

array([[-0.27699896,  0.19462785,  0.0823711 ],
       [-1.54128563,  0.04673247,  1.49455316],
       [ 0.38926205, -0.99496934,  0.6057073 ],
       [ 0.84924828, -0.5767113 , -0.27253699]])
1
2
3
4

#使用特殊的np.newaxis属性和“完整”切片来插入新轴
arr = np.zeros((4,4))
arr
1
2
3

array([[0., 0., 0., 0.],
       [0., 0., 0., 0.],
       [0., 0., 0., 0.],
       [0., 0., 0., 0.]])
1
2
3
4

arr_3d = arr[:,np.newaxis,:]
arr_3d
1
2

array([[[0., 0., 0., 0.]],

       [[0., 0., 0., 0.]],

       [[0., 0., 0., 0.]],

       [[0., 0., 0., 0.]]])
1
2
3
4
5
6
7

arr_3d.shape
1

(4, 1, 4)
1

arr_1d = np.random.normal(size=3)
arr_1d[:,np.newaxis]
1
2

array([[-0.44142019],
       [ 0.19138049],
       [ 1.70465573]])
1
2
3

arr_1d[np.newaxis,:]
1

array([[-0.44142019,  0.19138049,  1.70465573]])
1

arr = np.random.randn(3,4,5)
arr
1
2

array([[[ 0.10223077, -1.53873895, -0.99946213,  0.71598751,
         -0.90498114],
        [-0.01548156,  0.30273138,  0.34831772,  1.64086735,
          0.52801345],
        [-1.31620627, -0.79570758, -1.34854625, -2.63311809,
         -1.11911915],
        [-0.80136175, -1.94967438, -0.28787123,  0.33664872,
          0.16180744]],

       [[ 1.77507844, -0.6858868 , -0.53739313,  1.33779554,
          1.53855697],
        [ 1.9271013 ,  0.58314326, -0.73893003,  0.67052899,
         -0.00530868],
        [-0.19838128, -0.92396483, -0.72747217,  0.8346707 ,
          0.44643892],
        [-0.37615445,  1.8688799 , -0.55484319,  0.50585597,
         -0.26799842]],

       [[ 0.57238033, -0.17529308, -0.72637569, -2.89489543,
         -0.01108801],
        [-0.17406094, -0.79553743, -0.64445857, -1.0084828 ,
          0.59183829],
        [-0.60375821,  0.15761849,  0.25371104, -0.60639911,
         -1.20483347],
        [ 0.70185761, -0.90187431,  0.45284624, -1.09157387,
          0.70808834]]])
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

depth_means = arr.mean(2)
depth_means
1
2

array([[-0.52499279,  0.56088967, -1.44253947, -0.50809024],
       [ 0.6856302 ,  0.48730697, -0.11374173,  0.23514796],
       [-0.64705438, -0.40614029, -0.40073225, -0.0261312 ]])
1
2
3

depth_means.shape
1

(3, 4)
1

demeaned = arr - depth_means[:,:,np.newaxis]
1

demeaned.mean(2)
1

array([[-2.22044605e-17,  4.44089210e-17, -2.22044605e-17,
        -2.22044605e-17],
       [-4.44089210e-17, -2.22044605e-17,  4.44089210e-17,
         0.00000000e+00],
       [-8.88178420e-17, -4.44089210e-17,  8.88178420e-17,
         2.22044605e-17]])
1
2
3
4
5
6

def demean_axis(arr,axis=0):
    means = arr.mean(axis)
    indexer = [slice(None)]*arr.ndim
    indexer[axis] = np.newaxis
    return arr - means[indexer]
1
2
3
4
5

A.3.2 通过广播设定数组的值

arr = np.zeros((4,3))
arr[:] = 5
arr
1
2
3

array([[5., 5., 5.],
       [5., 5., 5.],
       [5., 5., 5.],
       [5., 5., 5.]])
1
2
3
4

col = np.array([1.28,-0.42,0.44,1.6])
arr[:] = col[:,np.newaxis]
arr
1
2
3

array([[ 1.28,  1.28,  1.28],
       [-0.42, -0.42, -0.42],
       [ 0.44,  0.44,  0.44],
       [ 1.6 ,  1.6 ,  1.6 ]])
1
2
3
4

arr[:2] = [[-1.37],[0.509]]
1

arr
1

array([[-1.37 , -1.37 , -1.37 ],
       [ 0.509,  0.509,  0.509],
       [ 0.44 ,  0.44 ,  0.44 ],
       [ 1.6  ,  1.6  ,  1.6  ]])
1
2
3
4

A.4 高阶ufunc用法

A.4.1 ufunc实例方法

• NumPy的每个二元ufunc（通用函数）都有特殊的方法来执行某些特殊的向量化操作。
• ufunc方法

arr = np.arange(10)
arr
1
2

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
1

#reduce方法接收单个数组并通过执行一系列二元操作在可选的轴向上对数组的值进行聚合。
#起始值（对于add方法是0）取决于ufunc。如果传递了一个轴，则沿该轴执行缩聚。
np.add.reduce(arr)
1
2
3

45
1

arr.sum()
1

45
1

#使用np.logical_and来检查数组的每一行中的值是否被排序
np.random.seed(12346)
arr = np.random.randn(5,5)
arr
1
2
3
4

array([[-8.99822478e-02,  7.59372617e-01,  7.48336101e-01,
        -9.81497953e-01,  3.65775545e-01],
       [-3.15442628e-01, -8.66135605e-01,  2.78568155e-02,
        -4.55597723e-01, -1.60189223e+00],
       [ 2.48256116e-01, -3.21536673e-01, -8.48730755e-01,
         4.60468309e-04, -5.46459347e-01],
       [ 2.53915229e-01,  1.93684246e+00, -7.99504902e-01,
        -5.69159281e-01,  4.89244731e-02],
       [-6.49092950e-01, -4.79535727e-01, -9.53521432e-01,
         1.42253882e+00,  1.75403128e-01]])
1
2
3
4
5
6
7
8
9
10

#对行排序
arr[::2].sort(1)
1
2

arr[:,:-1]
1

array([[-9.81497953e-01, -8.99822478e-02,  3.65775545e-01,
         7.48336101e-01],
       [-3.15442628e-01, -8.66135605e-01,  2.78568155e-02,
        -4.55597723e-01],
       [-8.48730755e-01, -5.46459347e-01, -3.21536673e-01,
         4.60468309e-04],
       [ 2.53915229e-01,  1.93684246e+00, -7.99504902e-01,
        -5.69159281e-01],
       [-9.53521432e-01, -6.49092950e-01, -4.79535727e-01,
         1.75403128e-01]])
1
2
3
4
5
6
7
8
9
10

arr[:,1:]
1

array([[-8.99822478e-02,  3.65775545e-01,  7.48336101e-01,
         7.59372617e-01],
       [-8.66135605e-01,  2.78568155e-02, -4.55597723e-01,
        -1.60189223e+00],
       [-5.46459347e-01, -3.21536673e-01,  4.60468309e-04,
         2.48256116e-01],
       [ 1.93684246e+00, -7.99504902e-01, -5.69159281e-01,
         4.89244731e-02],
       [-6.49092950e-01, -4.79535727e-01,  1.75403128e-01,
         1.42253882e+00]])
1
2
3
4
5
6
7
8
9
10

arr[:,:-1] < arr[:,1:]
1

array([[ True,  True,  True,  True],
       [False,  True, False, False],
       [ True,  True,  True,  True],
       [ True, False,  True,  True],
       [ True,  True,  True,  True]])
1
2
3
4
5

#请注意，logical_and.reduce等价于
np.logical_and.reduce(arr[:,:-1] < arr[:,1:],axis=1)
1
2

array([ True, False,  True, False,  True])
1

#accumulate与reduce是相关的，就像cumsum与sum相关一样。
#accumulate生成一个数组，其尺寸与中间“累计”值相同
arr = np.arange(15).reshape((3,5))
arr
1
2
3
4

array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])
1
2
3

np.add.reduce(arr,axis=1)
1

array([10, 35, 60])
1

np.add.accumulate(arr,axis=1)
1

array([[ 0,  1,  3,  6, 10],
       [ 5, 11, 18, 26, 35],
       [10, 21, 33, 46, 60]], dtype=int32)
1
2
3

np.add.reduce(arr,axis=0)
1

array([15, 18, 21, 24, 27])
1

np.add.accumulate(arr,axis=0)
1

array([[ 0,  1,  2,  3,  4],
       [ 5,  7,  9, 11, 13],
       [15, 18, 21, 24, 27]], dtype=int32)
1
2
3

#outer在两个数组之间执行成对的交叉乘积
arr = np.arange(3).repeat([1,2,2])
arr
1
2
3

array([0, 1, 1, 2, 2])
1

np.multiply.outer(arr,np.arange(5))
1

array([[0, 0, 0, 0, 0],
       [0, 1, 2, 3, 4],
       [0, 1, 2, 3, 4],
       [0, 2, 4, 6, 8],
       [0, 2, 4, 6, 8]])
1
2
3
4
5

#outer的输出的维度等于输入的维度总和
x,y = np.random.randn(3,4),np.random.randn(5)
x
1
2
3

array([[-1.1049211 ,  0.7239073 , -0.95465401,  0.24438966],
       [-0.14528732, -0.12229477,  0.49165039, -1.55720967],
       [ 0.11172771, -0.26132992,  0.27843076, -0.10798888]])
1
2
3

y
1

array([ 0.11090105, -0.37904993,  2.60555583, -1.02235214,  0.26172618])
1

result = np.subtract.outer(x,y)
result.shape
1
2

(3, 4, 5)
1

#reduceat方法接受一系列的“箱体边缘”，这些箱体边缘表示如何分隔以及聚合数据值
#结果是在arr[0:5]、arr[5:8]和arr[8:]上执行了缩聚（此处是加和）
arr = np.arange(10)
print(arr)
np.add.reduceat(arr,[0,5,8])
1
2
3
4
5

[0 1 2 3 4 5 6 7 8 9]





array([10, 18, 17], dtype=int32)
1
2
3
4
5
6
7

#你可以传递一个axis参数
arr = np.multiply.outer(np.arange(4),np.arange(5))
arr
1
2
3

array([[ 0,  0,  0,  0,  0],
       [ 0,  1,  2,  3,  4],
       [ 0,  2,  4,  6,  8],
       [ 0,  3,  6,  9, 12]])
1
2
3
4

np.add.reduceat(arr,[0,2,4],axis=1)
1

array([[ 0,  0,  0],
       [ 1,  5,  4],
       [ 2, 10,  8],
       [ 3, 15, 12]], dtype=int32)
1
2
3
4

A.4.2 使用Python编写新的ufunc方法

• numpy.frompyfunc函数接收一个具有特定数字输入和输出的函数。

def add_elements(x,y):
    return x+y
1
2

#frompyfunc(func, nin, nout, *[, identity])
#第一个参数函数名，第二个表示输入，第三个表示输出
add_them = np.frompyfunc(add_elements,2,1)
add_them
1
2
3
4

<ufunc 'add_elements (vectorized)'>
1

add_them(np.arange(8),np.arange(8))
1

array([0, 2, 4, 6, 8, 10, 12, 14], dtype=object)
1

#另一个函数numpy.vectorize允许指定输出的类型（但功能稍差）
add_them = np.vectorize(add_elements,otypes=[np.float64])
add_them
1
2
3

<numpy.vectorize at 0x15fa8974a60>
1

add_them(np.arange(8),np.arange(8))
1

array([ 0.,  2.,  4.,  6.,  8., 10., 12., 14.])
1

arr = np.random.randn(10000)
%timeit add_them(arr,arr)
1
2

902 µs ± 6.35 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1

#这种会快很多
%timeit np.add(arr,arr)
1
2

2.7 µs ± 2.16 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
1

A.5 结构化和记录数组

• ndarray是一个同构数据的容器。也就是说，它表示一个内存块，其中每个元素占用相同数量的字节，由dtype确定

dtype = [('x',np.float64),('y',np.int32)]
sarr = np.array([(1.5,6),(np.pi,-2)],dtype=dtype)
sarr
1
2
3

array([(1.5       ,  6), (3.14159265, -2)],
      dtype=[('x', '<f8'), ('y', '<i4')])
1
2

#一种典型的方式是使用(field_name, field_data_type)作为元组的列表
sarr[0]
1
2

(1.5, 6)
1

sarr[0]['y']
1

6
1

sarr['x']
1

array([1.5       , 3.14159265])
1

A.5.1 嵌套dtype和多维字段

• 当指定结构化的dtype时，你可以另外传递一个形状（以int或元组的形式）

dtype = [('x',np.float64,3),('y',np.int32)]
arr = np.zeros(4,dtype=dtype)
arr
1
2
3

array([([0., 0., 0.], 0), ([0., 0., 0.], 0), ([0., 0., 0.], 0),
       ([0., 0., 0.], 0)], dtype=[('x', '<f8', (3,)), ('y', '<i4')])
1
2

#在这种情况下，x字段引用的是每条记录中长度为3的数组
arr[0]['x']
1
2

array([0., 0., 0.])
1

arr['x']
1

array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]])
1
2
3
4

dtype = [('x',[('a','f8'),('b','f4')]),('y',np.int32)]
data = np.array([((1,2),5),((3,4),6)],dtype=dtype)
1
2

data['x']
1

array([(1., 2.), (3., 4.)], dtype=[('a', '<f8'), ('b', '<f4')])
1

data['y']
1

array([5, 6])
1

data['x']['a']
1

array([1., 3.])
1

A.5.2 为什么要使用结构化数组

• 结构化数组提供了一种将内存块解释为具有任意复杂嵌套列的表格结构的方法。
• 由于数组中的每个元素都在内存中表示为固定数量的字节，因此结构化数组提供了读/写磁盘（包括内存映射）数据，以及在网络上传输数据和其他此类用途的非常快速有效的方法。

A.6 更多关于排序的内容

• ndarray的sort实例方法是一种原位排序，意味着数组的内容进行了重排列，而不是生成了一个新的数组

arr = np.random.randn(6)
arr
1
2

array([ 0.51034093, -1.21799778, -0.27034648, -1.33534252, -0.78528729,
       -1.10908521])
1
2

arr.sort()
1

arr
1

array([-1.33534252, -1.21799778, -1.10908521, -0.78528729, -0.27034648,
        0.51034093])
1
2

#在进行数组原位排序时，请记住如果数组是不同ndarray的视图的话，原始数组将会被改变
arr = np.random.randn(3,5)
arr
1
2
3

array([[-0.00369513, -0.15297778, -0.46090167, -0.42008296, -0.91017112],
       [-1.05144731,  1.41433111,  0.22343751,  1.98200412, -0.11843381],
       [-1.71099598, -0.77901664,  1.9175701 , -0.36801273,  0.35893302]])
1
2
3

#对第一列的值原位排序
arr[:,0].sort()
1
2

#只有第一列数据排序有变化
arr
1
2

array([[-1.71099598, -0.15297778, -0.46090167, -0.42008296, -0.91017112],
       [-1.05144731,  1.41433111,  0.22343751,  1.98200412, -0.11843381],
       [-0.00369513, -0.77901664,  1.9175701 , -0.36801273,  0.35893302]])
1
2
3

#numpy.sort产生的是一个数组的新的、排序后的副本
arr = np.random.randn(5)
arr
1
2
3

array([ 0.83175214,  0.0981957 , -0.16337765,  1.57507692,  1.20540736])
1

np.sort(arr)
1

array([-0.16337765,  0.0981957 ,  0.83175214,  1.20540736,  1.57507692])
1

#经过np.sort()数组是不会改变的
arr
1
2

array([ 0.83175214,  0.0981957 , -0.16337765,  1.57507692,  1.20540736])
1

#所有这些排序方法都有一个axis参数，用于独立地沿着传递的轴对数据部分进行排序
arr = np.random.randn(3,5)
arr
1
2
3

array([[ 0.48623846,  1.40501429,  0.21771959, -0.6147521 , -1.03729051],
       [ 0.00466416,  1.31854631, -0.09256828, -1.03503114,  0.70669487],
       [-0.06967569, -0.55095404,  0.87325007, -1.9579896 , -0.10276109]])
1
2
3

#按照行排序，会改变arr
arr.sort(axis=1)
1
2

arr
1

array([[-1.03729051, -0.6147521 ,  0.21771959,  0.48623846,  1.40501429],
       [-1.03503114, -0.09256828,  0.00466416,  0.70669487,  1.31854631],
       [-1.9579896 , -0.55095404, -0.10276109, -0.06967569,  0.87325007]])
1
2
3

arr.sort(axis=0)
1

#按照行排序，会改变arr
arr
1
2

array([[-1.9579896 , -0.6147521 , -0.10276109, -0.06967569,  0.87325007],
       [-1.03729051, -0.55095404,  0.00466416,  0.48623846,  1.31854631],
       [-1.03503114, -0.09256828,  0.21771959,  0.70669487,  1.40501429]])
1
2
3

• 你可能会注意到所有的排序方法都没有降序排列的选项。
• 这是一个实践中的问题，因为数组切片会产生视图，因此不需要生成副本也不需要任何计算工作。

arr[:,::-1]
1

array([[ 0.87325007, -0.06967569, -0.10276109, -0.6147521 , -1.9579896 ],
       [ 1.31854631,  0.48623846,  0.00466416, -0.55095404, -1.03729051],
       [ 1.40501429,  0.70669487,  0.21771959, -0.09256828, -1.03503114]])
1
2
3

A.6.1 间接排序：argsort和lexsort

• pandas的方法，比如Series和DataFrame的sort_values方法是对这些方法的变相实现（这些方法也必须要考虑缺失值）

values = np.array([5,0,1,3,2])
indexer = values.argsort()
indexer
1
2
3

array([1, 2, 4, 3, 0], dtype=int64)
1

values[indexer]
1

array([0, 1, 2, 3, 5])
1

#对一个二维数组按照它的第一行进行重新排序
arr = np.random.randn(3,5)
arr[0] = values
arr
1
2
3
4

array([[ 5.        ,  0.        ,  1.        ,  3.        ,  2.        ],
       [ 1.01782863, -1.18082614,  0.66861266, -1.51142124, -0.91934196],
       [ 1.16468714,  0.12410901,  1.69151564,  0.8931546 ,  0.16763928]])
1
2
3

arr[:,arr[0].argsort()]
1

array([[ 0.        ,  1.        ,  2.        ,  3.        ,  5.        ],
       [-1.18082614,  0.66861266, -0.91934196, -1.51142124,  1.01782863],
       [ 0.12410901,  1.69151564,  0.16763928,  0.8931546 ,  1.16468714]])
1
2
3

#lexsort类似于argsort，但它对多键数组执行间接字典排序
first_name = np.array(['Bob','Jane','Steve','Bill','Barbara'])
last_name = np.array(['Jone','Arnold','Arnold','Jone','Walters'])
sorter = np.lexsort((first_name,last_name))
sorter
1
2
3
4
5

array([1, 2, 3, 0, 4], dtype=int64)
1

first_name[sorter]
1

array(['Jane', 'Steve', 'Bill', 'Bob', 'Barbara'], dtype='<U7')
1

last_name[sorter]
1

array(['Arnold', 'Arnold', 'Jone', 'Jone', 'Walters'], dtype='<U7')
1

zip(first_name[sorter],last_name[sorter])
1

<zip at 0x15fa89182c0>
1

A.6.2 其他的排序算法

values = np.array(['2:first','2:second','1:first','1:second','1:third'])
key = np.array([2,2,1,1,1])
indexer = key.argsort(kind='mergesort')
indexer
1
2
3
4

array([2, 3, 4, 0, 1], dtype=int64)
1

values.take(indexer)
1

array(['1:first', '1:second', '1:third', '2:first', '2:second'],
      dtype='<U8')
1
2

A.6.3 数组的部分排序

• 排序的目标之一可以是确定数组中最大或最小的元素。
• NumPy已经优化了方法numpy. partition和np.argpartition，用于围绕第k个最小元素对数组进行分区

np.random.seed(12345)
arr = np.random.randn(20)
arr
1
2
3

array([-0.20470766,  0.47894334, -0.51943872, -0.5557303 ,  1.96578057,
        1.39340583,  0.09290788,  0.28174615,  0.76902257,  1.24643474,
        1.00718936, -1.29622111,  0.27499163,  0.22891288,  1.35291684,
        0.88642934, -2.00163731, -0.37184254,  1.66902531, -0.43856974])
1
2
3
4

#在调用partition(arr,3)之后，结果中的前三个元素是最小的三个值，并不是特定的顺序。
np.partition(arr,3)
1
2

array([-2.00163731, -1.29622111, -0.5557303 , -0.51943872, -0.37184254,
       -0.43856974, -0.20470766,  0.28174615,  0.76902257,  0.47894334,
        1.00718936,  0.09290788,  0.27499163,  0.22891288,  1.35291684,
        0.88642934,  1.39340583,  1.96578057,  1.66902531,  1.24643474])
1
2
3
4

#numpy.argpartition类似于numpy.argsort排序，它返回的是将数据重新排列为等价顺序的索引
indices = np.argpartition(arr,3)
indices
1
2
3

array([16, 11,  3,  2, 17, 19,  0,  7,  8,  1, 10,  6, 12, 13, 14, 15,  5,
        4, 18,  9], dtype=int64)
1
2

arr.take(indices)
1

array([-2.00163731, -1.29622111, -0.5557303 , -0.51943872, -0.37184254,
       -0.43856974, -0.20470766,  0.28174615,  0.76902257,  0.47894334,
        1.00718936,  0.09290788,  0.27499163,  0.22891288,  1.35291684,
        0.88642934,  1.39340583,  1.96578057,  1.66902531,  1.24643474])
1
2
3
4

A.6.4 numpy.searchsorted：在已排序的数组寻找元素

• searchsorted是一个数组方法，它对已排序数组执行二分搜索，返回数组中需要插入值的位置以保持排序

arr = np.array([0,1,7,12,15])
arr.searchsorted(9)
1
2

3
1

#你还可以传递一个值数组来获取一个索引数组
#对于0元素，searchsorted返回0。这是因为默认行为是返回一组相等值左侧的索引
arr.searchsorted([0,8,11,16])
1
2
3

array([0, 3, 3, 5], dtype=int64)
1

arr = np.array([0,0,0,1,1,1,1])
arr.searchsorted([0,1])
1
2

array([0, 3], dtype=int64)
1

arr.searchsorted([0,1],side='right')
1

array([3, 7], dtype=int64)
1

#作为searchsorted的另一个应用，假设我们有一个介于0和10,000之间的数值，
#以及我们想用来分隔数据的单独的“桶边界”数组
data = np.floor(np.random.uniform(0,10000,size=50))
data
1
2
3
4

array([9940., 6768., 7908., 1709.,  268., 8003., 9037.,  246., 4917.,
       5262., 5963.,  519., 8950., 7282., 8183., 5002., 8101.,  959.,
       2189., 2587., 4681., 4593., 7095., 1780., 5314., 1677., 7688.,
       9281., 6094., 1501., 4896., 3773., 8486., 9110., 3838., 3154.,
       5683., 1878., 1258., 6875., 7996., 5735., 9732., 6340., 8884.,
       4954., 3516., 7142., 5039., 2256.])
1
2
3
4
5
6

bins = np.array([0,100,1000,5000,100000])
labels = bins.searchsorted(data)
labels
1
2
3

array([4, 4, 4, 3, 2, 4, 4, 2, 3, 4, 4, 2, 4, 4, 4, 4, 4, 2, 3, 3, 3, 3,
       4, 3, 4, 3, 4, 4, 4, 3, 3, 3, 4, 4, 3, 3, 4, 3, 3, 4, 4, 4, 4, 4,
       4, 3, 3, 4, 4, 3], dtype=int64)
1
2
3

#可以和pandas的groupby一起被用于分箱数据
pd.Series(data).groupby(labels).count()
1
2

2     4
3    18
4    28
dtype: int64
1
2
3
4

A.7 使用Numba编写快速NumPy函数

• Numba（http://numba.pydata.org）是一个开源项目，可为使用CPU、GPU或其他硬件的NumPy类型的数据创建快速函数。
• Numba不能编译所有的Python代码，但它支持纯Python代码的重要子集，这些代码对于编写数值算法最为有用

#该函数使用for循环计算表达式(x - y).mean()的值
import numpy as np
def mean_distance(x,y):
    nx = len(x)
    result = 0.0
    count = 0
    for i in range(nx):
        result+= x[i] - y[i]
        count+=1
    return result/count
1
2
3
4
5
6
7
8
9
10

x = np.random.randn(1000000)
y = np.random.randn(1000000)
%timeit mean_distance(x,y)
1
2
3

232 ms ± 1.45 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
1

%timeit (x-y).mean()
1

1.51 ms ± 16.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1

#可以使用numba.jit函数将这个函数编译成Numba函数
import numba as nb
numba_mean_distance = nb.jit(mean_distance)
1
2
3

#可以写成装饰器的形式
@nb.jit
def mean_distance(x,y):
    nx = len(x)
    result = 0.0
    count = 0
    for i in range(nx):
        result+= x[i] - y[i]
        count+=1
    return result/count
1
2
3
4
5
6
7
8
9
10

%timeit numba_mean_distance(x,y)
1

670 µs ± 48.8 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
1

from numba import float64,njit
@njit(float64(float64[:],float64[:]))
def mean_distance(x,y):
    return (x-y).mean()
1
2
3
4

A.7.1 使用Numba创建自定义numpy.ufunc对象

• numba.vectorize函数创建了编译好的NumPy ufunc，其行为也和内建的ufunc类似。

from numba import vectorize
@vectorize
def nb_add(x,y):
    return x+y
1
2
3
4

x = np.arange(10)
nb_add(x,x)
1
2

array([ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18], dtype=int64)
1

A.8 高阶数组输入和输出

A.8.1 内存映射文件

• 内存映射文件是一种与磁盘上的二进制数据交互的方法，就像它是存储在内存数组中一样
• NumPy实现了一个memmap对象，它是ndarray型的，允许对大型文件以小堆栈的方式进行读取和写入，而无须将整个数组载入内存。
• 此外，memmap还有和内存数组相同的方法，因此可以替代很多算法中原本要填入的ndarray。

mmap = np.memmap('mymmap',dtype='float64',mode='w+',shape=(10000,10000))
mmap
1
2

memmap([[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]])
1
2
3
4
5
6
7

#对memmap切片返回的是硬盘上数据的视图
section = mmap[:5]
section
1
2
3

memmap([[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]])
1
2
3
4
5

#如果你将数据赋值给这些切片，它将会在内存中缓冲（类似于一个Python文件对象），但你可以调用flush将数据写入硬盘
section[:] = np.random.randn(5,10000)
mmap.flush()
1
2
3

mmap
1

memmap([[ 0.41110843,  0.58204806,  1.2463012 , ...,  0.06582078,
         -0.34734378,  0.62280733],
        [-2.21583571,  0.29678775,  0.57086919, ...,  0.07007184,
         -0.26204433, -0.30061136],
        [ 0.77817885,  0.74008809,  0.49653126, ..., -0.51072764,
          1.11806763,  0.09285284],
        ...,
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ]])
1
2
3
4
5
6
7
8
9
10
11
12
13

del mmap
1

mmap = np.memmap('mymmap',dtype='float64',shape=(10000,10000))
mmap
1
2

memmap([[ 0.41110843,  0.58204806,  1.2463012 , ...,  0.06582078,
         -0.34734378,  0.62280733],
        [-2.21583571,  0.29678775,  0.57086919, ...,  0.07007184,
         -0.26204433, -0.30061136],
        [ 0.77817885,  0.74008809,  0.49653126, ..., -0.51072764,
          1.11806763,  0.09285284],
        ...,
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ]])
1
2
3
4
5
6
7
8
9
10
11
12
13

A.8.2 HDF5和其他数组存储选择

• PyTables和h5py是两个为NumPy提供友好接口的Python项目，用于以高效和可压缩的HDF5格式存储数组数据（HDF代表分层数据格式，Hierarchical Data Format）

A.9 性能技巧

• 注意事项
  • 将Python循环和条件逻辑转换为数组操作和布尔数组操作
  • 尽可能使用广播
  • 使用数组视图（切片）来避免复制数据
  • 使用ufunc和ufunc方法

A.9.1 连续内存的重要性

arr_c = np.ones((1000,1000),order='c')
arr_f = np.ones((1000,1000),order='F')
arr_c.flags
1
2
3

  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False
1
2
3
4
5
6
7

arr_f.flags
1

  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False
1
2
3
4
5
6
7

%timeit arr_c.sum(1)
1

241 µs ± 1.42 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1

%timeit arr_f.sum(1)
1

238 µs ± 589 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1

arr_f.copy('C').flags
1

  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False
1
2
3
4
5
6
7

#在数组上构建视图时，请记住结果并不能保证是连续的
arr_c[:50].flags.contiguous
1
2

True
1

arr_c[:,:50].flags
1

  C_CONTIGUOUS : False
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False
1
2
3
4
5
6
7

附录B更多IPython系统相关内容

• IPython维护一个小的磁盘数据库，其中包含你执行的每条命令的文本。这些文本有多种用途：
  • 以最少的打字搜索、完成并执行先前执行过的命令
  • 在会话之间保持命令历史记录
  • 将输入/输出历史日志，记录到文件

B.1.1 搜索和复用命令历史

B.1.2 输入和输出变量

• 前两个输出分别存储在_（一个下划线）和__（两个下划线）变量中

2**27
1

134217728
1

_
1

134217728
1

• 输入变量存储在名为_iX的变量中，其中X是输入行号。对于每个输入变量，都有一个对应的输出变量_X。

foo = 'bar'
foo
1
2

'bar'
1

_i137
1

"foo = 'bar'\nfoo"
1

_137
1

'bar'
1

#由于输入变量是字符串，因此可以使用Python exec关键字再次执行它们
exec(_i27)
1
2

• % hist可以用包含或不包含行号的形式打印全部或部分输入历史记录。
• % reset用于清除交互式命名空间以及可选的输入和输出缓存。
• % xdel魔术函数用于从IPython机器中移除对特定对象的所有引用。

%hist
1

B.2 与操作系统交互

B.2.1 shell命令及其别名

• 通过将以！转义的表达式赋值给变量，你可以把命令行的shell输出存储在一个变量中。
• % alias魔术函数可以为shell命令定义自定义快捷键
• 你会注意到IPython会在会话关闭后“忘记”所有你在交互中定义的别名。要创建永久别名，你需要使用配置系统。

B.2.2 目录书签系统

• %bookmark
• 使用%bookmark和-l选项，将列出你所有的书签

B.3 软件开发工具

B.3.1 交互式调试器

• Python调试器命令

B.3.1.1 调试器的其他用途

• 第一个函数set_trace是非常简单的。你可以在代码的任何部分使用set_trace来临时停止，以便更仔细地检查代码
• 按c（continue，继续）将导致代码恢复正常，不会造成任何损害。

B.3.2 对代码测时：%time和%timeit

• %time一次运行一条语句，并报告总执行时间。
• 给定任意的语句，%timeit有多次运行语句以产生更准确的平均运行时间的功能

B.3.3 基础分析：%prun和%run -p

B.3.4 逐行分析函数

B.4 使用IPython进行高效代码开发的技巧

B.4.1 重载模块依赖项

B.4.2 代码设计技巧

B.4.2.1 保持相关对象和数据的存在

B.4.2.2 扁平优于嵌套

B.4.2.3 克服对长文件的恐惧

B.5 高阶IPython特性

B.5.1 使你自定义的类对IPython友好

B.5.2 配置文件与配置

• 下面这些事情都可以通过配置来完成：
  • 更改颜色主题
  • 更改输入输出的外观，或者去除Out之后和下一个In之前的空白行
  • 执行任意的Python语句列表（例如，导入你总是使用的库，或者是其他你希望每次你启动IPython就运行的程序）
  • 始终启用IPython扩展，如line_profiler中的% lprun魔术函数
  • 激活Jupyter拓展
  • 自定义魔术函数或系统别名