Python字符串模糊匹配:thefuzz_模糊匹配 fuzz

在查询数据时，字符串匹配经常用到模糊匹配，这时就要用到模糊匹配算法，如Levenshtein Distance 算法，计算编辑距离，这里Python的thefuzz包实现了模糊匹配功能。

安装

pip install thefuzz
1

使用

简单匹配

from thefuzz import fuzz

rt = fuzz.ratio("我在山东", "我在山东省")
print(rt)  # 89
1
2
3
4

非完全匹配

非完全匹配，精度较高。

from thefuzz import fuzz

rt = fuzz.partial_ratio("我在山东", "我在山东省")
print(rt)  # 100
1
2
3
4

忽略顺序匹配（Token Sort Ratio）

from thefuzz import fuzz

rt = fuzz.token_sort_ratio("我在 山东", "山东 我在")
print(rt)  # 100
1
2
3
4

去重子集匹配（Token Set Ratio）

from thefuzz import fuzz

rt = fuzz.token_set_ratio("我在 山东", "山东 我在 山东")
print(rt)

1
2
3
4
5

extract提取多条数据

from thefuzz import process

choices = ["山东", "山东省", "青岛市", "青岛"]
ls = process.extract("青岛", choices, limit=2)
print(ls)
# [('青岛', 100), ('青岛市', 90)]
1
2
3
4
5
6

如果只提取一条，limit参数可以改为1，或者用extractOne()。

from thefuzz import process

choices = ["山东", "山东省", "青岛市", "青岛"]
ls = process.extractOne("青岛", choices)
print(ls)

1
2
3
4
5
6

Pandas Dataframe字段对比

from thefuzz import process


# 模糊匹配

def fuzzy_merge(df_1, df_2, key1, key2, threshold=90, limit=2):
    """
    :param df_1: the left table to join
    :param df_2: the right table to join
    :param key1: key column of the left table
    :param key2: key column of the right table
    :param threshold: how close the matches should be to return a match, based on Levenshtein distance
    :param limit: the amount of matches that will get returned, these are sorted high to low
    :return: dataframe with boths keys and matches
    """
    s = df_2[key2].tolist()

    m = df_1[key1].apply(lambda x: process.extract(x, s, limit=limit))
    df_1['matches'] = m

    m2 = df_1['matches'].apply(
        lambda x: [i[0] for i in x if i[1] >= threshold][0] if len([i[0] for i in x if i[1] >= threshold]) > 0 else '')
    df_1['matches'] = m2

    return df_1


df = fuzzy_merge(df1, df2, '字段1', '字段2', threshold=90)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29