《精通Python自然语言处理（ Deepti Chopra)》读书笔记（第十章）：NLP系统评估_len(label1.intersection(label2))) / len(

《精通Python自然语言处理》

Deepti Chopra(印度)
王威 译

---

第十章 NLP系统评估：性能分析

---

10.1 NLP系统评估要点

创建黄金标准注释语料库是一项主要的任务，而且其实成本也是非常昂贵的。它通过手工标注给定的测试数据来完成该操作。以这种方式筛选的标记被视为标准标记，其可用于表示大范围的信息。

10.1.1 NLP工具的评估（词性标注器、词干提取器及形态分析器）

训练一个一元语法标注器：

import nltk
from nltk.corpus import brown
sentences=brown.tagged_sents (categories= 'news')
sent=brown.sents (categories='news' )
unigram_sent = nltk.UnigramTagger (sentences)
print(unigram_sent.tag (sent [2008]))
print(unigram_sent.evaluate (sentences))     
1
2
3
4
5
6
7

使用分离的数据对一元语法标注器执行训练和测试：

import nltk
from nltk.corpus import brown
sentences = irown.tagged_sents (categories= 'news’)
sz=int (len (sentences)*0.8)
print(sz)
training_sents = sentences[:sz]
testing_sentssentences [sz:]
unigram_tagger = nltk.UnigramTagger (training_sents)
print(unigram_cagger.evaluate (testing_sents))
1
2
3
4
5
6
7
8
9

使用bigram（二元语法）标注器：

import nltk
from nltk.corpus import brown
sentences = brcwn.tagged_sents (categories= 'news')
s2 = int (len (sentences)*0.8)
training_sents = sentences[:sz]
testing_sents = sentences[sz:]
bigram_tagger = nltk.UnigramTagger (training_sents)
bigram_tagger = nltk.BigramTagger (training_sents)
print(bigram_tagger.tag (sentences[2008]))

un_sent = sentences [4203]
print(bigram_tagger.tag(un_sent))

print(bigram_tagger.evaluate (testing sents))
1
2
3
4
5
6
7
8
9
10
11
12
13
14

实现组合标注器：

import nltk
from nltk.corpus import brown
sentence = brown.tagged_sents (categories = ' news')
sz=int (len(sentences)*0.8)
training_sents = sentences[:sz]
tesling_sents = sentences[sz:]
s0 = nltk.DefaultTagger('NNP')
s1 = nltk.UnLgramTagger (training_sents, backoff = s0)
s2 = n1k.BigramTagger (training_sents,backoff = s1)
print(s2.evaluate(testing_sents))
1
2
3
4
5
6
7
8
9
10

语块解析器的评估:

import nltk
chunkparser = nltk.RegexpParser ("")
print (nltk.chunk.accuracy(chunkparser, nltk.corpus.con112000.chunked_sents(
	'train.txt', chunk_types=('NP',))))
1
2
3
4

朴素语块解释器的评估：

import nltk
grammar = r"NP: (< [CDJNP]. *>+}"
cp = nltk.RegexpParser (grammar)
print(nltk.chunk.accuracy(cp, nltk.corpus.con112000.chunked_sents(
		'train.txt', chunk_types = ('NP',))))
1
2
3
4
5

计算分块数据的条件频率分布：

def chunk_tags(train) :
	"""Generate a following tags list chat appears inside chunks"""
	cfreqdist = nltk.ConditionalFreqDist()
	for t in train:
		for word, tag, chunktag in nltk.chunk.tree2conlltags(t):
			if chtag == "O":
				cfreqdist[tag].inc (False)
			else:
				cfreqdist[tag].inc (True)
	return [tag for tag in cfreqdist.conditions() if cfreqdist [tag] .max() == True]
training_sents = nltk.corpus.conll2000.chunked_sents('train.txt', chunk_types = ('NP',))
print(chunked_tags (train_sents))
1
2
3
4
5
6
7
8
9
10
11
12

执行chunker评估：

import nltk
correct = nltk.chunk.tagstr2tree ("[ the/DT little/JJ cat/NN ] sat/VBD on/IN [ the/DT mat/NN ]")
print (correct. flatten())
grammar = r"NP: {< [CDJNP] . *>+}”
cp = nltk.RegexpParser (grammar)
grammar = r"NP: {<PRP|DT| POS| JJ|CD|N.*>+)”
chunk_parser = nltk.RegexpParser (grammar)
tagged_tok = [("the", "DT"), (“little", "JJ"), ( "cat","NN"), ("sat", "VBD"), ("on","IN"), ("the", "DT"), "mat", "NN")]
chunkscore = nltk.chunk.ChunkScore()
guesaed = cp.parse(correct.flatten())
chunkscore.score(correct, guessed)
print (chunkscore) 
1
2
3
4
5
6
7
8
9
10
11
12

评估一元语法chunker和二元语法chunker：

chunker_data = [[(t,c) for w, t, c in nltk.chunk.tree2conlltags (chtree)]
			for chtree in nltk.corpus.conll2000.chunked_sents('train.txt')]
unigram chunk = nltk.UnigramTagger (chunker_data)
print (nItk.tag.accuracy (unigram_chunk, chunker_data))

bigram_chunk = nltk.BigramTagger(chunker_data, backoff_unigram_chunker)
print( nltk. tag.accuracy (bigram_chunk, churker_data))
1
2
3
4
5
6
7

使用一个特征提取函数来检测给定的单词中所呈现的后缀，并用后缀来确定词性标记：

from nltk.corpus import brown
suffix_freqdist = nltk.FreqDist()
for wrd in brown.words() :
	wrd = wrd.lower()
	suffix_freqdist [wrd[-1:]] += 1
	suffix_fdist[wrd[-2:]] += 1
	suffix_fdist[wrd[-3:]] += 1
common_suffixes = [suffix for (suffix, count) in suffix_freqdist.most _common(100) ]
print (common_suffixes)

def pos_feature (wrd) :
	feature = { }
	for suffix in common_suffixes:
		feature['endswith({}) '. format(suffix)] = wrd.lower.endswith (suffix)
	return feature
tagged_wrds = brown.taged_wrds (categories = 'news ')
featureset = [(pos_feature(n), g)  for  (n,g)  in  tagged_wrds]
size = int (len (featureset) * 0.1)
train_set, test_set = featureset[size:], featureset[:size]
classifier1 = nltk.DecisionTreeClassifier. train(train_set)
print(nltk.classify.accuracy (classifier1, test_set))

classifier.classify(pos_features( 'cats'))
'NNS '

print (classifier.pseudocode (depth=4) )
if endswith(,) == True: return ','
if endswith(,) == False:
	if endswithlthe) == True: return 'AT'
	if endswith(the) == False:
		if endswith(s) == True:
			if endswith(is) == True: return ' BEZ'
			if endswith(is) == False: return 'VBZ'
		if endswith(s) == False:
			if endswith(.) == True: return' ·'
			if endswith(.) == False: return 'NN'
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

构建一个正则表达式标注器，并基于匹配模式进行标记分配：

import nltk
from nltk.corpus import brown
sentences = brown.tagged_sents (categories = 'news' )
sent = brown.sents (categories= 'news')
pattern= [
	(r’ .*ing$', 'VBG'),  				# for gerunds
	(z'. *eds', 'VBD'),  				# for simple past
	(r' . *es$', 'VBZ')，  				#for 3rd singular present
	(r'.*oulds', 'MD')，  				#for rodals
	(z'.*\'s$', 'NNS'),  				#for possessive nouns
	(r'. *s$'，'NNS'),  				#for plural nouns
	(r'^-?[0-9] + (.[0-9]+)?$','CD'), 	#for cardinal numbers
	(r' .*’, 'NN')  					#for nouns (default)
	]
regexpr_tagger = nitk .RegexpTagger (pattern)
print(regexpr_tagger.tag(sent[3]))

print(regexp_tagger.evaluate (sentences))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

构建查找标注器（由于一些单词不再最常用的单词列表中，需要分配一个None标记）：

import nltk
from nltk.corpus import brown
freqd = nltk.FreqDist (brown.words (categories = 'news'))
cfreqd = n1tk.ConditicnalFreqDist (brown.tagged_words (categories = 'news')
mostfreq_words = freqd.most_common(100) 
likelytags = dict (word, cfreqdlwordj.max()) for (word,_ )  in mostfreq_words)
baselinetagger = nItk.UnigramTagger (model = likelytags)
print(baselinetagger.evaluate (brown tagged sents))

sent = brown.sents(categories = 'news') [3]
print(baselinetagger.tag(sent))

baselinetagger = nltk.UnigramTagger (model=likely_ tags, 	backoff=nltk.

				DefaultTagger('NN'))
def performance (cfreqd, wordlist) :
	It = dict( (word, cfreqd[word] ,max()) for word in wordlist)
	Baseline_tagger = nltk.UnigramTagger (model = lt, backoff = nltk.DefaultTagger('NN'))
	return baseline_tagger.evaluate (brown.tagged_sents (categories = 'news')) 

def display() :
	import pylab
	word_freqs = nltk.FreqDist (browm.words (categories = 'news')).most_common ()
	words_by_freq = [w for (w, ) in word_freqs]
	cfd = nltk.ConditionalFreqDist (brown.agged_words (categories = 'news')) 
	sizes = 2 ** pylab.arange(15)
	perfs = lperformance(cfd, words_by_freq[:size]) for size in sizes]
	pylab.plot(sizes, perfs, '-bo')
	pylab.title('Lookup Tagger Performance with Varying Model Size')
	pylab.xlabel('Model Size')
	pylab.ylabel('Performance')
	pylab.show()
display()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

使用lancasterstemmer进行词干提取（使用黄金测试数据来完成一个stemmer的评估）：

import nltk 
from nltk.stem.lancaster import Lancasterstemmer
stri=LancasterStemmer ()
print(stri.stem('achievement'))
1
2
3
4

使用最大熵分类器设计一个基于分类的chunker：

class conseNPChunkagger( nltk.laggerI):
	def __init__(self, train_sente):
		train_set = []
		for tagsent in train_sents:
			untagsent = nitk.cag.untag(tagsent)
			history= []
			for i, (word, tag) in enumerate (tagsent):
				featureset = mpchunk_features (untagsent, i, history)
				train_set.append( (featureset, tag) )
				histcry.append(tag)
		self.classifier = nltk.MaxentClassifier.train(train_set, algorithm = ' megam', trace = 0)

	def tag(self, sentence) ;
		history = []
		for i, word in enumerate (sentence) :
			featureset = npchunk_features(sentence, i, history)
			tag = self.classifier.classify (featureset)
			histcry.append(tag)
		return zip(sentence, history)

class ConseNPChunker (nltk，ChunkParserI): [4]
	def__init__(self, train_sents) :
		tagsent = [ [ (w,t),c) for (w, t, c) in nltk.chunk.tree2conlltags(sent) ]
			for sent in train_sents]
		self.tagger = ConseNPChunkTagger (tagsent〉

	def parse (self, sentence) :
		tagsent= self.tagger.tag (sentence)
		conlltags = [(w, t, c) for (w, t), c] in tagsent]
		return nltk.chunk.conlltags2tree (conlltags)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

使用一个特征提取器执行chunker评估：

def npchunk_features (sentence, i, history) :
…	word, pos = sentence[i]
…	return {"pos": pos}
	chunker = ConseNPChunker (train_sents)
	print (chunker.evaluate (test_sents) )
1
2
3
4
5

Chunker评估（类似于二元语法chunker）:

def npchunk features (sentence, i, history):
…	word, pos = sentence[i]
…	if i == 0:
…		previword, previpos = "<START>", "<START>"
…	else:
…		previword, previpos = sentence[i - 1]
…	return {"pos": pos, "previpos": previpos}
	chunker = ConseNPChunker (train_sents)
	print (chunker.evaluate(test_sents)) 
1
2
3
4
5
6
7
8
9

Chunker评估(添加当前单词的特征以便提高chunker的性能)：

def npchunk_features (sentence, i, history) :
…	word, pos = sentence[i]
…	if I == 0:
…		previword, previpos = "<START>", "<START>"
…	else:
…		previword, previpos = sentence[i-1]
...  return {"pos"; pos, "word": word, "previpos": previpos}
	chunker = ConseNPChunker (train_sents)
	print (chunker.evaluate(test_sents))
1
2
3
4
5
6
7
8
9

Chunker评估(添加特征集以便提高chunker的性能)：

def npchunk features (sentence, i，history):
	word, pos = sentence[i]
	if i == 0:
		previword, previpos = "<START>", "<START>”
	else:
		previword, previpos = sentence[i-1]
	if i == len(sentence) - 1:
		nextword, nextpos = "<END>", "<END>"
	else:
		nextword, nextpos = sentence[i+1]
	return {"pos": pos,
			"word": word,
			"previpos": previpos,
			"nextpos": nextpos,
			"previpost + pos": "%s + %s" % (previpos, pos) ,
			"pos+nextpos": "%s + %s" % (pos, nextpos),
			"tags-since-dt": tags_since_dt(sentence, 1) }
def tags_since_dt (sentence, i) :
	tags = set()
	for word, pos in sentence[:i] :
		if pos =='DT':
			tags = set()
		else:
			tags.add (pos)
	return ‘+' .join (sorted(tags))

chunker = ConsecutiveNPChunker (train_sents)
print (chunker.evaluate (test_sents))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

10.1.2使用黄金数据执行解析器评估

以下两个手段可用于解析器性能的评估：

1. 标记的依恋评估（Labelled Attachment Score，LAS）
2. 标记的精确匹配（Labelled Exact Match，LEM）

---

10.2 IR系统的评估

IR系统评估需考虑如下几个方面：

• 所需资源、
• 文档的表述、
• 市场评估或用户黏性、
• 检索速度、
• 构建查询时的协助、
• 查找所需文档的能力。

（通常使用精确率、召回率、F值来评估IR系统）

---

10.3错误识别指标

错误识别是一个非常重要的可影响NLP系统性能的方面，可能涉及以下术语：

（通常使用精确率、召回率、F值来评估这个指标）

---

10.4基于词汇搭配的指标

为了检测一个给定的单词是否存在于文档中，构建一个特征提取器：

from nltk.corpus inport movie_reviews
docs = [(list(movie_reviews.words(fileid)),category)
			for category in movie_reviews.categories()
			for fileid in movie_reviews.fileids(category)]
random. shuffle (docs)
all_ wrds = nltk.FreqDist(w.lower() for w in movie_reviews.words())
word_features = list(all_wrds) [:2000]

def doc_features (doc) :
	doc_words.set (doc)
	features = {}
	for word in word_ features:
		features['contains({}) '.format (word)] = (word in doc_words)
	return features
print (doc_features (movie_reviews.words ('pos/cv957_8737.txt')))
print (nltk.classify.aecuracy(classifier, test_set))

classifier.show_most_informative_features(5)
Most Informative Features
	contains (outstanding) = True pos : neg = 11.1 :1.0
		contains (seagal) = True neg : pos = 7.7 :1.0
	contains (wonderfully) = True pos : neg = 6.8 :1.0
		contains(damon) = True pos : neg = 5.9 :1.0
		contains(wasted) = True neg : pos = 5.8 :1.0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

用来确定给定的输出与预期的输出是否相同的指标：

from __future__ import print_function
from __future__ import division
def _edit_dist_init(len1, len2):
	lev = []
	for i in range (len1):
		lev. append([0] * len2) 		#initialization of 2D array to zero
	for i in range (len1):
		lev[i][0] = I	# column 0: 0,1,2,3….
	for j in range(len2):
		lev[0][j] = j 	#row 0: 0,1,2,3,4…
	return  lev

def _edit_dist_step(lev, i, j, s1, s2, transpositions = False):
	c1 = s1[i – 1]
	c2 = s2[j – 1]
	# skipping a character in s1
	a = lev[i - 1][j] + 1
	# skipping a character in s2
	b = lev[i][j – 1] + 1
	#substicution
	c = lev[i - 1][j – 1] + (c1 != c2)
	#transposition
	d=c+1	# never picked by default
	if transpositions and i> 1 and j > 1:
		if s1[i - 2] = c2 and s2[j – 2] == c1:
			d = lev[i – 2][j - 2] + 1
	# pick the cheapest
	lev[i][j] = min(a, b, c, d)

def edit_distance(s1, s2, transpositions=False) :
	#set up a 2-D array
	len1 = len(s1)
	len2 = len(s2)
	lev =_edit_dist_init(len1 + 1, 1en2 + 1)

	# iterate over the array
	for i in range(len1):
		for j in range(len2) :
			_edit_dist_step(lev, i + 1, j +1, s1, s2,transpositions = transpositions)
	return lev[len1] [len2]

def binary.distance (label1, label2) :
	"""Simple equality test .
	0.0 if the labels are identical, 1.0 if they are diferent.

from nltk.metrics import
print(binary_distance(1,2))
print(binary_distance(1,3)
	“””
	return 0.0 if label1 == label2 else 1.0

def jaccard_distance(label1, label2):
	“""Distance metric comparing set-similarity.”””
	return  (len(label1.union(labe12) - len(label1.intersection(label2))) / 
				len(labell.union (label2))

def masi_ distance (labell, label2):
	len_intersection = len(label1. intersection (label2))
	len_union = len (label1.union (label2))
	len_label1 = len(labell)
	len_label2 = len(label2)
	if len_label1 == len_label2 and len_ label1 == len_intersection:
		m=1
	elif len_intersection = min(len_label1, len_label2) :
		m =0.67
	elif len_intersection > 0:
		m= 0.33
	else:
		m = 0
	return 1 - (len_intersection / len_union) * m

def interval_distance (label1, label2):
	try:
		return pow(label1 - label2, 2)
#		return pow(list(label1) [0] – list(label2) [0],2)
	except:
		print ("non-numeric labels not supported with intervaldistance")

def presence(label) :
	return lambda x, y: 1.0 * ((label in x) == (label in y))

def fractional_presence(label):
	return lambda x, y: \ abs(((1.0 / len(x)) - (1.0 / len(y)))) * (label in x and label in y) \ or 0.0 * 			(label not in x and label not in y) \ or abs((1.0 / len(x))) * (label in x and label not in y) \ 		or ((1.0 / len(y))) * (label not in x and label in y)

def custom_distance (file) :
	data =[]
	with open(file, 'p') as infile:
		for l in infile:
			labelA, labelB, diat = l.strip().split("\t")
			labelA = frozenset ( [labelA])
			labelB = frozenset( [labelB])
			data[frozenset([labelA, labelB])] = float(dist)
	return lambda x,y:data [frozenset([x,y])]

def demo():
	edit_distance_examples = [
		("rain", "shine"), ("abcdef", "acbdef"), ("language","lnaguaeg"),
		("language", "Inaugage"), ("language", "lngauage")]
	for s1, s2 in edit_distance_examples:
		print("Edit distance between '%s' and '%s' :" % (s1, s2), edit_distance (s1, s2))
	for s1, s2 in edit_distance_examples:
		print("Edit distance with transpositions between '%s' and'%s':" % (s1, s2), edit_ distance(s1, s2, transpositions = True))

	s1 = set([1, 2, 3, 4])
	s2 = set([3, 4, 5])
	print("s1:", s1)
	print("s2:", s2)
	print("Binary distance:", binary_distance(s1, s2))
	print ("Jaccard distance:", jaccard_distance(s1, s2))
	print ("MASI distance:", masi_distance(s1, s2))

if __name__ == __ main__' :
	demo ()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

---

10.5基于句法匹配的指标

句法匹配可以通过执行分块任务来完成。NLTK中提供了个叫作nitk. chunk.api的模块，其有助于识别语块并为给定的语块序列返回一个解析树。

句法匹配:

import  nltk 
from nltk.tree import Tree
print (Tree(1, [2, Tree(3, [4]) ,5])) 
ct=Tree('VP', [Tree('V',['gave']) ,Tree('NP',['her'])])
sent=Tree('S', [Tree('NP',['I']),ct])
print (sent)
print (sent[1])
print (sent[1,1])
t1 = Tree.from string("(S(NP I) (VP (V gave) (NP her))")
print(sent == t1)
t1[1][1].set_label('X')
t1[1][1].label ()
print (t1)
t1[0],t1[1,1] = t1[1,1], t1[0]
print (t1)
len(t1)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

---

10.6使用浅层语义匹配的指标

计算wordnet相似度：

print(wordnet.N[‘dog’][0].path_similarity(wordnet.N[‘cat’][0]))

print(wordnet.V[‘run’][0].path_similarity(wordnet.N[‘walk’][0]))
1
2
3

---

（文章末尾有前面几张的链接，如果需要的话。）

---

“”"***笔者的话：整理了《精通Python自然语言处理》的第十章内容：NLP系统评估。本书的最后一张内容。介绍了有关NLP系统评估的内容，主要还是计算系统的准确度，本博客记录了书中的每段代码。希望对阅读这本书的人有所帮助。ＦＩＧＨＴＩＮＧ．．．（热烈欢迎大家批评指正，互相讨论）
（"Catch the moments of your life. Catch them while you are young and quick." --《我们这一天》） ***"""

---

（第九章）：语篇分析（https://blog.csdn.net/cjx14060307101/article/details/88623202）
（第八章）：信息检索（https://blog.csdn.net/cjx14060307101/article/details/88595396）
（第七章）：情感分析（https://blog.csdn.net/cjx14060307101/article/details/88580981）
（第六章）：语义分析（https://blog.csdn.net/cjx14060307101/article/details/88541214）
（第五章）：语法分析（https://blog.csdn.net/cjx14060307101/article/details/88378177）
（第四章）：词性标注（https://blog.csdn.net/cjx14060307101/article/details/88357016）
（第三章）：形态学（https://blog.csdn.net/cjx14060307101/article/details/88316108）
（第二章）：统计语言建模（https://blog.csdn.net/cjx14060307101/article/details/88087305）
（第一章）：字符串操作（https://blog.csdn.net/cjx14060307101/article/details/87980631）