python.nlp随笔（四）简单的全文检索系统_nlp实验 信息检索 代码

实现一个简单的电影评论语料库的全文检索系统

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Wed Apr  4 15:28:11 2018
@author: dag
"""
#coding:utf-8  
import nltk
import re
def raw(file):
 
    contents = open(file).read()
 
    contents = re.sub(r'<.*?>', ' ', contents)
 
    contents = re.sub('\s+', ' ', contents)
 
    return contents
 
 
 
def snippet(doc, term): # buggy
 
    text = ' '*30 + raw(doc) + ' '*30
 
    pos = text.index(term)
 
    return text[pos-30:pos+30]
 
 
 
print ("Building Index...")
 
files = nltk.corpus.movie_reviews.abspaths()
 
idx = nltk.Index((w, f) for f in files for w in raw(f).split())
 
 
 
query = ''
 
while query != "quit":
 
    query = input("query> ")
 
    if query in idx:
 
        for doc in idx[query]:
 
            print (snippet(doc, query))
 
    else:
 
        print ("Not found")

output:

runfile('/Users/dag/.spyder-py3/filesearch.py', wdir='/Users/dag/.spyder-py3')
Building Index...
 
query> natural
drug usage as being a hip and natural part of the art scene 
twister , this is yet another natural disaster movie that do
, such as bull durham and the natural . no curve balls here 
great abandon . rejecting her natural abilities , she has sp
 due to the exhausting of our natural resources . each membe
rn kentucky . it's beauty and natural goodness is being slow
ilming in order to get a more natural expression of fear out