devbox
运维做开发
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

全文检索django-haystack+jieba+whoosh_haystack结合jieba

作者：运维做开发 | 2024-07-21 08:34:04

赞

踩

haystack结合jieba

全文检索里的组件简介

　　1、什么是haystack？

　　　　1. haystack是django的开源搜索框架，该框架支持Solr,Elasticsearch,Whoosh, *Xapian*搜索引擎，不用更改代码，直接切换引擎，减少代码量。

　　　　2. 搜索引擎使用Whoosh，这是一个由纯Python实现的全文搜索引擎，没有二进制文件等，比较小巧，配置比较简单，当然性能自然略低。

　　　　3. 中文分词Jieba，由于Whoosh自带的是英文分词，对中文的分词支持不是太好，故用jieba替换whoosh的分词组件。

　　2、什么是jieba？

　　　　1、很多的搜索引擎对中的支持不友好，jieba作为一个中文分词器就是加强对中文的检索功能

　　3、Whoosh是什么？

　　　　1、Python的全文搜索库，Whoosh是索引文本及搜索文本的类和函数库

　　　　2、Whoosh 自带的是英文分词，对中文分词支持不太好，使用 jieba 替换 whoosh 的分词组件。

 

haystack配置使用（前后端分离）

安装工具

pip install django-haystack
pip install whoosh
pip install jieba

在setting.py中配置

  1. '''注册app '''
  2. INSTALLED_APPS = [
  3.     'django.contrib.admin',
  4.     'django.contrib.auth',
  5.     'django.contrib.contenttypes',
  6.     'django.contrib.sessions',
  7.     'django.contrib.messages',
  8.     'django.contrib.staticfiles',
  9.     # haystack要放在应用的上面
  10.     'haystack',
  11.     'jsapp',  # 这个jsapp是自己创建的app
  12. ]
  13.  
  14.  
  15. ''' 模板路径 '''
  16. TEMPLATES = [
  17.     {
  18.         'DIRS': [os.path.join(BASE_DIR,'templates')],
  19.  
  20.     },
  21. ]
  22.  
  23.  
  24. '''配置haystack '''
  25. # 全文检索框架配置
  26. HAYSTACK_CONNECTIONS = {
  27.     'default': {
  28.         # 指定whoosh引擎
  29.         'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
  30.         # 'ENGINE': 'jsapp.whoosh_cn_backend.WhooshEngine',      # whoosh_cn_backend是haystack的whoosh_backend.py改名的文件为了使用jieba分词
  31.         # 索引文件路径
  32.         'PATH': os.path.join(BASE_DIR, 'whoosh_index'),
  33.     }
  34. }
  35. # 添加此项，当数据库改变时，会自动更新索引，非常方便
  36. HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'

定义数据库 ( jsapp/models.py)

  1. from django.db import models
  2.  
  3. # Create your models here.
  4. class UserInfo(models.Model):
  5.     name = models.CharField(max_length=254)
  6.     age = models.IntegerField()
  7.  
  8.  
  9. class ArticlePost(models.Model):
  10.     author = models.ForeignKey(UserInfo,on_delete=models.CASCADE)
  11.     title = models.CharField(max_length=200)
  12.     desc = models.SlugField(max_length=500)
  13.     body = models.TextField()

索引文件生成

　1）在子应用下创建索引文件

　　在子应用的目录下，创建一个名为 jsapp/search_indexes.py 的文件

  1. from haystack import indexes
  2. from .models import ArticlePost
  3.  
  4. # 修改此处，类名为模型类的名称+Index，比如模型类为GoodsInfo,则这里类名为GoodsInfoIndex(其实可以随便写)
  5. class ArticlePostIndex(indexes.SearchIndex, indexes.Indexable):
  6.     # text为索引字段
  7.     # document = True，这代表haystack和搜索引擎将使用此字段的内容作为索引进行检索
  8.     # use_template=True 指定根据表中的那些字段建立索引文件的说明放在一个文件中
  9.     text = indexes.CharField(document=True, use_template=True)
  10.  
  11.     # 对那张表进行查询
  12.     def get_model(self):  # 重载get_model方法，必须要有！
  13.         # 返回这个model
  14.         return ArticlePost
  15.  
  16.     # 建立索引的数据
  17.     def index_queryset(self, using=None):
  18.         # 这个方法返回什么内容，最终就会对那些方法建立索引，这里是对所有字段建立索引
  19.         return self.get_model().objects.all()

2）指定索引模板文件　　　

# 创建文件路径命名必须这个规范：templates/search/indexes/应用名称/模型类名称_text.txt
# templates/search/indexes/jsapp/articlepost_text.txt

 

templates/search/indexes/jsapp/articlepost_text.txt

  1. {{ object.title }}
  2. {{ object.author.name }}
  3. {{ object.body }}

3）使用命令创建索引

python manage.py rebuild_index  # 建立索引文件

替换成jieba分词

　　1）将haystack源码复制到项目中并改名

'''1.复制源码中文件并改名 '''
将 C:\python37\Lib\site-packages\haystack\backends\whoosh_backend.py文件复制到项目中
并将 whoosh_backend.py改名为 whoosh_cn_backend.py 放在APP中如：jsapp\whoosh_cn_backend.py

'''2.修改源码中文件'''
# 在全局引入的最后一行加入jieba分词器
from jieba.analyse import ChineseAnalyzer

# 修改为中文分词法
查找
analyzer=StemmingAnalyzer()
改为
analyzer=ChineseAnalyzer()

whoosh_cn_backend.py

  1. # encoding: utf-8
  2.  
  3. from __future__ import absolute_import, division, print_function, unicode_literals
  4.  
  5. import json
  6. import os
  7. import re
  8. import shutil
  9. import threading
  10. import warnings
  11.  
  12. from django.conf import settings
  13. from django.core.exceptions import ImproperlyConfigured
  14. from django.utils import six
  15. from django.utils.datetime_safe import datetime
  16. from django.utils.encoding import force_text
  17.  
  18. from haystack.backends import BaseEngine, BaseSearchBackend, BaseSearchQuery, EmptyResults, log_query
  19. from haystack.constants import DJANGO_CT, DJANGO_ID, ID
  20. from haystack.exceptions import MissingDependency, SearchBackendError, SkipDocument
  21. from haystack.inputs import Clean, Exact, PythonData, Raw
  22. from haystack.models import SearchResult
  23. from haystack.utils import log as logging
  24. from haystack.utils import get_identifier, get_model_ct
  25. from haystack.utils.app_loading import haystack_get_model
  26.  
  27. try:
  28.     import whoosh
  29. except ImportError:
  30.     raise MissingDependency(
  31.         "The 'whoosh' backend requires the installation of 'Whoosh'. Please refer to the documentation.")
  32.  
  33. # Handle minimum requirement.
  34. if not hasattr(whoosh, '__version__') or whoosh.__version__ < (2, 5, 0):
  35.     raise MissingDependency("The 'whoosh' backend requires version 2.5.0 or greater.")
  36.  
  37. # Bubble up the correct error.
  38. from whoosh import index
  39. from whoosh.analysis import StemmingAnalyzer
  40. from whoosh.fields import ID as WHOOSH_ID
  41. from whoosh.fields import BOOLEAN, DATETIME, IDLIST, KEYWORD, NGRAM, NGRAMWORDS, NUMERIC, Schema, TEXT
  42. from whoosh.filedb.filestore import FileStorage, RamStorage
  43. from whoosh.highlight import highlight as whoosh_highlight
  44. from whoosh.highlight import ContextFragmenter, HtmlFormatter
  45. from whoosh.qparser import QueryParser
  46. from whoosh.searching import ResultsPage
  47. from whoosh.writing import AsyncWriter
  48.  
  49. DATETIME_REGEX = re.compile(
  50.     '^(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})T(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2})(\.\d{3,6}Z?)?$')
  51. LOCALS = threading.local()
  52. LOCALS.RAM_STORE = None
  53.  
  54.  
  55. class WhooshHtmlFormatter(HtmlFormatter):
  56.     """
  57.     This is a HtmlFormatter simpler than the whoosh.HtmlFormatter.
  58.     We use it to have consistent results across backends. Specifically,
  59.     Solr, Xapian and Elasticsearch are using this formatting.
  60.     """
  61.     template = '<%(tag)s>%(t)s</%(tag)s>'
  62.  
  63.  
  64. class WhooshSearchBackend(BaseSearchBackend):
  65.     # Word reserved by Whoosh for special use.
  66.     RESERVED_WORDS = (
  67.         'AND',
  68.         'NOT',
  69.         'OR',
  70.         'TO',
  71.     )
  72.  
  73.     # Characters reserved by Whoosh for special use.
  74.     # The '\\' must come first, so as not to overwrite the other slash replacements.
  75.     RESERVED_CHARACTERS = (
  76.         '\\', '+', '-', '&&', '||', '!', '(', ')', '{', '}',
  77.         '[', ']', '^', '"', '~', '*', '?', ':', '.',
  78.     )
  79.  
  80.     def __init__(self, connection_alias, **connection_options):
  81.         super(WhooshSearchBackend, self).__init__(connection_alias, **connection_options)
  82.         self.setup_complete = False
  83.         self.use_file_storage = True
  84.         self.post_limit = getattr(connection_options, 'POST_LIMIT', 128 * 1024 * 1024)
  85.         self.path = connection_options.get('PATH')
  86.  
  87.         if connection_options.get('STORAGE', 'file') != 'file':
  88.             self.use_file_storage = False
  89.  
  90.         if self.use_file_storage and not self.path:
  91.             raise ImproperlyConfigured(
  92.                 "You must specify a 'PATH' in your settings for connection '%s'." % connection_alias)
  93.  
  94.         self.log = logging.getLogger('haystack')
  95.  
  96.     def setup(self):
  97.         """
  98.         Defers loading until needed.
  99.         """
  100.         from haystack import connections
  101.         new_index = False
  102.  
  103.         # Make sure the index is there.
  104.         if self.use_file_storage and not os.path.exists(self.path):
  105.             os.makedirs(self.path)
  106.             new_index = True
  107.  
  108.         if self.use_file_storage and not os.access(self.path, os.W_OK):
  109.             raise IOError("The path to your Whoosh index '%s' is not writable for the current user/group." % self.path)
  110.  
  111.         if self.use_file_storage:
  112.             self.storage = FileStorage(self.path)
  113.         else:
  114.             global LOCALS
  115.  
  116.             if getattr(LOCALS, 'RAM_STORE', None) is None:
  117.                 LOCALS.RAM_STORE = RamStorage()
  118.  
  119.             self.storage = LOCALS.RAM_STORE
  120.  
  121.         self.content_field_name, self.schema = self.build_schema(
  122.             connections[self.connection_alias].get_unified_index().all_searchfields())
  123.         self.parser = QueryParser(self.content_field_name, schema=self.schema)
  124.  
  125.         if new_index is True:
  126.             self.index = self.storage.create_index(self.schema)
  127.         else:
  128.             try:
  129.                 self.index = self.storage.open_index(schema=self.schema)
  130.             except index.EmptyIndexError:
  131.                 self.index = self.storage.create_index(self.schema)
  132.  
  133.         self.setup_complete = True
  134.  
  135.     def build_schema(self, fields):
  136.         schema_fields = {
  137.             ID: WHOOSH_ID(stored=True, unique=True),
  138.             DJANGO_CT: WHOOSH_ID(stored=True),
  139.             DJANGO_ID: WHOOSH_ID(stored=True),
  140.         }
  141.         # Grab the number of keys that are hard-coded into Haystack.
  142.         # We'll use this to (possibly) fail slightly more gracefully later.
  143.         initial_key_count = len(schema_fields)
  144.         content_field_name = ''
  145.  
  146.         for field_name, field_class in fields.items():
  147.             if field_class.is_multivalued:
  148.                 if field_class.indexed is False:
  149.                     schema_fields[field_class.index_fieldname] = IDLIST(stored=True, field_boost=field_class.boost)
  150.                 else:
  151.                     schema_fields[field_class.index_fieldname] = KEYWORD(stored=True, commas=True, scorable=True,
  152.                                                                          field_boost=field_class.boost)
  153.             elif field_class.field_type in ['date', 'datetime']:
  154.                 schema_fields[field_class.index_fieldname] = DATETIME(stored=field_class.stored, sortable=True)
  155.             elif field_class.field_type == 'integer':
  156.                 schema_fields[field_class.index_fieldname] = NUMERIC(stored=field_class.stored, numtype=int,
  157.                                                                      field_boost=field_class.boost)
  158.             elif field_class.field_type == 'float':
  159.                 schema_fields[field_class.index_fieldname] = NUMERIC(stored=field_class.stored, numtype=float,
  160.                                                                      field_boost=field_class.boost)
  161.             elif field_class.field_type == 'boolean':
  162.                 # Field boost isn't supported on BOOLEAN as of 1.8.2.
  163.                 schema_fields[field_class.index_fieldname] = BOOLEAN(stored=field_class.stored)
  164.             elif field_class.field_type == 'ngram':
  165.                 schema_fields[field_class.index_fieldname] = NGRAM(minsize=3, maxsize=15, stored=field_class.stored,
  166.                                                                    field_boost=field_class.boost)
  167.             elif field_class.field_type == 'edge_ngram':
  168.                 schema_fields[field_class.index_fieldname] = NGRAMWORDS(minsize=2, maxsize=15, at='start',
  169.                                                                         stored=field_class.stored,
  170.                                                                         field_boost=field_class.boost)
  171.             else:
  172.                 schema_fields[field_class.index_fieldname] = TEXT(stored=True, analyzer=ChineseAnalyzer(),
  173.                                                                   field_boost=field_class.boost, sortable=True)
  174.  
  175.             if field_class.document is True:
  176.                 content_field_name = field_class.index_fieldname
  177.                 schema_fields[field_class.index_fieldname].spelling = True
  178.  
  179.         # Fail more gracefully than relying on the backend to die if no fields
  180.         # are found.
  181.         if len(schema_fields) <= initial_key_count:
  182.             raise SearchBackendError(
  183.                 "No fields were found in any search_indexes. Please correct this before attempting to search.")
  184.  
  185.         return (content_field_name, Schema(**schema_fields))
  186.  
  187.     def update(self, index, iterable, commit=True):
  188.         if not self.setup_complete:
  189.             self.setup()
  190.  
  191.         self.index = self.index.refresh()
  192.         writer = AsyncWriter(self.index)
  193.  
  194.         for obj in iterable:
  195.             try:
  196.                 doc = index.full_prepare(obj)
  197.             except SkipDocument:
  198.                 self.log.debug(u"Indexing for object `%s` skipped", obj)
  199.             else:
  200.                 # Really make sure it's unicode, because Whoosh won't have it any
  201.                 # other way.
  202.                 for key in doc:
  203.                     doc[key] = self._from_python(doc[key])
  204.  
  205.                 # Document boosts aren't supported in Whoosh 2.5.0+.
  206.                 if 'boost' in doc:
  207.                     del doc['boost']
  208.  
  209.                 try:
  210.                     writer.update_document(**doc)
  211.                 except Exception as e:
  212.                     if not self.silently_fail:
  213.                         raise
  214.  
  215.                     # We'll log the object identifier but won't include the actual object
  216.                     # to avoid the possibility of that generating encoding errors while
  217.                     # processing the log message:
  218.                     self.log.error(u"%s while preparing object for update" % e.__class__.__name__,
  219.                                    exc_info=True, extra={"data": {"index": index,
  220.                                                                   "object": get_identifier(obj)}})
  221.  
  222.         if len(iterable) > 0:
  223.             # For now, commit no matter what, as we run into locking issues otherwise.
  224.             writer.commit()
  225.  
  226.     def remove(self, obj_or_string, commit=True):
  227.         if not self.setup_complete:
  228.             self.setup()
  229.  
  230.         self.index = self.index.refresh()
  231.         whoosh_id = get_identifier(obj_or_string)
  232.  
  233.         try:
  234.             self.index.delete_by_query(q=self.parser.parse(u'%s:"%s"' % (ID, whoosh_id)))
  235.         except Exception as e:
  236.             if not self.silently_fail:
  237.                 raise
  238.  
  239.             self.log.error("Failed to remove document '%s' from Whoosh: %s", whoosh_id, e, exc_info=True)
  240.  
  241.     def clear(self, models=None, commit=True):
  242.         if not self.setup_complete:
  243.             self.setup()
  244.  
  245.         self.index = self.index.refresh()
  246.  
  247.         if models is not None:
  248.             assert isinstance(models, (list, tuple))
  249.  
  250.         try:
  251.             if models is None:
  252.                 self.delete_index()
  253.             else:
  254.                 models_to_delete = []
  255.  
  256.                 for model in models:
  257.                     models_to_delete.append(u"%s:%s" % (DJANGO_CT, get_model_ct(model)))
  258.  
  259.                 self.index.delete_by_query(q=self.parser.parse(u" OR ".join(models_to_delete)))
  260.         except Exception as e:
  261.             if not self.silently_fail:
  262.                 raise
  263.  
  264.             if models is not None:
  265.                 self.log.error("Failed to clear Whoosh index of models '%s': %s", ','.join(models_to_delete),
  266.                                e, exc_info=True)
  267.             else:
  268.                 self.log.error("Failed to clear Whoosh index: %s", e, exc_info=True)
  269.  
  270.     def delete_index(self):
  271.         # Per the Whoosh mailing list, if wiping out everything from the index,
  272.         # it's much more efficient to simply delete the index files.
  273.         if self.use_file_storage and os.path.exists(self.path):
  274.             shutil.rmtree(self.path)
  275.         elif not self.use_file_storage:
  276.             self.storage.clean()
  277.  
  278.         # Recreate everything.
  279.         self.setup()
  280.  
  281.     def optimize(self):
  282.         if not self.setup_complete:
  283.             self.setup()
  284.  
  285.         self.index = self.index.refresh()
  286.         self.index.optimize()
  287.  
  288.     def calculate_page(self, start_offset=0, end_offset=None):
  289.         # Prevent against Whoosh throwing an error. Requires an end_offset
  290.         # greater than 0.
  291.         if end_offset is not None and end_offset <= 0:
  292.             end_offset = 1
  293.  
  294.         # Determine the page.
  295.         page_num = 0
  296.  
  297.         if end_offset is None:
  298.             end_offset = 1000000
  299.  
  300.         if start_offset is None:
  301.             start_offset = 0
  302.  
  303.         page_length = end_offset - start_offset
  304.  
  305.         if page_length and page_length > 0:
  306.             page_num = int(start_offset / page_length)
  307.  
  308.         # Increment because Whoosh uses 1-based page numbers.
  309.         page_num += 1
  310.         return page_num, page_length
  311.  
  312.     @log_query
  313.     def search(self, query_string, sort_by=None, start_offset=0, end_offset=None,
  314.                fields='', highlight=False, facets=None, date_facets=None, query_facets=None,
  315.                narrow_queries=None, spelling_query=None, within=None,
  316.                dwithin=None, distance_point=None, models=None,
  317.                limit_to_registered_models=None, result_class=None, **kwargs):
  318.         if not self.setup_complete:
  319.             self.setup()
  320.  
  321.         # A zero length query should return no results.
  322.         if len(query_string) == 0:
  323.             return {
  324.                 'results': [],
  325.                 'hits': 0,
  326.             }
  327.  
  328.         query_string = force_text(query_string)
  329.  
  330.         # A one-character query (non-wildcard) gets nabbed by a stopwords
  331.         # filter and should yield zero results.
  332.         if len(query_string) <= 1 and query_string != u'*':
  333.             return {
  334.                 'results': [],
  335.                 'hits': 0,
  336.             }
  337.  
  338.         reverse = False
  339.  
  340.         if sort_by is not None:
  341.             # Determine if we need to reverse the results and if Whoosh can
  342.             # handle what it's being asked to sort by. Reversing is an
  343.             # all-or-nothing action, unfortunately.
  344.             sort_by_list = []
  345.             reverse_counter = 0
  346.  
  347.             for order_by in sort_by:
  348.                 if order_by.startswith('-'):
  349.                     reverse_counter += 1
  350.  
  351.             if reverse_counter and reverse_counter != len(sort_by):
  352.                 raise SearchBackendError("Whoosh requires all order_by fields"
  353.                                          " to use the same sort direction")
  354.  
  355.             for order_by in sort_by:
  356.                 if order_by.startswith('-'):
  357.                     sort_by_list.append(order_by[1:])
  358.  
  359.                     if len(sort_by_list) == 1:
  360.                         reverse = True
  361.                 else:
  362.                     sort_by_list.append(order_by)
  363.  
  364.                     if len(sort_by_list) == 1:
  365.                         reverse = False
  366.  
  367.             sort_by = sort_by_list
  368.  
  369.         if facets is not None:
  370.             warnings.warn("Whoosh does not handle faceting.", Warning, stacklevel=2)
  371.  
  372.         if date_facets is not None:
  373.             warnings.warn("Whoosh does not handle date faceting.", Warning, stacklevel=2)
  374.  
  375.         if query_facets is not None:
  376.             warnings.warn("Whoosh does not handle query faceting.", Warning, stacklevel=2)
  377.  
  378.         narrowed_results = None
  379.         self.index = self.index.refresh()
  380.  
  381.         if limit_to_registered_models is None:
  382.             limit_to_registered_models = getattr(settings, 'HAYSTACK_LIMIT_TO_REGISTERED_MODELS', True)
  383.  
  384.         if models and len(models):
  385.             model_choices = sorted(get_model_ct(model) for model in models)
  386.         elif limit_to_registered_models:
  387.             # Using narrow queries, limit the results to only models handled
  388.             # with the current routers.
  389.             model_choices = self.build_models_list()
  390.         else:
  391.             model_choices = []
  392.  
  393.         if len(model_choices) > 0:
  394.             if narrow_queries is None:
  395.                 narrow_queries = set()
  396.  
  397.             narrow_queries.add(' OR '.join(['%s:%s' % (DJANGO_CT, rm) for rm in model_choices]))
  398.  
  399.         narrow_searcher = None
  400.  
  401.         if narrow_queries is not None:
  402.             # Potentially expensive? I don't see another way to do it in Whoosh...
  403.             narrow_searcher = self.index.searcher()
  404.  
  405.             for nq in narrow_queries:
  406.                 recent_narrowed_results = narrow_searcher.search(self.parser.parse(force_text(nq)),
  407.                                                                  limit=None)
  408.  
  409.                 if len(recent_narrowed_results) <= 0:
  410.                     return {
  411.                         'results': [],
  412.                         'hits': 0,
  413.                     }
  414.  
  415.                 if narrowed_results:
  416.                     narrowed_results.filter(recent_narrowed_results)
  417.                 else:
  418.                     narrowed_results = recent_narrowed_results
  419.  
  420.         self.index = self.index.refresh()
  421.  
  422.         if self.index.doc_count():
  423.             searcher = self.index.searcher()
  424.             parsed_query = self.parser.parse(query_string)
  425.  
  426.             # In the event of an invalid/stopworded query, recover gracefully.
  427.             if parsed_query is None:
  428.                 return {
  429.                     'results': [],
  430.                     'hits': 0,
  431.                 }
  432.  
  433.             page_num, page_length = self.calculate_page(start_offset, end_offset)
  434.  
  435.             search_kwargs = {
  436.                 'pagelen': page_length,
  437.                 'sortedby': sort_by,
  438.                 'reverse': reverse,
  439.             }
  440.  
  441.             # Handle the case where the results have been narrowed.
  442.             if narrowed_results is not None:
  443.                 search_kwargs['filter'] = narrowed_results
  444.  
  445.             try:
  446.                 raw_page = searcher.search_page(
  447.                     parsed_query,
  448.                     page_num,
  449.                     **search_kwargs
  450.                 )
  451.             except ValueError:
  452.                 if not self.silently_fail:
  453.                     raise
  454.  
  455.                 return {
  456.                     'results': [],
  457.                     'hits': 0,
  458.                     'spelling_suggestion': None,
  459.                 }
  460.  
  461.             # Because as of Whoosh 2.5.1, it will return the wrong page of
  462.             # results if you request something too high. :(
  463.             if raw_page.pagenum < page_num:
  464.                 return {
  465.                     'results': [],
  466.                     'hits': 0,
  467.                     'spelling_suggestion': None,
  468.                 }
  469.  
  470.             results = self._process_results(raw_page, highlight=highlight, query_string=query_string,
  471.                                             spelling_query=spelling_query, result_class=result_class)
  472.             searcher.close()
  473.  
  474.             if hasattr(narrow_searcher, 'close'):
  475.                 narrow_searcher.close()
  476.  
  477.             return results
  478.         else:
  479.             if self.include_spelling:
  480.                 if spelling_query:
  481.                     spelling_suggestion = self.create_spelling_suggestion(spelling_query)
  482.                 else:
  483.                     spelling_suggestion = self.create_spelling_suggestion(query_string)
  484.             else:
  485.                 spelling_suggestion = None
  486.  
  487.             return {
  488.                 'results': [],
  489.                 'hits': 0,
  490.                 'spelling_suggestion': spelling_suggestion,
  491.             }
  492.  
  493.     def more_like_this(self, model_instance, additional_query_string=None,
  494.                        start_offset=0, end_offset=None, models=None,
  495.                        limit_to_registered_models=None, result_class=None, **kwargs):
  496.         if not self.setup_complete:
  497.             self.setup()
  498.  
  499.         field_name = self.content_field_name
  500.         narrow_queries = set()
  501.         narrowed_results = None
  502.         self.index = self.index.refresh()
  503.  
  504.         if limit_to_registered_models is None:
  505.             limit_to_registered_models = getattr(settings, 'HAYSTACK_LIMIT_TO_REGISTERED_MODELS', True)
  506.  
  507.         if models and len(models):
  508.             model_choices = sorted(get_model_ct(model) for model in models)
  509.         elif limit_to_registered_models:
  510.             # Using narrow queries, limit the results to only models handled
  511.             # with the current routers.
  512.             model_choices = self.build_models_list()
  513.         else:
  514.             model_choices = []
  515.  
  516.         if len(model_choices) > 0:
  517.             if narrow_queries is None:
  518.                 narrow_queries = set()
  519.  
  520.             narrow_queries.add(' OR '.join(['%s:%s' % (DJANGO_CT, rm) for rm in model_choices]))
  521.  
  522.         if additional_query_string and additional_query_string != '*':
  523.             narrow_queries.add(additional_query_string)
  524.  
  525.         narrow_searcher = None
  526.  
  527.         if narrow_queries is not None:
  528.             # Potentially expensive? I don't see another way to do it in Whoosh...
  529.             narrow_searcher = self.index.searcher()
  530.  
  531.             for nq in narrow_queries:
  532.                 recent_narrowed_results = narrow_searcher.search(self.parser.parse(force_text(nq)),
  533.                                                                  limit=None)
  534.  
  535.                 if len(recent_narrowed_results) <= 0:
  536.                     return {
  537.                         'results': [],
  538.                         'hits': 0,
  539.                     }
  540.  
  541.                 if narrowed_results:
  542.                     narrowed_results.filter(recent_narrowed_results)
  543.                 else:
  544.                     narrowed_results = recent_narrowed_results
  545.  
  546.         page_num, page_length = self.calculate_page(start_offset, end_offset)
  547.  
  548.         self.index = self.index.refresh()
  549.         raw_results = EmptyResults()
  550.  
  551.         searcher = None
  552.         if self.index.doc_count():
  553.             query = "%s:%s" % (ID, get_identifier(model_instance))
  554.             searcher = self.index.searcher()
  555.             parsed_query = self.parser.parse(query)
  556.             results = searcher.search(parsed_query)
  557.  
  558.             if len(results):
  559.                 raw_results = results[0].more_like_this(field_name, top=end_offset)
  560.  
  561.             # Handle the case where the results have been narrowed.
  562.             if narrowed_results is not None and hasattr(raw_results, 'filter'):
  563.                 raw_results.filter(narrowed_results)
  564.  
  565.         try:
  566.             raw_page = ResultsPage(raw_results, page_num, page_length)
  567.         except ValueError:
  568.             if not self.silently_fail:
  569.                 raise
  570.  
  571.             return {
  572.                 'results': [],
  573.                 'hits': 0,
  574.                 'spelling_suggestion': None,
  575.             }
  576.  
  577.         # Because as of Whoosh 2.5.1, it will return the wrong page of
  578.         # results if you request something too high. :(
  579.         if raw_page.pagenum < page_num:
  580.             return {
  581.                 'results': [],
  582.                 'hits': 0,
  583.                 'spelling_suggestion': None,
  584.             }
  585.  
  586.         results = self._process_results(raw_page, result_class=result_class)
  587.  
  588.         if searcher:
  589.             searcher.close()
  590.  
  591.         if hasattr(narrow_searcher, 'close'):
  592.             narrow_searcher.close()
  593.  
  594.         return results
  595.  
  596.     def _process_results(self, raw_page, highlight=False, query_string='', spelling_query=None, result_class=None):
  597.         from haystack import connections
  598.         results = []
  599.  
  600.         # It's important to grab the hits first before slicing. Otherwise, this
  601.         # can cause pagination failures.
  602.         hits = len(raw_page)
  603.  
  604.         if result_class is None:
  605.             result_class = SearchResult
  606.  
  607.         facets = {}
  608.         spelling_suggestion = None
  609.         unified_index = connections[self.connection_alias].get_unified_index()
  610.         indexed_models = unified_index.get_indexed_models()
  611.  
  612.         for doc_offset, raw_result in enumerate(raw_page):
  613.             score = raw_page.score(doc_offset) or 0
  614.             app_label, model_name = raw_result[DJANGO_CT].split('.')
  615.             additional_fields = {}
  616.             model = haystack_get_model(app_label, model_name)
  617.  
  618.             if model and model in indexed_models:
  619.                 for key, value in raw_result.items():
  620.                     index = unified_index.get_index(model)
  621.                     string_key = str(key)
  622.  
  623.                     if string_key in index.fields and hasattr(index.fields[string_key], 'convert'):
  624.                         # Special-cased due to the nature of KEYWORD fields.
  625.                         if index.fields[string_key].is_multivalued:
  626.                             if value is None or len(value) is 0:
  627.                                 additional_fields[string_key] = []
  628.                             else:
  629.                                 additional_fields[string_key] = value.split(',')
  630.                         else:
  631.                             additional_fields[string_key] = index.fields[string_key].convert(value)
  632.                     else:
  633.                         additional_fields[string_key] = self._to_python(value)
  634.  
  635.                 del (additional_fields[DJANGO_CT])
  636.                 del (additional_fields[DJANGO_ID])
  637.  
  638.                 if highlight:
  639.                     sa = StemmingAnalyzer()
  640.                     formatter = WhooshHtmlFormatter('em')
  641.                     terms = [token.text for token in sa(query_string)]
  642.  
  643.                     whoosh_result = whoosh_highlight(
  644.                         additional_fields.get(self.content_field_name),
  645.                         terms,
  646.                         sa,
  647.                         ContextFragmenter(),
  648.                         formatter
  649.                     )
  650.                     additional_fields['highlighted'] = {
  651.                         self.content_field_name: [whoosh_result],
  652.                     }
  653.  
  654.                 result = result_class(app_label, model_name, raw_result[DJANGO_ID], score, **additional_fields)
  655.                 results.append(result)
  656.             else:
  657.                 hits -= 1
  658.  
  659.         if self.include_spelling:
  660.             if spelling_query:
  661.                 spelling_suggestion = self.create_spelling_suggestion(spelling_query)
  662.             else:
  663.                 spelling_suggestion = self.create_spelling_suggestion(query_string)
  664.  
  665.         return {
  666.             'results': results,
  667.             'hits': hits,
  668.             'facets': facets,
  669.             'spelling_suggestion': spelling_suggestion,
  670.         }
  671.  
  672.     def create_spelling_suggestion(self, query_string):
  673.         spelling_suggestion = None
  674.         reader = self.index.reader()
  675.         corrector = reader.corrector(self.content_field_name)
  676.         cleaned_query = force_text(query_string)
  677.  
  678.         if not query_string:
  679.             return spelling_suggestion
  680.  
  681.         # Clean the string.
  682.         for rev_word in self.RESERVED_WORDS:
  683.             cleaned_query = cleaned_query.replace(rev_word, '')
  684.  
  685.         for rev_char in self.RESERVED_CHARACTERS:
  686.             cleaned_query = cleaned_query.replace(rev_char, '')
  687.  
  688.         # Break it down.
  689.         query_words = cleaned_query.split()
  690.         suggested_words = []
  691.  
  692.         for word in query_words:
  693.             suggestions = corrector.suggest(word, limit=1)
  694.  
  695.             if len(suggestions) > 0:
  696.                 suggested_words.append(suggestions[0])
  697.  
  698.         spelling_suggestion = ' '.join(suggested_words)
  699.         return spelling_suggestion
  700.  
  701.     def _from_python(self, value):
  702.         """
  703.         Converts Python values to a string for Whoosh.
  705.         Code courtesy of pysolr.
  706.         """
  707.         if hasattr(value, 'strftime'):
  708.             if not hasattr(value, 'hour'):
  709.                 value = datetime(value.year, value.month, value.day, 0, 0, 0)
  710.         elif isinstance(value, bool):
  711.             if value:
  712.                 value = 'true'
  713.             else:
  714.                 value = 'false'
  715.         elif isinstance(value, (list, tuple)):
  716.             value = u','.join([force_text(v) for v in value])
  717.         elif isinstance(value, (six.integer_types, float)):
  718.             # Leave it alone.
  719.             pass
  720.         else:
  721.             value = force_text(value)
  722.         return value
  723.  
  724.     def _to_python(self, value):
  725.         """
  726.         Converts values from Whoosh to native Python values.
  728.         A port of the same method in pysolr, as they deal with data the same way.
  729.         """
  730.         if value == 'true':
  731.             return True
  732.         elif value == 'false':
  733.             return False
  734.  
  735.         if value and isinstance(value, six.string_types):
  736.             possible_datetime = DATETIME_REGEX.search(value)
  737.  
  738.             if possible_datetime:
  739.                 date_values = possible_datetime.groupdict()
  740.  
  741.                 for dk, dv in date_values.items():
  742.                     date_values[dk] = int(dv)
  743.  
  744.                 return datetime(date_values['year'], date_values['month'], date_values['day'], date_values['hour'],
  745.                                 date_values['minute'], date_values['second'])
  746.  
  747.         try:
  748.             # Attempt to use json to load the values.
  749.             converted_value = json.loads(value)
  750.  
  751.             # Try to handle most built-in types.
  752.             if isinstance(converted_value, (list, tuple, set, dict, six.integer_types, float, complex)):
  753.                 return converted_value
  754.         except:
  755.             # If it fails (SyntaxError or its ilk) or we don't trust it,
  756.             # continue on.
  757.             pass
  758.  
  759.         return value
  760.  
  761.  
  762. class WhooshSearchQuery(BaseSearchQuery):
  763.     def _convert_datetime(self, date):
  764.         if hasattr(date, 'hour'):
  765.             return force_text(date.strftime('%Y%m%d%H%M%S'))
  766.         else:
  767.             return force_text(date.strftime('%Y%m%d000000'))
  768.  
  769.     def clean(self, query_fragment):
  770.         """
  771.         Provides a mechanism for sanitizing user input before presenting the
  772.         value to the backend.
  774.         Whoosh 1.X differs here in that you can no longer use a backslash
  775.         to escape reserved characters. Instead, the whole word should be
  776.         quoted.
  777.         """
  778.         words = query_fragment.split()
  779.         cleaned_words = []
  780.  
  781.         for word in words:
  782.             if word in self.backend.RESERVED_WORDS:
  783.                 word = word.replace(word, word.lower())
  784.  
  785.             for char in self.backend.RESERVED_CHARACTERS:
  786.                 if char in word:
  787.                     word = "'%s'" % word
  788.                     break
  789.  
  790.             cleaned_words.append(word)
  791.  
  792.         return ' '.join(cleaned_words)
  793.  
  794.     def build_query_fragment(self, field, filter_type, value):
  795.         from haystack import connections
  796.         query_frag = ''
  797.         is_datetime = False
  798.  
  799.         if not hasattr(value, 'input_type_name'):
  800.             # Handle when we've got a ``ValuesListQuerySet``...
  801.             if hasattr(value, 'values_list'):
  802.                 value = list(value)
  803.  
  804.             if hasattr(value, 'strftime'):
  805.                 is_datetime = True
  806.  
  807.             if isinstance(value, six.string_types) and value != ' ':
  808.                 # It's not an ``InputType``. Assume ``Clean``.
  809.                 value = Clean(value)
  810.             else:
  811.                 value = PythonData(value)
  812.  
  813.         # Prepare the query using the InputType.
  814.         prepared_value = value.prepare(self)
  815.  
  816.         if not isinstance(prepared_value, (set, list, tuple)):
  817.             # Then convert whatever we get back to what pysolr wants if needed.
  818.             prepared_value = self.backend._from_python(prepared_value)
  819.  
  820.         # 'content' is a special reserved word, much like 'pk' in
  821.         # Django's ORM layer. It indicates 'no special field'.
  822.         if field == 'content':
  823.             index_fieldname = ''
  824.         else:
  825.             index_fieldname = u'%s:' % connections[self._using].get_unified_index().get_index_fieldname(field)
  826.  
  827.         filter_types = {
  828.             'content': '%s',
  829.             'contains': '*%s*',
  830.             'endswith': "*%s",
  831.             'startswith': "%s*",
  832.             'exact': '%s',
  833.             'gt': "{%s to}",
  834.             'gte': "[%s to]",
  835.             'lt': "{to %s}",
  836.             'lte': "[to %s]",
  837.             'fuzzy': u'%s~',
  838.         }
  839.  
  840.         if value.post_process is False:
  841.             query_frag = prepared_value
  842.         else:
  843.             if filter_type in ['content', 'contains', 'startswith', 'endswith', 'fuzzy']:
  844.                 if value.input_type_name == 'exact':
  845.                     query_frag = prepared_value
  846.                 else:
  847.                     # Iterate over terms & incorportate the converted form of each into the query.
  848.                     terms = []
  849.  
  850.                     if isinstance(prepared_value, six.string_types):
  851.                         possible_values = prepared_value.split(' ')
  852.                     else:
  853.                         if is_datetime is True:
  854.                             prepared_value = self._convert_datetime(prepared_value)
  855.  
  856.                         possible_values = [prepared_value]
  857.  
  858.                     for possible_value in possible_values:
  859.                         terms.append(filter_types[filter_type] % self.backend._from_python(possible_value))
  860.  
  861.                     if len(terms) == 1:
  862.                         query_frag = terms[0]
  863.                     else:
  864.                         query_frag = u"(%s)" % " AND ".join(terms)
  865.             elif filter_type == 'in':
  866.                 in_options = []
  867.  
  868.                 for possible_value in prepared_value:
  869.                     is_datetime = False
  870.  
  871.                     if hasattr(possible_value, 'strftime'):
  872.                         is_datetime = True
  873.  
  874.                     pv = self.backend._from_python(possible_value)
  875.  
  876.                     if is_datetime is True:
  877.                         pv = self._convert_datetime(pv)
  878.  
  879.                     if isinstance(pv, six.string_types) and not is_datetime:
  880.                         in_options.append('"%s"' % pv)
  881.                     else:
  882.                         in_options.append('%s' % pv)
  883.  
  884.                 query_frag = "(%s)" % " OR ".join(in_options)
  885.             elif filter_type == 'range':
  886.                 start = self.backend._from_python(prepared_value[0])
  887.                 end = self.backend._from_python(prepared_value[1])
  888.  
  889.                 if hasattr(prepared_value[0], 'strftime'):
  890.                     start = self._convert_datetime(start)
  891.  
  892.                 if hasattr(prepared_value[1], 'strftime'):
  893.                     end = self._convert_datetime(end)
  894.  
  895.                 query_frag = u"[%s to %s]" % (start, end)
  896.             elif filter_type == 'exact':
  897.                 if value.input_type_name == 'exact':
  898.                     query_frag = prepared_value
  899.                 else:
  900.                     prepared_value = Exact(prepared_value).prepare(self)
  901.                     query_frag = filter_types[filter_type] % prepared_value
  902.             else:
  903.                 if is_datetime is True:
  904.                     prepared_value = self._convert_datetime(prepared_value)
  905.  
  906.                 query_frag = filter_types[filter_type] % prepared_value
  907.  
  908.         if len(query_frag) and not isinstance(value, Raw):
  909.             if not query_frag.startswith('(') and not query_frag.endswith(')'):
  910.                 query_frag = "(%s)" % query_frag
  911.  
  912.         return u"%s%s" % (index_fieldname, query_frag)
  913.  
  914.  
  915. class WhooshEngine(BaseEngine):
  916.     backend = WhooshSearchBackend
  917.     query = WhooshSearchQuery
  918.  
  919.  
  920. '''2.修改源码中文件'''
  921. # 在全局引入的最后一行加入jieba分词器
  922. from jieba.analyse import ChineseAnalyzer
  923.  
  924. # 修改为中文分词法
  925. # 查找
  926. # analyzer=StemmingAnalyzer()
  927. # 改为
  928. analyzer=ChineseAnalyzer()
'
运行



2）Django内settings内修改相应的haystack后台文件名
（setting.py）

  1. # 全文检索框架配置
  2. HAYSTACK_CONNECTIONS = {
  3.     'default': {
  4.         # 指定whoosh引擎
  5.         'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
  6.         # 'ENGINE': 'jsapp.whoosh_cn_backend.WhooshEngine',      #article.whoosh_cn_backend便是你刚刚添加的文件
  7.         # 索引文件路径
  8.         'PATH': os.path.join(BASE_DIR, 'whoosh_index'),
  9.     }
  10. }
  11. # 添加此项，当数据库改变时，会自动更新索引，非常方便
  12. HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'

索引文件使用

jsapp/urls.py

  1. from django.conf.urls import url
  2. from . import views as view
  3.  
  4. urlpatterns=[
  5.     url(r'abc/$', view.basic_search),
  6.  
  7. ]

 jsapp/views.py

  1. from django.shortcuts import render
  2.  
  3. # Create your views here.
  4. import json
  5. from django.conf import settings
  6. from django.core.paginator import InvalidPage, Paginator
  7. from django.http import Http404, HttpResponse,JsonResponse
  8. from haystack.forms import  ModelSearchForm
  9. from haystack.query import EmptySearchQuerySet
  10. RESULTS_PER_PAGE = getattr(settings, 'HAYSTACK_SEARCH_RESULTS_PER_PAGE', 20)
  11.  
  12.  
  13.  
  14. def basic_search(request, load_all=True, form_class=ModelSearchForm, searchqueryset=None, extra_context=None, results_per_page=None):
  15.     query = ''
  16.     results = EmptySearchQuerySet()
  17.     if request.GET.get('q'):
  18.         form = form_class(request.GET, searchqueryset=searchqueryset, load_all=load_all)
  19.  
  20.         if form.is_valid():
  21.             query = form.cleaned_data['q']
  22.             results = form.search()
  23.     else:
  24.         form = form_class(searchqueryset=searchqueryset, load_all=load_all)
  25.  
  26.     paginator = Paginator(results, results_per_page or RESULTS_PER_PAGE)
  27.     try:
  28.         page = paginator.page(int(request.GET.get('page', 1)))
  29.     except InvalidPage:
  30.         result = {"code": 404, "msg": 'No file found！', "data": []}
  31.         return HttpResponse(json.dumps(result), content_type="application/json")
  32.  
  33.     context = {
  34.         'form': form,
  35.         'page': page,
  36.         'paginator': paginator,
  37.         'query': query,
  38.         'suggestion': None,
  39.     }
  40.     if results.query.backend.include_spelling:
  41.         context['suggestion'] = form.get_suggestion()
  42.  
  43.     if extra_context:
  44.         context.update(extra_context)
  45.  
  46.  
  47.     jsondata = []
  48.     print(len(page.object_list))
  49.     for result in page.object_list:
  50.         data = {
  51.             'pk': result.object.pk,
  52.             'title': result.object.title,
  53.             'content': result.object.body,
  54.  
  55.         }
  56.         jsondata.append(data)
  57.     result = {"code": 200, "msg": 'Search successfully！', "data": jsondata}
  58.     return JsonResponse(result, content_type="application/json")

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/运维做开发/article/detail/860226
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号