当前位置:   article > 正文

Python开发——logging模块详解

valueerror: dictionary doesn't specify a version

logging模块详解

logging简介

logging是python的内置库,主要用于进行格式化内容输出,可将格式化内容输出到文件,也可输出到屏幕。我们在开发过程中常用print函数来进行调试,但是实际应用部署时我们要将日志的信息要输出到文件中,方便后续查找以及备份。在我们使用日志管理时,我们还可以将日志格式化成json对象转存到ELK中方便图形化查看及管理。前面说的这些,我们都可以通过logging所包含的功能以及提供扩展的方式来完成。

logging工作流程

这是从网上查到一个关于python logging模块的工作流程图,非常遗憾没有找到出处。

可以看到图中,几个关键的类

Logger

用于记录日志的对象。

通过流程图可以看到

  1. 判断是否enabled,实质就是看记录的level(logger.info,logger.debug等)和当前logger对象设置的level是否满足(比如logger设置的lever是Info,记录时使用的logger.debug,那么就会不满足,所以不会记录日志)
  2. 查看logger的过滤器是否满足。filter通过之后,交给logger的handler来记录日志,一个logger是可以设置多个handler。
  3. 交给Handlers实际记录日志

注:在整个应用中可以有多个logger,使用logging.getLogger时通过指定name来获取对象,实际logging中还存在一个Manager类,由Manager来进行多logger的单例模式管理。

Handler

用于记录日志到具体的文件或者输出流或其他的管道。

  1. 查看记录日志是否满足过滤器
  2. 满足过滤器,按照设置的Formatter生成字符串
  3. 将内容写入到具体的文件或者输出流

不同的Handler可能有不同的处理,但是底层原理还是做这三件事情。

Filter

用于过滤用户记录日志时,是否满足记录标准

Formatter

用于格式化,输出的字符串格式

这是原生自带的格式

  1. %(name)s Name of the logger (logging channel)
  2. %(levelno)s Numeric logging level for the message (DEBUG, INFO,
  3. WARNING, ERROR, CRITICAL)
  4. %(levelname)s Text logging level for the message ("DEBUG", "INFO",
  5. "WARNING", "ERROR", "CRITICAL")
  6. %(pathname)s Full pathname of the source file where the logging
  7. call was issued (if available)
  8. %(filename)s Filename portion of pathname
  9. %(module)s Module (name portion of filename)
  10. %(lineno)d Source line number where the logging call was issued
  11. (if available)
  12. %(funcName)s Function name
  13. %(created)f Time when the LogRecord was created (time.time()
  14. return value)
  15. %(asctime)s Textual time when the LogRecord was created
  16. %(msecs)d Millisecond portion of the creation time
  17. %(relativeCreated)d Time in milliseconds when the LogRecord was created,
  18. relative to the time the logging module was loaded
  19. (typically at application startup time)
  20. %(thread)d Thread ID (if available)
  21. %(threadName)s Thread name (if available)
  22. %(process)d Process ID (if available)
  23. %(message)s The result of record.getMessage(), computed just as
  24. the record is emitted
  25. 复制代码

LogRecord

我们每一次的 logger.info logger.debug logger.error等实际上都是进行一次LogRecord的处理

包含一些基本的输出日志内容,以及内容中参数,还有附带的函数名称,行数等信息

logging源码阅读

这是我这边基于字典的一个logging配置

  1. import logging
  2. import logging.config
  3. import os
  4. basedir = os.path.abspath(os.path.dirname(__file__))
  5. log_path = os.path.join(basedir, '..', 'logs')
  6. service_name = "test_log"
  7. if not os.path.isdir(log_path):
  8. os.mkdir(log_path)
  9. class InfoFilter(logging.Filter):
  10. def filter(self, record):
  11. """
  12. 只筛选出INFO级别的日志
  13. """
  14. if logging.INFO <= record.levelno < logging.ERROR:
  15. return super().filter(record)
  16. else:
  17. return 0
  18. class ErrorFilter(logging.Filter):
  19. def filter(self, record):
  20. """
  21. 只筛选出ERROR级别的日志
  22. """
  23. if logging.ERROR <= record.levelno < logging.CRITICAL:
  24. return super().filter(record)
  25. else:
  26. return 0
  27. LOG_CONFIG_DICT = {
  28. 'version': 1,
  29. 'disable_existing_loggers': False,
  30. 'formatters': {
  31. 'simple': {
  32. 'class': 'logging.Formatter',
  33. 'format': '%(asctime)s %(levelname)s %(name)s %(filename)s %(module)s %(funcName)s '
  34. '%(lineno)d %(thread)d %(threadName)s %(process)d %(processName)s %(message)s'
  35. },
  36. # json模式, 方便ELK收集处理
  37. 'json': {
  38. 'class': 'logging.Formatter',
  39. 'format': '{"time:":"%(asctime)s","level":"%(levelname)s","logger_name":"%(name)s",'
  40. '"file_name":"%(filename)s","module":"%(module)s","func_name":"%(funcName)s",'
  41. '"line_number":"%(lineno)d","thread_id":"%(thread)d","thread_name":"%(threadName)s",'
  42. '"process_id":"%(process)d","process_name":"%(processName)s","message":"%(message)s"}'}
  43. },
  44. # 过滤器
  45. 'filters': {
  46. 'info_filter': {
  47. '()': InfoFilter
  48. },
  49. 'error_filter': {
  50. '()': ErrorFilter
  51. }
  52. },
  53. # 处理器
  54. 'handlers': {
  55. # 控制台输出
  56. 'console': {
  57. 'class': 'logging.StreamHandler',
  58. 'level': 'INFO',
  59. 'formatter': 'simple'
  60. },
  61. # info文件输出
  62. 'info_file': {
  63. 'level': 'INFO',
  64. 'formatter': 'json',
  65. 'class': 'logging.handlers.TimedRotatingFileHandler',
  66. 'filename': '{0}/{1}_info.log'.format(log_path, service_name),
  67. 'when': "d",
  68. 'interval': 1,
  69. 'encoding': 'utf8',
  70. 'backupCount': 30,
  71. 'filters': ['info_filter']
  72. },
  73. # error文件输出
  74. 'error_file': {
  75. 'level': 'ERROR',
  76. 'formatter': 'json',
  77. 'class': 'logging.handlers.TimedRotatingFileHandler',
  78. 'filename': '{0}/{1}_error.log'.format(log_path, service_name),
  79. 'when': "d",
  80. 'interval': 1,
  81. 'encoding': 'utf8',
  82. 'backupCount': 30,
  83. 'filters': ['error_filter']
  84. }
  85. },
  86. # 记录器
  87. 'loggers': {
  88. 'full_logger': {
  89. 'handlers': ['console', 'info_file', 'error_file'],
  90. 'level': 'INFO'
  91. },
  92. 'only_console_logger': {
  93. 'handlers': ['console'],
  94. 'level': 'INFO'
  95. },
  96. 'only_file_logger': {
  97. 'handlers': ['info_file', 'error_file']
  98. }
  99. }
  100. }
  101. logging.config.dictConfig(LOG_CONFIG_DICT)
  102. 复制代码

下面我们就基于config第一次配置时整个logging的工作原理。结合代码进行分析logging的工作之路。

我的python版本是Python3.6

从logging.config开始

  1. dictConfigClass = DictConfigurator
  2. def dictConfig(config):
  3. """Configure logging using a dictionary."""
  4. dictConfigClass(config).configure()
  5. 复制代码

实质上是实例化logging.config.DictConfigurator类的对象,然后执行其configure方法。

  1. class DictConfigurator(BaseConfigurator):
  2. """
  3. Configure logging using a dictionary-like object to describe the
  4. configuration.
  5. """
  6. def configure(self):
  7. """Do the configuration."""
  8. config = self.config
  9. if 'version' not in config:
  10. raise ValueError("dictionary doesn't specify a version")
  11. if config['version'] != 1:
  12. raise ValueError("Unsupported version: %s" % config['version'])
  13. incremental = config.pop('incremental', False)
  14. EMPTY_DICT = {}
  15. logging._acquireLock()
  16. try:
  17. if incremental:
  18. handlers = config.get('handlers', EMPTY_DICT)
  19. for name in handlers:
  20. if name not in logging._handlers:
  21. raise ValueError('No handler found with '
  22. 'name %r' % name)
  23. else:
  24. try:
  25. handler = logging._handlers[name]
  26. handler_config = handlers[name]
  27. level = handler_config.get('level', None)
  28. if level:
  29. handler.setLevel(logging._checkLevel(level))
  30. except Exception as e:
  31. raise ValueError('Unable to configure handler '
  32. '%r: %s' % (name, e))
  33. loggers = config.get('loggers', EMPTY_DICT)
  34. for name in loggers:
  35. try:
  36. self.configure_logger(name, loggers[name], True)
  37. except Exception as e:
  38. raise ValueError('Unable to configure logger '
  39. '%r: %s' % (name, e))
  40. root = config.get('root', None)
  41. if root:
  42. try:
  43. self.configure_root(root, True)
  44. except Exception as e:
  45. raise ValueError('Unable to configure root '
  46. 'logger: %s' % e)
  47. else:
  48. disable_existing = config.pop('disable_existing_loggers', True)
  49. logging._handlers.clear()
  50. del logging._handlerList[:]
  51. # Do formatters first - they don't refer to anything else
  52. formatters = config.get('formatters', EMPTY_DICT)
  53. for name in formatters:
  54. try:
  55. formatters[name] = self.configure_formatter(
  56. formatters[name])
  57. except Exception as e:
  58. raise ValueError('Unable to configure '
  59. 'formatter %r: %s' % (name, e))
  60. # Next, do filters - they don't refer to anything else, either
  61. filters = config.get('filters', EMPTY_DICT)
  62. for name in filters:
  63. try:
  64. filters[name] = self.configure_filter(filters[name])
  65. except Exception as e:
  66. raise ValueError('Unable to configure '
  67. 'filter %r: %s' % (name, e))
  68. handlers = config.get('handlers', EMPTY_DICT)
  69. deferred = []
  70. for name in sorted(handlers):
  71. try:
  72. handler = self.configure_handler(handlers[name])
  73. handler.name = name
  74. handlers[name] = handler
  75. except Exception as e:
  76. if 'target not configured yet' in str(e):
  77. deferred.append(name)
  78. else:
  79. raise ValueError('Unable to configure handler '
  80. '%r: %s' % (name, e))
  81. # Now do any that were deferred
  82. for name in deferred:
  83. try:
  84. handler = self.configure_handler(handlers[name])
  85. handler.name = name
  86. handlers[name] = handler
  87. except Exception as e:
  88. raise ValueError('Unable to configure handler '
  89. '%r: %s' % (name, e))
  90. root = logging.root
  91. existing = list(root.manager.loggerDict.keys())
  92. existing.sort()
  93. child_loggers = []
  94. #now set up the new ones...
  95. loggers = config.get('loggers', EMPTY_DICT)
  96. for name in loggers:
  97. if name in existing:
  98. i = existing.index(name) + 1 # look after name
  99. prefixed = name + "."
  100. pflen = len(prefixed)
  101. num_existing = len(existing)
  102. while i < num_existing:
  103. if existing[i][:pflen] == prefixed:
  104. child_loggers.append(existing[i])
  105. i += 1
  106. existing.remove(name)
  107. try:
  108. self.configure_logger(name, loggers[name])
  109. except Exception as e:
  110. raise ValueError('Unable to configure logger '
  111. '%r: %s' % (name, e))
  112. _handle_existing_loggers(existing, child_loggers,
  113. disable_existing)
  114. # And finally, do the root logger
  115. root = config.get('root', None)
  116. if root:
  117. try:
  118. self.configure_root(root)
  119. except Exception as e:
  120. raise ValueError('Unable to configure root '
  121. 'logger: %s' % e)
  122. finally:
  123. logging._releaseLock()
  124. 复制代码

我们对incremental不进行设置,即使用全量的方式进行配置,对于所有的handler重置并处理

看下来基本上分为4步来走。

  1. 配置Formatter
  2. 配置Filter
  3. 配置Handler
  4. 配置Logger

Formatter配置

源码部分

  1. def configure_formatter(self, config):
  2. """Configure a formatter from a dictionary."""
  3. if '()' in config:
  4. factory = config['()'] # for use in exception handler
  5. try:
  6. result = self.configure_custom(config)
  7. except TypeError as te:
  8. if "'format'" not in str(te):
  9. raise
  10. config['fmt'] = config.pop('format')
  11. config['()'] = factory
  12. result = self.configure_custom(config)
  13. else:
  14. fmt = config.get('format', None)
  15. dfmt = config.get('datefmt', None)
  16. style = config.get('style', '%')
  17. cname = config.get('class', None)
  18. if not cname:
  19. c = logging.Formatter
  20. else:
  21. c = _resolve(cname)
  22. result = c(fmt, dfmt, style)
  23. return result
  24. 复制代码

可以使用自己的Formatter,默认使用logging.Formatter

Filter配置

  1. def configure_filter(self, config):
  2. """Configure a filter from a dictionary."""
  3. if '()' in config:
  4. result = self.configure_custom(config)
  5. else:
  6. name = config.get('name', '')
  7. result = logging.Filter(name)
  8. return result
  9. 复制代码

设置filter,可以参考我上面配置的字典,可以自己定义Filter也可以使用系统内置Filter

Handler配置

  1. def configure_handler(self, config):
  2. """Configure a handler from a dictionary."""
  3. config_copy = dict(config) # for restoring in case of error
  4. formatter = config.pop('formatter', None)
  5. if formatter:
  6. try:
  7. formatter = self.config['formatters'][formatter]
  8. except Exception as e:
  9. raise ValueError('Unable to set formatter '
  10. '%r: %s' % (formatter, e))
  11. level = config.pop('level', None)
  12. filters = config.pop('filters', None)
  13. if '()' in config:
  14. c = config.pop('()')
  15. if not callable(c):
  16. c = self.resolve(c)
  17. factory = c
  18. else:
  19. cname = config.pop('class')
  20. klass = self.resolve(cname)
  21. #Special case for handler which refers to another handler
  22. if issubclass(klass, logging.handlers.MemoryHandler) and\
  23. 'target' in config:
  24. try:
  25. th = self.config['handlers'][config['target']]
  26. if not isinstance(th, logging.Handler):
  27. config.update(config_copy) # restore for deferred cfg
  28. raise TypeError('target not configured yet')
  29. config['target'] = th
  30. except Exception as e:
  31. raise ValueError('Unable to set target handler '
  32. '%r: %s' % (config['target'], e))
  33. elif issubclass(klass, logging.handlers.SMTPHandler) and\
  34. 'mailhost' in config:
  35. config['mailhost'] = self.as_tuple(config['mailhost'])
  36. elif issubclass(klass, logging.handlers.SysLogHandler) and\
  37. 'address' in config:
  38. config['address'] = self.as_tuple(config['address'])
  39. factory = klass
  40. props = config.pop('.', None)
  41. kwargs = dict([(k, config[k]) for k in config if valid_ident(k)])
  42. try:
  43. result = factory(**kwargs)
  44. except TypeError as te:
  45. if "'stream'" not in str(te):
  46. raise
  47. kwargs['strm'] = kwargs.pop('stream')
  48. result = factory(**kwargs)
  49. if formatter:
  50. result.setFormatter(formatter)
  51. if level is not None:
  52. result.setLevel(logging._checkLevel(level))
  53. if filters:
  54. self.add_filters(result, filters)
  55. if props:
  56. for name, value in props.items():
  57. setattr(result, name, value)
  58. return result
  59. 复制代码

可以看到,前面顺序先配置formatter和filter是因为这里需要引用的到

值得注意的是,我们初始化时传递的一个字典,在整个配置过程中,字典里面的值会随着我们每次的配置变化而变化,所以我们在每个元素配置之后,在使用上一个字典元素时,就是配置完成之后的元素,为了方便理解,将配置filter之前和配置filter之后,config中的filter变化列出来

  1. config == > before filters {'info_filter': {'()': <class 'utils.log.InfoFilter'>}, 'error_filter': {'()': <class 'utils.log.ErrorFilter'>}}
  2. config == > after filters {'info_filter': <utils.log.InfoFilter object at 0x00000000028EB208>, 'error_filter': <utils.log.ErrorFilter object at 0x00000000028EB240>}
  3. 复制代码

配置前是我们配置文件中的内容,配置完成之后filter已经是一组对象了,所以在配置handler时我们就可以直接使用对象add_filter了。

Logger配置

  1. def common_logger_config(self, logger, config, incremental=False):
  2. level = config.get('level', None)
  3. if level is not None:
  4. logger.setLevel(logging._checkLevel(level))
  5. if not incremental:
  6. #Remove any existing handlers
  7. for h in logger.handlers[:]:
  8. logger.removeHandler(h)
  9. handlers = config.get('handlers', None)
  10. if handlers:
  11. self.add_handlers(logger, handlers)
  12. filters = config.get('filters', None)
  13. if filters:
  14. self.add_filters(logger, filters)
  15. def configure_logger(self, name, config, incremental=False):
  16. """Configure a non-root logger from a dictionary."""
  17. logger = logging.getLogger(name)
  18. self.common_logger_config(logger, config, incremental)
  19. propagate = config.get('propagate', None)
  20. if propagate is not None:
  21. logger.propagate = propagate
  22. 复制代码

这就比较容易理解了,将我们上面配置过的filters和handlers添加到我们的logger中。

这里需要注意的一点是logger = logging.getLogger(name),看下logger.getLogger源码

  1. root = RootLogger(WARNING)
  2. Logger.root = root
  3. Logger.manager = Manager(Logger.root)
  4. def getLogger(name=None):
  5. if name:
  6. return Logger.manager.getLogger(name)
  7. else:
  8. return root
  9. class Manager(object):
  10. def __init__(self, rootnode):
  11. self.root = rootnode
  12. self.disable = 0
  13. self.emittedNoHandlerWarning = False
  14. self.loggerDict = {}
  15. self.loggerClass = None
  16. self.logRecordFactory = None
  17. def getLogger(self, name):
  18. rv = None
  19. if not isinstance(name, str):
  20. raise TypeError('A logger name must be a string')
  21. _acquireLock()
  22. try:
  23. if name in self.loggerDict:
  24. rv = self.loggerDict[name]
  25. if isinstance(rv, PlaceHolder):
  26. ph = rv
  27. rv = (self.loggerClass or _loggerClass)(name)
  28. rv.manager = self
  29. self.loggerDict[name] = rv
  30. self._fixupChildren(ph, rv)
  31. self._fixupParents(rv)
  32. else:
  33. rv = (self.loggerClass or _loggerClass)(name)
  34. rv.manager = self
  35. self.loggerDict[name] = rv
  36. self._fixupParents(rv)
  37. finally:
  38. _releaseLock()
  39. return rv
  40. 复制代码

可以看到,logging使用Mangaer进行logger的单例管理

截止到这里,基本上我们使用前的准备,logging都替我们准备好了,下面就是我们的使用

获取logger并记录日志

  1. class Logger(Filterer):
  2. def info(self, msg, *args, **kwargs):
  3. if self.isEnabledFor(INFO):
  4. self._log(INFO, msg, args, **kwargs)
  5. def error(self, msg, *args, **kwargs):
  6. if self.isEnabledFor(ERROR):
  7. self._log(ERROR, msg, args, **kwargs)
  8. def isEnabledFor(self, level):
  9. if self.manager.disable >= level:
  10. return False
  11. return level >= self.getEffectiveLevel()
  12. def _log(self, level, msg, args, exc_info=None, extra=None, stack_info=False):
  13. sinfo = None
  14. if _srcfile:
  15. try:
  16. fn, lno, func, sinfo = self.findCaller(stack_info)
  17. except ValueError: # pragma: no cover
  18. fn, lno, func = "(unknown file)", 0, "(unknown function)"
  19. else: # pragma: no cover
  20. fn, lno, func = "(unknown file)", 0, "(unknown function)"
  21. if exc_info:
  22. if isinstance(exc_info, BaseException):
  23. exc_info = (type(exc_info), exc_info, exc_info.__traceback__)
  24. elif not isinstance(exc_info, tuple):
  25. exc_info = sys.exc_info()
  26. record = self.makeRecord(self.name, level, fn, lno, msg, args,
  27. exc_info, func, extra, sinfo)
  28. self.handle(record)
  29. def handle(self, record):
  30. if (not self.disabled) and self.filter(record):
  31. self.callHandlers(record)
  32. def callHandlers(self, record):
  33. c = self
  34. found = 0
  35. while c:
  36. for hdlr in c.handlers:
  37. found = found + 1
  38. if record.levelno >= hdlr.level:
  39. hdlr.handle(record)
  40. if not c.propagate:
  41. c = None #break out
  42. else:
  43. c = c.parent
  44. if (found == 0):
  45. if lastResort:
  46. if record.levelno >= lastResort.level:
  47. lastResort.handle(record)
  48. elif raiseExceptions and not self.manager.emittedNoHandlerWarning:
  49. sys.stderr.write("No handlers could be found for logger"
  50. " \"%s\"\n" % self.name)
  51. self.manager.emittedNoHandlerWarning = True
  52. class StreamHandler(Handler):
  53. terminator = '\n'
  54. def __init__(self, stream=None):
  55. """
  56. Initialize the handler.
  57. If stream is not specified, sys.stderr is used.
  58. """
  59. Handler.__init__(self)
  60. if stream is None:
  61. stream = sys.stderr
  62. self.stream = stream
  63. def flush(self):
  64. self.acquire()
  65. try:
  66. if self.stream and hasattr(self.stream, "flush"):
  67. self.stream.flush()
  68. finally:
  69. self.release()
  70. def emit(self, record):
  71. try:
  72. msg = self.format(record)
  73. stream = self.stream
  74. stream.write(msg)
  75. stream.write(self.terminator)
  76. self.flush()
  77. except Exception:
  78. self.handleError(record)
  79. def __repr__(self):
  80. level = getLevelName(self.level)
  81. name = getattr(self.stream, 'name', '')
  82. if name:
  83. name += ' '
  84. return '<%s %s(%s)>' % (self.__class__.__name__, name, level)
  85. 复制代码

根据代码可以看到符合我们流程图看到的流程,我们细化一下就是

  1. 查看记录日志是否满足过滤器,然后准备logrecord中的信息并生成logrecord
  2. 将logrecord交给所有handlers处理
  3. 在handler中确定是否满足handler过滤器,满足的话按照配置的Formatter生成字符串
  4. 将内容写入到具体的文件或者输出流
  1. logger = logging.getLogger("full_logger")
  2. logger.info("111")
  3. # 根据我们的配置 这里会输出到流中,以及记录到test_log_info.log文件中
  4. logger.error("error")
  5. # 根据我们的配置 这里会输出到流中,以及记录到test_log_error.info文件中
  6. 复制代码

有兴趣的同学可以继续试试其他好玩的处理

也可以关注我的公众号共同学习。

转载于:https://juejin.im/post/5cee61d86fb9a07eff006b2c

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/很楠不爱3/article/detail/590639
推荐阅读
相关标签
  

闽ICP备14008679号