2
\$\begingroup\$

I'm running a package that downloads data from the internet. This package makes logging calls from different modules.

Suppose I have a package 'demopackage'. Here is how my logging is currently set up. Is it possible to improve this way of working? The main.py script runs triggered by a cron job.

demopackage/init.py

""" __init__.py """

import logging
from demopackage.core import Logs

Logs.config_root_logger()

I set up 3 handlers, a console handler with color, a file handler, and an email handler for critical log entries (Amazon SES handler). I would prefer the stream and file handler to inherit a handler like the email handler.

demopackage/core.py

import boto.ses
import coloredlogs


class Logs:
    fs_head = '%(asctime)s.%(msecs)03d %(levelname)1.1s L%(lineno)d '
    fs = fs_head + '%(filename)s %(funcName)s %(name)s | %(message)s'
    dfs = '%d/%m/%y %H:%M:%S'
    formatter = demopackage.logging.Formatter(fmt=fs, datefmt=dfs)

    @staticmethod
    def config_root_logger():
        log = demopackage.logging.getLogger()
        # configure file handler
        logpath = demopackage.os.path.join('demopackage.log')
        rfh = demopackage.logging.handlers.RotatingFileHandler(logpath)
        rfh.maxBytes = 1000000000  # about 1 GB
        rfh.backupCount = 2
        rfh.mode = 'a'
        rfh.setFormatter(Logs.formatter)
        rfh.setLevel('INFO')
        log.addHandler(rfh)

        # configure email handler
        smtp_handler = Logs.demopackage_SESHandler()
        smtp_handler.setFormatter(Logs.formatter)
        smtp_handler.setLevel('CRITICAL')
        log.addHandler(smtp_handler)
        # configure root logger StreamHandler
        # blue, cyan, green, magenta, red, white and yellow
        level_dict = {'debug': {'color': 'blue'},
                      'info': {'color': 'green'},
                      'warning': {'color': 'yellow'},
                      'error': {'color': 'red', 'bold': True},
                      'critical': {'color': 'magenta', 'bold': True}}
        field_dict = {'asctime': {'color': 'green'},
                      'levelname': {'color': 'white', 'bold': True},
                      'lineno': {'color': 'red'},
                      'filename': {'color': 'cyan'},
                      'funcName': {'color': 'magenta'},
                      'name': {'color': 'green'}}
        coloredlogs.install(fmt=Logs.fs, datefmt=Logs.dfs,
                            level_styles=level_dict,
                            field_styles=field_dict)
        log.info('stream handler initialized with coloredlogs module')
        log.info('root logger file handler to {}'.format(logpath))
        log.info('root logger smtp handler for AWS SES')

    class demopackage_SESHandler(demopackage.logging.Handler):
        """
        A handler class which sends an email to Amazon SES for logging events.
        """
        def __init__(self):
            """ Initialize the handler. """
            demopackage.logging.Handler.__init__(self)

        def emit(self, record):
            """
            Emit a record.
            Format the record and send it to the specified addressees.
            """
            try:
                key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
                conn = boto.ses.connect_to_region(
                    'eu-west-1',
                    aws_access_key_id='XXXXXXXXXXXXXXXXXXXXXXXXXXXX',
                    aws_secret_access_key=key)
                message = self.format(record)
                subject = 'Critical error in ' + record.pathname
            conn.send_email('from@sender.com', subject, message,
                            'to@receiver.com')
            except Exception:
                self.handleError(record)

logging calls in any module:

import demopackage

log = demopackage.logging.getLogger('dwh.get_data.py')

log.info('some text')
log.error('some error raising an exception')
log.critical('critical error with email alert')
\$\endgroup\$
  • \$\begingroup\$ It would be good if you'll post your demopackage.logging.getLogger method here. \$\endgroup\$ – Montreal Jan 29 at 5:34
  • \$\begingroup\$ the demopackage.logging.getLogger method is the method from the standard python logging module: docs.python.org/3/library/logging.html#logging.getLogger It is available in this namespace since I've imported logging in init.py \$\endgroup\$ – Mathijs303 Jan 30 at 12:23

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.