2
\$\begingroup\$

I'm running a package that downloads data from the internet. This package makes logging calls from different modules.

Suppose I have a package 'demopackage'. Here is how my logging is currently set up. Is it possible to improve this way of working? The main.py script runs triggered by a cron job.

demopackage/init.py

""" __init__.py """

import logging
from demopackage.core import Logs

Logs.config_root_logger()

I set up 3 handlers, a console handler with color, a file handler, and an email handler for critical log entries (Amazon SES handler). I would prefer the stream and file handler to inherit a handler like the email handler.

demopackage/core.py

import boto.ses
import coloredlogs


class Logs:
    fs_head = '%(asctime)s.%(msecs)03d %(levelname)1.1s L%(lineno)d '
    fs = fs_head + '%(filename)s %(funcName)s %(name)s | %(message)s'
    dfs = '%d/%m/%y %H:%M:%S'
    formatter = demopackage.logging.Formatter(fmt=fs, datefmt=dfs)

    @staticmethod
    def config_root_logger():
        log = demopackage.logging.getLogger()
        # configure file handler
        logpath = demopackage.os.path.join('demopackage.log')
        rfh = demopackage.logging.handlers.RotatingFileHandler(logpath)
        rfh.maxBytes = 1000000000  # about 1 GB
        rfh.backupCount = 2
        rfh.mode = 'a'
        rfh.setFormatter(Logs.formatter)
        rfh.setLevel('INFO')
        log.addHandler(rfh)

        # configure email handler
        smtp_handler = Logs.demopackage_SESHandler()
        smtp_handler.setFormatter(Logs.formatter)
        smtp_handler.setLevel('CRITICAL')
        log.addHandler(smtp_handler)
        # configure root logger StreamHandler
        # blue, cyan, green, magenta, red, white and yellow
        level_dict = {'debug': {'color': 'blue'},
                      'info': {'color': 'green'},
                      'warning': {'color': 'yellow'},
                      'error': {'color': 'red', 'bold': True},
                      'critical': {'color': 'magenta', 'bold': True}}
        field_dict = {'asctime': {'color': 'green'},
                      'levelname': {'color': 'white', 'bold': True},
                      'lineno': {'color': 'red'},
                      'filename': {'color': 'cyan'},
                      'funcName': {'color': 'magenta'},
                      'name': {'color': 'green'}}
        coloredlogs.install(fmt=Logs.fs, datefmt=Logs.dfs,
                            level_styles=level_dict,
                            field_styles=field_dict)
        log.info('stream handler initialized with coloredlogs module')
        log.info('root logger file handler to {}'.format(logpath))
        log.info('root logger smtp handler for AWS SES')

    class demopackage_SESHandler(demopackage.logging.Handler):
        """
        A handler class which sends an email to Amazon SES for logging events.
        """
        def __init__(self):
            """ Initialize the handler. """
            demopackage.logging.Handler.__init__(self)

        def emit(self, record):
            """
            Emit a record.
            Format the record and send it to the specified addressees.
            """
            try:
                key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
                conn = boto.ses.connect_to_region(
                    'eu-west-1',
                    aws_access_key_id='XXXXXXXXXXXXXXXXXXXXXXXXXXXX',
                    aws_secret_access_key=key)
                message = self.format(record)
                subject = 'Critical error in ' + record.pathname
            conn.send_email('from@sender.com', subject, message,
                            'to@receiver.com')
            except Exception:
                self.handleError(record)

logging calls in any module:

import demopackage

log = demopackage.logging.getLogger('dwh.get_data.py')

log.info('some text')
log.error('some error raising an exception')
log.critical('critical error with email alert')
\$\endgroup\$
  • \$\begingroup\$ It would be good if you'll post your demopackage.logging.getLogger method here. \$\endgroup\$ – Montreal Jan 29 at 5:34
  • \$\begingroup\$ the demopackage.logging.getLogger method is the method from the standard python logging module: docs.python.org/3/library/logging.html#logging.getLogger It is available in this namespace since I've imported logging in init.py \$\endgroup\$ – Mathijs303 Jan 30 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.