Finally got around to finishing this rather long but very intriguing article on the history of the corporation. The points near the end on time are excellent. Must remember this.

The first important thing to realize about the logging package is that there are four types of objects that each play an important role in getting the logging system working:

• Loggers – used by application code to send messages to the logging system.
• Formatters – format the message for output.
• Filters – provide fine-grained output control.
• Handlers – send formatted output to a destination such as a file.

Of these four types, filters are probably the least important to understand, at least initially. I have rarely used filters. On the other hand, they can help to accomplish things that are hard if not impossible to do without them. For example, in multi-threading network-oriented code, they can be a useful way of getting contextual information into the log (e.g. remote IP address or username, or database-connection-specific info). Also, see this post on how to configure logging for Web applications – filters are used as part of the solution.

The thing that probably causes most frustration is knowing how, when, and where to use each of these types in an application, framework, or library. Any complete, stand-alone application must use at least three of the four to get proper logging output. The standard library documentation is fairly clear on how to do this. It may be as simple as calling logging.basicConfig or as involved as reading configuration from a YAML file, merging in sensible defaults, and feeding the result to logging.config.dictConfig.

Most of the time library and framework authors should only use Logger objects. Get a logger and use it to publish log events. That is all. Do not set the logging level. Do not setup formatters or handlers. Otherwise go directly to jail; do not pass Go, do not collect $200. Oh, and fix your logging code while you’re sitting in jail.

One of the best ways to get a logger is like this:

log = logging.getLogger(__name__)

This gets a logger with a name that indicates exactly which module a given logging event originated from (assuming this logger is only used in the module in which it is defined). In general, all modules that need to log should have a module-level logger instantiated with __name__, and code in that module uses that logger. You can also have child loggers of that module logger if you need more fine-grained control. However, since loggers are singletons there is no need to hold a reference to a logger in a class instance. Loggers are singletons for a good reason: it must be possible to reference each logger in order to globally configure and control it. This would not be possible if loggers were not singletons.

Earlier I implied that an application is a stand-alone program. I will take a moment to distinguish between libraries and frameworks for the purposes of this article. A library is something like Beaker, which provides a particular set of functionality such as a caching system. A framework is something like Django—a collection of libraries and glue code that work together to make it easier to write an application. It is imperative that libraries and frameworks use the logging package properly when they choose to use it. Improper use can make a library or framework difficult, not to mention intensely frustrating, to incorporate into a larger application.

With one exception, libraries have no business configuring logging levels, formatters, or handlers. If you are a library author and are tempted to do one or more of those as a convenience for your users, just don’t do it. You will likely frustrate them more than you help them. As for that one exception, modules that wish to use logging, but do not wish to emit a warning if the library user has not configured logging will want to add a NullHandler to the logger at the top-level of the logging namespace used by the library:

import logging
log = logging.getLogger(__name__)
log.addHandler(logging.NullHandler())

This configures a do-nothing handler that prevents the “No handlers could be found for …” warning when your library triggers logging events in contexts where handlers have not been otherwise configured. See the Logging HOWTO for more on this subject.

Note that it is acceptable for libraries to include APIs that configure all the various parts of the logging system, including logging levels, formatters, handlers, and filters. A key point here is that any such API should be well documented and possible to avoid. With the noted exception above, this rule expressly prohibits any import-time configuration. This allows the developer to roll her own logging configuration if the provided API does not result in a satisfactory configuration.

As with libraries, it is equally important to follow sound practices when using logging in a framework. A framework may include a bootstrap system, or means with which to get an application running. This often includes some configuration mechanism through which various features are enabled or configured. Logging levels, filters, formats, and handlers are among the things that a framework may want to configure. If your framework must setup a specific handler and/or formatter, do it in a way that can be easily disabled, and clearly document how to disable it. By “disable it” I mean disable every single code branch where a filter, handler, or formatter is instantiated or a logging level is set (with setLevel) or a handler is added to a logger (with addHandler). You may want to setup a specific set of handlers and formatters, for example, because you anticipate that most of your users will want a very specific type of logging output. One example of this would be a web framework like CherryPy, which uses the logging system to generate standard access and error logs similar to those generated by Apache. This can be nice when getting a simple web application up and running. However, in more advanced use cases it can be very frustrating if there is no easy way (i.e., other than monkey-patching) to override or disable logging configurations done within a framework.

Logging for an application is most often configured once at startup (although more elaborate scenarios are possible and can be quite useful). This configuration process can be annoyingly hampered by a framework or library doing its own logging configuration in addition to the configuration done by the application developer. In some cases the first configuration wins (e.g., logging.basicConfig), in others you’ll end up with two competing configurations (e.g., two handlers sending output to the same file resulting in duplicate messages), and in yet other situations the last one wins (e.g., setting the logging level of a given logger or handler). In cases where logging is configured at import-time (generally something to avoid), the order in which modules are imported can change logging behavior in strange and non-intuitive ways. All this is to say that the behavior of the logging package can be very confusing when used improperly. This is compounded when frameworks and libraries configure logging when they should not, and it is often hard to track down such misconfiguration.

As a developer of a framework or library you should always allow or provide a single point of configuration for logging levels, filters, formats, and handlers. When developing a library, just get a logger and log stuff—don’t mess with any other parts of the logging system. When developing a framework, provide a way to override or disable anything that can be configured by logging.config.dictConfig. Following these simple rules will make your library or framework vastly more convenient for anyone trying to setup advanced logging configurations.

Special thanks to Vinay Sajip for reviewing this article and providing valuable comments.