Overview of Mandrel

Overview of Mandrel

Mandrel attempts to provide a reasonable means of managing a few common concerns:

• General configuration: file layout, ease of retrieval, flexibility.
• Logging configuration: a single point of entry for configuring the python logging utility and for retrieving logging.Logger objects.
• Bootstrapping: a straightforward means by which the details of the above can be adjusted once at the proper time, to avoid complexity.
• Runners: scripts to act as entry points into your apps that are unified with the bootstrap mechanism to give a consistent CLI across a system.

Why?

Unless a team makes an active effort to approach these problems in a unified way, the management of a complex system grows chaotic. Mandrel is this renewed effort on our team. (And what team is that? Not tellin'.)

Beyond the need to provide consistency in how this problem is approached, there's another motivation: the tool or application developer is primarily interested in building something to solve a specific human-world problem, and needs to give the real problems the majority of focus.

Hopefully by using Mandrel, people can stay focused on the right thing and be able to handle logging and configuration in a sane way with a minimum of thought.

Bootstrapping

Mandrel's bootstrap step determines a number of critical things:

• The root path of your application (mandrel.bootstrap.ROOT_PATH)
• The search path for configuration files (mandrel.bootstrap.SEARCH_PATH)
• What kinds of configuration files Mandrel knows how to read (mandrel.config.LOADERS)
• Where logging configuration can be found (mandrel.bootstrap.LOGGING_CONFIG_BASENAME)
• What logging configuration applies in the absence of configuration file.

Basic Idea

Bootstrapping needs to happen before any configuration is loaded, either for logging or for application config. This allows the bootstrap step to affect how the configuration works, avoiding the chicken/egg problem.

Therefore, similar to the approach we see in other popular projects (Git, Ruby's bundler, or Ruby's Rake), Mandrel expects the root of your project to contain a magical file that both identifies the project root and provides opportunity to customize Mandrel itself before any configuration loading happens.

The name of that file is, not surprisingly, Mandrel.py.

When you do anything with Mandrel that results in an import of mandrel.bootstrap, the bootstrap process begins, and a search for Mandrel.py begins from your current working directory upwards.

The first directory found to contain Mandrel.py is determined to be the mandrel.bootstrap.ROOT_PATH.

If not found, your stuff will explode with a charming mandrel.exception.MissingBootstrapException.

Once found, the bootstrap file (the path of which is preserved at mandrel.bootstrap.BOOTSTRAP_FILE) is evaluated by python having both mandrel.bootstrap and mandrel.config in scope (as bootstrap and config, respectively).

The bootstrap file can do whatever it wants. The notion being it uses this opportunity to alter Mandrel's behavior, via the SEARCH_PATH, the logging configuration settings, etc.

Because this file is guaranteed to be processed before any Mandrel configuration/logging is applied, you get a guarantee of consistency in how things work.

A Note on Style

Attributes or variables named with ALL_CAPS are conventionally treated as "constants". It's understandable that altering such variables in the mandrel.bootstrap module may feel like a horrible faux pas.

This was considered and the faux pas embraced. We view these as constants post-bootstrap, meaning we expect that they will be treated in a hands-off manner outside the bootstrap process, and we also expect that the bootstrap process can do whatever it wants.

The point being: for the general run time of your application, these constants should indeed be thought of as constants.

For more

See Bootstrapping for more on this subject.

Configuration

Mandrel address three aspects of the problem of configuration:

1. Where do configuration files live?
2. How do we read them?
3. How do we represent the loaded configuration?

Where files live?

When you ask Mandrel for configuration, it uses its search path (mandrel.bootstrap.SEARCH_PATHS) to look for the relevant configuration file. This is exactly analogous to the library search path (sys.path) or your shell (bash, right?) PATH variable.

The paths on the search path may be absolute, or relative. When relative, they're assumed to be relative to the bootstrapper's ROOT_PATH (see Bootstrapping for more). They are searched in order, with the first match winning (in normal use).

How do we read them?

The bootstrapper contains a mutable list of "readers", with each entry being a tuple of (file_extension, reader_callable). The order of extensions determines the order of extensions applied to the configuration file search.

The reader callable is assumed to know how to read the file type, and is expected to return a dictionary.

By default, the only reader is for simple YAML, with extension ".yaml".

Thus one asks for configuration "foo", and the configuration reader looks across the search path for "foo.yaml". Upon finding the first matching file, the YAML file path is handed to the reader, the reader parses it, and hands back the resulting structure.

Nothing formally enforces the contract that the resulting structure be a dict, other than the fact that the mandrel.config.Configuration class will blow up if you try to use it with a not-dict.

How do we represent loaded configuration?

In the simplest case, you can ask Mandrel for a given configuration, and it'll give you the resulting dict.

However, the mandrel.config module provides Configuration and ForgivingConfiguration classes to enable smarter representation.

A Configuration subclass, when provided with a NAME string, will use that NAME both for its configuration dict and for its logger name. It will wrap a configuration dictionary within an object that allows access to the underlying dict via object attributes, and gives the app/component developer a natural means of enforcing defaults, applying transforms to the config values, etc.

The ForgivingConfiguration is potentially more helpful, as it will use an empty configuration dict when no config can be found by the loader. An app/component developer would do well to extend ForgivingConfiguration and design the subclass so it works well (particularly for local development mode, perhaps) purely from defaults.

For more details

For a discussion of configuration in greater detail, check out Configuration.

Logging

The standard python logging utility is pretty flexible, but can get difficult in a complex system since logging manages global state pertaining to the total log configuration and extant logging.Logger instances. It can be a little fussy to combine logging configuration files, so it's often easier to have a big unified file. However, at the same time, each piece of a system needs logging and needs some means of achieving a sane logging configuration independent of the larger system.

So Mandrel's approach to this:

• Let Mandrel manage the logging configuration for you.
• Let Mandrel do that by finding the logging configuration file on your search path, when first configuring logging.
• Use Mandrel to get your logging.Logger objects, so that Mandrel can consistently apply the logging configuration.
• Provide a sane fallback when no logging configuration exists.

Bootstrap Integration

Mandrel's logging configuration is tightly integrated with Mandrel's bootstrapping. While logging configuration is applied on-demand (lazy-load style), the means of determining that configuration is wrapped up in mandrel.bootstrap.

The following "constants" in mandrel.bootstrap determine how logging configuration works:

• LOGGING_CONFIG_BASENAME:

  This is the file basename for the single logging configuration file, which is assumed to be in the format supported by logging.config.fileConfig.

  Its value determines the file we search for across the regular configuration search paths (mandrel.bootstrap.SEARCH_PATHS), when looking for the logging configuration to load upon first access.

  This defaults to logging.cfg.

• DEFAULT_LOGGING_CALLBACK:

  A callable that mandrel.bootstrap will invoke when configuring logging in the event that the LOGGING_CONFIG_BASENAME file cannot be found on the search path.

  This should be an actual callable, not the name of it. By default, it is set to mandrel.bootstrap.initialize_simple_logging, which sets up minimal logging to sys.stderr and is intended to be reasonable for local development.

• DISABLE_EXISTING_LOGGERS:

  Determines the value used for the disable_existing_loggers keyword parameter when applying the configuration file to the logging system.

  Defaults to True.

Functions

The functions for controlling logging configuration and retrieving loggers are provided by the mandrel.bootstrap module itself, as well. This is perhaps a little surprising and may change in the future, but if it does, it'll change with ample warning.

The most important one from the perspective of an app developer is get_logger(name), which will return a logging.Logger for name based on the best logging configuration available; if logging is not yet configured, it'll configure it at this time.

That means: get your logger from mandrel.bootstrap.get_logger and your system will behave the way you want.

For more

See Logging for more on this subject.

Runners

Mandrel provides two runner scripts as part of a pip install. They act as entry points into the system, provide a common set of options for altering Mandrel's setup via the command line, and therefore allow for a uniform interface for launching stuff with control over where your configuration lives.

• mandrel-runner will launch any callable target identified by a fully qualified name (the first positional parameter).
• mandrel-script will launch any arbitrary python script identified by a resolvable path (the first positional parameter).

See Runners for more on this subject.