Option and Configuration Handling

Option Management

Command-line options are often also set in configuration files for Flake8. While not all options are meant to be parsed from configuration files, many default options are also parsed from configuration files as well as most plugin options.

In Flake8 2, plugins received a optparse.OptionParser instance and called optparse.OptionParser.add_option() to register options. If the plugin author also wanted to have that option parsed from config files they also had to do something like:

parser.config_options.append('my_config_option')
parser.config_options.extend(['config_opt1', 'config_opt2'])

This was previously undocumented and led to a lot of confusion about why registered options were not automatically parsed from configuration files.

Since Flake8 3 was rewritten from scratch, we decided to take a different approach to configuration file parsing. Instead of needing to know about an undocumented attribute that pep8 looks for, Flake8 3 now accepts a parameter to add_option, specifically parse_from_config which is a boolean value.

Flake8 does this by creating its own abstractions on top of argparse. The first abstraction is the flake8.options.manager.Option class. The second is the flake8.options.manager.OptionManager. In fact, we add three new parameters:

• parse_from_config
• comma_separated_list
• normalize_paths

The last two are not specifically for configuration file handling, but they do improve that dramatically. We found that there were options that, when specified in a configuration file, often necessitated being spit multiple lines and those options were almost always comma-separated. For example, let’s consider a user’s list of ignored error codes for a project:

[flake8]
ignore =
    # Reasoning
    E111,
    # Reasoning
    E711,
    # Reasoning
    E712,
    # Reasoning
    E121,
    # Reasoning
    E122,
    # Reasoning
    E123,
    # Reasoning
    E131,
    # Reasoning
    E251

It makes sense here to allow users to specify the value this way, but, the standard library’s configparser.RawConfigParser class does returns a string that looks like

"\nE111,  \nE711,  \nE712,  \nE121,  \nE122,  \nE123,  \nE131,  \nE251  "

This means that a typical call to str.split() with ',' will not be sufficient here. Telling Flake8 that something is a comma-separated list (e.g., comma_separated_list=True) will handle this for you. Flake8 will return:

["E111", "E711", "E712", "E121", "E122", "E123", "E131", "E251"]

Next let’s look at how users might like to specify their exclude list. Presently OpenStack’s Nova project has this line in their tox.ini:

exclude = .venv,.git,.tox,dist,doc,*openstack/common/*,*lib/python*,*egg,build,tools/xenserver*,releasenotes

We think we can all agree that this would be easier to read like this:

exclude =
    .venv,
    .git,
    .tox,
    dist,
    doc,
    *openstack/common/*,
    *lib/python*,
    *egg,
    build,
    tools/xenserver*,
    releasenotes

In this case, since these are actually intended to be paths, we would specify both comma_separated_list=True and normalize_paths=True because we want the paths to be provided to us with some consistency (either all absolute paths or not).

Now let’s look at how this will actually be used. Most plugin developers will receive an instance of OptionManager so to ease the transition we kept the same API as the optparse.OptionParser object. The only difference is that add_option() accepts the three extra arguments we highlighted above.

Configuration File Management

In Flake8 2, configuration file discovery and management was handled by pep8. In pep8’s 1.6 release series, it drastically broke how discovery and merging worked (as a result of trying to improve it). To avoid a dependency breaking Flake8 again in the future, we have created our own discovery and management. As part of managing this ourselves, we decided to change management/discovery for 3.0.0. We have done the following:

• User files (files stored in a user’s home directory or in the XDG directory inside their home directory) are the first files read. For example, if the user has a ~/.flake8 file, we will read that first.
• Project files (files stored in the current directory) are read next and merged on top of the user file. In other words, configuration in project files takes precedence over configuration in user files.
• New in 3.0.0 The user can specify --append-config <path-to-file> repeatedly to include extra configuration files that should be read and take precedence over user and project files.
• New in 3.0.0 The user can specify --config <path-to-file> to so this file is the only configuration file used. This is a change from Flake8 2 where pep8 would simply merge this configuration file into the configuration generated by user and project files (where this takes precedence).
• New in 3.0.0 The user can specify --isolated to disable configuration via discovered configuration files.

To facilitate the configuration file management, we’ve taken a different approach to discovery and management of files than pep8. In pep8 1.5, 1.6, and 1.7 configuration discovery and management was centralized in 66 lines of very terse python which was confusing and not very explicit. The terseness of this function (Flake8’s authors believe) caused the confusion and problems with pep8’s 1.6 series. As such, Flake8 has separated out discovery, management, and merging into a module to make reasoning about each of these pieces easier and more explicit (as well as easier to test).

Configuration file discovery is managed by the ConfigFileFinder object. This object needs to know information about the program’s name, any extra arguments passed to it, and any configuration files that should be appended to the list of discovered files. It provides methods for finding the files and similar methods for parsing those fles. For example, it provides local_config_files() to find known local config files (and append the extra configuration files) and it also provides local_configs() to parse those configuration files.

Note

local_config_files also filters out non-existent files.

Configuration file merging and managemnt is controlled by the MergedConfigParser. This requires the instance of OptionManager that the program is using, the list of appended config files, and the list of extra arguments. This object is currently the sole user of the ConfigFileFinder object. It appropriately initializes the object and uses it in each of

• parse_cli_config()
• parse_local_config()
• parse_user_config()

Finally, merge_user_and_local_config() takes the user and local configuration files that are parsed by parse_local_config() and parse_user_config(). The main usage of the MergedConfigParser is in aggregate_options().

Aggregating Configuration File and Command Line Arguments

aggregate_options() accepts an instance of OptionManager and does the work to parse the command-line arguments passed by the user necessary for creating an instance of MergedConfigParser.

After parsing the configuration file, we determine the default ignore list. We use the defaults from the OptionManager and update those with the parsed configuration files. Finally we parse the user-provided options one last time using the option defaults and configuration file values as defaults. The parser merges on the command-line specified arguments for us so we have our final, definitive, aggregated options.

API Documentation

flake8.options.aggregator.aggregate_options(manager, config_finder, argv)

Aggregate and merge CLI and config file options.

Parameters:

• manager (flake8.options.manager.OptionManager) – The instance of the OptionManager that we’re presently using.
• config_finder (flake8.options.config.ConfigFileFinder) – The config file finder to use.
• argv (list) – The list of remaining command-line argumentsthat were unknown during preliminary option parsing to pass to manager.parse_args.

Returns:

Tuple of the parsed options and extra arguments returned by manager.parse_args.

Return type:

tuple(argparse.Namespace, list)

class flake8.options.manager.Option(short_option_name=<_ARG.NO: 1>, long_option_name=<_ARG.NO: 1>, action=<_ARG.NO: 1>, default=<_ARG.NO: 1>, type=<_ARG.NO: 1>, dest=<_ARG.NO: 1>, nargs=<_ARG.NO: 1>, const=<_ARG.NO: 1>, choices=<_ARG.NO: 1>, help=<_ARG.NO: 1>, metavar=<_ARG.NO: 1>, callback=<_ARG.NO: 1>, callback_args=<_ARG.NO: 1>, callback_kwargs=<_ARG.NO: 1>, required=<_ARG.NO: 1>, parse_from_config=False, comma_separated_list=False, normalize_paths=False)

Our wrapper around an argparse argument parsers to add features.

__init__(short_option_name=<_ARG.NO: 1>, long_option_name=<_ARG.NO: 1>, action=<_ARG.NO: 1>, default=<_ARG.NO: 1>, type=<_ARG.NO: 1>, dest=<_ARG.NO: 1>, nargs=<_ARG.NO: 1>, const=<_ARG.NO: 1>, choices=<_ARG.NO: 1>, help=<_ARG.NO: 1>, metavar=<_ARG.NO: 1>, callback=<_ARG.NO: 1>, callback_args=<_ARG.NO: 1>, callback_kwargs=<_ARG.NO: 1>, required=<_ARG.NO: 1>, parse_from_config=False, comma_separated_list=False, normalize_paths=False)

Initialize an Option instance.

The following are all passed directly through to argparse.

Parameters:

• short_option_name (str) – The short name of the option (e.g., -x). This will be the first argument passed to ArgumentParser.add_argument
• long_option_name (str) – The long name of the option (e.g., --xtra-long-option). This will be the second argument passed to ArgumentParser.add_argument
• default – Default value of the option.
• dest – Attribute name to store parsed option value as.
• nargs – Number of arguments to parse for this option.
• const – Constant value to store on a common destination. Usually used in conjuntion with action="store_const".
• choices (iterable) – Possible values for the option.
• help (str) – Help text displayed in the usage information.
• metavar (str) – Name to use instead of the long option name for help text.
• required (bool) – Whether this option is required or not.

The following options may be passed directly through to argparse but may need some massaging.

Parameters:

• type – A callable to normalize the type (as is the case in argparse). Deprecated: you can also pass through type strings such as 'int' which are handled by optparse.
• action (str) – Any action allowed by argparse. Deprecated: this also understands the action='callback' action from optparse.
• callback (callable) – Callback used if the action is "callback". Deprecated: please use action= instead.
• callback_args (iterable) – Additional positional arguments to the callback callable. Deprecated: please use action= instead (probably with functools.partial).
• callback_kwargs (dictionary) – Keyword arguments to the callback callable. Deprecated: please use action= instead (probably with functools.partial).

The following parameters are for Flake8’s option handling alone.

Parameters:

• parse_from_config (bool) – Whether or not this option should be parsed out of config files.
• comma_separated_list (bool) – Whether the option is a comma separated list when parsing from a config file.
• normalize_paths (bool) – Whether the option is expecting a path or list of paths and should attempt to normalize the paths to absolute paths.

normalize(value, *normalize_args)

Normalize the value based on the option configuration.

to_argparse()

Convert a Flake8 Option to argparse add_argument arguments.

class flake8.options.manager.OptionManager(prog, version, usage='%(prog)s [options] file file ...', parents=None)

Manage Options and OptionParser while adding post-processing.

__init__(prog, version, usage='%(prog)s [options] file file ...', parents=None)

Initialize an instance of an OptionManager.

Parameters:

• prog (str) – Name of the actual program (e.g., flake8).
• version (str) – Version string for the program.
• usage (str) – Basic usage string used by the OptionParser.
• parents (argparse.ArgumentParser) – A list of ArgumentParser objects whose arguments should also be included.

__weakref__

list of weak references to the object (if defined)

add_option(*args, **kwargs)

Create and register a new option.

See parameters for Option for acceptable arguments to this method.

Note

short_option_name and long_option_name may be specified positionally as they are with argparse normally.

extend_default_ignore(error_codes)

Extend the default ignore list with the error codes provided.

Parameters:error_codes (list) – List of strings that are the error/warning codes with which to extend the default ignore list.

extend_default_select(error_codes)

Extend the default select list with the error codes provided.

Parameters:error_codes (list) – List of strings that are the error/warning codes with which to extend the default select list.

generate_epilog()

Create an epilog with the version and name of each of plugin.

generate_versions(format_str='%(name)s: %(version)s', join_on=', ')

Generate a comma-separated list of versions of plugins.

group(name)

Attach options to an argparse group during this context.

parse_args(args=None, values=None)

Proxy to calling the OptionParser’s parse_args method.

parse_known_args(args=None)

Parse only the known arguments from the argument values.

Replicate a little argparse behaviour while we’re still on optparse.

register_plugin(name, version, local=False)

Register a plugin relying on the OptionManager.

Parameters:

• name (str) – The name of the checker itself. This will be the name attribute of the class or function loaded from the entry-point.
• version (str) – The version of the checker that we’re using.
• local (bool) – Whether the plugin is local to the project/repository or not.

remove_from_default_ignore(error_codes)

Remove specified error codes from the default ignore list.

Parameters:error_codes (list) – List of strings that are the error/warning codes to attempt to remove from the extended default ignore list.

update_version_string()

Update the flake8 version string.

class flake8.options.config.ConfigFileFinder(program_name, extra_config_files=None, config_file=None, ignore_config_files=False)

Encapsulate the logic for finding and reading config files.

__init__(program_name, extra_config_files=None, config_file=None, ignore_config_files=False)

Initialize object to find config files.

Parameters:

• program_name (str) – Name of the current program (e.g., flake8).
• extra_config_files (list) – Extra configuration files specified by the user to read.
• config_file (str) – Configuration file override to only read configuraiton from.
• ignore_config_files (bool) – Determine whether to ignore configuration files or not.

__weakref__

list of weak references to the object (if defined)

cli_config(files)

Read and parse the config file specified on the command-line.

generate_possible_local_files()

Find and generate all local config files.

local_config_files()

Find all local config files which actually exist.

Filter results from generate_possible_local_files() based on whether the filename exists or not.

Returns:List of files that exist that are local project config files with extra config files appended to that list (which also exist). Return type:[str]

local_configs()

Parse all local config files into one config object.

local_configs_with_files()

Parse all local config files into one config object.

Return (config, found_config_files) tuple.

user_config()

Parse the user config file into a config object.

class flake8.options.config.MergedConfigParser(option_manager, config_finder)

Encapsulate merging different types of configuration files.

This parses out the options registered that were specified in the configuration files, handles extra configuration files, and returns dictionaries with the parsed values.

GETBOOL_ACTIONS = {'store_false', 'store_true'}

Set of actions that should use the getbool() method.

__init__(option_manager, config_finder)

Initialize the MergedConfigParser instance.

Parameters:

• option_manager (flake8.options.manager.OptionManager) – Initialized OptionManager.
• config_finder (flake8.options.config.ConfigFileFinder) – Initialized ConfigFileFinder.

__weakref__

list of weak references to the object (if defined)

config_finder = None

Our instance of our ConfigFileFinder

config_options = None

Mapping of configuration option names to Option instances

is_configured_by(config)

Check if the specified config parser has an appropriate section.

merge_user_and_local_config()

Merge the parsed user and local configuration files.

Returns:Dictionary of the parsed and merged configuration options. Return type:dict

option_manager = None

Our instance of flake8.options.manager.OptionManager

parse()

Parse and return the local and user config files.

First this copies over the parsed local configuration and then iterates over the options in the user configuration and sets them if they were not set by the local configuration file.

Returns:Dictionary of parsed configuration options Return type:dict

parse_cli_config(config_path)

Parse and return the file specified by –config.

parse_local_config()

Parse and return the local configuration files.

parse_user_config()

Parse and return the user configuration files.

program_name = None

The prog value for the cli parser