Option and Configuration Handling
Option Management
Command-line options are often also set in configuration files for Flake8. While not all options are meant to be parsed from configuration files, many default options are also parsed from configuration files as well as most plugin options.
In Flake8 2, plugins received a optparse.OptionParser
instance and
called optparse.OptionParser.add_option()
to register options. If the
plugin author also wanted to have that option parsed from config files they
also had to do something like:
parser.config_options.append('my_config_option')
parser.config_options.extend(['config_opt1', 'config_opt2'])
This was previously undocumented and led to a lot of confusion about why registered options were not automatically parsed from configuration files.
Since Flake8 3 was rewritten from scratch, we decided to take a different
approach to configuration file parsing. Instead of needing to know about an
undocumented attribute that pep8 looks for, Flake8 3 now accepts a parameter
to add_option
, specifically parse_from_config
which is a boolean
value.
Flake8 does this by creating its own abstractions on top of argparse
.
The first abstraction is the flake8.options.manager.Option
class. The
second is the flake8.options.manager.OptionManager
. In fact, we add
three new parameters:
parse_from_config
comma_separated_list
normalize_paths
The last two are not specifically for configuration file handling, but they do improve that dramatically. We found that there were options that, when specified in a configuration file, often necessitated being split across multiple lines and those options were almost always comma-separated. For example, let’s consider a user’s list of ignored error codes for a project:
[flake8]
ignore =
# Reasoning
E111,
# Reasoning
E711,
# Reasoning
E712,
# Reasoning
E121,
# Reasoning
E122,
# Reasoning
E123,
# Reasoning
E131,
# Reasoning
E251
It makes sense here to allow users to specify the value this way, but, the
standard library’s configparser.RawConfigParser
class does returns a
string that looks like
"\nE111, \nE711, \nE712, \nE121, \nE122, \nE123, \nE131, \nE251 "
This means that a typical call to str.split()
with ','
will not be
sufficient here. Telling Flake8 that something is a comma-separated list
(e.g., comma_separated_list=True
) will handle this for you. Flake8 will
return:
["E111", "E711", "E712", "E121", "E122", "E123", "E131", "E251"]
Next let’s look at how users might like to specify their exclude
list.
Presently OpenStack’s Nova project has this line in their tox.ini:
exclude = .venv,.git,.tox,dist,doc,*openstack/common/*,*lib/python*,*egg,build,tools/xenserver*,releasenotes
We think we can all agree that this would be easier to read like this:
exclude =
.venv,
.git,
.tox,
dist,
doc,
*openstack/common/*,
*lib/python*,
*egg,
build,
tools/xenserver*,
releasenotes
In this case, since these are actually intended to be paths, we would specify
both comma_separated_list=True
and normalize_paths=True
because we
want the paths to be provided to us with some consistency (either all absolute
paths or not).
Now let’s look at how this will actually be used. Most plugin developers
will receive an instance of OptionManager
so
to ease the transition we kept the same API as the
optparse.OptionParser
object. The only difference is that
add_option()
accepts the three
extra arguments we highlighted above.
Configuration File Management
In Flake8 2, configuration file discovery and management was handled by pep8. In pep8’s 1.6 release series, it drastically broke how discovery and merging worked (as a result of trying to improve it). To avoid a dependency breaking Flake8 again in the future, we have created our own discovery and management in 3.0.0. In 4.0.0 we have once again changed how this works and we removed support for user-level config files.
Project files (files stored in the current directory) are read next and merged on top of the user file. In other words, configuration in project files takes precedence over configuration in user files.
New in 3.0.0 The user can specify
--append-config <path-to-file>
repeatedly to include extra configuration files that should be read and take precedence over user and project files.New in 3.0.0 The user can specify
--config <path-to-file>
to so this file is the only configuration file used. This is a change from Flake8 2 where pep8 would simply merge this configuration file into the configuration generated by user and project files (where this takes precedence).New in 3.0.0 The user can specify
--isolated
to disable configuration via discovered configuration files.
To facilitate the configuration file management, we’ve taken a different approach to discovery and management of files than pep8. In pep8 1.5, 1.6, and 1.7 configuration discovery and management was centralized in 66 lines of very terse python which was confusing and not very explicit. The terseness of this function (Flake8 3.0.0’s authors believe) caused the confusion and problems with pep8’s 1.6 series. As such, Flake8 has separated out discovery, management, and merging into a module to make reasoning about each of these pieces easier and more explicit (as well as easier to test).
Configuration file discovery and raw ini reading is managed by
load_config()
. This produces a loaded
RawConfigParser
and a config directory (which will be
used later to normalize paths).
Next, parse_config()
parses options using the
types in the OptionManager
.
Most of this is done in aggregate_options()
.
Aggregating Configuration File and Command Line Arguments
aggregate_options()
accepts an instance of
OptionManager
and does the work to parse the
command-line arguments.
After parsing the configuration file, we determine the default ignore list. We use the defaults from the OptionManager and update those with the parsed configuration files. Finally we parse the user-provided options one last time using the option defaults and configuration file values as defaults. The parser merges on the command-line specified arguments for us so we have our final, definitive, aggregated options.
API Documentation
- flake8.options.aggregator.aggregate_options(manager, cfg, cfg_dir, argv)[source]
Aggregate and merge CLI and config file options.
- Parameters:
manager (OptionManager)
cfg (RawConfigParser)
cfg_dir (str)
- Return type:
- class flake8.options.manager.Option(short_option_name=_ARG.NO, long_option_name=_ARG.NO, action=_ARG.NO, default=_ARG.NO, type=_ARG.NO, dest=_ARG.NO, nargs=_ARG.NO, const=_ARG.NO, choices=_ARG.NO, help=_ARG.NO, metavar=_ARG.NO, required=_ARG.NO, parse_from_config=False, comma_separated_list=False, normalize_paths=False)[source]
Our wrapper around an argparse argument parsers to add features.
- Parameters:
short_option_name (str | _ARG)
long_option_name (str | _ARG)
action (str | type[argparse.Action] | _ARG)
default (Any | _ARG)
type (Callable[..., Any] | _ARG)
dest (str | _ARG)
const (Any | _ARG)
choices (Sequence[Any] | _ARG)
help (str | _ARG)
metavar (str | _ARG)
required (bool | _ARG)
parse_from_config (bool)
comma_separated_list (bool)
normalize_paths (bool)
- __init__(short_option_name=_ARG.NO, long_option_name=_ARG.NO, action=_ARG.NO, default=_ARG.NO, type=_ARG.NO, dest=_ARG.NO, nargs=_ARG.NO, const=_ARG.NO, choices=_ARG.NO, help=_ARG.NO, metavar=_ARG.NO, required=_ARG.NO, parse_from_config=False, comma_separated_list=False, normalize_paths=False)[source]
Initialize an Option instance.
The following are all passed directly through to argparse.
- Parameters:
short_option_name (str | _ARG) – The short name of the option (e.g.,
-x
). This will be the first argument passed toArgumentParser.add_argument
long_option_name (str | _ARG) – The long name of the option (e.g.,
--xtra-long-option
). This will be the second argument passed toArgumentParser.add_argument
default (Any | _ARG) – Default value of the option.
dest (str | _ARG) – Attribute name to store parsed option value as.
nargs (int | str | _ARG) – Number of arguments to parse for this option.
const (Any | _ARG) – Constant value to store on a common destination. Usually used in conjunction with
action="store_const"
.choices (Sequence[Any] | _ARG) – Possible values for the option.
help (str | _ARG) – Help text displayed in the usage information.
metavar (str | _ARG) – Name to use instead of the long option name for help text.
required (bool | _ARG) – Whether this option is required or not.
parse_from_config (bool)
comma_separated_list (bool)
normalize_paths (bool)
- Return type:
None
The following options may be passed directly through to
argparse
but may need some massaging.- Parameters:
type (Callable[[...], Any] | _ARG) – A callable to normalize the type (as is the case in
argparse
).action (str | type[Action] | _ARG) – Any action allowed by
argparse
.short_option_name (str | _ARG)
long_option_name (str | _ARG)
default (Any | _ARG)
dest (str | _ARG)
const (Any | _ARG)
help (str | _ARG)
metavar (str | _ARG)
required (bool | _ARG)
parse_from_config (bool)
comma_separated_list (bool)
normalize_paths (bool)
- Return type:
None
The following parameters are for Flake8’s option handling alone.
- Parameters:
parse_from_config (bool) – Whether or not this option should be parsed out of config files.
comma_separated_list (bool) – Whether the option is a comma separated list when parsing from a config file.
normalize_paths (bool) – Whether the option is expecting a path or list of paths and should attempt to normalize the paths to absolute paths.
short_option_name (str | _ARG)
long_option_name (str | _ARG)
default (Any | _ARG)
dest (str | _ARG)
const (Any | _ARG)
help (str | _ARG)
metavar (str | _ARG)
required (bool | _ARG)
- Return type:
None
- class flake8.options.manager.OptionManager(*, version, plugin_versions, parents, formatter_names)[source]
Manage Options and OptionParser while adding post-processing.
- Parameters:
- __init__(*, version, plugin_versions, parents, formatter_names)[source]
Initialize an instance of an OptionManager.
- __weakref__
list of weak references to the object
- add_option(*args, **kwargs)[source]
Create and register a new option.
See parameters for
Option
for acceptable arguments to this method.Note
short_option_name
andlong_option_name
may be specified positionally as they are with argparse normally.
- extend_default_ignore(error_codes)[source]
Extend the default ignore list with the error codes provided.
- extend_default_select(error_codes)[source]
Extend the default select list with the error codes provided.
- flake8.options.config.load_config(config, extra, *, isolated=False)[source]
Load the configuration given the user options.
in
isolated
mode, return an empty configurationif a config file is given in
config
use that, otherwise attempt to discover a configuration usingtox.ini
/setup.cfg
/.flake8
finally, load any
extra
configuration files
- flake8.options.config.parse_config(option_manager, cfg, cfg_dir)[source]
Parse and normalize the typed configuration options.
- Parameters:
option_manager (OptionManager)
cfg (RawConfigParser)
cfg_dir (str)
- Return type: