How Checks are Run¶
In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 Flake8
takes on that responsibility. This has allowed for simpler
handling of the --jobs
parameter (using multiprocessing
) and
simplified our fallback if something goes awry with concurency.
At the lowest level we have a FileChecker
. Instances of FileChecker
are
created for each file to be analyzed by Flake8. Each instance, has a copy
of all of the plugins registered with setuptools in the flake8.extension
entry-point group.
The FileChecker
instances are managed by an instance of Manager
. The
Manager
instance handles creating sub-processes with
multiprocessing
module and falling back to running checks in serial if
an operating system level error arises. When creating FileChecker
instances,
the Manager
is responsible for determining if a particular file has been
excluded.
Processing Files¶
Unfortunately, since Flake8 took over check running from pep8/pycodestyle,
it also had to take over parsing and processing files for the checkers
to use. Since it couldn’t reuse pycodestyle’s functionality (since it did not
separate cleanly the processing from check running) that function was isolated
into the FileProcessor
class. We moved
several helper functions into the flake8.processor
module (see also
Processor Utility Functions).
API Reference¶
-
class
flake8.checker.
FileChecker
(filename, checks, options)[source]¶ Manage running checks for a file and aggregate the results.
-
process_tokens
()[source]¶ Process tokens and trigger checks.
This can raise a
flake8.exceptions.InvalidSyntax
exception. Instead of using this directly, you should useflake8.checker.FileChecker.run_checks()
.
-
-
class
flake8.checker.
Manager
(style_guide, arguments, checker_plugins)[source]¶ Manage the parallelism and checker instances for each plugin and file.
This class will be responsible for the following:
- Determining the parallelism of Flake8, e.g.:
- Do we use
multiprocessing
or is it unavailable? - Do we automatically decide on the number of jobs to use or did the user provide that?
- Do we use
- Falling back to a serial way of processing files if we run into an
OSError related to
multiprocessing
- Organizing the results of each checker so we can group the output together and make our output deterministic.
-
is_path_excluded
(path)[source]¶ Check if a path is excluded.
Parameters: path (str) – Path to check against the exclude patterns. Returns: True if there are exclude patterns and the path matches, otherwise False. Return type: bool
-
report
()[source]¶ Report all of the errors found in the managed file checkers.
This iterates over each of the checkers and reports the errors sorted by line number.
Returns: A tuple of the total results found and the results reported. Return type: tuple(int, int)
-
run
()[source]¶ Run all the checkers.
This will intelligently decide whether to run the checks in parallel or whether to run them in serial.
If running the checks in parallel causes a problem (e.g., https://gitlab.com/pycqa/flake8/issues/74) this also implements fallback to serial processing.
-
start
(paths=None)[source]¶ Start checking files.
Parameters: paths (list) – Path names to check. This is passed directly to make_checkers()
.
- Determining the parallelism of Flake8, e.g.:
-
class
flake8.processor.
FileProcessor
(filename, options, lines=None)[source]¶ Processes a file and holdes state.
This processes a file by generating tokens, logical and physical lines, and AST trees. This also provides a way of passing state about the file to checks expecting that state. Any public attribute on this object can be requested by a plugin. The known public attributes are:
blank_before
blank_lines
checker_state
indent_char
indent_level
line_number
logical_line
max_line_length
max_doc_length
multiline
noqa
previous_indent_level
previous_logical
previous_unindented_logical_line
tokens
file_tokens
total_lines
verbose
-
blank_before
= None¶ Number of preceding blank lines
-
blank_lines
= None¶ Number of blank lines
-
checker_state
= None¶ Current checker state
-
file_tokens
¶ Return the complete set of tokens for a file.
Accessing this attribute may raise an InvalidSyntax exception.
Raises: flake8.exceptions.InvalidSyntax
-
generate_tokens
()[source]¶ Tokenize the file and yield the tokens.
Raises: flake8.exceptions.InvalidSyntax – If a tokenize.TokenError
is raised while generating tokens.
-
hang_closing
= None¶ User provided option for hang closing
-
indent_char
= None¶ Character used for indentation
-
indent_level
= None¶ Current level of indentation
-
keyword_arguments_for
(parameters, arguments=None)[source]¶ Generate the keyword arguments for a list of parameters.
-
line_number
= None¶ Line number in the file
-
logical_line
= None¶ Current logical line
-
max_doc_length
= None¶ Maximum docstring / comment line length as configured by the user
-
max_line_length
= None¶ Maximum line length as configured by the user
-
multiline
= None¶ Whether the current physical line is multiline
-
next_logical_line
()[source]¶ Record the previous logical line.
This also resets the tokens list and the blank_lines count.
-
noqa
= False¶ always
False
, included for compatibility
-
previous_indent_level
= None¶ Previous level of indentation
-
previous_logical
= None¶ Previous logical line
-
previous_unindented_logical_line
= None¶ Previous unindented (i.e. top-level) logical line
-
should_ignore_file
()[source]¶ Check if
flake8: noqa
is in the file to be ignored.Returns: True if a line matches defaults.NOQA_FILE
, otherwise FalseReturn type: bool
-
split_line
(token)[source]¶ Split a physical line’s line based on new-lines.
This also auto-increments the line number for the caller.
-
statistics
= None¶ Statistics dictionary
-
tokens
= None¶ Current set of tokens
-
total_lines
= None¶ Total number of lines in the file
-
verbose
= None¶ Verbosity level of Flake8
Utility Functions¶
-
flake8.processor.
count_parentheses
(current_parentheses_count, token_text)[source]¶ Count the number of parentheses.
-
flake8.processor.
expand_indent
(line)[source]¶ Return the amount of indentation.
Tabs are expanded to the next multiple of 8.
>>> expand_indent(' ') 4 >>> expand_indent('\t') 8 >>> expand_indent(' \t') 8 >>> expand_indent(' \t') 16