How Checks are Run

In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 Flake8 takes on that responsibility. This has allowed for simpler handling of the --jobs parameter (using multiprocessing) and simplified our fallback if something goes awry with concurency. At the lowest level we have a FileChecker. Instances of FileChecker are created for each file to be analyzed by Flake8. Each instance, has a copy of all of the plugins registered with setuptools in the flake8.extension entry-point group.

The FileChecker instances are managed by an instance of Manager. The Manager instance handles creating sub-processes with multiprocessing module and falling back to running checks in serial if an operating system level error arises. When creating FileChecker instances, the Manager is responsible for determining if a particular file has been excluded.

Processing Files

Unfortunately, since Flake8 took over check running from pep8/pycodestyle, it also had to take over parsing and processing files for the checkers to use. Since it couldn’t reuse pycodestyle’s functionality (since it did not separate cleanly the processing from check running) that function was isolated into the FileProcessor class. We moved several helper functions into the flake8.processor module (see also Processor Utility Functions).

API Reference

class flake8.checker.FileChecker(filename, checks, options)

Manage running checks for a file and aggregate the results.

check_physical_eol(token)

Run physical checks if and only if it is at the end of the line.

handle_newline(token_type)

Handle the logic when encountering a newline token.

process_tokens()

Process tokens and trigger checks.

This can raise a flake8.exceptions.InvalidSyntax exception. Instead of using this directly, you should use flake8.checker.FileChecker.run_checks().

report(error_code, line_number, column, text, line=None)

Report an error by storing it in the results list.

run_ast_checks()

Run all checks expecting an abstract syntax tree.

run_check(plugin, **arguments)

Run the check in a single plugin.

run_checks()

Run checks against the file.

run_logical_checks()

Run all checks expecting a logical line.

run_physical_checks(physical_line, override_error_line=None)

Run all checks for a given physical line.

A single physical check may return multiple errors.

class flake8.checker.Manager(style_guide, arguments, checker_plugins)

Manage the parallelism and checker instances for each plugin and file.

This class will be responsible for the following:

• Determining the parallelism of Flake8, e.g.:
  • Do we use multiprocessing or is it unavailable?
  • Do we automatically decide on the number of jobs to use or did the user provide that?
• Falling back to a serial way of processing files if we run into an OSError related to multiprocessing
• Organizing the results of each checker so we can group the output together and make our output deterministic.

is_path_excluded(path)

Check if a path is excluded.

Parameters:path (str) – Path to check against the exclude patterns. Returns:True if there are exclude patterns and the path matches, otherwise False. Return type:bool

make_checkers(paths=None)

Create checkers for each file.

report()

Report all of the errors found in the managed file checkers.

This iterates over each of the checkers and reports the errors sorted by line number.

Returns:A tuple of the total results found and the results reported. Return type:tuple(int, int)

run()

Run all the checkers.

This will intelligently decide whether to run the checks in parallel or whether to run them in serial.

If running the checks in parallel causes a problem (e.g., https://gitlab.com/pycqa/flake8/issues/74) this also implements fallback to serial processing.

run_parallel()

Run the checkers in parallel.

run_serial()

Run the checkers in serial.

start(paths=None)

Start checking files.

Parameters:paths (list) – Path names to check. This is passed directly to make_checkers().

stop()

Stop checking files.

class flake8.processor.FileProcessor(filename, options, lines=None)

Processes a file and holdes state.

This processes a file by generating tokens, logical and physical lines, and AST trees. This also provides a way of passing state about the file to checks expecting that state. Any public attribute on this object can be requested by a plugin. The known public attributes are:

• blank_before
• blank_lines
• checker_state
• indent_char
• indent_level
• line_number
• logical_line
• max_line_length
• max_doc_length
• multiline
• noqa
• previous_indent_level
• previous_logical
• previous_unindented_logical_line
• tokens
• file_tokens
• total_lines
• verbose

blank_before = None

Number of preceding blank lines

blank_lines = None

Number of blank lines

build_ast()

Build an abstract syntax tree from the list of lines.

build_logical_line()

Build a logical line from the current tokens list.

build_logical_line_tokens()

Build the mapping, comments, and logical line lists.

check_physical_error(error_code, line)

Update attributes based on error code and line.

checker_state = None

Current checker state

delete_first_token()

Delete the first token in the list of tokens.

file_tokens

Return the complete set of tokens for a file.

Accessing this attribute may raise an InvalidSyntax exception.

Raises:flake8.exceptions.InvalidSyntax

generate_tokens()

Tokenize the file and yield the tokens.

Raises:flake8.exceptions.InvalidSyntax – If a tokenize.TokenError is raised while generating tokens.

hang_closing = None

User provided option for hang closing

indent_char = None

Character used for indentation

indent_level = None

Current level of indentation

inside_multiline(line_number)

Context-manager to toggle the multiline attribute.

keyword_arguments_for(parameters, arguments=None)

Generate the keyword arguments for a list of parameters.

line_for(line_number)

Retrieve the physical line at the specified line number.

line_number = None

Line number in the file

logical_line = None

Current logical line

max_doc_length = None

Maximum docstring / comment line length as configured by the user

max_line_length = None

Maximum line length as configured by the user

multiline = None

Whether the current physical line is multiline

next_line()

Get the next line from the list.

next_logical_line()

Record the previous logical line.

This also resets the tokens list and the blank_lines count.

noqa = None

Whether or not we’re observing NoQA

previous_indent_level = None

Previous level of indentation

previous_logical = None

Previous logical line

previous_unindented_logical_line = None

Previous unindented (i.e. top-level) logical line

read_lines()

Read the lines for this file checker.

read_lines_from_filename()

Read the lines for a file.

read_lines_from_stdin()

Read the lines from standard in.

reset_blank_before()

Reset the blank_before attribute to zero.

should_ignore_file()

Check if flake8: noqa is in the file to be ignored.

Returns:True if a line matches defaults.NOQA_FILE, otherwise False Return type:bool

split_line(token)

Split a physical line’s line based on new-lines.

This also auto-increments the line number for the caller.

statistics = None

Statistics dictionary

strip_utf_bom()

Strip the UTF bom from the lines of the file.

tokens = None

Current set of tokens

total_lines = None

Total number of lines in the file

update_checker_state_for(plugin)

Update the checker_state attribute for the plugin.

update_state(mapping)

Update the indent level based on the logical line mapping.

verbose = None

Verbosity level of Flake8

visited_new_blank_line()

Note that we visited a new blank line.

Utility Functions

flake8.processor.count_parentheses(current_parentheses_count, token_text)

Count the number of parentheses.

flake8.processor.expand_indent(line)

Return the amount of indentation.

Tabs are expanded to the next multiple of 8.

>>> expand_indent('    ')
4
>>> expand_indent('\t')
8
>>> expand_indent('       \t')
8
>>> expand_indent('        \t')
16

flake8.processor.is_eol_token(token)

Check if the token is an end-of-line token.

flake8.processor.is_multiline_string(token)

Check if this is a multiline string.

flake8.processor.log_token(log, token)

Log a token to a provided logging object.

flake8.processor.mutate_string(text)

Replace contents with ‘xxx’ to prevent syntax matching.

>>> mute_string('"abc"')
'"xxx"'
>>> mute_string("'''abc'''")
"'''xxx'''"
>>> mute_string("r'abc'")
"r'xxx'"

flake8.processor.token_is_newline(token)

Check if the token type is a newline token type.