How Checks are Run

In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 Flake8 takes on that responsibility. This has allowed for simpler handling of the --jobs parameter (using multiprocessing) and simplified our fallback if something goes awry with concurency. At the lowest level we have a FileChecker. Instances of FileChecker are created for each file to be analyzed by Flake8. Each instance, has a copy of all of the plugins registered with setuptools in the flake8.extension entry-point group.

The FileChecker instances are managed by an instance of Manager. The Manager instance handles creating sub-processes with multiprocessing module and falling back to running checks in serial if an operating system level error arises. When creating FileChecker instances, the Manager is responsible for determining if a particular file has been excluded.

Processing Files

Unfortunately, since Flake8 took over check running from pep8/pycodestyle, it also had to take over parsing and processing files for the checkers to use. Since it couldn’t reuse pycodestyle’s functionality (since it did not separate cleanly the processing from check running) that function was isolated into the FileProcessor class. We moved several helper functions into the flake8.processor module (see also Processor Utility Functions).

API Reference

class flake8.checker.FileChecker(filename, checks, options)

Manage running checks for a file and aggregate the results.

check_physical_eol(token)

Run physical checks if and only if it is at the end of the line.

handle_comment(token, token_text)

Handle the logic when encountering a comment token.

handle_newline(token_type)

Handle the logic when encountering a newline token.

process_tokens()

Process tokens and trigger checks.

This can raise a flake8.exceptions.InvalidSyntax exception. Instead of using this directly, you should use flake8.checker.FileChecker.run_checks().

report(error_code, line_number, column, text, line=None)

Report an error by storing it in the results list.

run_ast_checks()

Run all checks expecting an abstract syntax tree.

run_check(plugin, **arguments)

Run the check in a single plugin.

run_checks(results_queue, statistics_queue)

Run checks against the file.

run_logical_checks()

Run all checks expecting a logical line.

run_physical_checks(physical_line, override_error_line=None)

Run all checks for a given physical line.

class flake8.checker.Manager(style_guide, arguments, checker_plugins)

Manage the parallelism and checker instances for each plugin and file.

This class will be responsible for the following:

• Determining the parallelism of Flake8, e.g.:
  • Do we use multiprocessing or is it unavailable?
  • Do we automatically decide on the number of jobs to use or did the user provide that?
• Falling back to a serial way of processing files if we run into an OSError related to multiprocessing
• Organizing the results of each checker so we can group the output together and make our output deterministic.

is_path_excluded(path)

Check if a path is excluded.

Parameters:path (str) – Path to check against the exclude patterns. Returns:True if there are exclude patterns and the path matches, otherwise False. Return type:bool

make_checkers(paths=None)

Create checkers for each file.

report()

Report all of the errors found in the managed file checkers.

This iterates over each of the checkers and reports the errors sorted by line number.

Returns:A tuple of the total results found and the results reported. Return type:tuple(int, int)

run()

Run all the checkers.

This will intelligently decide whether to run the checks in parallel or whether to run them in serial.

If running the checks in parallel causes a problem (e.g., https://gitlab.com/pycqa/flake8/issues/74) this also implements fallback to serial processing.

run_parallel()

Run the checkers in parallel.

run_serial()

Run the checkers in serial.

start(paths=None)

Start checking files.

Parameters:paths (list) – Path names to check. This is passed directly to make_checkers().

stop()

Stop checking files.

class flake8.processor.FileProcessor(filename, options, lines=None)

Processes a file and holdes state.

This processes a file by generating tokens, logical and physical lines, and AST trees. This also provides a way of passing state about the file to checks expecting that state. Any public attribute on this object can be requested by a plugin. The known public attributes are:

• blank_before
• blank_lines
• checker_state
• indent_char
• indent_level
• line_number
• logical_line
• max_line_length
• multiline
• noqa
• previous_indent_level
• previous_logical
• tokens
• total_lines
• verbose

build_ast()

Build an abstract syntax tree from the list of lines.

build_logical_line()

Build a logical line from the current tokens list.

build_logical_line_tokens()

Build the mapping, comments, and logical line lists.

check_physical_error(error_code, line)

Update attributes based on error code and line.

delete_first_token()

Delete the first token in the list of tokens.

generate_tokens()

Tokenize the file and yield the tokens.

Raises:flake8.exceptions.InvalidSyntax – If a tokenize.TokenError is raised while generating tokens.

inside_multiline(line_number)

Context-manager to toggle the multiline attribute.

keyword_arguments_for(parameters, arguments=None)

Generate the keyword arguments for a list of parameters.

line_for(line_number)

Retrieve the physical line at the specified line number.

next_line()

Get the next line from the list.

next_logical_line()

Record the previous logical line.

This also resets the tokens list and the blank_lines count.

read_lines()

Read the lines for this file checker.

read_lines_from_filename()

Read the lines for a file.

read_lines_from_stdin()

Read the lines from standard in.

reset_blank_before()

Reset the blank_before attribute to zero.

should_ignore_file()

Check if # flake8: noqa is in the file to be ignored.

Returns:True if a line matches defaults.NOQA_FILE, otherwise False Return type:bool

split_line(token)

Split a physical line’s line based on new-lines.

This also auto-increments the line number for the caller.

strip_utf_bom()

Strip the UTF bom from the lines of the file.

update_checker_state_for(plugin)

Update the checker_state attribute for the plugin.

update_state(mapping)

Update the indent level based on the logical line mapping.

visited_new_blank_line()

Note that we visited a new blank line.

Utility Functions

flake8.processor.count_parentheses(current_parentheses_count, token_text)

Count the number of parentheses.

flake8.processor.expand_indent(line)

Return the amount of indentation.

Tabs are expanded to the next multiple of 8.

>>> expand_indent('    ')
4
>>> expand_indent('\t')
8
>>> expand_indent('       \t')
8
>>> expand_indent('        \t')
16

flake8.processor.is_eol_token(token)

Check if the token is an end-of-line token.

flake8.processor.is_multiline_string(token)

Check if this is a multiline string.

flake8.processor.log_token(log, token)

Log a token to a provided logging object.

flake8.processor.mutate_string(text)

Replace contents with ‘xxx’ to prevent syntax matching.

>>> mute_string('"abc"')
'"xxx"'
>>> mute_string("'''abc'''")
"'''xxx'''"
>>> mute_string("r'abc'")
"r'xxx'"

flake8.processor.token_is_comment(token)

Check if the token type is a comment.

flake8.processor.token_is_newline(token)

Check if the token type is a newline token type.