How Checks are Run

In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 Flake8 takes on that responsibility. This has allowed for simpler handling of the --jobs parameter (using multiprocessing) and simplified our fallback if something goes awry with concurrency. At the lowest level we have a FileChecker. Instances of FileChecker are created for each file to be analyzed by Flake8. Each instance, has a copy of all of the plugins registered with setuptools in the flake8.extension entry-point group.

The FileChecker instances are managed by an instance of Manager. The Manager instance handles creating sub-processes with multiprocessing module and falling back to running checks in serial if an operating system level error arises. When creating FileChecker instances, the Manager is responsible for determining if a particular file has been excluded.

Processing Files

Unfortunately, since Flake8 took over check running from pep8/pycodestyle, it also had to take over parsing and processing files for the checkers to use. Since it couldn’t reuse pycodestyle’s functionality (since it did not separate cleanly the processing from check running) that function was isolated into the FileProcessor class. We moved several helper functions into the flake8.processor module (see also Processor Utility Functions).

API Reference

class flake8.checker.FileChecker(*, filename, plugins, options)

Manage running checks for a file and aggregate the results.

Parameters

• filename (str) –

• plugins (Checkers) –

• options (Namespace) –

check_physical_eol(token, prev_physical)

Run physical checks if and only if it is at the end of the line.

Parameters

• token (TokenInfo) –

• prev_physical (str) –

Return type

None

handle_newline(token_type)

Handle the logic when encountering a newline token.

Parameters

token_type (int) –

Return type

None

process_tokens()

Process tokens and trigger checks.

Instead of using this directly, you should use flake8.checker.FileChecker.run_checks().

Return type

None

report(error_code, line_number, column, text)

Report an error by storing it in the results list.

Parameters

• error_code (Optional[str]) –

• line_number (int) –

• column (int) –

• text (str) –

Return type

str

run_ast_checks()

Run all checks expecting an abstract syntax tree.

Return type

None

run_check(plugin, **arguments)

Run the check in a single plugin.

Parameters

• plugin (LoadedPlugin) –

• arguments (Any) –

Return type

Any

run_checks()

Run checks against the file.

Return type

Tuple[str, List[Tuple[str, int, int, str, Optional[str]]], Dict[str, int]]

run_logical_checks()

Run all checks expecting a logical line.

Return type

None

run_physical_checks(physical_line)

Run all checks for a given physical line.

A single physical check may return multiple errors.

Parameters

physical_line (str) –

Return type

None

class flake8.checker.Manager(style_guide, plugins)

Manage the parallelism and checker instances for each plugin and file.

This class will be responsible for the following:

• Determining the parallelism of Flake8, e.g.:

  • Do we use multiprocessing or is it unavailable?

  • Do we automatically decide on the number of jobs to use or did the user provide that?

• Falling back to a serial way of processing files if we run into an OSError related to multiprocessing

• Organizing the results of each checker so we can group the output together and make our output deterministic.

Parameters

• style_guide (StyleGuideManager) –

• plugins (Checkers) –

make_checkers(paths=None)

Create checkers for each file.

Parameters

paths (Optional[List[str]]) –

Return type

None

report()

Report all of the errors found in the managed file checkers.

This iterates over each of the checkers and reports the errors sorted by line number.

Returns

A tuple of the total results found and the results reported.

Return type

Tuple[int, int]

run()

Run all the checkers.

This will intelligently decide whether to run the checks in parallel or whether to run them in serial.

If running the checks in parallel causes a problem (e.g., #117) this also implements fallback to serial processing.

Return type

None

run_parallel()

Run the checkers in parallel.

Return type

None

run_serial()

Run the checkers in serial.

Return type

None

start(paths=None)

Start checking files.

Parameters

paths (Optional[List[str]]) – Path names to check. This is passed directly to make_checkers().

Return type

None

stop()

Stop checking files.

Return type

None

class flake8.processor.FileProcessor(filename, options, lines=None)

Processes a file and holdes state.

This processes a file by generating tokens, logical and physical lines, and AST trees. This also provides a way of passing state about the file to checks expecting that state. Any public attribute on this object can be requested by a plugin. The known public attributes are:

• blank_before

• blank_lines

• checker_state

• indent_char

• indent_level

• line_number

• logical_line

• max_line_length

• max_doc_length

• multiline

• noqa

• previous_indent_level

• previous_logical

• previous_unindented_logical_line

• tokens

• file_tokens

• total_lines

• verbose

Parameters

• filename (str) –

• options (Namespace) –

• lines (Optional[List[str]]) –

blank_before

Number of preceding blank lines

blank_lines

Number of blank lines

build_ast()

Build an abstract syntax tree from the list of lines.

Return type

AST

build_logical_line()

Build a logical line from the current tokens list.

Return type

Tuple[str, str, List[Tuple[int, Tuple[int, int]]]]

build_logical_line_tokens()

Build the mapping, comments, and logical line lists.

Return type

Tuple[List[str], List[str], List[Tuple[int, Tuple[int, int]]]]

checker_state: Dict[Any, Any]

Current checker state

delete_first_token()

Delete the first token in the list of tokens.

Return type

None

property file_tokens: List[TokenInfo]

Return the complete set of tokens for a file.

generate_tokens()

Tokenize the file and yield the tokens.

Return type

Generator[TokenInfo, None, None]

hang_closing

User provided option for hang closing

indent_char: Optional[str]

Character used for indentation

indent_level

Current level of indentation

indent_size

Number of spaces used for indentation

inside_multiline(line_number)

Context-manager to toggle the multiline attribute.

Parameters

line_number (int) –

Return type

Generator[None, None, None]

keyword_arguments_for(parameters, arguments)

Generate the keyword arguments for a list of parameters.

Parameters

• parameters (Dict[str, bool]) –

• arguments (Dict[str, Any]) –

Return type

Dict[str, Any]

line_number

Line number in the file

logical_line

Current logical line

max_doc_length

Maximum docstring / comment line length as configured by the user

max_line_length

Maximum line length as configured by the user

multiline

Whether the current physical line is multiline

next_line()

Get the next line from the list.

Return type

str

next_logical_line()

Record the previous logical line.

This also resets the tokens list and the blank_lines count.

Return type

None

noqa = False

always False, included for compatibility

noqa_line_for(line_number)

Retrieve the line which will be used to determine noqa.

Parameters

line_number (int) –

Return type

Optional[str]

previous_indent_level

Previous level of indentation

previous_logical

Previous logical line

previous_unindented_logical_line

Previous unindented (i.e. top-level) logical line

read_lines()

Read the lines for this file checker.

Return type

List[str]

read_lines_from_filename()

Read the lines for a file.

Return type

List[str]

read_lines_from_stdin()

Read the lines from standard in.

Return type

List[str]

reset_blank_before()

Reset the blank_before attribute to zero.

Return type

None

should_ignore_file()

Check if flake8: noqa is in the file to be ignored.

Returns

True if a line matches defaults.NOQA_FILE, otherwise False

Return type

bool

split_line(token)

Split a physical line’s line based on new-lines.

This also auto-increments the line number for the caller.

Parameters

token (TokenInfo) –

Return type

Generator[str, None, None]

statistics

Statistics dictionary

strip_utf_bom()

Strip the UTF bom from the lines of the file.

Return type

None

tokens: List[TokenInfo]

Current set of tokens

total_lines

Total number of lines in the file

update_checker_state_for(plugin)

Update the checker_state attribute for the plugin.

Parameters

plugin (LoadedPlugin) –

Return type

None

update_state(mapping)

Update the indent level based on the logical line mapping.

Parameters

mapping (List[Tuple[int, Tuple[int, int]]]) –

Return type

None

verbose

Verbosity level of Flake8

visited_new_blank_line()

Note that we visited a new blank line.

Return type

None

Utility Functions

flake8.processor.count_parentheses(current_parentheses_count, token_text)

Count the number of parentheses.

Parameters

• current_parentheses_count (int) –

• token_text (str) –

Return type

int

flake8.processor.expand_indent(line)

Return the amount of indentation.

Tabs are expanded to the next multiple of 8.

>>> expand_indent('    ')
4
>>> expand_indent('\t')
8
>>> expand_indent('       \t')
8
>>> expand_indent('        \t')
16

Parameters

line (str) –

Return type

int

flake8.processor.is_eol_token(token)

Check if the token is an end-of-line token.

Parameters

token (TokenInfo) –

Return type

bool

flake8.processor.is_multiline_string(token)

Check if this is a multiline string.

Parameters

token (TokenInfo) –

Return type

bool

flake8.processor.mutate_string(text)

Replace contents with ‘xxx’ to prevent syntax matching.

>>> mutate_string('"abc"')
'"xxx"'
>>> mutate_string("'''abc'''")
"'''xxx'''"
>>> mutate_string("r'abc'")
"r'xxx'"

Parameters

text (str) –

Return type

str

flake8.processor.token_is_newline(token)

Check if the token type is a newline token type.

Parameters

token (TokenInfo) –

Return type

bool