gradema.test

Submodules

Package Contents

Classes

TestReporter

A TestReporter is passed to all tests. This exposes a public API for tests to use and report what they are doing.

CommandData

StdioCommandData

Represents what stdout and stderr should be set to on a command.

Test

This is the abstract class inherited by all tests.

FractionalTestResult

PercentTestResult

RustProgram

Functions

create_python_test(→ gradema.test.Test)

Creates a python test that runs the given module and is successful depending on the successful exit code of running the given module.

create_python_pytest(→ gradema.test.Test)

Creates a pytest test.

create_python_format_check(→ gradema.test.Test)

create_python_format_check_from_path(→ gradema.test.Test)

create_python_type_check(→ gradema.test.Test)

create_mypy_command(→ gradema.test.Test)

create_python_stdio_test(→ gradema.test.Test)

create_python_traditional_stdio_test(→ gradema.test.Test)

Creates a traditional stdio test

create_python_traditional_arg_test(→ gradema.test.Test)

dummy_test(→ gradema.test.Test)

create_file_exists_test(→ gradema.test.Test)

Attributes

TestResult

class gradema.test.TestReporter

Bases: abc.ABC

A TestReporter is passed to all tests. This exposes a public API for tests to use and report what they are doing. A TestReporter’s implementation may log what the test is doing to the console, or even perform an action depending on how the TestReporter is configured.

abstract log(message: str) → None

Log an informative message to the user.

Parameters:

message – The message to log. No formatting is support

abstract log_unexpected_exception(exception: BaseException) → None

Log an unexpected exception

Parameters:

exception – The exception to log

abstract resolve_command(command: Sequence[gradema.test.argument.Argument], test_identifier: str) → tuple[Sequence[str], Mapping[tuple[int, gradema.test.argument.Argument], gradema.test.argument.ResolvedArgumentInfo]]

Resolves a list of Arguments by returning a list of strings.

The returned value contains the resolved command and an argument info dictionary. The key of the argument info dictionary is a tuple of (index, Argument) where index is the index of the Argument in the command parameter.

If you are wondering why command’s type is Sequence[Argument] rather than a list[Argument], it is because the Sequence type is covariant. That means that you can assign a Sequence[str] to a Sequence[Argument], which is necessary to make sure mypy type checking is happy, and correctly documents how the command argument is used: it is not modified. More information here: https://mypy.readthedocs.io/en/stable/common_issues.html#variance

Parameters:

  • test_identifier – The identifier of the test being run

  • command – The command to resolve

Returns:

A tuple of (resolved command, argument info dictionary)

Raises:

ValueError – A ValueError may be risen if the parameters supplied to this function do not give enough information to resolve a particular Argument. This exception is currently not risen in any known implementation, but it’s possible that optional parameters will be added in the future that aid in resolving an Argument that is not created as of writing this docstring

abstract report_command(command: Sequence[str]) → CommandData

Reports that the given command WILL be run

Parameters:

command – The command that will be run

Returns:

A CommandData object containing parameters that determine where the stdout and stdin of the command should go.

abstract report_stdio_command(command: Sequence[str], test_identifier: str, input_file: pathlib.Path | None, stdout_as_output: bool) → StdioCommandData

Reports that a stdio test command WILL be run.

This is similar to report_command, but it is expected that the caller will run the command and direct its output to that specified by the returned object.

Parameters:

  • command – The base command that is being run (not including any I/O redirection)

  • test_identifier – An identifier for the test. This is not recommended to contain spaces or upper case characters

  • input_file – The input file that will be directed into the command’s stdin, or None if no specific input file is being used.

  • stdout_as_output – True to use the command’s stdout as an output, False to not

Returns:

A StdioCommandData that contains the file to direct stdout to and the TextIO to direct stderr to.

abstract maybe_launch_debugger(command: Sequence[str]) → None

Calling this function will launch the debugger by running the given command only if we are in debugging mode. The implementation will likely prompt the user to make sure they want to launch the debugger

Parameters:

command – The command to launch the debugger

abstract report_diff_result(test_identifier: str, goal_file: pathlib.Path, output_file: pathlib.Path, similarity: float, fuzzy: bool) → None

Reports the result of a diff and possibly opens a diff viewer depending on the configuration of this reporter.

Parameters:

  • test_identifier – The identifier for the test

  • goal_file – The goal file

  • output_file – The output file that is supposed to contain the same contents of the goal file

  • similarity – A number from 0 to 1 representing the similarity (1 is exactly the same). Non-fuzzy diffs should only pass 0 or 1.

  • fuzzy – True if fuzzy, false otherwise

abstract report_file_exists_test(file: pathlib.Path, expected: str) → None

Reports that a file existence test WILL BE performed.

Parameters:

  • file

  • expected

Returns:

abstract report_file_exists_test_result(file: pathlib.Path, expected: str, is_correct: bool, actual: str | None) → None

Reports that a file existence test has been performed and that the result is this.

Parameters:

  • file – The file being tested for its existence

  • expected – The expected type of the file (basically the output of file <file>

  • is_correct – True if the actual matches the expected

  • actual – The actual type of the file, or None if the file does not exist

Returns:

class gradema.test.CommandData
stdout: TextIO

The stdout to pass to subprocess.run

stderr: TextIO

The stderr to pass to subprocess.run

class gradema.test.StdioCommandData

Represents what stdout and stderr should be set to on a command. stdout is represented by a file path because it is expected that the caller wants to be able to know how to read from that file after the command is executed.

stdout_file: pathlib.Path | None

The file that stdout should be directed to

stdout: TextIO

The stdout to pass to subprocess.run assuming that stdout_file is None

stderr: TextIO

The stderr to pass to subprocess.run

class gradema.test.Test

Bases: abc.ABC

This is the abstract class inherited by all tests.

abstract run(reporter: gradema.test.TestReporter) → TestResult
class gradema.test.FractionalTestResult
success_count: int
total_count: int
class gradema.test.PercentTestResult
percent: float
gradema.test.TestResult
gradema.test.create_python_test(module_name: str) → gradema.test.Test

Creates a python test that runs the given module and is successful depending on the successful exit code of running the given module.

Please note that this kind of test is susceptible to “early exit attacks”. Details here: https://gradema.readthedocs.io/en/latest/develop/assignment/security.html#early-exit-attacks

Parameters:

module_name – The name of the module to run

Returns:

A test whose success is based on the exit code of running the specified module

gradema.test.create_python_pytest(file_argument: str) → gradema.test.Test

Creates a pytest test.

Currently, does not have a debugging option. See source code of this function for detailed comment.

Parameters:

file_argument – The argument to pass to pytest that is the file and optionally has a ::test_function_name tacked on at the end

Returns:

A pytest test

gradema.test.create_python_format_check() → gradema.test.Test
gradema.test.create_python_format_check_from_path(path: str) → gradema.test.Test
gradema.test.create_python_type_check() → gradema.test.Test
gradema.test.create_mypy_command(arguments: Sequence[str]) → gradema.test.Test
gradema.test.create_python_stdio_test(module_name: str, test_identifier: str, args: Sequence[gradema.test.argument.Argument], input_file: pathlib.Path | None, goal_file: pathlib.Path, stdout_as_output: bool) → gradema.test.Test
gradema.test.create_python_traditional_stdio_test(module_name: str, test_identifier: str, input_file: pathlib.Path, goal_file: pathlib.Path) → gradema.test.Test

Creates a traditional stdio test

Parameters:

  • module_name

  • test_identifier

  • input_file

  • goal_file

Returns:

gradema.test.create_python_traditional_arg_test(module_name: str, test_identifier: str, args: Sequence[gradema.test.argument.Argument], goal_file: pathlib.Path) → gradema.test.Test
gradema.test.dummy_test() → gradema.test.Test
gradema.test.create_file_exists_test(files: list[tuple[str | pathlib.Path | os.PathLike[str], str]], mime: bool = False) → gradema.test.Test
class gradema.test.RustProgram(main_location: str | os.PathLike[str] | pathlib.Path, binary_location: str | os.PathLike[str] | pathlib.Path)
create_compile_step() → gradema.test.Test
create_stdio_test(test_identifier: str, args: Sequence[gradema.test.argument.Argument], input_file: pathlib.Path | None, goal_file: pathlib.Path, stdout_as_output: bool) → gradema.test.Test
create_traditional_stdio_test(test_identifier: str, input_file: pathlib.Path, goal_file: pathlib.Path) → gradema.test.Test
create_traditional_arg_test(test_identifier: str, args: Sequence[gradema.test.argument.Argument], goal_file: pathlib.Path) → gradema.test.Test
create_format_check() → gradema.test.Test

Creates a rustfmt check.

Note that the resulting command will be something like rustfmt --check src/main.rs. Even though this only includes the main file, all files referenced by main will also be format checked – it’s not necessary to list all the files here! https://github.com/rust-lang/rustfmt/issues/2426

Returns:

A test that completes successfully when there are no formatting errors

create_cargo_command(arguments: list[str]) → gradema.test.Test