Design Decisions

This file contains design decisions and why they were made.

Project Structure

  • gradema

    • Contains all the code for gradema

  • example

    • Contains example usages of gradema

  • tests

    • Contains unittests to test the functionality of gradema.

    • This is the only place unit tests exist for the project itself. A “unit test” in another location is likely just an example unit test that does not test the functionality of gradema itself

Build system of Gradema

We had a couple of different options for choosing how to build Gradema. Here’s a bullet point list of some of the considerations.

Ultimately, poetry was chosen because of its lock file and it made it easy to publish packages. It’s also one of the most popular and well supported build tools for Python.

Python Unit Tests

When creating an autograder to grade Python programs, it’s worth considering the different options we have for writing unit tests.

(Thanks https://www.softwaretestinghelp.com/python-testing-frameworks/)

  • unittest - is built-in and standard

    • Can easily use a unittest.TextTestRunner to run individual test cases

  • pytest - makes writing tests very easy.

  • robot - an automation testing framework

    • Not what we need. This is less of a unit test library and more of an automation testing library

  • nose2 - an extension for unittest

    • This project makes unittest tests feel more like pytest tests - you can use assert!

      • It gives useful info when an assert statement fails

    • nose2 is also not easy to run on its own

  • Testify - an extension for unittest that provides useful things that makes this feel like a testing library you would use in Java

    • This is deprecated

One of the things that we debated using is autodiscovery of tests. When using autodiscovery, we don’t usually have a way to programmatically run individual tests. One possible workaround would be to not allow the easy execution of individual tests within the autograder itself, and then just use the success/failure counts to assign a grade to the unit test portion of the

Additionally, it’s worth mentioning that in assignments utilizing grade.sh, unit tests are basic files with if __name__ == "__main__" entry points. The benefit to those is that they are very easy to use a debugger such as pudb for.

Ultimately, pytest is what we are recommending. It allows for the easy writing of tests, its output is not too verbose, and its output makes it easy to see what has happened on a failing assert statement.

Using rich library

Using the rich library makes it very easy to have cross platform colored outout.

Useful documentation for colors: https://github.com/Textualize/rich/blob/c034c6869bb574dcc2ad6d94dcb0870b50165da5/rich/color.py#L49

Escaping in rich

You might see the use of something like this from time to time:

from rich import markup

console.print(markup.escape(some_message_here))

We need to escape some_message_here, because there’s a chance that it contains characters that Rich would interpret as color codes. Remember that a string such as "[red]Hello, World![/red]" would print Hello, World! in the color red. So anytime there is an untrusted string, you need to make sure to escape it.

Escaping sure feels weird, especially when we wouldn’t have to do it if we weren’t using rich, right!? Well, yes. If you really don’t like escaping that much, then take a look at an example like this:

console.print(Text.assemble("Actually ", ("read ALL the output generated by this script!", "cyan")))

If you use Text.assemble, you don’t have to escape the string you pass to it! It might be worth considering doing this as default, but it’s not currently an issue yet.

Try guessing what this prints:

console.print("Hello \\ there. [red]How are \\\\ you today?[/red]")

It outputs this (pretend the color is there):

Hello \ there. How are \ you today?

You can see that backslash escaping is weird. Inside of [red], you need to escape the backslash, whereas outside of it, you don’t need to. On top of all this, you have to remember that you have to escape backslashes in Python strings!

Custom Themes

It would be worth looking into Style Themes in the future. This would allow us to do something like this:

from rich.console import Console
from rich.theme import Theme
custom_theme = Theme({
    "command": "magenta",
})
console = Console(theme=custom_theme)
console.print("[command]{markup.escape(shlex.join(command))}[/command]")

This has the advantage of us being able to easily change the color that commands are output in a single place.

Defining Dependencies in Assignment Repositories

The obvious choice for defining dependencies in assignment repositories is a simple requirements.txt. However, pyproject.toml with a simple pip install . has a few advantages:

  • The definition of metadata

  • Ability to define tool specific configuration (for tools like black, mypy, pytest).

Considering pyproject.toml

As we mentioned above, pyproject.toml has some nice advantages.

Poetry would have also worked, but we can’t ask students to install poetry on their machines, especially when the assignment isn’t a Python assignment.

So, since we like the features that pyproject.toml gets us, let’s see what using one would look like.

[project]
name = "binary-converter-autograder"
description = "Binary converter autograder"
version = "1.0.0"
authors = [
    {name = "Lavender Shannon", email = "jdsfz4@mst.edu"},
]
readme = "README.md"
requires-python = ">=3.10"

dependencies = [
    "gradema==0.0.4"
]

With that we have defined so metadata, and most importantly, the dependencies section. The metadata really isn’t all that important, but it’s cool to have.

Now, if you need to install dependencies (just gradema in this case), simply run pip install . (in a virtual environment!!!). Keep in mind that this has the downside of installing the given project: https://stackoverflow.com/questions/62408719/download-dependencies-declared-in-pyproject-toml-using-pip More discussion here: https://github.com/pypa/pip/issues/11584#issuecomment-1306769428 Although the project is being installed, it doesn’t matter too much since we should always be in a virtual environment. The biggest downside is that it will put something inside of src/project-name.egg-info, which is seriously annoying (especially in Rust projects). The really cool thing about pip install . here is that it will fail if your Python version is less than the minimum specified. pyproject.toml is really cool, but it might be worth reconsidering the usage of pip install . at least.

So, pip install . has its problems for our use case, but what about generating a requirements.txt from pyproject.toml? Well, this isn’t that easy to do unless you are using a tool like poetry. So, consider this: We use poetry to create the assignment, but generate a requirements.txt before releasing the assignment and commit that to version control. poetry.lock and pyproject.toml remain in the project, but are not as important now. This is one thing we could consider in the future, but for now it’s a little overkill. If we are really set on pinning dependencies, this makes a lot of sense, but we haven’t been pinning dependencies for a while and things seem to be fine.

Choosing requirements.txt

For the reasons mentioned above, we will choose requirements.txt for student assignment repositories. This means that a requirements.txt is all that is required, plus the addition of setup_venv.sh. A sample requirements.txt could look like this:

gradema==0.0.4

Now, a setup_venv.sh could look like this:

#!/usr/bin/env sh
set -e
cd "$(dirname "$0")"

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

We use the python3 binary because we only expect this script to be used on Linux, where python3 is almost always available to point to the python installation.

A setup_venv.bat could look like this:

@echo off
setlocal

:: Change directory to the location of the batch script
cd /d "%~dp0"

:: Create virtual environment
python -m venv .venv
if errorlevel 1 (
    echo Failed to create virtual environment
    exit /b 1
)

:: Activate virtual environment
call .venv\Scripts\activate.bat
if errorlevel 1 (
    echo Failed to activate virtual environment
    exit /b 1
)

:: Install requirements
pip install -r requirements.txt
if errorlevel 1 (
    echo Failed to install requirements
    exit /b 1
)

exit /b 0