Security of Autograder

While developing the autograder, we need to worry about students doing things on purpose or by accident to bypass the intended flow of the autograder.

Changing Autograder Files

This exploit has always existed. If you can change the autograder files, it’s pretty easy to get the grade you want. We need to reimplement the hashing functionality for Gradema in order to prevent this attack.

Early exit attacks

In Python, calling sys.exit(0) will stop the program and spit out an exit code of 0. This is a problem if you are using something like create_python_test(), which uses the exit code directly.

To stop this from being a potential exploit in your grading program, you need to understand how sys.exit(0) works. All it does is raise a SystemExit. So, to prevent this from happening, either except SystemExit explicitly, or have a except BaseException to handle unknown exceptions.

Remember that the autograder itself won’t be the thing exiting early, but rather the tests inside the student’s program. This is most important for unit tests. Lucky for us, if you use pytest you won’t have this problem, as pytest will catch SystemExits.

Modification of results.txt

However unlikely, if a student program alters results.txt and then somehow gets the autograder to crash or not write to results.txt, then a program can get a perfect score. The easiest way to do this is not to get the autograder to crash, but to instead change the permissions of results.txt after writing to it. Take this code:

$ echo 100 > results.txt
$ chmod 400 results.txt
$ echo 40 > results.txt
bash: results.txt: Permission denied

Although we cannot write to the file now, we can actually still remove the file, which we can take advantage of:

$ rm -f results.txt

Fully solving this problem will require an understanding of the .gitlab-ci.yml such as this:

image: "git-docker.mst.edu/os/container"

generaltester:
  script:
    - ./grade.sh
  timeout: 2 hours 30 minutes
  artifacts:
    when: always
    paths:
      - results.txt
    expire_in: 1 month

With the current configuration, after all the steps in the script section complete, the artifact will be uploaded (assuming it exists). If we look at artifacts:when, we can see that the only conditions we can have are on_success, on_failure, always. That means that unless we want the successfulness of the job to help determine if the autograder has run successfully, it has no use to us. (Remember that we currently use the successfulness of the job to tell the student if they have > 70% on the assignment). artifacts:paths is just as you would expect.

Knowing all this, I don’t think we even need to worry. As long as we make sure we remove results.txt before writing to it, we won’t get permission errors within Gradema. The only potential exploit is if a student is able to make the Gradema autograder code crash after writing to results.txt.

Additional Exploits to Consider

With Gradema and the recommended way to set up assignments, there’s a few other considerations:

• Assuming we use the virtual environment when running the CI/CD, it could be possible to commit a malicious .venv