Wrong answer. Now what?

Trying again is how students learn. We want to give them something useful to work with on their next attempt: visualizations, hints, and feedback that help explain what the code actually did without solving the problem for them.

The basic idea

Keep the problem connected to what students submit

Forge knows how the problem is put together. Keeping that information attached when the problem reaches a course gives us more to work with than a verdict alone.

Describe the problem itself

Tell Forge about the inputs, constraints, test data, and solutions, instead of scattering that information across a collection of scripts.

Use the same problem in class

Put it in an assessment and collect submissions without turning it into a separate, unrelated version of the problem.

Learn from the attempts

This is where we want to use what Appledore knows about the problem to make feedback more useful for students and instructors.

The not-all-finished part

What we hope better feedback looks like

Some of this exists. Some of it is still an idea with a suspicious number of notes attached. This is the direction, not a list of promises about what Appledore can do today.

The Forge part

Okay, so why build another problem editor?

Most problem development tools leave you with a statement, a validator, a generator, some test data, and several scripts. Each one knows a slightly different piece of the same problem.

Forge asks you to describe the problem itself. If the input contains a graph, you say that it is a graph, how large it can be, and what properties it has. Forge can then use that same description in more than one place.

Validators and basic generators are the obvious uses. The more interesting use is understanding the test data and, eventually, which kinds of cases students get wrong.

This does mean doing things the Forge way. Forge has opinions. That is mostly the point.

A fairly normal problem package

statement.pdf

validator.cpp

generate.py

one_more_script.sh

What Forge would rather have

n is an integer

G is a graph with n vertices

G must be connected

What this makes possible

validation

basic generation

fuzzing

mutation testing

test coverage

constraint analysis

edge-case search

known counterexamples

known tricky cases

common wrong approaches

test case classification

sample checking

statement help

counterexample search

submission analysis

student trouble spots

hints

visualizations

…and other possibilities

That is the plan, anyway.

Build decent tools for programming courses, keep the student experience pleasant, and make wrong answers a little less useless.

Wrong answer. Now what?

Keep the problem connected to what students submit

What we hope better feedback looks like

See what your code does

Give a nudge

Name the kind of failure

Notice class patterns

Okay, so why build another problem editor?

That is the plan, anyway.