A result card another person can rerun
Keep the task, split, baseline, error counts and reproduction details with the score.
State the task before the score
A reader needs to know what was predicted, which rows were evaluated and how a correct prediction was counted. “81% accuracy” alone leaves all three questions unanswered. Use the following card to make a small experiment reviewable.
| Field | Worked example |
|---|---|
| Purpose | Teach the separation between fitting, selection and final evaluation |
| Data | 1,000 synthetic rows with three uniform features; noisy binary target |
| Availability | All three features are generated before each label; no real feature pipeline is tested |
| Split | 600 training / 200 validation / 200 final test; independent rows, not a time split |
| Candidates | Four scoring rules, eleven thresholds per rule |
| Fitting | Highest training accuracy; lower threshold wins a tie |
| Selection | Highest validation accuracy; rule name breaks a tie |
| Baseline | Constant zero, the training majority class |
| Selected rule | Predict one when a >= 0.5 |
| Final result | 162/200 correct (81.0%); baseline 125/200 (62.5%) |
| Error counts | TP 64, TN 98, FP 27, FN 11 |
| Reproduction | Seed 247022; retained script, dataset digest and machine-readable result |
| Limits | One synthetic split; no real-world performance or deployment claim |
Keep provenance exact
Record the data reference and its version, the script revision or checksum, the interpreter version, the settings and the output. A random seed alone is not a complete reproduction package: it does not identify a changed generator or an edited scoring rule. For a real dataset, keep the access procedure and retention restrictions with the reference. Do not put private records in a public result card.
Describe what was searched
Report the four rule families and eleven thresholds, rather than presenting the winning rule as the only option considered. If a later experiment adds features or changes the split after inspecting the test, create a new result card and explain that development step. Preserve the previous result instead of silently replacing it.
Give the reader the next useful action
For this exercise, rerun the accompanying script and compare the JSON result. For an operational task, ask the owner to review whether the population, feature timing and error costs match the intended decision. The synthetic result card cannot answer those questions for a real service.
Run the example
Download the Python script and run python heuristic_experiment.py with Python 3.13. The script prints JSON and uses only the standard library. Compare it with the retained result.
Script SHA256: ee45280b846e2a21f5ffd4ad5fdddf507e45fcb473a8f728412c172a565e5f68. Seed: 247022. Dataset SHA256: ee6b7e746c1c11f6df5127cd64115e7c3d58227910cdf8d65451ecd717f14c1a.