Select a heuristic without spending the final test
A runnable synthetic example with separate fitting, selection and final evaluation.
Start with a small decision
You have four simple scoring rules. Each rule turns a row into a number, and a threshold turns that number into a class. The question is which rule to retain. Decide how to select it before looking at the final test.
Make the exercise inspectable
The accompanying Python script creates 1,000 rows with three features, a, b and c. Each feature is sampled uniformly between zero and one and rounded to six decimal places. The label is one when 0.7*a + 0.3*b + noise > 0.55; noise is uniform between −0.25 and 0.25. Feature c does not enter that formula. These choices deliberately create a teachable classification task. They do not represent a measured population.
Freeze the split
The first 600 rows are training data, the next 200 are validation data and the last 200 are the final test. The rows are independently generated under the same process; their order is not a timeline. This split does not demonstrate how a model behaves under temporal drift or repeated users. A real dataset needs a split that reflects its deployment setting.
Fit each threshold on training data
The rules score a, b, c, or the mean of a and b. For each rule, try thresholds from 0.25 to 0.75 in steps of 0.05, predict one for scores at or above the threshold, and retain the threshold with the highest training accuracy. A tie goes to the lower threshold. The constant baseline predicts the majority class from training data: zero. Neither a threshold nor that baseline class is fitted on the test rows.
Choose the rule on validation data
Compare the four fitted rules using validation accuracy. Ties are resolved by rule name. Both the search grid and tie policies are fixed in the script. There are 44 training threshold trials and four validation comparisons; the final test is not a fifth candidate-selection step.
| Rule | Fitted threshold | Training accuracy | Validation accuracy |
|---|---|---|---|
a_only | 0.50 | 81.83% | 82.50% |
b_only | 0.60 | 63.50% | 63.50% |
c_only | 0.55 | 53.33% | 54.50% |
equal_a_b | 0.55 | 79.67% | 79.00% |
Open the final test after selection
Validation selects a_only at 0.50. On the 200 test rows it gets 162 predictions right: 81.0% accuracy. The constant baseline gets 125 right: 62.5%. The difference is 18.5 percentage points on this one synthetic test. It is not an estimate of commercial benefit or a result from the former project.
| Test result | Selected rule | Constant baseline |
|---|---|---|
| True positives | 64 | 0 |
| True negatives | 98 | 125 |
| False positives | 27 | 0 |
| False negatives | 11 | 75 |
Keep the limitations with the score
Accuracy treats the two kinds of error equally. That choice would need review where errors have different costs. The example uses one seed, one split, four handpicked rules and a known synthetic label process. It does not test robustness across populations, drift, feature availability or production software. Repeatedly changing the rules after inspecting these test numbers would make this test part of development. Further tuning needs a new evaluation plan.
Run the example
Download the Python script and run python heuristic_experiment.py with Python 3.13. The script prints JSON and uses only the standard library. Compare it with the retained result.
Script SHA256: ee45280b846e2a21f5ffd4ad5fdddf507e45fcb473a8f728412c172a565e5f68. Seed: 247022. Dataset SHA256: ee6b7e746c1c11f6df5127cd64115e7c3d58227910cdf8d65451ecd717f14c1a.