Build the baseline before tuning the model
A comparison is useful when both sides get the same test.
Choose a baseline that fits the task
For classification, a constant prediction can reveal how far an accuracy score is driven by class frequency. A simple rule already used by the business can offer a second reference. For a time-based task, compare against an appropriate simple historical rule using only information available at prediction time. State the baseline precisely enough to reproduce it.
Choose the consequence before the metric
In a fictional manual-review queue, staff capacity might limit the number of records reviewed each day. Evaluate how many relevant cases are found within that capacity, alongside false alarms and missed cases. A single aggregate metric does not describe every operating trade-off. Use a fixed evaluation rule and label the example as an exercise, not evidence of commercial performance.
Keep model selection separate
Use training and validation data to choose transformations, settings and thresholds. Keep a final holdout for the agreed evaluation. If you repeatedly inspect it and adjust the model, that set has become part of development. Record the change and create an appropriate new evaluation plan rather than continuing to call the old score untouched.
Publish a small result card
Include the task, population, period, split rule, baseline result, model result and known limitations. Add a few failure categories without exposing personal information. Preserve the exact configuration and data reference. A clear result card lets another person assess what improved and what the experiment did not test.
References provide background. The working methods and fictional examples here are our editorial guidance, not an endorsement by the referenced organisations.
Put this into practice: open the worksheet →
Continue reading
Run a complete synthetic comparison and keep its result card.