When verified failures are scarce, test narrower claims separately. You may be able to measure review workload, replay predictions honestly and check warning timing without establishing how often a model misses failures. A quiet dashboard is not evidence of reliable detection. Design the evaluation around independent assets, information available at prediction time and outcomes that are actually known.
The useful result may be permission to continue an advisory evaluation, rather than a claim that the model is ready to guide maintenance. State that distinction before looking at a performance score.
Define the claim and the scoring unit
Consider a hypothetical fleet of twelve ventilation fans in a fabrication workshop. A reliability engineer wants to evaluate a bearing-warning model. Evaluation record EV13-01 holds the protocol, dataset version, partitions, outcome decisions and eventual report. This is a proposed study, with no trained model or measured results.
- Target event
- A confirmed bearing-related loss of required fan function, supported by reviewed findings.
- Prediction unit
- One eligible fan-day, predicting that event within the next seven calendar days.
- Operational question
- Does a warning arrive with useful review time and manageable attention demand?
- Claim boundary
- Results apply to the observed conditions and tested assets, with unresolved outcomes disclosed.
The seven-day horizon and preceding 24-hour input lookback are invented study choices, not maintenance intervals. Define eligibility and how repeated warnings form one review episode before scoring. A hundred overlapping windows from each of twelve fans yield 100 × 12 = 1,200 windows, but still only twelve assets.
Shared conditions link assets; repeated windows do not establish independent trials.
Preserve incomplete outcomes before counting errors
Outcome review asks what happened after each prediction, not whether a maintenance record exists. The engineer applies the target definition to findings without seeing model scores where feasible. EV13-01 keeps the evidence and reason for every outcome assignment.
| Record state | Evaluation meaning | Do not infer |
|---|---|---|
| Bearing-related functional loss confirmed | Target event at the supported event time | Every earlier warning was useful |
| Fan observed without an event until study end | Later failure time is right-censored | The fan will never fail |
| Bearing replaced preventively | Intervention recorded; target failure remains unconfirmed | Replacement proves a fault or prevented failure |
| Report awaiting inspection findings | Outcome unresolved | No target event occurred |
The NIST/SEMATECH handbook's censoring explanation distinguishes observed survival time from an unobserved later failure. Its reliability discussion also notes that few failures limit available information.
Check follow-up for unalerted cases as well as alerted cases. If only warnings trigger inspection, outcomes will be better verified on one side of the comparison. Show unresolved fractions separately rather than interpreting the difference as predictive skill.
A fully observed seven-day prediction window can support a no-event label for that window. A window extending beyond observation cannot automatically receive the same label; retain its observed duration and reason for ending. Missing follow-up is different from documented survival, and a repair ends the prior condition history rather than revealing its untreated future.
Separate unfamiliar assets from later periods
Evaluate later periods and unfamiliar fans separately.
| Partition | Assets and time | Purpose |
|---|---|---|
| Development | F01-F09, weeks 1-8 | Fit model and preprocessing |
| Validation | F01-F09, weeks 10-12 | Choose settings before final tests |
| Familiar-asset test | F01-F09, weeks 14-18 | Assess later periods of known fans |
| Held-asset test | F10-F12, weeks 14-18 | Assess assets excluded from development |
The scikit-learn guide covers grouped and temporal splits.
Final predictions stop in week 18, but their seven-day outcomes need follow-up through week 19. Windows lacking that follow-up remain incomplete. Three held-out fans cannot represent fan types or operating conditions missing from the study.
Exclude F10-F12 from fitting/settings. Trim development/validation windows whose input history or outcome horizon crosses a partition boundary. Week 19 is outcome-only. Freeze final-test settings; report tests separately.
Replay only information available at the time
Later findings must not become earlier inputs.
- Fix the prediction time and identify inputs actually available by then.
- Use development-fitted preprocessing.
- Save the prediction and warning under the frozen settings.
- Review later findings solely to assign the outcome.
- Record interventions and incomplete follow-up alongside the score.
The guide requires training-only transformations. Backdated findings remain unavailable before actual entry.
Intervention also changes what can be learned. If a warning leads to preventive replacement, the unmaintained failure outcome is unavailable. Do not score the replacement as a detected failure, or claim the absence of failure proves prevention. Preserve that intervention branch separately. Condition-triggered maintenance can make censoring depend on deterioration; do not assume independent withdrawal in a survival analysis.
Compare workload and useful warning time
Evaluate the model beside the existing manual review process and a simple operational rule fixed before the final tests. Apply the same asset eligibility, information cutoff, outcome definitions and review schedule. A baseline cannot use richer future information than the model. Use the manual process as recorded, not an engineer's hindsight reconstruction of what staff should have noticed.
- Attention demand
- Review episodes per observed eligible fan-day, plus total staff review time.
- Warning usefulness
- Time from warning availability to confirmed event, compared with the agreed response need.
- Outcome coverage
- Confirmed events, fully observed no-event windows, interventions and unresolved windows.
- Comparison
- Model and baselines on the same cases, with differences and missing evidence shown.
Count recurring notices from one ongoing concern as defined review episodes, while retaining repeat-notice counts. Otherwise alert suppression can make a noisy method look quiet. Include days when the service could not produce a prediction in coverage reporting.
For a concrete matching rule, suppose EV13-01 requires 24 hours for review. Count a confirmed event as detected only with an available warning at least 24 hours and at most seven days beforehand. Match each event once; list late warnings separately. This invented timing requirement needs local agreement.
Before either test, EV13-01 fixes its eligible assets and daily prediction calendar in weeks 14-18, accounting for every confirmed event on those assets during weeks 14-19. Recall eligibility starts 24 hours after the first scheduled prediction and ends seven days after the last; the calendar must contain a scheduled prediction 24 hours to seven days before the event.
Apply this rule independently of actual prediction availability, retaining all confirmed events in that cohort, including service outages. Record outside-opportunity events separately with reasons, including initial test-edge events lacking a possible warning. Never move eligibility because a prediction is missing. Week 19 remains outcome-only follow-up.
Thus an event twelve hours after the first planned prediction is outside opportunity even when the service ran; an otherwise eligible event during an outage stays in the denominator. Show both with their eligibility reasons in the event ledger.
Early is useful only in context. A warning presented after the event offers no advance notice; a long warning may still lack usable evidence or reach staff too late. For verified events, report warning availability and response opportunity, not just the timestamp of a backfilled signal.
State what scarce positives cannot establish
Google's classification metrics lesson defines recall as correctly detected positives divided by all actual positives. Here, the denominator includes all confirmed target events in the predeclared asset and event-calendar cohort. A detection requires the qualifying pre-event warning; an arbitrary positive sensor window is not a detected event.
| Evidence state | Reportable | Unsupported |
|---|---|---|
| Zero confirmed eligible target events | Observed workload and outcome coverage | 100% recall or a demonstrated low miss rate |
| Few confirmed events | Individual event and warning accounts | Reliable detection across unrepresented failure modes |
| Correlated windows | Window and asset counts | Assumed independent trials |
With zero confirmed eligible events, recall has a zero denominator and is undefined, not 100%. Unknown outcomes cannot become negative examples. Eligible confirmed events without predictions remain in the denominator; outside-opportunity events stay visible in the separate accounting.
Any uncertainty interval needs a defensible sampling model, event ascertainment and treatment of dependence. This worksheet supplies no confidence bounds. Even correctly calculated bounds would not establish coverage of failure types absent from the test.
Choose the next evidence rather than a bigger claim
Rig tests and simulated faults can expose a model to controlled conditions. They can test pipeline behavior and responses to specified fault patterns. They do not establish performance on the workshop's field population, operating mix or naturally developing failures.
- Reject a compromised test and repair its partitions or outcome definitions.
- Use controlled cases to investigate specific technical weaknesses, labeling their origin.
- Continue prospective advisory observation when burden is workable but detection evidence remains insufficient.
- Revisit the claimed use only when suitable independent outcomes support it.
Keep normal maintenance and physical operating permissions outside this evaluation. The engineer records the tested scope, unresolved cases and evidence needed next. A useful conclusion might be that the study can measure review burden but cannot yet support a missed-failure claim. That is an actionable result, provided the next evaluation has a defined question.