A temperature curve without nulls may represent continuous measurements—or ten minutes of lost communication filled with the last value. The numbers can look identical while their evidential strength differs. Industrial data engineering should explain where each value came from, what time interval it represents and whether it is usable now. This design concerns low-frequency process monitoring and feature construction; high-speed vibration needs separate signal-processing decisions. All values and thresholds are constructed examples, not site measurements.
Define measurement semantics before deciding what a minute-long row means
Retain at least the asset, measurement point, original event timestamp, system availability timestamp, value, unit, quality code and acquisition batch. Event time identifies when measurement occurred; availability identifies when the feature service could read it. Preserve traceable handling of clock offsets, retransmissions and late arrivals instead of overwriting event time with ingestion time.
Distinguish instantaneous quantities, interval statistics, cumulative counters and states. Mean temperature, incremental electricity consumption, meter totals and switch states cannot share one aggregation rule. Detect meter resets or replacements before differencing. Temperature remains physically meaningful during a shutdown, while zero production and unobserved production are different states.
One row per minute could mean the average of observations within that minute, the latest trustworthy state at a timestamp, or a time-weighted interval mean. These meanings differ. pandas resample exposes closed, label and origin for bucket membership, output labels and grouping origin.[1] A row labeled 10:01 does not inherently contain only information available by 10:01.
Specify interval closure, timezone, bucket width, observation cutoff and late-arrival revisions in the data contract. For example, [10:00,10:01) excludes an event exactly at 10:01. Cross-device features also need an allowable clock-error policy: uncorrected alignment can falsely order measurements from the same physical process.
Constructed counterexample: averaging rows is not always averaging over time
Suppose temperature is 20°C for the first 50 seconds of a minute and 80°C for the final ten seconds, with these states independently established. If the first segment contributes one reading and the second ten readings, the mean of eleven rows is approximately 74.55°C. The time-weighted mean is (20×50 + 80×10)÷60 = 30°C. Sampling density, not faulty arithmetic, explains the difference.
Real observations rarely establish the state so completely. A single temperature reading does not prove that temperature remained unchanged afterward. If using last-value hold to estimate a time-weighted mean, state the maximum trustworthy duration and behavior across operating-state boundaries. A linear-change assumption gives a different integral. Sparse data do not turn an estimation assumption into a physical fact.
Decide whether the task concerns a peak, interval heat exposure or state at a specific moment before choosing features. Retain a mean alongside the maximum, valid observation count and valid covered duration where relevant. A single mean may be dominated by densely sampled abnormal periods or, under a different sampling pattern, dilute a brief anomaly.
Count coverage can be defined as the fraction of expected samples validly received under a fixed sampling protocol, but event-triggered devices may have no comparable denominator. Time coverage is the fraction of the interval covered by trustworthy observations under an explicit validity-duration rule. Both require deduplication, quality-code and shutdown policies; many rows alone do not establish complete coverage.
Completing a time axis, interpolating and downsampling are different operations
Creating continuous time buckets creates coordinates, not measurements. Aggregation summarizes existing observations under a rule; interpolation estimates unobserved positions from known points and assumptions. Decide which original observations must survive before estimating target positions, rather than allowing defaults to silently change the evidence set.
The pandas Resampler.interpolate documentation says that it first reindexes to target timestamps and then interpolates nulls, warning of information loss for nonuniform or high-frequency data.[2] Suppose values at seconds 00 and 60 are both 20°C, with an 80°C observation at second 25. A target grid containing only seconds 00, 30 and 60 can discard the off-grid spike before filling an apparently stable middle value.
To estimate temperature at target instants, one option is bounded, elapsed-time interpolation over the union of original and target timestamps, followed by target selection. To construct minute-level anomaly features, calculate statistics such as the maximum directly from original observations. The former still need not preserve peaks; an interpolated curve cannot replace every feature. Retaining the original series and transformation version makes the difference explainable.
Do not downsample high-speed waveforms merely by taking every nth point. SciPy resample_poly combines a low-pass FIR filter with rate conversion, and its documentation makes boundary-extension assumptions explicit.[3] Low-pass filtering controls aliasing but also changes high-frequency information. Select the method for the target frequency band, acquisition conditions and endpoint treatment instead of copying slow-temperature filling rules.
Decide whether to fill from gap type and elapsed duration
First distinguish transmission delay, actual acquisition loss, sensor faults, equipment shutdown and inapplicable measurements. Retransmission can restore a real observation; interpolation cannot recover a lost fact. Preserve invalid quality codes and communication failure as distinct states. A repeated last reading is not automatically a valid new measurement.
Constructed policy: for temperature already shown to vary slowly, permit estimation only within the same stable operating state and across gaps no longer than 30 seconds. Beyond that boundary, preserve missingness and mark the feature unavailable. Thirty seconds is illustrative; process dynamics, alert deadlines and validation evidence must determine the actual limit. High-frequency vibration, switch transitions and temperatures around starts or stops cannot inherit the assumption.
Forward filling uses only past values and may suit state persistence with a trustworthy expiry, but it does not authorize indefinite use of stale readings. Two-sided linear interpolation depends on a later anchor and may suit some offline reviews without being suitable for real-time alerts. A smoother curve can erase precisely the transition that needs detection.
A tool’s limit commonly counts consecutive null positions rather than elapsed time, so changing the target frequency changes its temporal meaning.[2] Outside the filling function, also check anchor separation, age of the nearest real observation, maintenance or start/stop boundaries, and the length of the entire gap. Filling only the first few positions of a long gap must not cause them to be mislabeled as a valid short-gap estimate.
Suggested outputs include the original or aggregated value, estimated value, provenance, anchor timestamps, total gap span, current data age and rule version. Provenance should distinguish measurement, aggregation, last-value hold, two-sided interpolation and unavailability. Downstream consumers need the quality information too; adding a missingness column while ignoring it during evaluation does not establish robustness.
Keep real-time and retrospectively revised views separate
Suppose a feature is needed at 10:00. Only a 20°C reading from 09:59 is available, and a 40°C reading at 10:01 arrives at 10:02. Retrospective two-sided interpolation can construct 30°C at 10:00, but that value did not exist for the actual 10:00 decision. Writing the revised estimate into historical serving inputs lets evaluation see the future.
The real-time view uses only observations available by its cutoff and explicitly chooses to wait, use a bounded stale value, degrade or withhold a feature. An offline revised view may include later measurements and permitted estimates, but needs its own version and revision time. It must not overwrite the input snapshot actually supplied to the model.
This article addresses input semantics for resampling and missingness. Complete model evaluation should also follow data availability and retraining schedules in time-series backtesting, so training, prediction and scoring each use the information they are allowed to see. High completeness after offline filling is not evidence of online data availability.
Make fallback behavior task-specific. Some noncritical gaps may allow an output with quality flags; loss of a critical measurement may require withholding automated model judgments and handing over to established rules or operators. Do not classify no result as normal, and do not let preprocessing silently replace equipment safety protection.
Test whether transformations change information using counterexamples
Build a fixed set of counterexamples: bucket-edge observations, off-grid spikes, long continuous gaps, out-of-order retransmissions, duplicate timestamps, unit changes, meter resets and operating-state transitions. Specify expected bucket assignment, whether estimation is allowed and which quality flags must survive. Conflicting values at the same timestamp require source and revision rules, not arbitrary selection of the last row.
Next hide observations in held-out intervals with trustworthy complete measurements. Compare no filling, bounded last-value hold and bounded interpolation. Simulate random losses, contiguous outages and gaps near anomalies. Masking tests establish reconstruction behavior under constructed missingness, not that real fault-related losses follow the same distribution. Intervals without ground truth during failures remain an explicit limitation.
Report absolute reconstruction error in original units at hidden positions, peak preservation, changes in threshold events and the number of cases automatically used beyond the permitted gap. For maximum-based features, compare original window extrema; for state features, check transition timing. One mean-error score cannot substitute for every business objective.
Break results down by gap duration, operating condition, measurement type and estimation method. Also report actual model-usable coverage and fallback proportion. Permissive filling can raise availability while increasing false confidence; strict rejection creates operator workload. Acceptance must constrain both costs using limits tied to consequences.
Finally freeze the sampling contract, original snapshots, clock handling, imputation rules, software versions and expected counterexample results. This article checked behavior described in the online pandas 3.0.6 and SciPy 1.18.0 documentation; deployments should pin and verify their own versions. A tidy table is useful, but every estimate must remain identifiable, explainable and rejectable.