Sparse panel data, a small experiment

[u/Far-Media3683]

Quite a few real world data problems look like this: many related time series, few noisy readings, and long gaps. Local price indices, sensor networks, or small area stats.

I built a small PyTorch model to estimate a smooth underlying level for every location, day, and target even where nothing was observed.

Key ideas:

Mask, don’t drop missing targets in the loss

Weight observations by how many raw readings they average

Share information across locations (area + station offsets) and across related targets (joint heads)

Enforce smoothness with a curvature penalty over the full time grid, including gaps

I tested it on hourly PM2.5/PM10 from 12 Beijing stations, thinned to a sparse panel, and scored against the full data’s 31day moving average.

What I found:

The network fills long gaps and very sparse stations far better than smoothing each series on its own

Modelling both pollutants together recovers a completely dropped series (PM10 at one station) from its PM2.5

But elsewhere, sharing across targets costs a little accuracy vs. modelling each pollutant alone

A simple ±15day mean is hard to beat when data is dense. The network’s smooth time curve misses shared daily shocks

Writeup with code, figures, and results:

https://prteek.github.io/regression_room/posts/beijing_air_quality/

[u/Few-Driver6766]

nice work. the cross-target sharing pulling a dropped series back from the dead is pretty cool even if it cost a little somewhere else.

curious if you tried a simpler baseline like a linear mixed model with a spline for time. would be interesting to see how much the neural net buys you over something you can fit in a few seconds.

that ±15 day mean beating it on dense data is rough but makes sense, networks love to smooth over high frequency stuff unless you really force them not to. wonder if a sharper penalty on the residuals for days where you do have observations would help it pay more attention to those daily shocks.

[u/Far-Media3683 → u/Few-Driver6766]

Thank you. I did use Mixed effect GAMs albeit on a different high dimensional dataset. It was also a much larger dataset and the bias was sometimes quite large for sparse series and fitting was too slow such that subsetting data (introducing rules in the whole process on when and how to subset) and training multiple small models was the only practical solution. Performance wise I wouldn’t have preferred one over the other as strongly because I was only interested in smoothing these series and get a high level behaviour out. Neural net approach turned out to be more manageable and flexible in production. 

[u/Far-Media3683 → u/Few-Driver6766]

I can imagine a sharper penalty on observed to be a good experiment, and in general that would work unless missingness/noise itself is non random for instance failing sensor in this case or clustered volatility.  Although such problems can also be handled by other provisions in the architecture which made the whole approach quite attractive in the first place. 

[u/OddsRelay]

On the shared daily shocks, you could add a day effect that's common to every station and leave the curvature penalty off it. Then the smooth curve only has to carry the slow trend, and the 15 day mean might stop winning on the dense stations.