© Creative Commons

LOFAR produces enormous data sets. During calibration, it may be necessary to create several instances of the same data set: these are forward-predicted (model) data that allow direction-dependent calibration.

We've been compressing LOFAR data with Dysco compression to reduce the volume for archive and transfer, but it turns out that Dysco is not efficient for forward-predicted model data. This is because model data has different properties compared to the original data: it is smooth and noiseless. Therefore, we developed a novel compression framework called 'Sisco' (Simulated Signal Compression), that aims to compress model data. Unlike Dysco, compression is lossless, and compression ratios of nearly a factor of 8 can be reached on LOFAR data.

Sisco is somewhat similar to the FLAC compression that is used for compressing music or audio: it tries to estimate the next value based on the previous value, and uses efficient encoding for storing the residual. FLAC does this in one dimension (time), but Sisco extends this to two dimensions (time-frequency). Quite some effort went into applying this on standard measurement set ordering and integrating this into the Casacore I/O library, with the final result that the method can be applied transparently to our data, thereby reducing the intermediate I/O considerably.

The image shows compression results on MeerKAT (left) and LOFAR data (right), using various prediction methods to determine the most efficient ones. The method was accepted for publication in A&A: Offringa & Van Weeren (2026), https://arxiv.org/abs/2512.23490 .