© Henk Mulder

In the LOFAR 1 days, checking the spectrum meant taking over a station completely. You had to start it in a specific mode and use a collection of coding and expert knowledge just to see how the antennas, signals, and RFI looked at that moment. While doing so, the station was unavailable for observations. It was useful for verifying repairs, tracking down external RFI sources, or investigating suspicious antennas, but it was far from ideal. Some plots were automatically generated and available after observations, like the RTSM plots, allowing us to detect problems shortly after an observation finished. The downside was obvious: by the time you discovered a major issue, the next observation had already started.

We learned a lot since then.

In LOFAR2, the stations are always on and continuously generating data. That opened the door to piggybacking on live data streams without interrupting observations. Once I discovered we could generate plots in Jupyter that roughly resembled the old LOFAR1 spectrum plots, the next question became obvious: how do we make this available for every station, continuously, and in a way anyone can use? The first prototype was held together with pieces of code. It generated PNGs of spectra on individual stations, moved them into central MinIO storage, and displayed them in Grafana through HTML in text panels. It worked, but it depended on one Jupyter panel kicking off a whole chain of other Jupyter code every time we wanted updated plots in you Grafana webfrontend.

Eventually we rebuilt it properly.

Instead of treating spectra as images, we turned the raw data into a datasource we could subscribe to directly. From there, a single query could retrieve the spectrum for any station, antenna field, tile, antenna, or polarization, all selectable through simple dropdown menus in a web frontend. Suddenly anyone could open Grafana and watch a live spectrum updating every five seconds. That changed how we operated the system. Events like High Noise or Oscillations were no longer something we discovered afterward in logs or post-observation plots. We could watch them happen live in the spectrum itself. For the first time, we could track RFI sources in real time and compare multiple stations simultaneously to estimate where interference was coming from.

The next step was turning the idea into time-series data.

Because as far as I know, nobody want to spend 24/7 staring at a live spectrum. Once the data became historical as well as live, entirely new possibilities appeared. In the images attached, you can see the mowing of grass near two stations as it propagated over time, first appearing on one station and later on the next. Each line represents a different LBA X or Y connection, and the impact becomes immediately visible. That opened the door to filtering out non-operational antennas, detecting jitter noise automatically, identifying LBAs that fell over, calculating standard deviations, and correlating behavior across the array over long periods of time. In a constantly evolving system like LOFAR, this is probably just another step in the evolution. The interesting part is no longer whether we can see the spectrum, but what new things we will eventually learn to see inside it.