TL;DR; Summary
The CLOUDLAB data visualisation page displays near-real-time information of most measurement devices of the CLOUDLAB project. Under the hood, a server creates figures of daily / three-hourly / hourly / special periods with different height levels (many deliver profile information) and moves them to a web server. As the number of figures grew and in the end, 5.5 cloud radars (2x Ka-Band, 2x W-Band, 1x X-Band, 1x X-Band Microrainradar counting as half) were deployed so did the number of figures. As data were collected on-site, it made the most sense to create figures on-site as well and move these instead. Simultaneously, this meant that the sheer number of figures that were frequently updated required an API to ease the server load.
Background
For the CLOUDLAB project, aerosol particles are released to create ice crystals (read more at the project website). To evaluate the success, cloud radars scan certain parts of the cloud and the resulting measurement needs to be visualised as soon as possible. This meant via browser and required an interface. Thankfully, my previous experience with dolueg2 meant I could reuse some parts for the website but making the figures in the first place also took quite a bit of work.
As it was clear that the data from the various devices would not end up in a database due to the various formats (netCDF, ASCII, only sometimes other continuous) it meant that figures would have to be static – especially as data should not be transferred to the web server and the data server was on mobile internet *cough* . This gave the freedom to design the figures via Python and run the creation via cronjobs.
As not all devices are continuously running in the same configuration, it was also clear that a search for available figures was required, based on a sensible folder structure (Y/m/d) and a defined filename with a fixed position of the date in the filename, I implemented an API that searches folder based on the PHP request according to certain patterns and in a specific subfolder. Further refinements were the extraction of available dates, time and check of missing dates. This tied into the use of the jquery calendar to facilitate user handling (compared to an endless list since the project was running for 3 campaigns and more than 3 years).
One device – One page
As the various devices really do deliver different resolutions in time and space and serve different purposes, it was unavoidable that each has their own little quirks despite being similar. To give some example
- A microwave radiometer delivers profiles but also weather station data, i.e. a height selection is needed, as is a daily and an hourly overview of them and the time series of the weather station, i.e. 8 figures per time selection but only 2 changes per height selection
- A ceilometer delivers cloud base, planetary boundary layer height and backscatter. The first two are time-series but up to three are available. Backscatter is a profile measurement. The combination of them in one figure is sensible, but again at different height levels and at different time resolutions. However, only one figure has to be shown.
- Cloud radars can be used together with a model, a ceilometer and a microwave radiometer to derive a classification of clouds via cloudnetpy. Since we had several radars, we wanted to be able to compare them side by side, for each category for different heights. As such, three figures need to be shown side by side, but only in daily resolution in this case.
- Cloud radars provide many variables but on a daily resolution, small features are hard to distinguish even though daily figures give a great overview. Additionally, the cloud state is relevant to plan an experiment. This meant we had to have daily, hourly, last hourly and three hourly figures for different height levels. The resulting amount of figures takes up roughly half the amount of space and made the API necessary.
The number of potential combinations of figures usually also meant that not everything has to be done at the same time and that for some approaches the reading of the data could be done before individual figures were created. But for some devices like the cloud radars, it also meant that the number of data files could be reduced for e.g. hourly resolution and thus speed up production.
Transferability of figure script between similar devices
It becomes clear from the above that the hourly and daily figures share some similarities. Even between devices (like the different cloud radars), similarities could be exploited: Instead of creating a single bash and Python script for each configuration, a single file Python (the bash script anyway mainly just (r)synced to the webserver and made sure the Python script wasn’t called too often) I made a single file that was then linked to by the other device-scripts. This meant edits/bugfixes in the one single file would automatically apply to all. The only cost was to introduce a couple of dicts before the main function to figure out some internal switches.
Connect the device pages
To ease interaction with the website/single pages of each device, I built a wrapper page that unified the layout and included the single pages as iframes. As a result, rescaling the iframe size and focusing on it to access UI select elements was required. Especially the resizing went through different stages as setting the view section correctly took quite a bit of work.
As a result, however, other pages were easy to integrate, devices and their figures could be easily added, or also the table of flown mission (exported from Excel via pandas to HTML). Including the lovely MeteoSwiss station kiosks that display data of a single station in a compact form.
Forecasting
To avoid being on-site unnecessarily, we reached out to MeteoSwiss and subsequently received their forecast specific to our site (transects in two directions and Bise forecast). Initially, each had its own page similar to the devices but it became evident that the information in the Bise forecast was limited and hence I integrated the two forecasts (in 2 layout options to provide a better overview) for easier decision-making. Via JavaScript and knowing that the figure remains the same, I could even add the timing of the transect forecast onto the Bise forecast.
Ensuring data flow
To ensure data flow, I created two ways:
- A control page which lists in tabular form the device, the location of the last file the last modification time and the last timestamp of the data. Behind the scenes, a combination of bash checks for the last files of known devices and passes it to a Python reader that gets the last record of the various available formats. This could then be checked at will
- An email alert that groups devices together (e.g. remote sensing or aerosols) and links them to the responsible personnel. Every morning, files were checked for issues and an email went out based on whether the group of devices had an issue, including information about the checks and what is the issue.
These two ways meant that we barely had issues with devices when it came to data collection. Exceptions were of course when a device actually broke down because of an issue that required manual fixing.
Finishing up the project/ a look today
I’m very happy with the final state of the website, especially as we branched out to a second web server as a fallback. All API calls run just the same, devices are still running and provide continuous updates on the website.