Dolueg2

TL;DR; Summary

Dolueg2 is a published (in BAMS) framework that requires no to little maintenance and consists of a dynamic website part and several predefined plotting types. You can check out the code/publication here:

Background / status at the beginning

During my time at the University of Basel starting in 2014, I was put in charge of the website of the MCR (Meteorology, Climatology and Remote Sensing Lab). At the time, there was a typo3 site which contained the profile of people at the lab and a bit about the lab itself. Additionally, a visualisation page was available for the measurement stations in and around Basel. However, adding new stations, and setting up the pipelines for it was quite tedious as it required many manual changes of the HTML code and the IDL code to make the figures (which was a whole other construction site). Overall, the page had aged. Due to licencing issues, we also ran version 7.X for IDL, which meant that there was no SVG output, but the only semi-sensible choices were PNG (in a bad way, writing out to a plot window and rereading it to get an acceptable resolution – still shudders me to think about it) or (E)PS. This came in version 8, but that was too late.

So two tasks were identified. First, the website should be more dynamic, i.e. if new plots were made because a station was added or retired, it should require much less interaction – basically, only one file that has a list of stations. Secondly, the plots should become more attractive, contain more information, become readable (the PNGs resolution meant text was barely legible) and more sensible. This essentially meant they should be SVGs. No budget other than my work time and computer time was available of course so we had to do it all in-house.

Task 1 – The dynamic website

As we had a webspace available, which meant PHP, I started learning to create the central function – a file search to find PNG/SVG files in folders. The rest was creating a sensible folder structure and settling on some defaults, e.g. choosing categories for plots like numbers for stations in the filename, letters for variables (or groups of variables) and timespans to show (one day, one week, one month, one year). At some point, this involved also learning JavaScript to reduce the server load with the search as the number of plots drastically increased over the projects life time.

At some points we also wanted to include the previous projects so all the information would be in one place. Published is a skeleton version of the page which gives the basic functionality at the respective GitHub repo, you only have to add/create your own plots as described in the next section.

Task 2 – The new design of figures

As IDL was the only programming language I knew very well at the beginning of the project, it was natural to start with redesigning the plots. It was clear, however, that they should be now vector graphics as we expected more plots now (FOGNET with 11 new stations was coming along) and as most would be displaying simple time series vector graphics were far more sensible. So we took the only way – make PS figures with IDL, and convert them to SVG via LaTeX (which meant, save the PS figure, make a PDF of it, and then convert that to SVG). A rather slow, and tedious approach. But it also meant that we now had vector graphics! Within this part, I also rewrote the standard plots available that still exist largely in this form today (including details like aligned tickmarks when a second axis is added, the option to colour it, including meta information in the legend etc.)

A year later I started to learn Python for my other duties and revisited the whole pipeline as we also created a new function for the database access (that moved from binary files to MySQL). Of course, with the ability to get data from the MCR database, it was (in comparison) straightforward to create SVGs with Python and translation of the scripts itself from IDL to Python was relatively simple as averaging could now be done via pandas. The Python version of the figures is what has been published and is available at the GitHub repo of dolueg2figures.

Finishing up the project/ a look today

My time at the MCR came to an end several years ago but dolueg2 is still going strong, i.e. no adjustments were required, no updates other than adding/removing stations (which was planned like this from the beginning). As such I’m really happy with the state of the site and still use it privately to get the current measured weather. 🙂