VLBI CORRELATOR CENTRE
In 2009 the correlation was switched to the DiFX (Distributed FX) software correlator, after 30 years of operating hardware VLBI correlators; the reasons have been dramatically lower investment and extreme flexibility of a software-based solution. The DiFX correlator is Open Source maintained by a community of developers mostly located at NRAO, ASTRON, MPIfR and CSIRO.
DiFX is executed on a High Performance Computing cluster, which was upgraded in December 2015 after more than 7 years of operation. The upgrade was financed with funds from the MPG and a significant contribution from the German mapping authority BKG (Federal Agency for Cartography and Geodesy), our long-standing partner at the correlator. The upgrade went smoothly as planned with a correlator downtime of less than 2 weeks. The new cluster is roughly 10 times more powerful than the old one, and will be able to handle the increasing amount of wide-band VLBI observations in the mm-domain and geodesy for at least the next five years.
Reference values of the new VLBI cluster are:
• 68 compute nodes
• each node with 2 processors with together 20 compute cores which adds to 1360 compute cores total
• ∼1175 TB disk storage (16 RAIDS; ~944 TB for VLBI)
• 56 Gbps Infiniband fabric as interconnect (plus 1 GE Ethernet for service tasks)
• 48 TB disk space for correlated data
• 40 TB backup for correlated data and other important files
• 2 head nodes for correlation (several correlations can be run in parallel)
• 1 head node for other tasks
• 1 appliance computer for cluster installation and monitoring
• 15 Mark 5 units for playback of data into the correlator
• 6 Mark 6 units + 4 expansion chassis for 1mm VLBI data and VGOS data
• 1 Gbps connection to the Internet reserved for VLBI data transfers (soon 2 Gbps)
In addition to the standard DiFX versions (latest stable, development, etc.), which are used for “normal” correlation, a DiFX branch for the correlation of RadioAstron data was developed at MPIfR: DiFX-RA. The most significant part of this effort involved modifying the delay model server (Calc, from the Calc/Solve package) to be able to calculate delay information for telescopes with arbitrary coordinates and velocities (that is, not fixed on the ground), and changes to the DiFX metadata system to deal with the changing position and velocity of the spacecraft as a function of time. The delay model was also modified to correct for (general) relativistic effects as the highly elliptical orbit of the spacecraft results in large changes in velocity and gravitational potential compared to the terrestrial frame.
DiFX-RA was used extensively to correlate a number of RadioAstron projects at Bonn, but also to help debug the ASC correlator in Moscow (Kovalev et al. (2016), arXiv:1601.05806). RadioAstron correlation requires much more time than other projects for determining the clock offset and rate for each scan for the satellite. Sixteen RadioAstron projects have been correlated to date with another five waiting in the queue. All except one have collaborators from the scientific VLBI group, and five projects have a principal investigator from the group.
The cluster is used for correlation of GMVA observations, EHT data (load shared with Haystack), RadioAstron projects, other MPIfR-based observations, and tests, for instance of the DBBC2/3, while standard EVN observations are correlated at JIVE. The MPIfR correlator is at present also the major correlator for IVS (geodetic International VLBI Service); it carries the biggest load of the geodetic production correlation worldwide. Most notably the geodetic group at the correlator processed the CONT14 (2 weeks of geodetic continuous observations) with up to 20 antennas in record time. In the past such campaigns had beed correlated at USNO with the MK IV hardware correlator. The experience of the Bonn team with DiFX correlation has triggered a number of other institutes to send their personnel to Bonn to be trained in DiFX correlation for production operation at their home institutes.
Despite the increased load, due to a growing number of antennas per observation and increasing bandwidth, the correlation is now more efficient, stable and faster, with reduced personnel: only one operator is needed at present.
In preparation for very wide bandwidth recording, in 2015 direct playback from Mark 6 modules and handling of more than one module (data stream) per station were implemented by W. Brisken (NRAO, contracted by MPIfR), who is one of main contributors to the DiFX project, and H. Rottmann (MPIfR). These enhancements make it possible to correlate GMVA (with ALMA), EHT and VGOS data in a much more straightforward and efficient way, as for instance demanded by ALMA for VLBI.
DiFX offers paramount flexibility like extreme spectral resolution with a minimum of 0.5 Hz, or integration times of ms, or the ability to correlate mis-matched frequency bands using zoom-band mode, and the ability to invert mis-matched sidebands.
In addition to VLBI data correlation the cluster has been used for jet simulations, pulsar searches, LOFAR data reduction and other task by MPIfR personnel of various research departments.
Future plans aim at making DiFX compatible with data as produced by the next generation of digital backends (DBBC3H and BRAND; see below), which will be able to digitize and process up to 16 GHz of bandwidth.
W. Alef, H. Rottmann, G. Bruni, J. Anderson, A. Bertarini, H. Fuchs, H. Sturm, R. Märtens, A.L. Roy, D.A. Graham, M. Wunderlich, W. Brisken (part time contract, on leave from NRAO
T. Schüler, J. Ihde (BKG Frankfurt), A. Müskens, S. Bernhart, L. La Porta (University Bonn), J. Anderson (GFZ Potsdam), W. Brisken (NRAO, Socorro), DiFX developers community worldwide