ALMA Pipeline Heuristics
The Atacama Large Millimeter Array (ALMA) Pipeline Heuristics system is being developed to automatically reduce data taken with the standard observing modes such as single fields, mosaics or single dish "on-the-fly" maps. The goal is to make ALMA easily accessible also for astronomers who are not experts in radio interferometry. The Pipeline Heuristics must capture the expert knowledge required to provide data products that can be used for scientific analysis without further processing.
The ALMA Pipeline Heuristics system is implemented as a set of Python scripts using CASA (Common Astronomy Software Applications) as the data processing engine. It provides calibration, flagging and automatic line/continuum finding, baseline subtraction, automasking and cleaning to compute images / cubes. Several techniques are used to search for bad data. In the spectral domain edge detection algorithms are applied, while in the time and uv domains running mean methods are used. The pipeline is parallelized and it has been in regular use for processing ALMA interferometry and single dish data since 2015.
The pipeline functionality is provided via the CASA task interface mechanism. These tasks can also be used for offline processing in CASA by the PIs. Compared to plain CASA commands one gets the benefit of the automatic heuristics and the weblog output. The tasks are also being used and extended for other projects like VLA, VLASS (VLA Sky Survey), SRDP (Science Ready Data Products) and the 45m NAOJ telescope.
The pipeline processing results are presented as a hierarchical set of web pages facilitating navigation from high-level summaries down to detail pages illustrating the data structure and the reduction steps with plots and quality scores.
The quality of data processing and products is evaluated with numerical Quality Assessment (QA) scores ranging from 0.0 (bad) to 1.0 (perfect). A stop light system with green (1.0-0.66), yellow (0.66-0.33) and red (0.33-0.0) colors shows three major categories which are meant to guide the QA2 process by ALMA staff. QA scores are based on metrics such as RMS, S/N ratios, etc. Each step in a reduction recipe and each topic (calibration, flagging, imaging) is tagged with a score.
Geers, V, Davis, L., Hales, C.A., Kent, B., Kern, J., Kosugi, G, Muders, D., Nakazato, T., Sugimoto, K., Williams, S., Wyrowski, F., The ALMA Science Pipeline, ADASS XXVI ASP Conference Series, Vol. 521, 2019
Humphreys, E., Miura, R., Brogran, C., Hibbard, J., Hunter, T., Indebetouw, R., Kern, J., Videla, L., Villard, E., Chang, C.-S., Davis, L., Francke, H., Gallardo, J., Garcia, J., Geers, V., Gomez, L., Gonzalez, S., Kent. B., Kosugi, G., Mason, B., Morales, F., Morokuma-Matsui, K., Muders, D., Nakazato, T., Sugimoto, K., Williams, S., The ALMA Science Pipeline: Current Status, Proceedings of the 2016 ALMA Conference, 2016, id. 1
Kent, B., Masters, J., Chandler, C., Davis, L., Kern, J., Ott, J., Schinzel, F., Medlin, D., Muders, D., Williams, S., Geers, V., Momijan, E., Butler, B., Nakazato, T., Sugimoto, K., The Very Large Array Data Processing Pipeline, AAS Meeting #231, id. 342.14
Lightfoot, J., Wyrowski, F., Muders, D., Boone, F., Davis, L., Shepherd, D., Wilson, C., ALMA Pipeline Heuristics, ADASS XV ASP Conference Series, 2006
Muders, D. Wyrowski, F., Lightfoot, J., Williams, S., Nakazato, T., Kosugi, G., Davis, L., Kern, J., The ALMA Pipeline, ADASS XXIII ASP Conference Series, 2014