Compression of interferometric radio-astronomical data

The volume of radio-astronomical data is a considerable burden in the processing and storing of radio observations with high time and frequency resolutions and large bandwidths. Lossy compression of interferometric radio-astronomical data is considered to reduce the volume of visibility data and to speed up processing. A new compression technique named "Dysco" is introduced that consists of two steps: a normalization step, in which grouped visibilities are normalized to have a similar distribution; and a quantization and encoding step, which rounds values to a given quantization scheme using a dithering scheme. Several non-linear quantization schemes are tested and combined with different methods for normalizing the data. Four data sets with observations from the LOFAR and MWA telescopes are processed with different processing strategies and different combinations of normalization and quantization. The effects of compression are measured in image plane. The noise added by the lossy compression technique acts like normal system noise. The accuracy of Dysco is depending on the signal-to-noise ratio of the data: noisy data can be compressed with a smaller loss of image quality. Data with typical correlator time and frequency resolutions can be compressed by a factor of 6.4 for LOFAR and 5.3 for MWA observations with less than 1% added system noise. An implementation of the compression technique is released that provides a Casacore storage manager and allows transparent encoding and decoding. Encoding and decoding is faster than the read/write speed of typical disks. The technique can be used for LOFAR and MWA to reduce the archival space requirements for storing observed data. Data from SKA-low will likely be compressible by the same amount as LOFAR. The same technique can be used to compress data from other telescopes, but a different bit-rate might be required.Comment: Accepted for publication in A&A. 13 pages, 8 figures. Abstract was abridge

A morphological algorithm for improving radio-frequency interference detection

A technique is described that is used to improve the detection of radio-frequency interference in astronomical radio observatories. It is applied on a two-dimensional interference mask after regular detection in the time-frequency domain with existing techniques. The scale-invariant rank (SIR) operator is defined, which is a one-dimensional mathematical morphology technique that can be used to find adjacent intervals in the time or frequency domain that are likely to be affected by RFI. The technique might also be applicable in other areas in which morphological scale-invariant behaviour is desired, such as source detection. A new algorithm is described, that is shown to perform quite well, has linear time complexity and is fast enough to be applied in modern high resolution observatories. It is used in the default pipeline of the LOFAR observatory.Comment: Accepted for publication in A&

Post-correlation radio frequency interference classification methods

We describe and compare several post-correlation radio frequency interference classification methods. As data sizes of observations grow with new and improved telescopes, the need for completely automated, robust methods for radio frequency interference mitigation is pressing. We investigated several classification methods and find that, for the data sets we used, the most accurate among them is the SumThreshold method. This is a new method formed from a combination of existing techniques, including a new way of thresholding. This iterative method estimates the astronomical signal by carrying out a surface fit in the time-frequency plane. With a theoretical accuracy of 95% recognition and an approximately 0.1% false probability rate in simple simulated cases, the method is in practice as good as the human eye in finding RFI. In addition it is fast, robust, does not need a data model before it can be executed and works in almost all configurations with its default parameters. The method has been compared using simulated data with several other mitigation techniques, including one based upon the singular value decomposition of the time-frequency matrix, and has shown better results than the rest.Comment: 14 pages, 12 figures (11 in colour). The software that was used in the article can be downloaded from http://www.astro.rug.nl/rfi-software

Prospects for detecting the 21cm forest from the diffuse intergalactic medium with LOFAR

We discuss the feasibility of the detection of the 21cm forest in the diffuse IGM with the radio telescope LOFAR. The optical depth to the 21cm line has been derived using simulations of reionization which include detailed radiative transfer of ionizing photons. We find that the spectra from reionization models with similar total comoving hydrogen ionizing emissivity but different frequency distribution look remarkably similar. Thus, unless the reionization histories are very different from each other (e.g. a predominance of UV vs. x-ray heating) we do not expect to distinguish them by means of observations of the 21cm forest. Because the presence of a strong x-ray background would make the detection of 21cm line absorption impossible, the lack of absorption could be used as a probe of the presence/intensity of the x-ray background and the thermal history of the universe. Along a random line of sight LOFAR could detect a global suppression of the spectrum from z>12, when the IGM is still mostly neutral and cold, in contrast with the more well-defined, albeit broad, absorption features visible at lower redshift. Sharp, strong absorption features associated with rare, high density pockets of gas could be detected also at z~7 along preferential lines of sight.Comment: 12 pages, 13 figures. MNRAS, in pres

Calibration and Stokes Imaging with Full Embedded Element Primary Beam Model for the Murchison Widefield Array

15 pages, 11 figures. Accepted for publication in PASA. © Astronomical Society of Australia 2017The Murchison Widefield Array (MWA), located in Western Australia, is one of the low-frequency precursors of the international Square Kilometre Array (SKA) project. In addition to pursuing its own ambitious science program, it is also a testbed for wide range of future SKA activities ranging from hardware, software to data analysis. The key science programs for the MWA and SKA require very high dynamic ranges, which challenges calibration and imaging systems. Correct calibration of the instrument and accurate measurements of source flux densities and polarisations require precise characterisation of the telescope's primary beam. Recent results from the MWA GaLactic Extragalactic All-sky MWA (GLEAM) survey show that the previously implemented Average Embedded Element (AEE) model still leaves residual polarisations errors of up to 10-20 % in Stokes Q. We present a new simulation-based Full Embedded Element (FEE) model which is the most rigorous realisation yet of the MWA's primary beam model. It enables efficient calculation of the MWA beam response in arbitrary directions without necessity of spatial interpolation. In the new model, every dipole in the MWA tile (4 x 4 bow-tie dipoles) is simulated separately, taking into account all mutual coupling, ground screen and soil effects, and therefore accounts for the different properties of the individual dipoles within a tile. We have applied the FEE beam model to GLEAM observations at 200 - 231 MHz and used false Stokes parameter leakage as a metric to compare the models. We have determined that the FEE model reduced the magnitude and declination-dependent behaviour of false polarisation in Stokes Q and V while retaining low levels of false polarisation in Stokes U.Peer reviewedFinal Accepted Versio

First LOFAR results on galaxy clusters

Deep radio observations of galaxy clusters have revealed the existence of diffuse radio sources related to the presence of relativistic electrons and weak magnetic fields in the intracluster volume. The role played by this non-thermal intracluster component on the thermodynamical evolution of galaxy clusters is debated, with important implications for cosmological and astrophysical studies of the largest gravitationally bound structures of the Universe. The low surface brightness and steep spectra of diffuse cluster radio sources make them more easily detectable at low-frequencies. LOFAR is the first instrument able to detect diffuse radio emission in hundreds of massive galaxy clusters up to their formation epoch. We present the first observations of clusters imaged by LOFAR and the huge perspectives opened by this instrument for non-thermal cluster studies.Comment: Proceedings of the 2012 week of the French Society of Astronomy and Astrophysics (SF2A) held in Nice, June 5th-8t

The Murchison Widefield Array Transients Survey (MWATS). A search for low frequency variability in a bright Southern hemisphere sample

We report on a search for low-frequency radio variability in 944 bright (> 4Jy at 154 MHz) unresolved, extragalactic radio sources monitored monthly for several years with the Murchison Widefield Array. In the majority of sources we find very low levels of variability with typical modulation indices < 5%. We detect 15 candidate low frequency variables that show significant long term variability (>2.8 years) with time-averaged modulation indices M = 3.1 - 7.1%. With 7/15 of these variable sources having peaked spectral energy distributions, and only 5.7% of the overall sample having peaked spectra, we find an increase in the prevalence of variability in this spectral class. We conclude that the variability seen in this survey is most probably a consequence of refractive interstellar scintillation and that these objects must have the majority of their flux density contained within angular diameters less than 50 milli-arcsec (which we support with multi-wavelength data). At 154 MHz we demonstrate that interstellar scintillation time-scales become long (~decades) and have low modulation indices, whilst synchrotron driven variability can only produce dynamic changes on time-scales of hundreds of years, with flux density changes less than one milli-jansky (without relativistic boosting). From this work we infer that the low frequency extra-galactic southern sky, as seen by SKA-Low, will be non-variable on time-scales shorter than one year.Comment: 19 pages, 11 figure