A numerical study of 21-cm signal suppression and noise increase in
  direction-dependent calibration of LOFAR data
  We investigate systematic effects in direction dependent gain calibration in
the context of the Low-Frequency Array (LOFAR) 21-cm Epoch of Reionization
(EoR) experiment. The LOFAR EoR Key Science Project aims to detect the 21-cm
signal of neutral hydrogen on interferometric baselines of $50-250 \lambda$. We
show that suppression of faint signals can effectively be avoided by
calibrating these short baselines using only the longer baselines. However,
this approach causes an excess variance on the short baselines due to small
gain errors induced by overfitting during calibration. We apply a regularised
expectation-maximisation algorithm with consensus optimisation (sagecal-co) to
real data with simulated signals to show that overfitting can be largely
mitigated by penalising spectrally non-smooth gain solutions during
calibration. This reduces the excess power with about a factor 4 in the
simulations. Our results agree with earlier theoretical analysis of this
bias-variance trade off and support the gain-calibration approach to the LOFAR
21-cm signal data.
