Photometric Selection of z~5 Lyman Break Galaxies in the ESO Remote
  Galaxy Survey
  We describe the selection of a sample of photometrically-defined Lyman break
galaxies (LBGs) at z~5 using the multi-wavelength imaging data of the ESO
Remote Galaxy Survey (ERGS). The data is drawn from ten widely-separated fields
covering a total sky area of 275 arcmin squared. Starting with a simple colour
(R-I>1.3) and magnitude (I<26.3) cut to isolate the Lyman break and then
refining the sample by applying further optical and near-infrared photometric
criteria we identify a sample of 253 LBG candidates. We carefully model the
completeness of this sample and the factors that affect its reliability. There
is considerable overlap between this sample and a spectroscopically-confirmed
sample drawn from the same survey and this allows us to determine the
reliability of the optical photometric selection (~60 per cent) and to show
that the reliability can be significantly improved (to ~80 per cent) by
applying near-infrared waveband criteria to exclude very red contaminants. Even
this high level of reliability may compromise some statistical studies of LBG
properties. We show that over 30 per cent of the highest reliability candidates
have multiple UV-luminous components and/or disturbed morphology in HST
imaging, though it is unclear whether this represents multiple
interacting/merging sources or individual large sources with multiple UV bright
regions. Using this sample we confirm that the normalisation of the bright end
of the z=5 UV luminosity function (down to M*) is lower than the same at z=4 by
a factor of 3. Using a Schechter fit we determine M*uv=-20.9+/-0.2. We discuss
whether it is reasonable to expect the UV luminosity function to follow a
Schechter function, given the UV emission is short-lived and stochastic, and
does not necessarily trace the underlying mass of the galaxy.
