Q:

Is there a light-hearted word or phrase for a hypochondriac parent?

The title may not be entirely correct..
Is there a word or phrase that describes a mother/father who is overly anxious about their baby's health? 
Not a word to describe both parents together, a word to describe one parent. 
Also, a word that could be used in describing the parent to others as well as directly to them. Something like 'Worrying Wendy, Delirious Dad, Medicine Mad Mummy'. I suppose they must have coined a phrase for this in the medical profession.

A:

Given the clarification from the edit, here are a few less severe options:

worrisome — tending to worry: having a tendency to worry
overprotective
tender — with gentle feeling: showing care, gentleness, and feeling
with or using "kid gloves"
precautious — using or displaying precaution

Here are two relevant terms that have some relation to your intended meaning:

helicopter parent — a parent who pays extremely close attention to a child's or children's experiences and problems, particularly at educational institutions

Helicopter parents are traditionally seen as overly worrisome or paranoid. This could certainly apply to medical concerns.
I wasn't able to find anything more specific but if you used the phrase "parental hypochondria" I suspect most English speakers would understand your intent.

One potentially relevant footnote is the term Münchausen syndrome by proxy.

Münchausen syndrome by proxy — a behavior pattern in which a caregiver deliberately exaggerates, fabricates, and/or induces physical, psychological, behavioral, and/or mental health problems in those who are in their care

Münchausen syndrome by proxy is a very complicated mental illness that is somewhat related to your desired term in the sense that the illness is through "proxy"; only instead of Münchausen syndrome you are looking for something like hypochondria.
It is worth noting that you should be careful to never confuse the two. Hypochondria is usually a worry of wellbeing while Münchausen syndrome by proxy is usually an attention seeking behavior.

A:

"Hypervigilant parent" may be as close as you can come to the term you're looking for. 
Granted, hypervigilance is a state of sensory arousal common in patients with PTSD, it is also used to describe parents who are always concerned with potential illness in children, and is thought by some to contribute to hypochondriasis.
One can be an over-attached parent, a fretful parent, or simply anxious parent.
There was a phrase we used to use to describe the anxiety of a parent whose child had a very serious illness, and due to this illness tended to be overly fretful, but this, it would seem, is not the situation you are describing.

A:

A worrywart \ˈwər-ē-ˌwȯrt\ is a term which may describe a parent who worries excessively about their child's health,
a person who tends to worry habitually and often needlessly.
Nervous Nelly, is perhaps the most light-hearted expression I am aware of, which describes a very fearful or nervous person .
You could call a mother a "nervous Nelly mom", this doesn't necessarily describes a parent who is hypochondriac by proxy but it could include this behaviour.

many new parents are nervous Nellies when it comes to the health of
  their babies

And a term which I think strikes the right balance between gentle teasing and description accuracy is 
over-parenting.
Kidspot parent exchange

OK, so we all have days where we are, like, over it, but
  over-parenting is another thing entirely. It's when you get too into
  the detail of your role as a parent that you can't see the wood for
  the trees.

