I'd rather answer on the grounds of the methodology itself, rather than how to "fix" the situation.  In another context, I assisted in working on a ratings and classification system, and found that inter-rater agreement was disappointingly low.  Two paths were considered


Change how rating agreements were defined and identify those who seemed to "understand" the task, or
Refine the definitions used, along with the guidance and examples provider to raters, so that they could more easily understand how to rate things.


In the first sceneario, the whole methodology and results could be rendered a waste simply because the inter-rater reliability was low.  It indicated that either the original definitions were bad or that the raters were given poor instructions.  If I proceeded along that path, I was sure to have problems.

In the second case, the agreement between raters was very good.  Since they rated quite a lot of items, they could also give feedback with when they thought the original definitions and guidance were inadequate.  In the end, the methodology was very reproducible.

Based on that, I would not yet modify your set of raters, but return to the original definitions and guidance.  Any tinkering after the rating is a problem, though it can be useful as a quality check.  There are sometimes raters that are going to do what they want, no matter the guidance given.  With good statistical methods, it is easy to identify them and weight their contributions appropriately.

Now, if I'm mistaken and you don't plan to do further collection, i.e. your data is already collected and done, what you may do is PCA or something like it, and see if you can get a sense of how the different doctors (or patients) cluster.

Were the patients exposed to all of the doctors at the same time (e.g. through a video recording) or were they exposed sequentially, and had a chance to modify their presentation with each interaction?  If the latter, then there could be issues with the patients, and not the doctors.

What to do in case of low inter-rater reliability (ICC)?
I shot at a concert for the first time. The pictures were pretty good for the capability of my equipment, but on some shots, the light has a very strange flare. It isn't the usual rays, but some soft, non-symmetric form which I find quite strange, and, on most shots, distracting. 



On this image, you can see it just above Gino's head and also behind his back, to the right of the mic stand. I am not sure if the slight blue lines on the left side of the picture (they form a triangle aimed at Chris's head, maybe not visible on all monitors) are part of the problem or if they come from a different light source. 

What makes them appear, and how do I avoid them? 

I shot with a D90 with a 18-200 lens, no filters or hoods attached. The settings were quite strained (iso 3200, often at max length and max aperture for the length. This specific picture is 1/125, f 5.3, 95 mm). 
