  Actually, the Spectres are there because that is where the Subtle Knife was created and was used there very often which led to the creation of many Spectres because they are created every time someone cuts into another world.  