You are quite immobile which can make you a prime target for chain CC and assassins. Aphelios relies on extended trades, so taking him out quickly will give the enemy team a huge advantage (especially if he has his Infernum W up when he dies).