The Dursleys only needed three days to realise that someone had put a spell on them. Having been fed and watered at Hogwarts, it took Harry two days to get hungry. After nearly breaking down in the garden, Petunia accused him that there was a spell at work.