The scholars interpreted this phrase and said that the brothers are saying that their father is mistaken in this one issue of this world and not to do with the religion.
We see that the brothers are not to the level of Yusuf; otherwise, they would not be speaking like this.