Everyone else that was in and out of the household were also either tracking the horcrux's, fear mongering in different streets, or working normal business to listen in on whatever conversation was new. People were still regularly in and out but between genuine business, terrorism and hiring new recruits, the Malfoy family and any other respected guests were the only ones who were allowed where they pleased and have the freedom of being in or out of meetings or events.