Exactly "kicked away", after He Xuan sorting, students didn’t expect anyone else outside their enclosed circle, so almost all the freshmen have already moved away and found a new place away from the former ghost. That is why when Ron Weasley joined them with a will to sit in the middle of the table, as if he makes a big excuse for all the Slytherin by just simple his being there, no more space left for social isolation and those little owners of the reptile brain didn’t have a better plan than just to ask older years to use magic and levitate the younger boy to the almost empty far end.