Hermione didn't know how Lucius did it, but he made it so that she couldn't cast most useful spells properly; Anything that she tried to transfigure, turned into a snake that would try to attack her - She cynically thought that this was Lucius" Slytherin gag to her - and it seemed that the more she tried to transfigure, the more dangerous the snake would become. Whenever Hermione tried to use a disillusionment charm, it would either last no longer than a minute, or the item she disillusioned would still be visible as it altered the objects being seen through it to look like a very poor Photoshop job.