Should I get informal with a professor if they didn't explicitly invite me to do so but the situation seems to suggest it's appropriate? Some years ago I completed my B.Sc. in a university. I was very young and the default behaviour for all the students (especially the young ones) was to formally address the professors. PhD students were more likely to get informal with them.

After my degree I spent some years working in a research institute and during this period I had some distant collaborations with some of the former professors and I got more familiar with some of them. Also some of my fellow students during the B.Sc. in the same period enrolled in a PhD and some of them became researchers in the same university.

Eventually I decided to enrol again in the same university to pursue a M.Sc. This put me in a weird situation. I was in a very informal relationship with my former fellow students and they were in a quite informal relationship with my professors but still when talking with my professors I was supposed (by these unwritten rules) to address them formally, even when we were all in the same room. Some of my professors were also very casual with me, in some cases one even gave me his personal phone number to have casual chats about technology and similar stuff and meet at an event and have dinner together, another one hugged me and kissed me when I won a competition, and another one invited me to spend a weekend with me (and other people) for a short holiday.

I want to stress that all this behaviours were in general not uncommon in this university and by my personal standard not at all invasive or "bad". They were all done in a very friendly manner and not hidden from other students or other professor. It was all fairly normal among professors, researchers and PhD students.

Said that, I still felt that I was not authorised to get too informal with them because of the relationship professor-student and they never told me to get more informal with them. 

I'm not sure if I came across as too cold, but I was not really sure what my behaviour should have been in a situation like that. Should I have been more informal with them by my own initiative?
I would suggest that you handle interactions more on the informal side when you are just around the professors and your former fellow students.

But when around the professors and your current fellow students I would stay on the formal side because otherwise you imply to the other students that every student can adress the professors like that (which the professors may not want).