I'm not sure what is meaning of no response during post-doc contact. I'm looking for post-doc position, and applied to a position. Here's brief flow of email converstaion. 

Possible PI: Well, it looks like your research interest and experience matches well with mine, let's have a interview. 

Me: Thank you. When can we have a discussion? 

Possible PI: (No response) 

3 days later

Me: Hello? Do we have interview? 

Possible PI: (No response) 

more 3 days later 

Me: Hello? Can I have interview with you? 

Possible PI: (No response) 

I don't know what is going on his side, but I never wrote something rude or stupid. Can I regard this sign as refusal? Or do I need to email him again? Or can I call to his office phone number? 

I don't think his reaction - no response - is polite at all, at least...
If they originally had interest in interviewing you, after the second reminder, there is a good chance it has evaporated. The mail frequency was too high, and there were too many repeats (unless you know the person really well). 

A PI would be probably reluctant to hire someone who appears to be insistively dependant on them or even "pushy". A person who wasn't "pushy" may not get a reply for various reasons (like the PI is trying to make it possible, but the whole idea fizzles out, or gets postponed by months and he doesn't want to give a straight no in case it ends up working, and then forgets about it). On the other hand, if an opportunity presents, they may reconsider and come back to you later. 

However, they are less likely to do that with a person that comes across as too aggressively pursuing their purposes, because they will infer that's how they will also behave during regular interaction.

Instead: What you should have done is to respond to the interview suggestion as: "Shall we agree on a date?" and then wait for a few weeks (two weeks, as suggested in the comments above, is a good number) and then try again. If it does not work out, it's effectively a no (although it may not be even a final no, see above comments).

Yes, a "no response" can be considered a bit "rude", but do understand that people are simply overworked, and whatever the most current pressure is, it pushes itself in front of the queue. Don't be the one adding to their pressure by overmailing, because then the rudeness has appeared on your side before it has had a chance presenting itself on theirs.