But Iâve also read that caffeine can lower your immune systemâs defenses and may make you more susceptible to fluâs and colds. Like most medical advice, itâs mixed. My dad split the difference and had one cup of coffee with his breakfast and one cup of tea with his dinner. He had water or a cup of juice with lunch.