Although it doesn't affect your utility, you won't be able to get enough items to impact fights later on which will cost your team multiple fights and objectives. Farming alone in the side lanes can be really dangerous for you as you are quite easy to get picks on.