Completing undergrad in 4.5 years I'm currently in my junior year, doing CS and math double major. For the next year, I have the choice of spending a semester abroad at a top 20 CS program. I have completed nearly all of my required courses, except one. 

The thing is that, if I cannot take an equivalent of that course in the semester abroad, I won't be able to graduate at the end of four years, I will need an extra semester just to take that. 

On the positive side, in the semester abroad, since I do not have much course requirements, I will be able to spend time doing research (the school I will be going has undergrad research program).

So, my question is, how much one extra semester (probably with only one course in it) would affect my chances in graduate school admissions (for US)?

Note: I know that the extra semester would cost me a whole year due to the admission cycles. In that semester, I can stay at my current home instution (maybe again working with a professor), because double major students are allowed/expected to complete in 5 years.
Extra time you spend doing research will almost certainly do more to help your chances of admission in graduate school than taking >4 years to graduate will hurt - especially if you don't have other strong research credentials (with the caveat that admissions committees are full of people, and you might run into one person who for some reason decides that taking >4 years to graduate is the mark of a failure - it isn't really worth your time worrying about each of these possible types of people).

However, you should keep in mind that graduate admissions, particularly for PhD programs in the US, typically occur only at one time in the year, for fall semester, so you should have a plan for what you intend to do the rest of the year. If you can get hired in a university lab in your field of interest, that would be ideal for graduate admissions.

This answer is based on my experience in Biological/Biomedical Sciences in the US, and advice may vary by field and country; I also have in mind primarily PhD admissions.