Yes, Jaime loves Brienne. Coster-Waldau's longing glances have thoroughly convinced us of that much. And given that we haven't see Jaime sleep around (or sleep with anyone that isn't Cersei, until now) we know that he probably sees sex as kind of a big deal. His plan to leave in the middle of the night is cowardly, and Brienne catches him, which likely makes Jaime feel more ashamed. We know he feels like he can’t escape his past, which suggests he might feel a little unworthy of the always-honorable Brienne. So while he does love her, and some part of him probably wants to be with her still, it didn't feel right to him. This is not a good enough excuse to be cruel to Brienne, but I guess that's Jaime logic.