The Ansible commands to ssh to remote hosts use -oIdentityFile=blah, however, ssh will still attempt to use keys it finds in ssh-agent and/or ~/.ssh/.
openssh server will typically drop a connection after 5 or 6 failed publickey attempts, causing Ansible to fail.
I have proposed https://review.openstack.org/#/c/374769/1 to also add "IdentitiesOnly yes" to the ssh.config.ansible file, for all hosts.