If you use the OpenSSH keyfile format (ssh-keygen -o or using Ed25519 keys) and using gnome-keyring you may get:
sr4l@Lars-Laptop:~$ ssh user@server
sign_and_send_pubkey: signing failed: agent refused operation
Permission denied (publickey).
This error message cost me a lot of time because on some PCs it does work on others it doesn't. Later I discovered that the problem only occurred if the keys have the default name ~/ssh/id_rsa. Later I found similar reports leading to gnome-keyring daemon.
Solutions:
- Deactivate gnome-keyring entirely
- Only deactivate the gnome-keyring SSH backend
- Safe your keyfiles with a non-default …