If you use the OpenSSH keyfile format (ssh-keygen -o or using Ed25519 keys) and using gnome-keyring you may get:

sr4l@Lars-Laptop:~$ ssh user@server
sign_and_send_pubkey: signing failed: agent refused operation
Permission denied (publickey).

This error message cost me a lot of time because on some PCs it does work on others it doesn't. Later I discovered that the problem only occurred if the keys have the default name ~/ssh/id_rsa. Later I found similar reports leading to gnome-keyring daemon.

Solutions:

• Deactivate gnome-keyring entirely
• Only deactivate the gnome-keyring SSH backend
• Safe your keyfiles with a non-default name and use ssh-add /path/to/file or ~/.ssh/config to use them

A good resource for detailed solutions can be found in the Arch Linux Wiki.

Notes for XFCE:

In XFCE gnome-keyring starts automatically if you have "Launch GNOME services on startup" under "Settings > Session and Startup > Advanced" checked. I unchecked it and enabled gnome-keyring by enabling gnome-keyring pkcs11 and secrets backend in "Settings > Session and Startup > Application Autostart"