If you use the OpenSSH keyfile format (ssh-keygen -o or using Ed25519 keys) and using gnome-keyring you may get:
sr4l@Lars-Laptop:~$ ssh user@server sign_and_send_pubkey: signing failed: agent refused operation Permission denied (publickey).
This error message cost me a lot of time because on some PCs it does work on others it doesn't. Later I discovered that the problem only occurred if the keys have the default name ~/ssh/id_rsa. Later I found similar reports leading to gnome-keyring daemon.
- Deactivate gnome-keyring entirely
- Only deactivate the gnome-keyring SSH backend
- Safe your keyfiles with a non-default name and use ssh-add /path/to/file or ~/.ssh/config to use them
A good resource for detailed solutions can be found in the Arch Linux Wiki.
Notes for XFCE:
In XFCE gnome-keyring starts automatically if you have "Launch GNOME services on startup" under "Settings > Session and Startup > Advanced" checked. I unchecked it and enabled gnome-keyring by enabling gnome-keyring pkcs11 and secrets backend in "Settings > Session and Startup > Application Autostart"