Yes, that is the hard part. But it can be done. Geoip blocking like only allow your country - blocking every China or Russian user etc. If you are selfhosting at home and worry about your SSH access, then you can do a lot of things to block then early. It is all about authentication. Lets say you require VPN access in - example Wireguard. You could require access only through somebody else, like Cloud flare tunnel. You could also do “port knocking” but that is not encrypted. You could require the user first has to be authenticated somewhere else, like require first Microsoft login and only then your ip is allowed.

I believe it is a good start with crowdsec but feels like it gives false protection. The blocking only happens after they have done a couple of attempts and not before.