You need the web site to use a certificate from the same root authority as your client certificate.
I’m not sure if I’ve misunderstood you, but I use Lets Encrypt for the server’s TLS, and then my own CA cert (which is only present on the webserver) for the client’s mTLS and everything works fine, since it’s the client that validates the server’s cert and the server that validates the client’s cert.
Only works with the web UI, no apps support it
Yeah that’s true.
you will get warnings when adding your root certificates to any device
It’s not a root certificate, and I’ve never seen any warnings.
Possibly mTLS, which you’d configure in your reverse proxy. You could email them the certificate and instructions on installing it. I believe for Chromium browsers on Windows you basically just double click the cert and click through the wizard. Firefox I know has a thing in the settings for importing the cert. Android you just tap on the cert and make sure it opens with ‘Certificate Installer’ if it gives you the option.
The Immich app (at least on Android) supports mTLS client certificates, I use that for my instance.