Your diagram is almost right, but I think it will help to understand more of the details. It’s important to understand the difference between DNS (domain name lookup) and IP routing.

To break your diagram down more, this is what happens when any computer looks up your website:

1. The device does a DNS lookup of “example.com” using their name server, which may forward it to another DNS server (most home routers do this). I won’t go into the multiple levels of DNS lookups and caching here.
2. Through looking it up by DNS, the device now has the final IP. DNS is now out of the picture and we’re doing IP routing.
3. The device tries to make an HTTP connection to your external IP. HTTP is a protocol that runs over TCP/IP (UDP is used for QUIC/HTTP3). To keep things simple I’ll stick with old fashioned HTTP over TCP without SSL. I am also skipping over NAT.
4. For TCP, it performs a handshake, which the reverse proxy will negotiate. Once the connection is established, the browser speaks HTTP to the reverse proxy. It looks something like:

GET / HTTP/1.1
Host: example.com
...lots more headers...
<blank line>

5. The reverse proxy then takes that request, maps it to an upstream (if any), and makes another request to it via the configured transport. If that part falls down, you will see a 503 error. Otherwise, you will see the response from the upstream, possibly with some modifications made by the reverse proxy (some will rewrite links and cookie paths, for example)
6. The reverse proxy sends that response back to the client.

That’s all very simplified, of course.

As others pointed out, things may seem to work differently from the “inside”, if hairpinning is not available or enabled. This is not related to DNS, but to IP routing. The firewall doing NAT can get confused and not know what to do when an internal request goes to an external IP that it itself has. When it turns that around and routes it back to the internal network, that’s called hairpinning.

One “fix” for this, often used in enterprises, is to use so-called split DNS. All that means is that if you’re asking your internal DNS server for an internal name, it will give you the internal address (192.168.1.123 for example), but an external client would get an external IP.

TL;DR: DNS and IP routing are separate concerns and happen at different parts of the TCP/IP stack.

Glad you got it working!

My hypothesis is that it was DNS (channeling Jeff Geerling here). Since Pihole is your DNS (makes sense), it may have recognized that address as its own and given you its IP. By resolving the naming collision, you fixed the problem because the name is now unambiguous.

These problems can happen very easily when you’re using DHCP and sharing a network and domain name between your clients and upstreams, so I think using home.arpa for one and your other domain for the other was a good idea.

What is your DNA setup like? A lot of dhcp clients are set up to register their name in DNS (if allowed). It could be your pihole server is hijacking it.

If you have multiple DNS servers (eg your home router and your lab) them you may not be getting the full picture.

Can you confirm that the DNS actually resolves to the NGINX IP address (and only that address) when you use PiHole’s FQDN? It sounds like it’s bypassing the proxy because it stopped working when you turned 443 off.