A big “It Depends” on that - plenty of applications of asymmetric crypto where you just hard-code the servers public key into the client and call it a day, and GPG has its own PKI scheme that is just kinda weird.

You also don’t have to use Diffie-Hellman - early versions of SSL just sent the ephemeral key (the symmetric key used for the actual AES session) directly. This works, but using DH also gives you “forward secrecy” - even if a malicious third party has captured the entire encrypted session, then later steals (or factors) your private key they still won’t be able to read the encrypted traffic because they can’t recover the ephemeral key because it wasn’t sent over the wire in the first place