The PostgreSQL database itself is hosted on a different remote server and can only be accessed through a specific whitelisted internet connection....My local PC is connected to the internet through VPN, which is also fast.
I can't say without more information, but the difference in performance can simply be attributed to having to send large amounts of data over the internet through a VPN.
You provided latency numbers, but that's not relevant for a sustained transfer of large amounts of data; I presume the CSV is large because you wouldn't be noticing a problem with a small CSV. What is relevant for COPY is sustained transfer rates. That's what you should be testing.
Latency is how long it takes to get a response. Transfer rates are how fast data can be pushed through. To use a plumbing analogy, latency is how long it takes after turning on the tap to get water, transfer rates are how much water can come through the pipes. Your transfer rates are only as fast as the slowest pipe, and that's probably the VPN.
Managed networks can have two data transfer rates: a fast rate for the initial data, and then a slower rate for more data. Small files transfer fast, large files start fast then get slower. Since most operations transfer a lot of small files (consider a web page with small images and text files) this makes the connection feel fast for most things. But transfer a large file, like a large CSV, and things slow down. This is to conserve bandwidth while keeping the connection feeling snappy.
What you should be looking into is the sustained transfer rates to the server. Try using scp to transfer the CSV file to the server and see how long that takes.