I am running the headless server, which requires some acrobatics to get it to run on boot, covered here: Headless Server 100% 1 CPU on Rpi4 (fixed) - #4 by ei4gnb
I have been keeping an eye on CPU usage and resources on the server end, and it does appear that there are no issues there. Even under heavy load, the Pi4 claims it’s coping, but if that (4 year old) thread’s issues have been blind-eye’d, as is likely, due to a crap chipset they are stuck with, then We have an answer: “…the Pi sucks a bit at networking, what do You expect for 30 quid?”
I have no attachment to the Pi4, i’ll swap it out for a Ryzen based box with a grown-up chipset, see how that goes. Would have been nice to have the Pi working though, given it’s super-low entry point position.