Further to my remote operation topic, I think I have now solved the routing part of remote operation but I am finding that the wfview connection is unreliable and suffers short and long drop-outs.
For the purposes of this test, I am using:
Server: Raspberry Pi connected to an IC-705 via USB. This instance of wfview seems to run just fine. The Pi is connected to my home LAN using Powerline ethernet (not Wifi). The router is a Netgear R7000 (this may be significant? I’m beginning to suspect that my router could be part of the problem). Ports 50001-50003 are forwarded to the Pi.
Client: Macbook connected to a Personal Hotspot running on my phone. Good solid EE 4G signal. I have tried various settings on wfview, but currently I’m on LPCM 1ch 8bit, 48000 sample rate, max latency.
Note that I am not using VPN and IP forwarding in this test. I’m keeping it as simple as possible.
For the purposes of testing, I’m listening to BBC Radio 2… which makes it easy to hear the drop-outs! Typically it is mostly good, but there are occasional momentary interruptions. Then every so often (5-10 minutes) there are much longer drop-outs, lasting around 1 minute.
I am watching the log on the server side, and sometimes it gets itself into a state where it is reporting many, many “INF audio Input Packet 0 arrived too late”. It is reporting these approximately every 20ms. The audio seems to be arriving fine whilst it is in this state, which continues until it hits a longer outage.
Sometimes it goes to a long outage from which it never recovers. Service can be restored by pressing “Disconnect” then “Connect”.
20ms is because of 50 Hz power system and powerline ethernet is a problem.
My experience is from tv streaming over powerline. Distance just 15m inside the appartment. Even on the same phase. We got several cutoffs and had often even to reset the tv box.
With powerline you get too many noise from everyone arround from all the motors, leds and so on.
Live streams are very sensitive. No handshake with udp protocoll. We tried Wifi at 2,4 GHz, still problems at some lower rate but 20 networks arround. Now with wifi mesh at 5 Ghz (Fritzbox and Fritz extenders) no problem any more.
Just to clarify, 20ms is because that is the length of audio that is sent in each UDP packet.
It is very difficult to get stable and reliable realtime audio streaming using powerline or even WiFi as they don’t have a predictable round-trip time and they are extremely susceptible to external influences which can cause packet loss. The nature of UDP packets means that it is ideally suited for realtime applications because it performs no handshaking, this can be an issue on less than perfect networks though.
What might be absolutely fine for web browsing or watching Youtube videos isn’t necessarily going to work well for realtime audio where one lost packet can result in an audio dropout. The Icom protocol (and wfview) have a retransmit request system but for audio this is almost completely useless, as the retransmitted audio will arrive too late to be useful.
Watching web based videos uses a completely different protocol (TCP) and will usually have a significant cache (probably 5+ seconds) which makes it unsuitable for realtime audio where we are aiming for 100ms or less latency.
My only suggestion if you are using wfview client and server is to absolutely use Opus rather than 8bit audio as the bandwidth requirements are significantly lower. It is also worth checking the CPU usage of both client and server machines as spikes in CPU can also cause delays in processing which can result in dropouts.
I am rather new to wfview, but I also occasionally observe these symptoms. My setup is a little different:
o Local - IC7300 USB-connected; Ubuntu 20.04; Server connected to Mesh WIFI.
o Remote; Ubuntu 20.04; connecting via my public IP but no VPN.
On the Local side, things seem to run flawlessly with wfview, wsjtx, fldigi, and (occasionally) flrig, using wfview’s rigctld. I can’t get cqrlog to cooperate with wfview’s rigctld, but that’s a topic for a different thread.
On the Remote side, I see these “dropouts” regardless whether I am connected to the home LAN (but to a different Mesh access point), to my public IP, or whether my VPN is active or not. Usually I am connected via the public IP without the VPN running. Sometimes the connection is uninterrupted for hours at a time, but occasionally there are momentary interrupts, and sometimes there are “freezes” that last minutes, requiring a server dis/re-connect.
Yes, I understand the difference between UDP and TCP, and indeed why UDP is the right choice for audio streaming in this application.
In my experience so far, performance is poor over Wifi, but acceptable over powerline and hard-wired. It also seems okay using a mobile hotspot, which involves a single-user Wifi hop (no contention).
I think there are several things going on here:
Short, intermittent drop-outs. This is probably to be expected, and can be lived with. It’s just a little bit of QRM, after all!
Long drop-outs of 30s to several minutes (which can be overcome by a forced disconnect/reconnect). A bit of a show-stopper for contest operating! I may be wrong, but this seems like a bug or design issue, since it is looks as though the transmission link is good (possibly after a short outage), but wfview has not managed to reconnect.
wfview getting into a state where it is streaming large quantities of INF messages to the log, over an extended period, again possibly after a short outage. Note that the link is working absolutely fine whilst this is happening. Again, this seems like a bug? Users wouldn’t be aware of this unless monitoring the log file in real-time.
No criticism intended when I point my finger at possible bugs, of course! I’ve got nothing but astonishment and admiration for what has been achieved, so far.
Phil - your recommendation regarding Opus is noted, and I’ll try to go back to that, but in my current testing bandwidth isn’t a problem. I should have 100 Mbit/s or more available end-to-end.