Testing "settings" branch for audio interruptions - success!

Further to the recent testing here: Comparison with RS-BA1 - wfview

…I have built today’s version of the “settings” branch, and it is a huge improvement!

For this test I have:

  • IC-705, with USB to…
  • Raspberry Pi running the new build, connected to the internet via a wireless hotspot and VPN to…
  • Raspberry Pi Zero wired to my home LAN router as the other end of the VPN, with iptables routing to…
  • Mac (MacOS), WiFi on my home LAN, running the wfview client

Codec is Opus, and latency is maximum.

I have been streaming BBC Radio 2 for over an hour - and the audio quality is surprisingly good, even for music! In that time I have experienced:

  • a couple of very brief audio stutters
  • three instances of outages of around 4 seconds

So it’s not perfect… but it is definitely now viable for remote station operation. My next step will be to get some other members of our contest group involved, to test for real on HF.

That’s great Russell!!

I am still improving things on the settings branch. The main cause of the complete stoppage was actually some badly written code (by me) that was taking a long time to process the contents of a buffer to check for missing packets (as part of the retransmit handling for missing UDP packets). In my tests, in the event of a missing packet, this could take up to 10 seconds to process the whole buffer. During this period, all server processing was suspended!

On my recent tests, the same process now takes between 1 and 3 ms so quite an improvement…

I suspect that the occasional outages that you are now seeing may be WiFi or cellular network related but if you can check the logs of both the server and client when the outages occurred, we may be able to see a pattern?

I still have work to do on recovering from a momentary network loss as I find that the audio buffer fills-up and needs to be flushed as otherwise audio latency can increase significantly.

73 Phil M0VSE

Phil,
would this improved version also give some improvements for the Mac version (as client, connected to the 705)?
I would be happy, because currently I still have interruptions probably due to WLAN overload on some hours of the day. My workaround in that case is to restart wfview on my Mac, which gives me again around 30 min, until the problem occurs again.

73 Peter DL5DLA

Hi Peter.

Yes I have also implemented the same buffering fix in the client side of wfview so once we produce a new MacOS build, this will (hopefully) be fixed.

73 Phil M0VSE

Great - looking forward for the new version.
Thanks.

73 Peter DL5DLA

Oh, dear - this is very frustrating.

I was about to open up my rig to a couple of other testers for true remote operation, and I thought I’d better just check the transmit audio. Good job I did, because it doesn’t seem to be working!

Server is set to alsa_input.usb-Burr-Brown…etc

Radio settings are USB everywhere, and the rig Connectors MOD Input is USB 100%

I have tried MacBook and iMac as clients with their internal microphones. Perhaps I should try a Windows client next? I can’t think of anything else to try!

Hi Russell.

When did you last ‘pull’ the settings branch? I discovered an issue with server TX audio where the level was set to zero internally. This should be OK now.

You are probably better reverting to the master branch now as that has been merged with the settings branch and will receive any future updates.

73 Phil M0VSE

Aha! Yes, that would explain it.

Okay - I’ll try a master branch build tomorrow.

Just to give an update on this:

I’m running client/server successfully at the moment on the current master branch. There are still occasional drop-outs: perhaps once per 20 minutes of 2-3 seconds duration. There are no log events (using standard logging) before, during or after the drop-out.

I have also learned that I have to turn off Mic audio in the IC-705 settings (or unplug the mic) for remote operation, otherwise the rig’s mic is live. Also, for FM operation it is a good idea to turn on squelch in the USB AF settings.

With debug on, here’s a log file which includes a short outage. I’m not sure exactly where the outage starts, but I stopped logging just after recovery.

Hi Russell, this would be the important bit:

2022-01-26 15:10:02.621 DBG audio: Output Attempting FEC on packet 136149 as last is 135766
2022-01-26 15:10:02.645 DBG udp: Too many missing packets, flushing buffer:  35674 missing= 242
2022-01-26 15:10:02.645 DBG udp: Too many missing packets, flushing buffer:  5078 missing= 382
2022-01-26 15:10:02.745 DBG udp: Too many missing packets, flushing buffer:  14127 missing= 77

The occasional missing packet is to be expected and wfview will deal with those, but the large numbers missing here will overwhelm wfview so it flushes the buffers and starts again.

To put this into perspective, assuming you are using the opus codec, 382 missing audio packets is 7.64 seconds of audio, which suggests something is blocking the network for this time?

I can’t see anything on the client or server that would cause this, but I am in the process of rewriting much of the server code to support (among other things) multiple radios connected to the same server. So when it’s ready, maybe my reworked code will be better?

73 Phil M0VSE

Thanks, Phil. Looking forward to the next full release, so that we can update the client machines as well; I’m not yet competent to build Windows & MacOS from source!