Escalating RX latency

Usage Questions
1

setup: server = 1.65 on ubuntu 22 , client = 1.64 on pop-os 22.04
rx codec: opus 1ch — tx codec: LPCM 1ch 16bit

I see in the status bar at the bottom of wfview client that the RX latency stays at about 120 to 140ms and then after 3 or 4 minutes of using the client it creeps up to 500, then 1000 and then stays roughly at 3300ms and never recovers.

I had pings going between client and server and they stay about 2ms to 6ms. always.

Whats causing the increasing latency? If i click “Disconnect from radio” on the client and then “Connect to Radio” then the problem goes away for awhile but might reappear after 5 to 10 mins

PS: The Shuttle Pro V2 is working great!! thank you

2

I have personally not seen the rise in latency issue, although it has been reported here before.

Have you tried the Opus codec, and also, have you tried each of the three audio systems (RT, QT, and Port)? Remember to disconnect, change the audio parameters, and then re-connect.

Also what do you have your latency sliders set to in the user interface?

–E
de W6EL

3

“Have you tried the Opus…”

Yes, I have Opus 1ch for the RX, and LPCM 1ch 16bit for the TX codec

Sliders: RX Latency: 110ms TX Latency : 130ms

4

Which of the three audio systems are you using? You may wish to vary both the server side and the client side (server side interacts with the radio’s USB audio source).

BTW what is the radio model?

–E
de W6EL

5

Why are you using Opus for RX and LPCM for TX? I would use Opus for both.

The different audio systems are the most important. I would use Qt if possible on both server and client, but try PA and RT to see if that makes any difference.

Phil

6

Creeping latency is also an indication that the server just can’t keep up, what server hardware are you using?

7

Elliott: I’m using pulse. rig is IC7300

Phil see below
server : Dell Precision M4400 laptop
client: Dell XPS 13 laptop

8

Dan,

Please try RT Audio and Qt Audio, on both client and server sides. Make sure to disconnect from the server before changing both sides.

Also as Phil mentioned, use the same codec for both Tx and Rx. Sometimes Opus is higher CPU usage, but on the other hand, Opus is much more tolerant of network glitches.

I would also set your sample rate to 16 khz, but you can try higher and lower.

–E
de W6EL

9

Update:
I have it stabilized sort of. On the client I’m using PortAudio and the LPCM codecs for both TX and RX.
Latency stays around 250ms which is borderline OK. It can and sometimes does exceed 500ms. At which point I just disconnect and reconnect client and that usually fixes it.

10

I’ve had similar issues as I posted on below link.
I cured it by running Qtaudio in both ends. Now it stays on 120mS, give or take 10mS, all the time.
I run Windows 11 on a decent PC. i3/i5. Dont remember what generation. But about 7 years old.

Latency build up over hours and days - Bugs - wfview

11

Started to notice this related issue myself this weekend: Lag of about 2 seconds for audio to appear on PTT after running for 30 minutes or so. This turns into about 7 seconds after 90 minutes, and at the 5 hour point, a complete failure of audio. I demonstrated this twice.

Kinda ruins any chance of a smooth FT8 session

Headless server, 7100, etc. etc.

Restarting the WFview client (windows) resets it.

Hope this is fixed in V2 :slight_smile:

12

I see this too. Quite often. But I just live with it. I have to hit the “Disconnect Radio” button in WF client and then choose “Connect” again. For me it takes about 30mins to 1hr for the issue to start appearing. I think its mainly the fact the remote WF server is on WiFi connection and that the wifi signal is only -70dBm and it could be much better. Like with a larger WiFI antenna. So check your wifi if thats what you’re using.
Would be nice if WF client could detect the RX latency at >2 secs and just self-restart the audio connection. But this may cause more problems.

13

I ran CAT6 between the Fibre Modem and the server, and have a decent switch at server-side also. I eliminated WIFI from the setup as it was a total crap-shoot.

This lag builds without any TX also - I can just listen for 30 minutes, then if i do a quick PTT or TUNE in JTDX I can see it has lost it’s responsiveness.

Yes, DISCONNECT/RECONNECT on WFview fixes it, and This can be done with a slow double-click without disturbing VAC or JTDX…

Is what it is, but it’s what it is. er…

14

I don’t personally have this issue, but it may well vanish with V2. Please make sure to try each of the three audio systems (Qt, RT, and Port). Use 16 KHz sample rate, single channel, and try Opus and LPCM. Use the same codec for both transmit and receive.

–E
de W6EL

15

@eliggett isn’t it strictly related to people who use the wfserver? I recall also seeing somewhere reports on the X6100 (and therefore X6200 too).

I also don’t have this issue as I am connected to the internal server(s) of the rig(s)

16

I have tried all combinations and permutations of systems for server & client, and the results are either a steady audio stream with building lag, or a completely useless one that glitches and drops out.

My tests for this were all using the TUNE tone in JTDX while monitoring the audio transmitted.

This is not a issue with resources on the server, or network capacity, both of them are well within acceptable limits - unless i’m missing something?

I would really like to know the exact setup that does work with no lag and no glitching, so maybe @eliggett could share this?

Not really feeling confident this issue will be fixed if it’s not been actively addressed for V2. It’s a real thing, and is quite a serious issue.

17

I haven’t seen any logs posted from you, as these will usually indicate what is causing the issue. UDP audio streaming is one of the most stressful things you can run on a network and will often expose weaknesses that you will not see in normal use. Can you describe exactly what constitutes your network and server? Things like ‘well within acceptable limits’ don’t really help us diagnose the issue. Also posting logs from both client and server to termbin will help us.

Many people are using the wfview server without this creeping latency (although we have seen other reports of it).

Phil

18

Can the SW Devs set the ToS field in the Ethernet Frames? Either that or some other QoS bit in the eth header?
Perhaps our routers at home can properly prioritize the UDP stream traffic if you set this bit.
Just a thought

19

I need to add, i noticed this while on the LAN at the server site, headless server, W10 Client, last 1.x both sides.

When i say “within acceptable limits”, i mean running mtr between the server and a local LAN connection, also a remote connection, and i see no losses or slow hops to be concerned about. In fact, since the server moved from a WiFi bridge to Cat6 to the switch, I see nothing going missing at all now. This is when running wfview standalone on the server, and nothing else bar Linux services.

I became fairly convinced any wireless jumps along the way will cause breakup and confusion in a most unpredictable manner.This was with both WfView and the Icom software - WiMAX or Wifi just trashed any hope of a reliable service for me, and as soon as i ran cable, things became predictable.

For a load test, I can stream 2 shoutcast audio streams, have 2 SSH sessions running mtr and htop, run (X11) NoMachine, all while running WFview on the server (standalone) and still mtr shows no drama in either direction, while htop will show no processors redlining, and nothing leaking. The same lag creeps in at the same rate. The server, i think, is innocent here.

I will be back at the server site (it’s 200 miles away) this evening, and i will grab some logs.

It would be really helpful if people who do not experience this issue posted their details, this has to be something simple and avoidable if there are those who do not experience it.

I am in NO way insisting on my personal workflow - I will do what it takes to make this happen, any hardware, and software.

20

Dan,

If you can supply some example C or C++ code to do this, I will take a look at it. It couldn’t hurt, and many routers and access points really do struggle with the real-time nature of what we’re doing.

That said, if there isn’t any other (or much other) traffic at the same time, it shouldn’t make much difference, correct?

—E
de W6EL