Latency build up over hours and days

Bugs
1

I have written about this before, but I wanted to collect more data before posting again. This is a continuation of this thread: Really? Can no one explain how to interpret the numbers in bottom right corner? - #4 by eliggett

I’m trying to understand if the latency number can help me determine why I get this strange delay.
This is a problem I have had all time since I started with wfview close to a year ago. Across all versions and upgrades.

The fault description is like this:
Let’s say wfview has been up and idle for a few hours. If I press TX and start talking into the mic, visually I can see that the remote and the client vfview goes into TX and thus the physical radio. But I’m still having receive audio in my headphones. It can take up to 4 seconds, or in extreme cases a lot longer, before rx audio is muted in the headphones. Similarly at the end of the tx sequnce, I press receive and the physical radio switches to rx, which is confirmed visually. But audio is still muted. After up to 4 seconds, the rx audio finally comes back in my headphones.
If, I quit wfview on both sides and restart, latency is low, around 200-300 ms. The latency is consequently building up again to 1000-4000 ms until next restart.
I have caught this behavior on a screen recording. It’s rather extreme, the latency here is 21 seconds, because I intentionally left the connection up for several days to illustrate the problem. Of course I can reset the connection every so often, but that only helps to get rid of the symptoms. It’s not curing the root cause.

This is what you can see in the screen recording:
5s mark: tx is clicked. I talk into the mic. One can see that visually on both the remote and the client. But I still have receive audio in my headphones. You can see that on the sound properties window also seen in this screen capture. The level bar indicates the audio I have in the headphones.
27s mark: Receive audio mutes which you can see in the sound properties window. In wfview you can see that I’m continuing to modulate tx.
43s mark: you can see how the client and remote returns to rx. But there is still no audio in the headphones. This is confirmed in the audio settings window.
1m:04s mark: I finally get RX audio back in the headphones, which is confirmed by the audio settings window.

The second screen recording shows the same thing but at 1400ms latency.

Setup:
Client: Windows 11 PC, i5 with 16gb RAM. Ethernet to fiber modem and 100Mb speed to ISP.
Remote: Windows 11 PC, i3 with 8gb RAM. Connected to router with wifi. Wifi signal strength is
-50dBm, considered to be “a good connection” by the Wifi-analyzer app. Router connected to ISP on 100MB fiber.
Radio: IC-7300

Here are links to the screen recordings (hope the sharing works)

Long version: https://1drv.ms/v/s!Ar4cpJa3-82mx55R-eu2v3bL8bUEoQ?e=pu3kgc
Short version: https://1drv.ms/v/s!Ar4cpJa3-82mx55Qq5IyOYUYs-bDjg?e=VnePfA

Here are tembins captured directly after the screen recording
https://termbin.com/j1b7 (remote)
https://termbin.com/9i7q (client)

1 Like
2

Your packet loss seems incredibly high.

Do you have any way to test without wifi?

–E
de W6EL

3

Yes, that’s the build-up since 5 days. It is usually lower because typically I restart every day. It’s an extrem case. Normal latency after a few hours is around 1000ms. But it is a cumulative process. The longer the connection is open, the latency increases…
I can’t really use Ethernet because the radio goes bananas on TX. Schack with radio and house with fiber modem are separated by 20 meters. I have tried to cure that with just about everything in the rfi-toolbox to no end. I even dug the cable down 15 cm, but didn’t help. So, I have closed that route.
Wifi connection is outside with AP and bridge directed towards each other.

4

Signal strength is just one attribute to gauge the quality of your wireless connection. I use a private clear channel via WiFi 6 on an enterprise class (not consumer) router with excellent throughput, and latency can still cause clipping (seemingly worsening over time). I haven’t taken notes in detail but qualitatively, performance improves reseting the client-server connection periodically. Your WiFi may not be causing a problem but likely is exacerbating one. To eliminate that variable you need to replicate the issue connected via wireline. On Linux, I use netperf to load test the connection and assess network throughput and performance.

1 Like
5

I think we may have an issue with latency build-up, but it is difficult for us to replicate it as it really is caused by large packet-loss.

If you can try to improve your wifi situation it will definitely help.

–E
de W6EL

6

From the logfiles, do I have a large packet-loss?
What is an acceptable level?

7

Quality shielded Cat5e cabling should eliminate the noise and you will be much happier with performance. Something comparable to

If you are convinced that you need wireless at the transceiver, you may also consider connecting the wireless access point nearby and have it backhaul over ethernet (ideally using PoE). From my experience you will be unable to get persistently low-latent connectivity over a 20m point-to-point 2.4Ghz connection unless you have good network equipment (i.e., Meraki, Unifi, Peplink) dialed in. For the time and expense I would try wireline again with better cabling.

8

OK. Let’s say the latency is because of a not so perfect wifi connection, how can it be that the latency builds up over time? Does wfview continue to ask for lost packets in eternity? Shouldn’t there be a time-out after x number of trials?

9

OK. I get a little bit inspired to try hard wire again. That being said, I used a shielded cat6 cable last time. But not of outdoor standard. Since then, I have also replaced the old router, which may have had close to 10 years on its neck, with a new one. So, some new factors there that may change the situation.
I have a friend that installs ethernet, I may be able to buy 20 meters of outdoor cable from him. Otherwise, it’s only available in rolls of 100 meters here.
Ulf

10

It’s a bug. We’re working on it.

1 Like
11

Just the clarify @eliggett’s comment. When a packet is lost/missed the remote end will request a retransmission. Each audio packet represents 20ms of audio and I suspect that these retransmissions and causing the server to delay sending subsequent packets (by an undefined amount), thus increasing latency. If you have minimal lost/retransmitted packets, you will have minimal additional latency.

Phil

12

For starters, I don’t think that 200ms or so is “good” latency, in particular if the ISPs are doing their job right. But then I looked at my own local setup and saw about 150ms, doing a disconnect and reconnect to my IC-9700 (without restarting the Wfview application) brought the latency back down to about 45ms (still merely “not terrible” in my view).

My local network is a mix of fibre and shielded copper, 10 Gbps backbone, PC has a fibre link to the switch that the IC-9700 is plugged into with shielded ethernet cable. So no issues with WiFi or ISPs in this test case.

OK, might be something in the rig or Wfview, I’ll see whether the latency goes up when I’m on HF - but I’d be surprised if anything in this network setup is resulting in packet loss unless somehow the switch is getting RF inside it. I made the mistake of disconnecting and reconnecting without noting the loss indication in Wfview, will keep an eye on this going forward.

13

Here’s another thought. The codecs in the rig/server and client/Wfview may not be running at the same rate, resulting in a backup of audio ring buffers.

As a point of reference, Thetis (the PC-based software for the Anan SDR) has variable ratio audio processing to compensate for slightly different rates of encoding/decoding between the SDR hardware and the PC. The ratio as I look at it at the moment is about one part in one hundred thousand off unity; I do see it alternating between slightly above 1.0 and slightly below as it compensates on an ongoing basis.

Perhaps the latency increase reflects the rig-side sending audio slightly faster than the client processes it.

14

Thank you. This i very helpful to understand what is going on!
/Ulf

15

What would you say is a good benchmark to aim at? After a fresh restart I’m on 170ms.
I thought that was good… :thinking:

16

my numbers here (YMMV)

7851 to router/modem – internet – <LTE CONN, 4G> – wifi – laptop

  • my rtt is about 48 ms
  • rx latency about 76/77ms, no drop-outs.

this is bypassing wfview as server, because not needed obvously, just like the 9700

lpcm 16 bit single channel, sample rate 48000, full duplex and rt-audio

17

After restarting the IC-9700 and then doing a connect from Wfview, I see an rtt fluctuating between 0 and 1ms usually, loss of 0, rx latency starts out at 45ms but before too long (varies, maybe on the order of an hour) gets up to 145ms and stays there.

My codec’s also LPCM 1ch 16bit. Just noticed that the sliders for RX/TX latency are set to about a third of the way to what appears to be the limit of 150ms (haven’t touched them). 48000 sample rate, don’t see controls for duplex and rt-audio but I’m using VB-Audio virtual audio cables for Audio Input/Output.

18

That usually depends on what the latency setting you have set. On a decent local LAN, you can set it pretty-low (the lowest is 30ms) although you may see audio glitches this low if the machine is busy. If you have set 150ms, then it will try to keep latency below this.

73 Phil

~WRD0001.jpg

19

I’ve set the sliders down to 75ms and there’s no problem with Wfview keeping below that.

Back to addressing the original question, even when I do see packet loss (must be due to RF getting into the switch, need to put more ferrites on it) I don’t see the latency climb. That’s despite Wfview running for days at a time.

20

You aren’t likely to see the latency climbing if connected to the IC9700 server, as we have already identified that the issue is with the wfview server.

Phil