There is still quite a bit of the command handling code in wfserver, which is not needed so I may be able to reduce the CPU consumption a small amount. This is on my list to look at.
The vast majority of CPU is taken with audio encoding/resampling so one option is to set the native sample rate of Linux audio to 48K and use 16bit 48K on your client. This should passthrough the audio “practically” untouched. You should see from the server log what the current native sample rate/format is.
Are you using Opus audio? This can sometimes result in lower CPU as it sends much smaller packets.
I guess it’s all about finding balance between network bandwidth and CPU. Opus is complex and does a lot of computational things to lower the network bandwidth requirements. PCM, on the other hand, is pretty raw. Lots of bandwidth but no processing.
uLaw may be a good option since it uses a lookup table to get considerably more than 8 but dynamic range out of an 8 bit stream. Very low CPU with it.