Debugging stuck tcp connection across localhost

[Ben Greear <greearb@candelatech.com>]

Hello,

I'm working on a strange problem, and could use some help if anyone has ideas.

On a heavily loaded system (500+ wifi station devices, VRF device per 'real' netdev,
traffic generation on the netdevs, etc), I see cases where two processes trying
to communicate across localhost with TCP seem to get a stuck network
connection:

[greearb@bendt7 ben_debug]$ grep 4004 netstat.txt |grep 127.0.0.1
tcp        0 7988926 127.0.0.1:4004          127.0.0.1:23184         ESTABLISHED
tcp        0  59805 127.0.0.1:23184         127.0.0.1:4004          ESTABLISHED

Both processes in question continue to execute, and as far as I can tell, they are properly
attempting to read/write the socket, but they are reading/writing 0 bytes (these sockets
are non blocking).  If one was stuck not reading, I would expect netstat
to show bytes in the rcv buffer, but it is zero as you can see above.

Kernel is 5.15.7+ local hacks.  I can only reproduce this in a big messy complicated
test case, with my local ath10k-ct and other patches that enable virtual wifi stations,
but my code can grab logs at time it sees the problem.  Is there anything
more I can do to figure out why the TCP connection appears to be stuck?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com