Mesh network not behaving as expected

From the failing trace, MP2 is never sending an Authentication commit frame. It is sending the first authentication commit frame in the MP2 → MG capture.

This tells me it is capable of initiating, but maybe not responding? We’re still digging through the traces, in the meantime I’d be interested to know what wpa_supplicant is reporting in the debuglogs. Looks like you’re running supplicant manually so just add -dd to your supplicant invocation.

I have captured the logs as mentioned. One in the failing state, exactly the same state as before. And one where it is directly connected to the MG.

Thanks for your continued support and I hope this helps!

Debug logs.zip (19.3 KB)

Hey @SanderV

Thanks for both logs. Comparing the two we see the successful logs getting:

nl80211: Event message available
nl80211: Drv Event 72 (NL80211_CMD_NEW_PEER_CANDIDATE) received for wlan0
nl80211: New peer candidate 00:c0:ca:b6:5c:be
wlan0: Event NEW_PEER_CANDIDATE (49) received
wlan0: new peer notification for 00:c0:ca:b6:5c:be

where the failing logs see

nl80211: Event message available
nl80211: BSS Event 59 (NL80211_CMD_FRAME) received for wlan0
nl80211: MLME event 59 (NL80211_CMD_FRAME) on wlan0(00:0a:52:09:18:ad) A1=00:0a:52:09:18:ad A2=be:d6:26:d0:92:21 on link_id=-1
nl80211: MLME event frame - hexdump(len=128): b0 00 ...
nl80211: Frame event
nl80211: RX frame da=00:0a:52:09:18:ad sa=be:d6:26:d0:92:21 bssid=be:d6:26:d0:92:21 freq=5180 ssi_signal=-100 fc=0xb0 seq_ctrl=0x120 stype=11 (WLAN_FC_STYPE_AUTH) len=128
wlan0: Event RX_MGMT (18) received
mgmt::auth
authentication: STA=be:d6:26:d0:92:21 auth_alg=3 auth_transaction=1 status_code=0 wep=0 seq_ctrl=0x120
SAE: Queue Authentication message from be:d6:26:d0:92:21 for processing (queue_len 0)
SAE: Process next available message from queue
authentication: STA=be:d6:26:d0:92:21 auth_alg=3 auth_transaction=1 status_code=0 wep=0 seq_ctrl=0x120 (from queue)
Mesh peer be:d6:26:d0:92:21 not yet known - drop Authentication frame

Now I believe the sequence of these two logs are a little different because in the successful case, MP2 is initiating the auth, whereas in the failing case MP1 is initiating the auth. However, the hint as to why MP1 is failing comes from the commit which added drop Authentication frame.

It came from upstream 09d96de - mesh: Drop Authentication frames from BLOCKED STA · MorseMicro/hostap@09d96de · GitHub. Indicating that MP1 is considered by MP2 to be BLOCKED.

The only reason I can think MP2 would block MP1 this early in the init sequence is the RSSI. Indicated in the log as ssi_signal=-100. This seems… wrong, especially as you have other logs showing the RSSI as a more sensible value.

As an experiment, can you set mesh_rssi_threshold=-110. Additionally, can you swap MP1 and MG. Does the failure follow MP1?

Thanks again and that makes sense

So I changed the RSSI threshold on MP2 to -110 and now I can ping it from MG. However, within 30 seconds after that I no longer can. I reduced the RSSI to -150 but that did not change anything. Nevertheless it seems like we are honing in on the problem!

Also, see the screenshot below taken from the web interface on MP1. We can see MG and MP2. However, MP2 shows a very low RX MCS despite having a good RSSI, but the MCS is no longer 0 for TX.

I hope this helps to narrow it down. Let me know if there is anything else I can try or if there are any other captures I can collect.

That looks like it’s actually connecting now.

The low RX MCS can just be the radio “idling”. If you were to send traffic to MP1 from MP2 the RX MCS should increase.

How was the RSSI when viewed from/to MG? I recall it being okay? RSSI issues would usually be an incorrect BCF or something going awry on the board RF traces or the antenna.

It initially connects, but after a few pings it disconnects. I am therefore also not able to start the video streaming from MP2.

From the MG I can only see MP1 which has a good RSSI, but I cannot see MP2. It is not shown in the list of stations. Unless MP2 is in direct range of MG, then it shows up and shows a correct RSSI. No problems connecting and streaming video in this case.

Any idea what might cause the disconnect after a succesful initial connect?

I didn’t pay close enough attention to your screenshot. The RSSI as measured by MP1 looks fine. Curious as to why it was reading such a low rssi on MP2, surely it isn’t asymmetrical :woozy_face:

I’m a little baffled…
More sniffer captures and wpa_supplicant logs of the disconnection are probably the best option here.

While associated, what does MP2 report the RSSI of the nodes it can see? Try watch -n1 iw dev wlan0 assoclist

I finally had some time to dive into this again and I now have a more or less functional setup.

What I didnt notice during the previous tests is that MP1 reboots about 30 seconds after a successful connect. I replaced it with another Heltec router and now it works as expected. Same device type, same power source, settings, and firmware. Heltec mentioned a reboot error in mesh mode in a previous firmware release, but I have updated both of them to the latest release already.

I purchased the Heltec routers for their low price and the fact that they can be powered over USB, but given the time I have already spent debugging them, I think I will switch to another type. Are you able to recommend one for my application?

That’s not good!

If you can afford the higher power budget, a HaLowLink 1 might be a sensible option.