I am setting up an 802.11s mesh but it is not behaving as expected in a multi‑hop scenario. Here is my setup:
Mesh gate (MG): This is either a Heltec or Alfa Halow router set up as a Mesh gate through the UI wizard. It runs a DHCP server to hand out IP addresses to all devices on the network
Laptop: It is connected to the Mesh Gate via ethernet no other router or upstream network.
Mesh Point 1 (MP1): This is a Heltec router. It is in range of the Mesh Gate via Halow.
Mesh Point 2 (MP2): This is my custom hardware. It uses a static IP. It is basically a camera that transmits video via Halow. It uses AsiaRF’s Halow module, and is controlled by an i.MX8M-Mini running Yocto.
What works
If MP1 and MP2 are both in range of the Mesh Gate, they associate directly to MG and everything works.
I can stream video from MP2 to the laptop via MG without issues.
What fails
When I move MP2 so that:
MP2 is out of range of MG,
but MP2 is in range of MP1
then:
MP2 associates to MP1 (shows up in MP1’s associated stations list),
but MG cannot ping MP2 and no traffic is forwarded through MP1.
On MG/MP1, MP2’s MAC shows up, but the UI never learns an IP for it (“?”)
Below is the wpa_supplicant_s1g file running on MP2.
I have tried a lot of different settings and configurations, but I am unable to get it to work. Perhaps my understanding of 802.11s is flawed, or somehow the hardware does not support it.
I’m definitely no mesh expert myself, but will run this by someone internally who may know more. There are a few things that stand out to me, and some things to consider in the meantime.
mesh_fwding might need to be included in the network block. Other configs I’ve seen place it there rather than the global section.
traffic will need to flow before MG builds the ARP table to map the IP to the MAC address. This could even be a DHCP exchange, but a ping will suffice (once the routing is working).
On the other nodes running OpenWrt, you can find their mesh configuration at /var/run/wpa_supplicant-wlanX.conf. It would be worth inspecting this and comparing your configuration.
The other thing I’m uncertain about is how mesh11sd may be interacting. It shouldn’t cause what you’re seeing, so have a look at the above first and then we can consider mesh11sd.
I moved the mesh_fwding property to the network block, but the behavior remains unchanged.
During testing I did notice that when I first connect MP2 to MG so it get an IP address, and then move MP2 out of range from MG, but in range of MP1 I can ping it just fine. So your theory on the ARP table seems likely.
Find below the wpa_supplicant on the routers used. The only difference between MG and MP1 here is the dot11MeshGateAnnouncements property.
Great, I found a few that might be worth applying. I won’t have time in the next couple of days, but I will check back in once I have done this with the results.
In the meantime, do you see the same behaviour if you put your custom hardware (MP2) in the middle? i.e. move MP1 around instead.
I’m curious if there is something awry with the Heltec router when it needs to forward frames in the mesh.
I replaced MP2 to also be a Heltec router. SO MP1 and MP2 are mesh point Heltec router and it works as expected. From my laptop I can ping MP2 even when it is outside the range of MG.
For you information. This is how I initialize the mesh point on my custom hardware. Maybe the static IP throws a wrench in things, but I doubt it.
#!/bin/sh
LOG=/var/log/rc.local.log
echo "rc.local starting at $(date)" >> "$LOG"
# Wait for wlan0
sleep 5
# Start S1G wpa_supplicant in background
wpa_supplicant_s1g -i wlan0 -c /etc/morse/wpa_supplicant_s1g.conf -D nl80211 -d -B \
|| echo "wpa_supplicant_s1g failed" >> "$LOG"
# Set static IP (ignore if already set)
ip addr add 192.168.3.11/24 dev wlan0 2>>"$LOG" \
|| echo "ip addr add 192.168.3.11 failed (maybe already exists)" >> "$LOG"
# Start your Python app with error logging
python3 /home/root/Main.py >> "$LOG" 2>&1 \
|| echo "python3 /home/root/Main.py failed" >> "$LOG"
echo "rc.local finished at $(date)" >> "$LOG"
exit 0
I highly doubt the IP address configuration would cause a problem here - static or dhcp shouldn’t matter. Though to be sure, on your custom hardware please share the output of ip route show to check if there is anything wrong with the routing config.
Can you also share the output of iw dev wlan0 mpath dump when in the “failing” state?
It may also be helpful to see the output of iw dev wlan0 station dump and iw dev wlan0 mpp dump
Lastly, if possible, it would be nice to see a tcpdump/wireshark capture of the wlan0 interface of your failing devixe, and concurrently perform a capture on the mesh gates interface.
Attempt to run a ping while the captures are running - this will tell us if layer 2 traffic (arps) are getting through the mesh.
Thanks @ajudge, I have captured the requested information. see below.
When connected directly to AP
root@ucm-imx8m-mini:~# ip route show
192.168.3.0/24 dev wlan0 proto kernel scope link src 192.168.3.11
root@ucm-imx8m-mini:~# iw dev wlan0 mpath dump
DEST ADDR NEXT HOP IFACE SN METRIC QLEN EXPTIME DTIM DRET FLAGS HOP_COUNT PATH_CHANGE
00:c0:ca:b6:5c:be 00:c0:ca:b6:5c:be wlan0 11 343 04092 0 0 0x15 1 1
root@ucm-imx8m-mini:~# iw dev wlan0 station dump
Station 00:c0:ca:b6:5c:be (on wlan0)
inactive time: 4 ms
rx bytes: 315762
rx packets: 3299
tx bytes: 18329
tx packets: 87
tx retries: 5
tx failed: 0
rx drop misc: 7
signal: -69 dBm
signal avg: -67 dBm
Toffset: 445697566 us
tx bitrate: 39.0 MBit/s VHT-MCS 4 VHT-NSS 1
tx duration: 0 us
rx bitrate: 52.0 MBit/s VHT-MCS 5 VHT-NSS 1
rx duration: 0 us
expected throughput: 23.253Mbps
mesh llid: 0
mesh plid: 0
mesh plink: ESTAB
mesh airtime link metric: 343
mesh connected to gate: yes
mesh connected to auth server: no
mesh local PS mode: ACTIVE
mesh peer PS mode: ACTIVE
mesh non-peer PS mode: ACTIVE
authorized: yes
authenticated: yes
associated: yes
preamble: long
WMM/WME: yes
MFP: yes
TDLS peer: no
DTIM period: 1
beacon interval:1000
connected time: 1598 seconds
associated at [boottime]: 18.762s
associated at: 1616581532375 ms
current time: 1616583127892 ms
root@ucm-imx8m-mini:~# iw dev wlan0 mpp dump
DEST ADDR PROXY NODE IFACE
84:69:93:68:02:f8 00:c0:ca:b6:5c:be wlan0
root@ucm-imx8m-mini:~#
In the failing state, so 2 hops away from AP
root@ucm-imx8m-mini:~# ip route show
192.168.3.0/24 dev wlan0 proto kernel scope link src 192.168.3.11
root@ucm-imx8m-mini:~# iw dev wlan0 mpath dump
iw dev wlan0 station dump
DEST ADDR NEXT HOP IFACE SN METRIC QLEN EXPTIME DTIM DRET FLAGS HOP_COUNT PATH_CHANGE
root@ucm-imx8m-mini:~# iw dev wlan0 station dump
root@ucm-imx8m-mini:~# iw dev wlan0 mpp dump
DEST ADDR PROXY NODE IFACE
root@ucm-imx8m-mini:~#
In the attached ZIP are the captures on the laptop connected to the mesh gate and the failing device MP2. I ran a ping from the laptop to MP2. The capture on MP2 is essentially empty. I did occasionally see the message below, but not during the capture:
Apologies, in that test I only booted it up while already out of range. So here it is again.
First I connected MP2 and MP1 while to the MG while they are both in range of MG. I then move away such that MP2 can only see MP1 and MP1 can see MG and MP1. I get successful ping, see below:
root@ucm-imx8m-mini:~# ping 192.168.3.201
PING 192.168.3.201 (192.168.3.201): 56 data bytes
64 bytes from 192.168.3.201: seq=2 ttl=64 time=312.702 ms
64 bytes from 192.168.3.201: seq=3 ttl=64 time=158.104 ms
64 bytes from 192.168.3.201: seq=4 ttl=64 time=32.746 ms
64 bytes from 192.168.3.201: seq=5 ttl=64 time=33.962 ms
64 bytes from 192.168.3.201: seq=6 ttl=64 time=32.856 ms
^C
--- 192.168.3.201 ping statistics ---
7 packets transmitted, 5 packets received, 28% packet loss
round-trip min/avg/max = 32.746/114.074/312.702 ms
root@ucm-imx8m-mini:~# ip route show
192.168.3.0/24 dev wlan0 proto kernel scope link src 192.168.3.11
iw dev wlan0 mpath dump
root@ucm-imx8m-mini:~# iw dev wlan0 mpath dump
DEST ADDR NEXT HOP IFACE SN METRIC QLEN EXPTIME DTIM DRET FLAGS HOP_COUNT PATH_CHANGE
5a:6e:0f:98:85:d4 5a:6e:0f:98:85:d4 wlan0 3 1546 0 0 0 0 0x14 1 1
00:c0:ca:b6:5c:be 5a:6e:0f:98:85:d4 wlan0 13 3956 0 3816 0 0 0x15 2 4
iw dev wlan0 station dump
root@ucm-imx8m-mini:~# iw dev wlan0 station dump
Station 5a:6e:0f:98:85:d4 (on wlan0)
inactive time: 1196 ms
rx bytes: 90774
rx packets: 922
tx bytes: 1740
tx packets: 17
tx retries: 0
tx failed: 0
rx drop misc: 10
signal: -102 dBm
signal avg: -102 dBm
Toffset: 31082083 us
tx bitrate: 6.5 MBit/s VHT-MCS 0 VHT-NSS 1
tx duration: 0 us
rx bitrate: 14.4 MBit/s VHT-MCS 1 short GI VHT-NSS 1
rx duration: 0 us
expected throughput: 5.126Mbps
mesh llid: 0
mesh plid: 0
mesh plink: ESTAB
mesh airtime link metric: 1546
mesh connected to gate: no
mesh connected to auth server: no
mesh local PS mode: ACTIVE
mesh peer PS mode: ACTIVE
mesh non-peer PS mode: ACTIVE
authorized: yes
authenticated: yes
associated: yes
preamble: long
WMM/WME: yes
MFP: yes
TDLS peer: no
DTIM period: 1
beacon interval:1000
connected time: 425 seconds
associated at [boottime]: 18.575s
associated at: 1616581532410 ms
current time: 1616581953894 ms
Station 00:c0:ca:b6:5c:be (on wlan0)
inactive time: 81824 ms
rx bytes: 67088
rx packets: 680
tx bytes: 5408
tx packets: 42
tx retries: 64
tx failed: 9
rx drop misc: 31
signal: -108 dBm
signal avg: -106 dBm
tx bitrate: 6.5 MBit/s VHT-MCS 0 VHT-NSS 1
tx duration: 0 us
rx bitrate: 72.2 MBit/s VHT-MCS 7 short GI VHT-NSS 1
rx duration: 0 us
mesh llid: 0
mesh plid: 0
mesh plink: ESTAB
mesh airtime link metric: 1705
mesh connected to gate: yes
mesh connected to auth server: no
mesh local PS mode: ACTIVE
mesh peer PS mode: ACTIVE
mesh non-peer PS mode: ACTIVE
authorized: yes
authenticated: yes
associated: yes
preamble: long
WMM/WME: yes
MFP: yes
TDLS peer: no
DTIM period: 1
beacon interval:1000
connected time: 424 seconds
associated at [boottime]: 18.743s
associated at: 1616581532578 ms
current time: 1616581953894 ms
root@ucm-imx8m-mini:~# iw dev wlan0 mpp dump
DEST ADDR PROXY NODE IFACE
00:c0:ca:b6:5c:bc 00:c0:ca:b6:5c:be wlan0
root@ucm-imx8m-mini:~# ping 192.168.3.201
PING 192.168.3.201 (192.168.3.201): 56 data bytes
64 bytes from 192.168.3.201: seq=0 ttl=64 time=47.146 ms
64 bytes from 192.168.3.201: seq=1 ttl=64 time=28.906 ms
^C
--- 192.168.3.201 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 28.906/38.026/47.146 ms
Leaving everything in the exact same location I reboot MP2 and leave it for a few mins. It does not reconnect and I get this:
root@ucm-imx8m-mini:~# ping 192.168.3.201
PING 192.168.3.201 (192.168.3.201): 56 data bytes
^C
--- 192.168.3.201 ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss
root@ucm-imx8m-mini:~# ip route show
192.168.3.0/24 dev wlan0 proto kernel scope link src 192.168.3.11
iw dev wlan0 mpath dump
root@ucm-imx8m-mini:~# iw dev wlan0 mapath dump
DEST ADDR NEXT HOP IFACE SN METRIC QLEN EXPTIME DTIM DRET FLAGS HOP_COUNT PATH_CHANGE
iw dev wlan0 station dump
root@ucm-imx8m-mini:~# iw dev wlan0 station dump
root@ucm-imx8m-mini:~# iw dev wlan0 mpp dump
DEST ADDR PROXY NODE IFACE
root@ucm-imx8m-mini:~# ping 192.168.3.201
PING 192.168.3.201 (192.168.3.201): 56 data bytes
^C
--- 192.168.3.201 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
As for the Heltec mesh11sd file it looks like this:
It looks like at the edge there the device is getting RSSI values of <-100 dBm. Your Heltec gateways are configured with a mesh_rssi_threshold of -85. I doubt these devices would boot you off the mesh as you move towards the edge, but once you’ve rebooted, they’re likely not letting you back on due to the rssi threshold.
Try reducing the RSSI threshold on the Heltec gateways to -110dB, and see if it resolves your issue in the same spot.
In saying that, I wouldn’t recommend an rssi threshold that low in a real deployment. You want to try to maximise the rate the links are operating to claim as much bandwidth as possible, as your link throughput will halve through each hop. But it’s a good experiment nonetheless.
Another test is to move the device to a location that it is with the -85 rssi threshold of MP1, but out of the range of MG. Does the behaviour you were seeing originally still exhibit in this instance?
I see. I assumed that was the signal strength of MG that MP2 sees. MP2 is only a few meters away from MP1 with only minor obstruction, so I doubt that that is its RSSI.
Anyway, I will check once I am back on site in about two weeks.
You are correct. I’m just assuming the signal strength is bidirectional (because it usually is). Do check your config for the rssi threshold of MP2. It will also reject joining mesh peers with low rssi.
If it’s possible, I’d be interested knowing the signal strength of MP2 that MP1 sees in that state.
For only a couple of meters away, I’m surprised by that rssi. It should be higher.
Below is a screenshot of the connected devices on MP1. We can see MG with a similar RSSI. And then below is MP2 with a good RSSI, but no IP address and MCS 0.
An RX rate of 0 MHz is pretty suspicious, I wonder if it hasn’t actually completed the authentication/association process (OpenWrt can show devices in the associated stations list early during the association process). The reported TX rate also looks invalid. This would align with you seeing it in OpenWrt, but nothing in the mesh paths or station dump commands.
I’d be interested to know what’s actually going on in the air. If you have a fourth device, you can put it into monitor mode (see https://www.morsemicro.com/resources/appnotes/MM_APPNOTE-36_How_to_do_a_Sniffer_Capture.pdf) and configure it for the same channel as the mesh (morse_cli -i wlan0 channel -o 1 -p 1 -c 904500 -n 0) , then boot your problematic device. Specifically wanting to capture the authentication and association process here.
root@HT-H7608-F017:~# iw dev wlan0 del
root@HT-H7608-F017:~# iw phy phy1 interface add mon0 type monitor
root@HT-H7608-F017:~# ifconfig mon0 up && ifconfig morse0 up
root@HT-H7608-F017:~# morse_cli -i mon0 channel -o 1 -p 1 -c 904500 -n 0
Full Channel Information
Operating Frequency: 904500 kHz
Operating BW: 1 MHz
Primary BW: 1 MHz
Primary Channel Index: 0
root@HT-H7608-F017:~#
I placed the sniffer close to MP1 so it is in range of all the devices. Attached is the pcap file. MAC addresses are MG (02:c0:ca:b6:5c:be), MP1 (e4:38:19:1f:f0:57) and MP2 (00:0a:52:09:18:ad).
I also did a test where MP2 is in range of MG to show what it looks like when they do connect. Followed by some Pings to confirm.
