MM8108 mesh multicast loss at light load — help distinguishing PHY errors from firmware drops

Software
1

Hi Morse Micro team,

We’re troubleshooting unreliable multicast delivery on a three-node OpenMANET mesh, intended primarily for PTT/voice and ATAK/CoT. We can reproduce loss below the host networking stack, but haven’t identified whether it comes from RF decoding, receive availability, or internal firmware behavior. We’d appreciate guidance on the next useful diagnostic.

For transparency, we used OpenAI Codex to assist with troubleshooting: reviewing driver source, writing and running test scripts, collecting captures and counters, and analyzing results. These measurements came from our physical radios, but the test tooling and interpretations were AI-assisted and may contain mistakes. We can provide the underlying captures, scripts, and logs for independent review; we are not presenting the suspected firmware involvement as a confirmed diagnosis.

Setup

  • Three Gateworks M.2 MM8108 cards, connected over USB, running OpenMANET 24.10 / 1.8.1, Linux 6.6.138.
  • Morse driver package 2.0.0-r2; firmware and CLI report rel_mm8108_2_0_0_2026_Apr_21.
  • Firmware and loaded boardtype 0807 BCF hashes match the published 2.0 files and bcf_mf15457.bin.
  • US channel 42: 923 MHz, 2 MHz operating bandwidth, normally 1 MHz primary bandwidth.
  • SAE mesh encryption, Morse rate control, enable_mcast_rate_control=1, multicast whitelist disabled, hardware crypto enabled, TX queue limit 256.
  • BATMAN handles forwarding; native 802.11s forwarding is disabled.
  • Radios are indoors and relatively close, with node 2 in another room. Antennas were checked/corrected before the selected comparisons.

An earlier node 2 outage was traced to a dead battery; we are not treating that as a firmware failure.

How we tested

We used two complementary workloads:

  1. Normal mesh multicast: generated voice-shaped UDP and synthetic CoT messages through the mesh, with simultaneous captures on wlh0, bat0, and br-ahwlan.
  2. Single-copy radio broadcasts: tagged raw Ethernet frames sent directly through wlh0, bypassing BATMAN forwarding, to isolate the underlying group-frame delivery.

A typical short radio test sends 500 voice-sized and 100 CoT-sized frames over 10 seconds. We matched packet IDs across captures, checked capture drops and boot continuity, and collected Morse CLI statistics.

We also captured USB traffic at the source and receiver. Source traces let us associate tagged submissions with USB completion and firmware TX-status records, including reported MCS/bandwidth/GI. We understand that a successful unacknowledged TX status does not prove successful reception or independently prove RF emission.

Main observations

Single-copy radio broadcasts commonly lose roughly 10–20% of frames. Normal mesh repetition recovers some losses, but message delivery remains variable.

A controlled SAE/open/SAE comparison held the source’s reported data rate at MCS2, 2 MHz, long GI for every tagged frame:

Configuration Node 2 missing Node 3 missing
SAE 104/600 — 17.3% 70/600 — 11.7%
Open 93/600 — 15.5% 67/600 — 11.2%
SAE restored 93/600 — 15.5% 65/600 — 10.8%

Transitions used cold boots. This does not exclude every encryption-related effect, but encryption is clearly not required to reproduce the loss.

In a receiver USB observation, node 2 received 487 of 600 tags. Those 487 exactly matched its host capture; the other 113 were already absent at the receiver USB boundary. All 600 source requests had USB completion and matching firmware TX status.

In a separate passive capture, we temporarily used node 3 as a monitor through the driver’s morse0 interface. We matched 90 of 109 frames missed by node 2 to observed source 802.11 headers without a reported BADFCS flag. Matching used addresses, sequence/fragment control, frame-control bits, and QoS TID. The capture snaplength did not permit independent FCS verification for every frame. This supports actual RF transmission for many packets that node 2 did not deliver. That monitor capture used SAE and a 192-byte snaplength; we have not yet combined open mode with a full-length over-the-air capture. Using node 3 as the monitor also changes the mesh topology.

Channel utilization

Using the firmware GET_CHANNEL_USAGE command, we measured approximately:

Three-node workload Receiver busy/listen ratio Message loss
Background only 2.4% —
5 voice + 1 CoT messages/sec 4–5% 0.6–1.1%
50 voice + 10 CoT messages/sec 15% ~6.3%
Same workload, fixed MCS3 17–20% ~18%

These are receiver-observed busy/listen ratios, not complete airtime accounting, particularly for each node’s own transmissions. We don’t see evidence of continuous channel saturation, although brief collisions or receive-unavailable periods remain possible.

Changes tried

We tested the following in bounded trials and restored baseline settings afterward:

  • Multicast rate control enabled/disabled.
  • Fixed rates, including MCS0, MCS2, and MCS3 in different comparisons.
  • 1 MHz versus 2 MHz transmitted data bandwidth.
  • Long versus short guard interval.
  • 1 MHz versus 2 MHz primary-channel width, holding tagged data at 2 MHz.
  • Reduced source power, including fixed-rate comparisons.
  • An alternate operating channel.
  • TX queue limit 256 versus 32.
  • CTS-to-self/RTS settings.
  • Receiver software crypto and an all-node open mesh.
  • Packet-spacing control.
  • Travelling pilots in a separate unicast control.
  • Suppression of receiver host-generated traffic to investigate contention.

None established a reliable mesh fix. Some effects were mixed: lower source power improved node 2 while worsening node 3; 1 MHz data frames improved raw broadcast delivery in one comparison series but worsened actual mesh-message delivery. We are not treating either as a recommended configuration.

Receive-error diagnostics

Signal-header CRC errors increase during the tests. In one run, node 2 accumulated 122 additional signal-header CRC errors while missing 113 tagged frames. These are aggregate counters including background traffic, so we cannot equate the counts or assign errors to individual tags.

The matching driver implements MORSE_CMD_ID_EVT_SIG_FIELD_ERROR (0x4003) with start/end timestamps. However, a complete 60-second normal-mode receiver USB trace captured ordinary CLI responses but no such events, despite the increasing error counter.

Questions

  1. Is there a supported way to enable per-reception PHY/SIG/FCS error reporting on production MM8108 2.0 firmware? Does 0x4003 require a special mode or diagnostic build?
  2. Which counters or traces best distinguish failed decoding, half-duplex receive unavailability, internal filtering, and firmware buffering/drop behavior?
  3. Are there known MM8108 2.0 issues matching group-frame loss at modest utilization, including strong-signal or 2 MHz reception conditions?
  4. Is a newer matched driver/firmware release recommended for this specific behavior, and is there a relevant fix we should test?
  5. What minimal controlled experiment would you recommend next?

We can provide packet captures, tagged USB/TX-status traces, before/after counters, firmware/BCF hashes, and reproduction scripts. We have not established a root cause and would particularly appreciate help obtaining evidence that distinguishes RF failure from internal firmware loss.

Morse driver settings and boot messages

We checked dmesg and the live /sys/module/morse/parameters/ values on all three nodes. The selected settings below match across all nodes.

Representative boot messages:

channelization_scheme = 3 (IEEE802.11-REVmf)
morse micro driver registration. Version 0-rel_mm8108_2_0_0_2026_Apr_21
Loaded firmware from morse/mm8108b2-rl.bin, size 488704, crc32 0xd5047b32
Loaded BCF from morse/bcf_boardtype_0807.bin, size 2510, crc32 0xc8af8c0d

Current runtime module parameters:

enable_mcast_rate_control       = Y
enable_mcast_whitelist          = N
enable_fixed_rate              = N
enable_sgi_rc                  = Y
enable_trav_pilot              = Y
enable_subbands                = 2
mcs_mask                       = 1023
mcs10_mode                     = 0
max_rate_tries                 = 1
max_rates                      = 4

max_txq_len                    = 256
max_aggregation_count          = 0
enable_amsdu_override          = -1
max_mc_frames                  = -1
tx_queued_lifetime_ms          = 1000
tx_status_lifetime_ms          = 15000

no_hwcrypt                     = N
enable_pv1                     = N
enable_cts_to_self             = N
enable_rts_8mhz                = N

enable_ps                      = 0
enable_pre_assoc_ps            = N
enable_dynamic_ps_offload      = N
enable_twt                     = N

enable_airtime_fairness        = 0
enable_auto_mpsw               = Y
enable_auto_duty_cycle         = Y
duty_cycle_mode                = 0
tx_max_power_mbm               = 0

A few interpretation details:

  • The boot log prints max_txq_len=32, while the current runtime value is 256 on all three nodes. Our test reports distinguish the queue configurations actually used.
  • enable_airtime_fairness=0 selects the driver’s kernel-dependent default; it does not mean airtime fairness is disabled.
  • no_hwcrypt=N means hardware crypto is enabled.
  • tx_max_power_mbm=0 is the module parameter readback, not a measured RF output of 0 dBm.
  • enable_auto_mpsw=Y does not establish active packet spacing; separate CLI baseline readbacks showed MPSW inactive.
2

Hey @73challenger

Appreciate your patience here, there is a bit to digest.

0x4003 should not require a special mode or diagnostic build. For short test builds, consider setting the debug_mask module parameter which will give you more output as event.c logs 0x4003 with

MORSE_DBG(mors, "Sig field error %llu - %llu\n",
				sig_field_error_evt->start_timestamp,
				sig_field_error_evt->end_timestamp);

The best stats to capture come straight from the firmware using morse_cli stats. Capture these stats periodically during your tests and share them here - or hand them to codex to infer what might be going on, the AI tools do a pretty good job of digesting the morse_cli stats output.

How strong is the “strong-signal”? If your devices are close together you can overload the AGC which would be consistent with CRC errors. You mentioned you tried reducing power, but can you also try separating devices by a couple of meters if you haven’t already.

2.1.0 did change some RX path handling which might help here…

As mentioned, it might be worth trying 2.1.0 here.

As above, I’d be collecting morse_cli stats periodically. Before capturing, give the stats a reset with morse_cli stats -r.