Mmc1: Timeout waiting for hardware interrupt

Software
1

Hi everyone!

For a project I am intefacing a halow module with an i.MX8M-Mini. The Halow Module is AsiaRF’s MM610x module, and I am using Compulab’s eval board for their SoM containing an i.MX8M-Mini.

We have compiled the driver and integrated it into the Yocto build provided by Compulab.

Regarding the HW I used an SD card breakout to connect the module to the eval board’s uSDHC2 interface. At first I suspected signal integrity issues, but I have reduced the frequency to 1 MHz and shortened the wires significantly, but the error remains. See photo of the setup below.

image
image964×1280 135 KB

However, I am now stuck on the error below. When I run chipreset.sh it seems to recognize the the SDIO device just fine, but then there is a timeout waiting for a HW interrupt. This entire sequence occurs at 1 MHz, so if it were signal integrity issues I would expect it to fail sooner.

root@ucm-imx8m-mini:~# [43665.118793] mmc1: new high speed SDIO card at address 0001
[43665.129791] Morse Micro Dot11ah driver registration. Version 0-rel_1_12_4_2024_Jun_11
[43665.162614] morse micro driver registration. Version 0-rel_1_12_4_2024_Jun_11
[43665.170211] morse_sdio mmc1:0001:1: sdio new func 1 vendor 0x325b device 0x306 block 0x8/0x8
[43665.179453] morse_sdio mmc1:0001:2: sdio new func 2 vendor 0x325b device 0x306 block 0x200/0x200
[43665.189669] morse_sdio mmc1:0001:2: Reading gpio pins configuration from device tree
[43665.200730] morse_sdio mmc1:0001:2: Loaded firmware from morse/mm6108.bin, size 433044, crc32 0x436ba524
[43665.210320] morse_sdio mmc1:0001:2: Loaded BCF from morse/bcf_default.bin, size 1077, crc32 0x1cbf23cb
[43675.740084] mmc1: Timeout waiting for hardware interrupt.
[43675.745496] mmc1: sdhci: ============ SDHCI REGISTER DUMP ===========
[43675.751937] mmc1: sdhci: Sys addr:  0x48f23000 | Version:  0x00000002
[43675.758376] mmc1: sdhci: Blk size:  0x00000200 | Blk cnt:  0x0000001f
[43675.764816] mmc1: sdhci: Argument:  0xac000080 | Trn mode: 0x00000023
[43675.771254] mmc1: sdhci: Present:   0x01f8818e | Host ctl: 0x00000013
[43675.777691] mmc1: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[43675.784128] mmc1: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
[43675.790566] mmc1: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[43675.797004] mmc1: sdhci: Int enab:  0x117f100b | Sig enab: 0x117f100b
[43675.803441] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000502
[43675.809878] mmc1: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000b400
[43675.816315] mmc1: sdhci: Cmd:       0x0000353a | Max curr: 0x00ffffff
[43675.822752] mmc1: sdhci: Resp[0]:   0x00000000 | Resp[1]:  0x00000000
[43675.829189] mmc1: sdhci: Resp[2]:   0x00000000 | Resp[3]:  0x00000000
[43675.835625] mmc1: sdhci: Host ctl2: 0x00000000
[43675.840065] mmc1: sdhci: ADMA Err:  0x00000007 | ADMA Ptr: 0x48f2320c
[43675.846502] mmc1: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP =========
[43675.854068] mmc1: sdhci-esdhc-imx: cmd debug status:  0x2100
[43675.859723] mmc1: sdhci-esdhc-imx: data debug status:  0x2220
[43675.865465] mmc1: sdhci-esdhc-imx: trans debug status:  0x2328
[43675.871295] mmc1: sdhci-esdhc-imx: dma debug status:  0x2400
[43675.876950] mmc1: sdhci-esdhc-imx: adma debug status:  0x25b4
[43675.882692] mmc1: sdhci-esdhc-imx: fifo debug status:  0x2680
[43675.888435] mmc1: sdhci-esdhc-imx: async fifo debug status:  0x2750
[43675.894697] mmc1: sdhci: ============================================
[43675.901791] morse_sdio mmc1:0001:2: sdio_memcpy_toio failed: 18446744073709551506
[43675.909298] morse_sdio mmc1:0001:2: morse_sdio_dm_write failed -110
[43676.544356] morse_sdio mmc1:0001:2: sdio writel failed 4294967280
[43676.550509] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551600
[43677.616485] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[43677.623206] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[43677.629058] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[43678.265609] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[43678.272338] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[43678.278226] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[43678.914872] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[43678.921592] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[43678.927472] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[43679.984083] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[43679.990803] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[43679.996689] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[43680.633412] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[43680.640136] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[43680.646023] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[43680.654308] morse_sdio mmc1:0001:2: morse_firmware_init failed: -5
[43681.288974] morse_sdio_probe failed. The driver has not been loaded!
[43681.295372] morse_sdio: probe of mmc1:0001:2 failed with error -5

Please see below the relevant part of the device tree:

&usdhc2 {
	#address-cells = <1>;
    	#size-cells = <0>;
	assigned-clocks = <&clk IMX8MM_CLK_USDHC2>;
	assigned-clock-rates = <1000000>;
	//clock-frequency = <400000>;
	pinctrl-names = "default"; 
	//, "state_100mhz", "state_200mhz";
	pinctrl-0 = <&pinctrl_usdhc2>, <&pinctrl_usdhc2_gpio>;
	//pinctrl-1 = <&pinctrl_usdhc2_100mhz>, <&pinctrl_usdhc2_gpio>;
	//pinctrl-2 = <&pinctrl_usdhc2_200mhz>, <&pinctrl_usdhc2_gpio>;
	bus-width = <4>;
	vmmc-supply = <&reg_usdhc2_vmmc>;
	status = "okay";
	/delete-property/ cd-gpios;
	keep-power-in-suspend;
	non-removable;
	wakeup-source;
	cap-sd-highspeed;
	cap-mmc-highspeed;
	disable-wp;
	cap-sdio-irq;
	fsl,sdio-interrupt-enabled;
	
	mm6108_sdio: mm6108_sdio@0 {
		//#address-cells = <2>;
         	//#size-cells = <1>;
		compatible = "morse,mm610x";
		//assigned-clock-rates = <5000000>;
		reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
		power-gpios = <&gpio3 24 GPIO_ACTIVE_LOW>, <&gpio3 20 GPIO_ACTIVE_HIGH>;
		status = "okay";
		reg = <2>;
		bus-width = <4>;
	};
};

Any suggestions on what to try next are much appreciated! mmc1: Timeout waiting for hardware interrupt

1 Like
4

Hi Sander,

I believe we’ve seen something similar on an iMX6. While not the same hardware, we’ll take a look at that first to see if we can reproduce the problem.
In the meantime, are you able to share a logic analyser capture of the SDIO lines?

2 Likes
5

Thanks! that would be very helpful!

See below a logic analyzer capture when running chipreset.sh.

The CLK rate is 1 MHz. Let me know if you would like to see any parts in more detail.

image
image1704×802 158 KB

6

Hi, just checking in to see if you have had any luck reproducing the issue?

Let me know if there is anything I can do!

7

Hi Sander, thanks for sending the logic analyser trace through! We’re still investigating and hope to have something concrete for you within the next week!

While we’re looking at the iMX6 hardware we have on hand, we’re also investigating the software/hardware configuration of the Compulab evaluation kit you are using. Unfortunately, we are having some issues downloading ucm-imx8m-mini_yocto-linux_2023-07-26.zip from Compulab. Are you able to verify that the bsp metalayer included in that package is identical to the following GitHub repository: GitHub - compulab-yokneam/meta-bsp-imx8mm: BSP for CompuLab SOMs iMX8M-mini?

I ask because there is a hint online that the issue may be related to an improper clock sequencing (or rather, a parent clock being disabled on account of no consumer claiming the child clock). This looks like it is fixed in 5.15.32, and while I suspect it would be causing issues with eMMC or the on board WiFi, it is an area we would be exploring first once our iMX6 is up and running.

For that kernel linked above, we also note that the fsl-imx-esdhc binding, containing the fsl-imx8mm-usdhc compatible, defines an fsl,sdio-async-interrupt-enabled property to enable DAT1 interrupt handling. It’s possible you may just need to simply update the fsl,sdio-interrupt-enabled accordingly.

Additionally, we would like to do further analysis on the traces you’ve captured. It looks like you’re using Logic, are you able to save the capture to a file and share the capture file so we can analyse the traces carefully?

Finally, it may be worth looking at the analog signal of DAT1, to ensure nothing else is impacting the line.

Hope something in here helps, and please let us know if anything does. We haven’t been able to try the above on our own hardware yet due to some minor build system setup issues we’re resolving first!

8

Hi! that is great to hear! Thanks for the suggestions!

Yes those are indeed the same. We started out with the latest 5.15.32, but later reverted to 5.10.35. However, we received the same error in both versions.

I tried the fsl,sdio-async-interrupt-enabled property, but unfortunately the error persists.

I am indeed using Logic. I have the capture file, but I am not allowed to attach it to this message. Not sure if it is against policy to share an email address here. I have just reached through LinkedIn.

Please see below a capture of DAT1. There is a little overshoot, but overall the signal looks quite clean to me.

image
image1101×687 65.9 KB

Let me know if anything else comes to mind I can try out!

9

Please find the capture file attached

chipreset on imx.sal (164.8 KB)

10

Hi Sander,

Thanks for your patience with us finding a resolution for you. We’ve had a few unrelated hardware issues with our iMX6 dev board which we will have resolved shortly and will hopefully be able to confirm a reproduction of the problem.

Reproduction aside, I have taken a look at your logic analyser output. There are some potentially unexpected CMD42s going from Host to Device. These look like SD Memory specific commands (as opposed to SDIO, see this link). I’m confirming with our digital teams on exactly how the MM6108 will react to this specific command. In the meantime, it would be worth disabling these commands and checking the outcome. You should be able to disable these commands by providing the no-sd device tree property to the usdhc2 node of your device.

Some digging through the mailing lists led me to the following thread: imx7d: Timeout waiting for hardware interrupt.
This thread helpfully defines the ADMA error you are seeing as a length mismatch. While there are a few commits mentioned in regards to that length mismatch, I can see that they are present in the 5.15.32 kernel and so unlikely to be your exact problem. However, it is worth noting that the Morse Micro Linux driver does have some alignment configuration which might be related. CONFIG_MORSE_SDIO_ALIGNMENT can be set at compile time of the driver to adjust the requested alignment between 2 and 8 bytes.

Finally, while your oscilloscope capture does show some IO operating at 3.3V, it might be worth trying with the no-1-8-v property set on your usdhc2 node in case the host controller has decided to switch to 1.8V.

Once our board is running, we will be trying the above options as well!

EDIT:
On closer inspection, the CMD42s I am seeing in your trace look like they may be misinterpreted command responses to CMD53. For high speed capable cards, there is an output delay between the rising edge, and the data transition of maximum 14ns. This will be lost in the resolution of the 12 MS/s rate of the logic analyser. Are you able to sample at a higher frequency?

11

Thanks for the continued support!

So I have included the no-sd property, but the error persists. Adding the no-1-8-v property did not change the behaviour.

Next, I performed another capture with the logic analyzer. see attached. I have increased the sample rate to 24MS/s. However, the CLK rate is set to only 1 MHz, so I do not expect this to make a difference.

I do still see the CMD42 commands in the capture. It should be noted that Logic does not support SDIO natively. I have used a library that I found online and incorporated it in the software.

Not sure if it is useful, but I have also performed a capture on a Raspberry Pi with the AsiaRF hat attached. See attached. Here I do not see the CMD42 command. So we may be on to something.

chipreset on rpi.sal (4.5 MB)

chipreset on imx_no-sd_24MSs.sal (749.3 KB)

12

I can confirm that we were able to reproduce this problem on a Variscite iMX6 board we have available to us. In the hope that it proves beneficial to you, I’m providing as much data as I can of our own debugging of the problem. However, we haven’t reached a conclusion as to what is causing the issue. Note that on our iMX6 boards we did not apply the dts properties no-1-8-v and no-sd.

See below, the error log on our iMX6 board, resembling yours:

[    8.429265] morse_sdio mmc0:0001:2: Loaded BCF from morse/bcf_mf08651_us.bin, size 1003, crc32 0xaf924dcf
[   19.049032] mmc0: Timeout waiting for hardware interrupt.
[   19.054532] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[   19.061016] mmc0: sdhci: Sys addr:  0x9c070004 | Version:  0x00000002
[   19.067503] mmc0: sdhci: Blk size:  0x00000200 | Blk cnt:  0x0000001b
[   19.073990] mmc0: sdhci: Argument:  0xac00001b | Trn mode: 0x00000023
[   19.080476] mmc0: sdhci: Present:   0x01f8858e | Host ctl: 0x00000013
[   19.086963] mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[   19.093447] mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000002f
[   19.099930] mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[   19.106415] mmc0: sdhci: Int enab:  0x117f100b | Sig enab: 0x117f100b
[   19.112901] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000302
[   19.119386] mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000b407
[   19.125872] mmc0: sdhci: Cmd:       0x0000353a | Max curr: 0x00ffffff
[   19.132356] mmc0: sdhci: Resp[0]:   0x00000000 | Resp[1]:  0x00000000
[   19.138841] mmc0: sdhci: Resp[2]:   0x00000000 | Resp[3]:  0x00000000
[   19.145323] mmc0: sdhci: Host ctl2: 0x00000000
[   19.149804] mmc0: sdhci: ADMA Err:  0x00000007 | ADMA Ptr: 0x9c070200
[   19.156292] mmc0: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP =========
[   19.163899] mmc0: sdhci-esdhc-imx: cmd debug status:  0x3100
[   19.169604] mmc0: sdhci-esdhc-imx: data debug status:  0x3220
[   19.175389] mmc0: sdhci-esdhc-imx: trans debug status:  0x3328
[   19.181264] mmc0: sdhci-esdhc-imx: dma debug status:  0x3402
[   19.186963] mmc0: sdhci-esdhc-imx: adma debug status:  0x35b4
[   19.192749] mmc0: sdhci-esdhc-imx: fifo debug status:  0x368c
[   19.198535] mmc0: sdhci-esdhc-imx: async fifo debug status:  0x3750
[   19.204843] mmc0: sdhci: ============================================
[   19.212949] morse_sdio mmc0:0001:2: sdio_memcpy_toio failed: 4294967186
[   19.220629] morse_sdio mmc0:0001:2: morse_sdio_dm_write failed -110

We’ve also captured some logic analyzer traces with our 250MS/s logic analyser to compare the bring up with our EKH01. Unfortunately the actual trace files are a bit large, so instead I’ve attached the data exported as a csv which should be sufficient. Unfortunately these csv files haven’t exported the full ARG value, which makes decoding a little bit difficult.
ekh01-boot-7-11-2024.csv (165.7 KB)
imx6-boot.csv (70.5 KB)

These captures do show a small difference in the iMX trying to trigger a voltage switch (CMD11). I expect that this wouldn’t be present with the no-1-8-v property applied.
Otherwise, there isn’t substantial difference in the CMD flow. We will continue to decode the arguments in these captures to confirm the host isn’t putting the chip into an unsupported state.

Noting ADMA errors in the log, we’ve tried to disable DMA by setting various SDHCI_QUIRKS both through the debug_quirks sdhci module parameter, and in the esdhc-imx driver source. Unfortunately, these didn’t give us anything useful, nor did they appear to disable ADMA.

We have several more things to try! In the mailing lists I’ve linked before, these timeout related error messages often appear to be related to parent clock configuration. Looking at the iMX6 device tree files which enable Wi-Fi on the Variscite SOM, attached, the next thing we intend to try is to configure the assigned-clock-parents property of our own device tree overlay.

&usdhc1 {
	#address-cells = <1>;
	#size-cells = <0>;
	pinctrl-names = "default", "state_100mhz", "state_200mhz";
	pinctrl-0 = <&pinctrl_usdhc1>, <&pinctrl_32k_clk>;
	pinctrl-1 = <&pinctrl_usdhc1_100mhz>, <&pinctrl_32k_clk>;
	pinctrl-2 = <&pinctrl_usdhc1_200mhz>, <&pinctrl_32k_clk>;
	assigned-clocks = <&clks IMX6UL_CLK_USDHC1_SEL>, <&clks IMX6UL_CLK_USDHC1>;
	assigned-clock-parents = <&clks IMX6UL_CLK_PLL2_PFD2>;
	assigned-clock-rates = <0>, <132000000>;
	fsl,tuning-step = <2>;
	keep-power-in-suspend;
	vmmc-supply = <&reg_sd1_vmmc>;
	status = "okay";

	brcmf: bcrmf@1 {
		reg = <1>;
		compatible = "brcm,bcm4329-fmac";
	};
};

However, unfortunately the Compulab device tree doesn’t appear to do something similar for the Wi-Fi on their board as seen in this patch. So perhaps this is not the right thing to do!

Looking at the linked patch, I note that Compulab add a wifi-host property to their Wi-Fi sdhci node. It looks like this is a legacy property, which enables SDHCI_QUIRK2_SDIO_IRQ_THREAD. Both of these appear to be legacy features of an old, 4.x NXP kernel, but perhaps it’s worth checking the wifi-host property is still valid in the Compulab distribution, and if so, enabling it.

We’ll continue to dig into the same issue on the iMX6, and get back to you with anything we do find.
Perhaps it’s worth checking with NXP as well? As this does look like it may be an iMX specific issue.

1 Like
13

Great! Happy to hear that the issue has been reproduced.

I have tried adding the wifi-host property, but it did not change or resolve the error.

I already started a ticket thread with NXP a while ago. So far this did not bring much, but it has now been escalated to a more specialized department. If you have any specific questions or clues you would like to share with them please let me know.

Good luck on the further investigations!

14

Hi @ajudge, I am just checking in to see if there have been any new findings?

On my end I have received custom designed PCBAs. Only missing the Halow modules which will arrive this week. Once assembled I will let you know the results. Although I do not expect the behavior to change.

15

Hi @SanderV
Nothing yet, unfortunately. We’re investigating some of the imx specific device tree parameters such as fsl,tuning and some deeper investigation into our hardware.

16

Thanks for the update!

In the meantime I have received my custom PCBAs, but as expected the the same error persists. See log below

At NXP the ticket has been escalated to a new team. I will keep you updated.

Thanks for your continued support! Hopefully we will find a solution soon.

[   61.634606] mmc1: new high speed SDIO card at address 0001
[   61.684447] morse_sdio mmc1:0001:1: sdio new func 1 vendor 0x325b device 0x306 block 0x8/0x8
[   61.693267] morse_sdio mmc1:0001:2: sdio new func 2 vendor 0x325b device 0x306 block 0x200/0x200
[   61.702465] morse_sdio mmc1:0001:2: Reading gpio pins configuration from device tree
[   61.712956] morse_sdio mmc1:0001:2: Loaded firmware from morse/mm6108.bin, size 433044, crc32 0x436ba524
[   61.722671] morse_sdio mmc1:0001:2: Loaded BCF from morse/bcf_default.bin, size 1077, crc32 0x1cbf23cb
[   66.853591] morse_sdio mmc1:0001:1: sdio removed func 1 vendor 0x325b device 0x306
[   72.222904] mmc1: Timeout waiting for hardware interrupt.
[   72.228317] mmc1: sdhci: ============ SDHCI REGISTER DUMP ===========
[   72.234760] mmc1: sdhci: Sys addr:  0x4a538000 | Version:  0x00000002
[   72.241200] mmc1: sdhci: Blk size:  0x00000200 | Blk cnt:  0x0000001f
[   72.247640] mmc1: sdhci: Argument:  0xac000080 | Trn mode: 0x00000023
[   72.254079] mmc1: sdhci: Present:   0x01f8818e | Host ctl: 0x00000013
[   72.260520] mmc1: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[   72.266961] mmc1: sdhci: Wake-up:   0x00000008 | Clock:    0x0000000f
[   72.273401] mmc1: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
[   72.279844] mmc1: sdhci: Int enab:  0x117f100b | Sig enab: 0x117f100b
[   72.286284] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000502
[   72.292725] mmc1: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000b400
[   72.299164] mmc1: sdhci: Cmd:       0x0000353a | Max curr: 0x00ffffff
[   72.305605] mmc1: sdhci: Resp[0]:   0x00000000 | Resp[1]:  0x00000000
[   72.312045] mmc1: sdhci: Resp[2]:   0x00000000 | Resp[3]:  0x00000000
[   72.318485] mmc1: sdhci: Host ctl2: 0x00000000
[   72.322929] mmc1: sdhci: ADMA Err:  0x00000007 | ADMA Ptr: 0x4a53820c
[   72.329370] mmc1: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP =========
[   72.336940] mmc1: sdhci-esdhc-imx: cmd debug status:  0x2100
[   72.342598] mmc1: sdhci-esdhc-imx: data debug status:  0x2220
[   72.348343] mmc1: sdhci-esdhc-imx: trans debug status:  0x2328
[   72.354173] mmc1: sdhci-esdhc-imx: dma debug status:  0x2400
[   72.359832] mmc1: sdhci-esdhc-imx: adma debug status:  0x25b4
[   72.365578] mmc1: sdhci-esdhc-imx: fifo debug status:  0x2680
[   72.371324] mmc1: sdhci-esdhc-imx: async fifo debug status:  0x2750
[   72.377590] mmc1: sdhci: ============================================
[   72.384444] morse_sdio mmc1:0001:2: sdio_memcpy_toio failed: 18446744073709551506
[   72.391961] morse_sdio mmc1:0001:2: morse_sdio_dm_write failed -110
[   73.027370] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[   73.034098] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[   73.040019] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[   74.099480] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[   74.106207] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[   74.112124] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[   74.748572] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[   74.755353] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[   74.761231] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[   75.397597] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[   75.404334] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[   75.410253] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[   76.467317] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[   76.474044] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[   76.479966] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[   77.116327] morse_sdio mmc1:0001:2: morse_sdio_set_func_address_base -16
[   77.123111] morse_sdio mmc1:0001:2: morse_sdio_get_func failed
[   77.128985] morse_sdio mmc1:0001:2: morse_sdio_reg32_write failed 18446744073709551611
[   77.137101] morse_sdio mmc1:0001:2: morse_firmware_init failed: -5
[   77.778390] morse_sdio: probe of mmc1:0001:2 failed with error -5
[   77.785079] mmc1: card 0001 removed
17

Hi Sander,

I hope NXP can get back to you shortly! I’ve exhausted a number of host side specific things I could find referenced online at this stage, and unfortunately testing small kernel changes is a slower process than I would like.

One particular oddity: the timeout is indicating an ADMA error - specifically an ADMA length mismatch, hinting that the sdhci driver is using ADMA: ADMA Err: 0x00000007

image
image867×541 59 KB

However, the capabilities reported by the driver are showing Caps: 0x07eb0000

image
image870×134 14 KB

which claims ADMA is not supported.

I will try to reach out to NXP as well to get some clarity on various register states and debug flags, but this does make me a little suspicious there may be some further memory alignment things to consider.

Really appreciate your patience with this. We will continue to investigate until this is resolved.

18

Interesting to see where this clue will take you.

The NXP team is verifying the DTS configuration. Will hopefully hear from them in a few days.

Let me know if there is anything I can test or check at my end.

Thanks for the commitment!

19

@SanderV the confusion came from a difference between the registers used by the iMX host controller driver, and the registers as documented in the SD host controller spec! The iMX driver corrects how these registers are reported to the SD Host Controller stack

0x07eb0000 indicates that ADMA1 is not used, but ADMA2 is used. This is manually manipulated in the iMX ESDHC driver.

		if (val & SDHCI_CAN_DO_ADMA1) {
			val &= ~SDHCI_CAN_DO_ADMA1;
			val |= SDHCI_CAN_DO_ADMA2;
		}

I have been able to disable ADMA2 by forcing this bit off to try fall back to SDMA. This still resulted in some timeouts at the time.

However! After inspecting more carefully I have noticed that the register dump on this imx6 board had changed when these timeouts occured. For some time now, I had been investigating a register dump with the following argument value, a CMD53 read of the MM chip ID.

Argument:  0x149a4004

This argument occurs earlier in the Morse Micro init sequence, and on closer inspection with the logic analyzer - this was identified as an unstable voltage supply to the Morse Micro hat.

imx-boot-28-11-2024
imx-boot-28-11-20241588×712 26.2 KB

Note in the above image, the instability in the clock, and the spurious transitions in the data lines.

Opting to connect the USB-C of the Morse Micro hat to the iMX6 baseboard for power, I was able to get back to the original error with argument

Argument:  0xac00001b

I realise this argument differs to yours slightly, but I expect this to be failing in a similar spot to yours - as this argument is a CMD53 write and should align with writing the firmware binary to the chip.

Reattempting the ADMA disabling, by forcing the value in esdhc_readl_le where SDHCI_CAPABILITIES is read.

		val &= ~SDHCI_CAN_DO_ADMA1;
		val &= ~SDHCI_CAN_DO_ADMA2;

I have managed to bring up the Morse Micro chip and have it register on the iMX6 as a netdev!

root@imx6ul-var-dart:~# ifconfig wlan0
wlan0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 0c:bf:74:00:02:9c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The ADMA change is necessary at this stage. The length mismatch error you are seeing (ADMA Err: 0x00000007) lines up with the SD Host Controller Spec, section 1.13.1.3 which states.

If total length of a descriptor were not multiple of block size, ADMA2 transfer might not be terminated. In this case, data timeout would occur and the transfer would be stopped by abort command. Therefore, total length should be multiple of block size.

The mmc request operations in the sdhci driver used by the iMX doesn’t split the data length into two parts of block multiples and one part for byte transfer, so the simplest thing to do is disable ADMA. The driver should still fall back to SDMA based transfers.

On our side, we still have some intermittent EILSEQ (-84) errors preventing us from bringing up a reliable interface, which we are still looking into. I suspect these are further signal integrity related issues or other forms of data corruption on the bus. Will be interested if you see the same thing.

20

Hi @ajudge,

Very happy to see that you have been able to resolve the timeout error!

I am by no means an expert on driver development, so I am going to need some help implementing this on my end.

How do I disable ADMA? Is it as simple as adding broken-adma2 = <0x1>; to the device tree?

Is there a need for me to edit the driver and recompile it?

21

There are possibly several ways to do this, and I will avoid making a strong claim as to what is the best approach.

I chose to modify how the iMX driver was reporting support for ADMA by modifying sdhci-esdhc-imx.c manually with the patch below.

--- a/drivers/mmc/host/sdhci-esdhc-imx.c
+++ b/drivers/mmc/host/sdhci-esdhc-imx.c
@@ -497,10 +497,8 @@ static u32 esdhc_readl_le(struct sdhci_host *host, int reg)
                 * quirk on MX25/35 platforms.
                 */
 
-               if (val & SDHCI_CAN_DO_ADMA1) {
-                       val &= ~SDHCI_CAN_DO_ADMA1;
-                       val |= SDHCI_CAN_DO_ADMA2;
-               }
+               val &= ~SDHCI_CAN_DO_ADMA1;
+               val &= ~SDHCI_CAN_DO_ADMA2;
        }
 
        if (unlikely(reg == SDHCI_CAPABILITIES_1)) {

You will probably want to put this into a .patch file and create a suitable Yocto layer. I found this article reasonably helpful for getting Yocto to do that.

I did try a few other things, with largely unsuccessful results. See below:

  • It looks like I should have been able to set the ESDHC_FLAG_ERR004536 flag in the correct soc data for the compatible used, which will communicate a broken ADMA quirk, but this did not appear to work!

  • There should be other ways to set the broken ADMA quirk, including adding the sdhci.debug_quirks=0x40 command line argument. This looks like it needed to be done in u-boot for my device, but devicetree also should have provisions for kernel boot arguments. Unfortunately adding the command line argument also didn’t work for me, and prevented the host from finding the Morse Micro device. If you decide to look into this path, their may be additional quirks to enable which are set by default in the esdhc driver.

1 Like