The last working kernel version was 6.1.y.
Checked with a simple overlay allo-piano-dac-pcm512x-audio and a single chip channel loads with a simplified controls. It leads me to believe, that the switch part is not functioning as it is meant to be.

517 is EPROBEDEFER, which means "not yet" - the kernel will try again later. It's nice if drivers suppress those error messages as they can be confusing, but otherwise I don['t see anything wrong there. The stall is probably caused by something failing to release a mutex.

Like most of the drivers we host in our kernel, we can't offer any support for them unless they reveal an underlying problem in the bits we do maintain. Our involvement consists of forward-porting the patches to new kernel versions, which is basically just a matter of getting them to compile again when the APIs change.

Allo seems to be long gone - your best hope would be that one of the users out there has some coding experience. Even a relative novice could learn quite a bit about the problem by scattering a few pr_info() calls around the driver.

Thank you for looking into this. Appreciated. Being a "novice" as you mention I can dive into the module, however will need a bit of guidance: mutex. What troubleshooting steps or actions would you suggest?

Here's a rough plan of action:

1. After booting your current kernel, run dmesg -l err,warn and look for any messages that might be relevant.
2. Once the error has occurred, run that dmesg again, and look for additions. Report your findings.
3. As you'll be building your own kernel you should use the fastest Pi you have, which looks to be a Pi 5. To avoid the need to set up a cross-compiler, I recommend you use a 64-bit image (to match the 64-bit kernel). Just in case something goes disastrously wrong, you might want to install a fresh image to a spare card - even a 64-bit Lite image would be enough.
4. Follow the kernel build instructions here: https://www.raspberrypi.com/documentation/computers/linux_kernel.html#building
   You need the "Natively build a kernel" instructions.
5. Once compilation has completed, follow the installation steps and verify that it boots.
6. Now we're going trace all the snd_soc_... calls. At the top of the file, after the last #include, add:

#define snd_soc_component_read(c, r) my_snd_soc_component_read(c, r, __LINE__)
#define snd_soc_component_write(c, r, v) my_snd_soc_component_write(c, r, v, __LINE__)
static unsigned int my_snd_soc_component_read(struct snd_soc_component *component,
				      unsigned int reg, int line);
static int my_snd_soc_component_write(struct snd_soc_component *component,
			    unsigned int reg, unsigned int val, int line);

7. At the end of the file (before MODULE_AUTHOR should be OK), put this:

#undef snd_soc_component_read
#undef snd_soc_component_write

static unsigned int my_snd_soc_component_read(struct snd_soc_component *component,
				      unsigned int reg, int line)
{
	unsigned int val;
	pr_info("< read %x (%d)\n", reg, line);
	val = snd_soc_component_read(component, reg);
	pr_info("> read %x -> %x (%d)\n", reg, val, line);
	return val;
}

static int my_snd_soc_component_write(struct snd_soc_component *component,
			    unsigned int reg, unsigned int val, int line)
{
	int ret;
	pr_info("< write %x = %x (%d)\n", reg, val, line);
	ret = snd_soc_component_write(component, reg, val);
	pr_info("> write %x = %x ret %d (%d)\n", reg, val, ret, line);
	return ret;
}

8. Rebuild and reinstall the kernel (it should be quick this time since you've only changed one file), and confirm that you are seeing messages (dmesg -l err).
9. Run it until you get a failure, then dmesg -l err,warn | pastebinit (you may need to sudo apt install pastebinit first).

Thank you for the steps. This round, the module was built against "stock" Pi kernel - 6.6.62-v7+ #1816 using cross-compiler.

load dtoverlay -> http://paste.debian.net/1339146/
after reboot on alsa test -> http://paste.debian.net/1339151/

I would expect a bit more output at this time, however the test was on Pi 3B+, moving to Pi5 or CM5 to speed-up the native build work.

Stock kernel:
uname -a
Linux foopi 6.6.62+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.62-1+rpt1 (2024-11-25) aarch64 GNU/Linux

dmesg -l err,warn

There are no new messages from test other than task alsactl:67778 blocked for more than xxx seconds.
On shutdown/reboot the alsa-restore.service stop job waits for over 5 min; hard reset.

Kernel from repo:
uname -a
Linux foopi 6.6.63-v8-16k+ #1 SMP PREEMPT Tue Dec 10 13:25:12 GMT 2024 aarch64 GNU/Linux

dmesg -l err,warn

Same behaviour as before.

Odd. Can you put another pr_err (any message you like) before the "return 0;" at the end of snd_allo_piano_dac_probe? That should confirm whether the driver initialisation is completing or not.

Kernel with updated module:
uname -a
Linux foopi 6.6.63-v8-16k+ #1 SMP PREEMPT Tue Dec 10 13:25:12 GMT 2024 aarch64 GNU/Linux

dmesg -l err,warn

Same behaviour as before.

Odd. Can you put another pr_err (any message you like) before the "return 0;" at the end of snd_allo_piano_dac_probe? That should confirm whether the driver initialisation is completing or not.

                if ((mute_gpio[0]) && (mute_gpio[1]))
                        snd_allo_piano_gpio_mute(&snd_allo_piano_dac);
                pr_err("DAC issue #6527\n");

                return 0;

Result: https://paste.debian.net/hidden/0aaf8bb3/

Looks like it does.

(I'll take your word that there is no change - you don't need to upload the logs each time)
Let's just double check that all the probe calls are failing - add another message at the start of snd_allo_piano_dac_probe, then if we only see it twice followed by two -517s then we know that the instantiation never succeeds.

sudo cat /sys/kernel/debug/devices_deferred should also tell you if there are still outstanding devices that are deferred. (The dependency will only be listed if the driver has used dev_err_probe though)

sudo cat /sys/kernel/debug/devices_deferred should also tell you if there are still outstanding devices that are deferred. (The dependency will only be listed if the driver has used dev_err_probe though)

Nothing, just empty:

foonerd@foopi:~ $ sudo cat /sys/kernel/debug/devices_deferred
foonerd@foopi:~ $ sudo ls -la /sys/kernel/debug/devices_deferred
-r--r--r-- 1 root root 0 Jan  1  1970 /sys/kernel/debug/devices_deferred

(The dependency will only be listed if the driver has used dev_err_probe though)

It doesn't.

add another message at the start of snd_allo_piano_dac_probe, then if we only see it twice followed by two -517s then we know that the instantiation never succeeds.

This is getting interesting - https://paste.debian.net/hidden/168f11aa/

Purely boot only, nothing touched... yet.

Well that's a bit of a relief - I was beginning to wonder how you could be getting so far if the probe() had never succeeded.

(The dependency will only be listed if the driver has used dev_err_probe though)

It doesn't.

The fact it is still deferred would be listed, just not what the dependency was.

Honestly, I'm scratching my head for past two weeks, doubting my sanity with this.
On the serious note - there is a quite clever switch in the code. Perhaps barking up the wrong tree - could the overlay as such need a revamp?

It seems unlikely, since it only provides a static configuration - by the time you've exited the probe function, the overlay plays no further part.

To add to the confusion, when using basic overlay: allo-piano-dac-pcm512x-audio the DAC works with limited functionality. This bypasses all channels separation 2xPCM, lowpass, etc.

Are you seeing any of my debug output when you use the controls? If not, which controls are available to you in this subset?

I think I made my last remark confusing.
Hardware is Piano 2.1

1. When using https://github.com/raspberrypi/linux/blob/rpi-6.6.y/sound/soc/bcm/allo-piano-dac.c via overlay https://github.com/raspberrypi/linux/blob/rpi-6.6.y/arch/arm/boot/dts/overlays/allo-piano-dac-pcm512x-audio-overlay.dts there are limited controls I was referring to. Audio is active via one of the pcm512x chip.

2. However, the DAC should be using https://github.com/raspberrypi/linux/blob/rpi-6.6.y/sound/soc/bcm/allo-piano-dac-plus.c via overlay https://github.com/raspberrypi/linux/blob/rpi-6.6.y/arch/arm/boot/dts/overlays/allo-piano-dac-plus-pcm512x-audio-overlay.dts and this is where debug output has been added.

Will add debug params to 1 and check.

The 1 seems to be producing events. Not sure how this is going to help though.

Comparing with working DAC on kernel 6.1.y, the node parameters is created under /sys/module/snd_soc_allo_piano_dac_plus/ and this is what alsa is using for hw2. On kernel 6.6.y node is missing.

A simple test, forcing bogus return 0; from snd_allo_piano_dac_probe and it is clear that all the params are not available to alsa. There is literally nothing in dmesg nor journalctl to follow up on. Any ideas?

Hey @pelwell,

This is getting interesting. After adding #include <linux/i2c.h> all of the sudden:

[   12.949688] < read 13d (306)
[   12.949693] > read 13d -> 30 (306)
[   12.949694] < read 13e (308)
[   12.949695] > read 13e -> 30 (308)
[   12.949697] < write 13e = 30 (328)
[   12.950447] > write 13e = 30 ret 0 (328)
[   12.950449] < write 13d = 30 (330)
[   12.951185] > write 13d = 30 ret 0 (330)
[   12.951187] < write 103 = 0 (332)
[   12.951916] > write 103 = 0 ret 0 (332)
[   12.951917] < write 103 = 0 (334)
[   12.952646] > write 103 = 0 ret 0 (334)

The output here (https://paste.debian.net/hidden/168f11aa/) showed it reaching the end of snd_allo_piano_dac_probe ("DAC issue #6527"), so why do you need to force it? And what changed between that run and the previous one (that produced much less output), other than the addition of the "DAC probe begin" message?

The only differences at this point were:

1. add another message at the start of snd_allo_piano_dac_probe
2. switched all power off, coffee break, power back on

Your log above shows the output from snd_allo_piano_dual_mode_put, turning dual_mode off. Do any other controls exist/work?

The amixer is used to set specific hwdevice with params. However, module once loaded renders alsa completely unresponsive and console hungs. No other controls are available. The continuous message kernel: INFO: task alsactl:1683 blocked for more than xxx seconds about 2 min apart.

Well, the DAC init is stuck here, indefinite loop:

[  484.320360] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  484.320363] task:alsactl         state:D stack:0     pid:750   ppid:1      flags:0x00400801
[  484.320369] Call trace:
[  484.320372]  __switch_to+0xe0/0x148
[  484.320381]  __schedule+0x31c/0xd60
[  484.320385]  schedule+0x64/0x108
[  484.320389]  schedule_preempt_disabled+0x2c/0x50
[  484.320393]  rwsem_down_write_slowpath+0x23c/0x6e0
[  484.320398]  down_write+0xa0/0xb0
[  484.320402]  snd_ctl_release+0x74/0x138 [snd]
[  484.320427]  __fput+0xc4/0x290
[  484.320432]  __fput_sync+0x58/0x70
[  484.320434]  __arm64_sys_close+0x40/0x90
[  484.320440]  invoke_syscall+0x50/0x128
[  484.320446]  el0_svc_common.constprop.0+0xd0/0xf8
[  484.320450]  do_el0_svc_compat+0x28/0x48
[  484.320455]  el0_svc_compat+0x30/0x88
[  484.320458]  el0t_32_sync_handler+0x98/0x148
[  484.320461]  el0t_32_sync+0x194/0x198

I think without addressing this part, we can't progress further.
Earlier on you mention that this may be related to mutex. Suggestions?

Thank you for looking into this.
The problem surfaces when setting volume above ca' 25% of threshold, after this level audio become distorted; similar to over-amplification. Hence why I think that this is linked.

If you can spare a bit of time, and take a look into the last week discussions in community thread.

dmesg:

[   16.550860] snd-allo-piano-dac-plus soc:sound: Failed to set master volume limit: -22
[   16.550884] snd-allo-piano-dac-plus soc:sound: Failed to set subwoofer volume limit: -22
[  314.091468] snd-allo-piano-dac-plus soc:sound: snd_soc_register_card() failed: -517
[  314.096541] snd-allo-piano-dac-plus soc:sound: snd_soc_register_card() failed: -517
[  314.116804] snd-allo-piano-dac-plus soc:sound: snd_soc_register_card() failed: -517
[  314.121478] snd-allo-piano-dac-plus soc:sound: Failed to set master volume limit: -22
[  314.121500] snd-allo-piano-dac-plus soc:sound: Failed to set subwoofer volume limit: -22

Hey @pelwell,

Let me know, which version/combination I can help testing. I still have the debug build from our initial investigation.
Noticed few things which may help:
soc:sound: snd_soc_register_card() failed: -517 happens when DSP firmware is loaded.
snd-allo-piano-dac-plus soc:sound: Failed to set master volume limit: -22 happens, when DAC is set in dual stereo mode.

Something I have not noticed before:
dmesg -w | grep pcm512x

[   31.056865] pcm512x 1-004c: No SCLK, using BCLK: -2
[   31.058415] pcm512x 1-004d: No SCLK, using BCLK: -2
[   31.084708] pcm512x 1-004c: No SCLK, using BCLK: -2
[   31.084719] pcm512x 1-004d: No SCLK, using BCLK: -2
[   31.269532] pcm512x 1-004c: No SCLK, using BCLK: -2
[   31.269542] pcm512x 1-004d: No SCLK, using BCLK: -2
[   31.392276] pcm512x 1-004c: No SCLK, using BCLK: -2
[   31.392285] pcm512x 1-004d: No SCLK, using BCLK: -2

Regarding audio stage itself, it is over-amplified. Raspberry Pi 4B, Pi5 2/4/8GB with the same results.
Pi Kernel 6.1.x: External AMP is set @ 50%, DAC @ 40 - 90% and no distortion.
Pi Kernel 6.6.x: External AMP is set @ 5%, DAC @ 25% and distortion is audible.

Hey @pelwell,

Question regarding snd_allo_piano_dac_init

Perhaps move to

Another place I am looking at do bolt on

compatible = "allo,piano-dac-plus";
digital_gain_0db_limit = <1>;

@pelwell,

The same system, only kernel is different. From amixer -c<num> scontents there is clear evidence that the digital_gain_0db_limit is ineffective and range 0 -> 255 is always set in kernel 6.6.y.

Perhaps setting ret = 0 avoids unintentional propagation of undefined or stale values.

The variable is initialised on the very next (non-blank) line. This isn't the cause of the problem.

Another place I am looking at do bolt on

I don't understand what you mean.

I'm back looking at this issue (it would be nice to be rid of it). It seems likely that my initial uneasiness ("I'm not sure if the code was arranged as it was for good reasons or just carelessness.") may have been justified.

Another place I am looking at do bolt on

I don't understand what you mean.

Sometimes it is possible to nudge with an overlay parameters. Ignore this please, makes no effect.

I can share module code via Pastezone if this helps.

Thanks, but that won't be necessary - it's completely reproducible here.

Hey @pelwell ,

Excellent result. No distortion, no dmesg errors. Kernel 6.6.y amixer -c0 scontents: http://paste.debian.net/1344094/

I will test with 24db_digital_gain and glb_mclk in dtoverlay, just to check if gain and Kali still works.

Hey @pelwell,

All checks out.

And that's with all three patches applied?

All three.

Code with extra traces (will remove in final build) here: https://dpaste.com/AYHDNNBNL
I think that change to pcm512x is cosmetic, bears no functional effect. Nonetheless, removes the message from dmesg.

Thanks - that's merged now.

Thank you. Will trigger community build and get this tested. Perhaps let's leave this open for a day or two, test results pending.

Based on community feedback, closing as resolved.