BackgroundI have a system that runs Linux. It's a NAS, and does not have any monitor, keyboard, or serial port. It does have a network port. I'm not happy with the software running on it, and am trying to get a different distro to run.What I haveOn the existing system, I can manage to get a new kernel and initrd booted, by using the web interface to upgrade the ROM, but that upgrade is a specially crafted image, that does nothing more than unpack a kernel, initrd, kexec, and runs kexec with the required arguments to boot the new kernel.The initrd sets up a network connection, starts an SSH server (dropbear), and waits for it to end. It then runs a different script. Using this, I can do some testing: I can boot into that kernel/initrd, log in via SSH, customise the stage-2 script, kill dropbear, and hope for the best.Using this approach, I have managed to install a functional OS on the hard disk. (For now, it's NixOS, if it matters, but I may change that in the future. My question isn't about any specific distro though.) I have intentionally not made it bootable. I wish to keep the flash memory as it is, so that aside from the data on the hard disk, the NAS remains official. I have, however, obtained the distro's own kernel and initrd, that I would want to put in an upgrade image.The problemUsing that kernel and initrd, the system does not start.My attemptsI have put together a shared initrd, containing the distro's setup as well as my own, which continues to boot into dropbear. In an SSH shell, I then attempt to run the distro's init scripts. This fails, however, because it relies on being run as PID 1.I then attempted to make PID 1 take arbitrary commands: I made it run scripts from a pipe, and wrote to that pipe from my remote shell, intending to inspect manually whether the command had the desired effect. However, that doesn't work: the init-shell (PID 1) sees EOF after the very first command, and immediately exits. Hello kernel panic.I also attempted to test what would happen if I made it not care whether it was being run as PID 1, by passing systemd's --system option, and then run the distro's init script from my shell. In that form, I was unable to reproduce the problem: it just worked.My question: What now?At this point I'm looking back at the pipe approach for an alternative, something that actually works to run more than one command. Preferably something that also lets me see the commands' output.Basically: I want to know how I can remotely see the output of the boot programs that run from PID 1, that cannot run in an SSH session, without actually attaching a monitor or serial cable.Answers that take a completely different approach are also welcome, but please keep in mind the system restrictions I'm dealing with. I don't just not have a monitor or serial cable, I don't even have a VGA or serial port. I have one USB port that I could plug a keyboard into, if needed, but I am of course not able to see anything I type.
How can I remotely debug a broken initrd?
linux;debugging;initrd
After starting an SSH session, other sessions can write there too, by writing to /dev/pts/<N>. So to see what is happening, since I'm already able to control what PID 1 will end up running, I just need to make it write there. Using exec 0<>/dev/pts/0 1<>/dev/pts/0 2<>/dev/pts/0 from PID 1 and making sure nothing else is reading/writing there I can see what's happening. When it eventually fails, the distro's init script prompts to ask what to do, and even properly responds to me entering R to reboot.The actual problem I was having seems to be caused by required kernel modules not getting loaded. Although I was able to get everything working manually, which obviously means the block, RAID and filesystem modules were available and getting loaded, the distro's initscript relies on udev, and it turns out that requires a few additional modules, in particular unix (which I'd never have chosen myself to not build into the kernel).