Date: Sat, 20 Dec 2025 08:10:59 -0700 From: Warner Losh <imp@bsdimp.com> To: A FreeBSD User <freebsd@walstatt-de.de> Cc: FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: CURRENT: havock: elf_load_section: truncated ELF file Message-ID: <CANCZdfo7SeJkYOO9eun%2Bfz-0yY5MW0OJ1G%2B--Ysnb3H2dR8qAA@mail.gmail.com> In-Reply-To: <20251220141124.1606aa7c@thor.sb211.local> References: <20251220141124.1606aa7c@thor.sb211.local>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Sat, Dec 20, 2025 at 6:12 AM A FreeBSD User <freebsd@walstatt-de.de> wrote: > Hello, > > recently a small server running recent CURRENT with a UFS basesd system > SSD (NVMe) and a data > graveyard based on RAID level 5 with ZFS (attached to a Fujitsu HBA > controler) gets corrupted > because of "loosing" a driver - this time the system reported TWO drives a > removed froma RAID > level 5 - which is like a death sentence. > > I guess this is a fallout of the recently changed timie parameters to the > CAM infrastructure > (I can't find any notes on this in man cam, so I feel lost). > Unlikely, but you can set this in the boot loader: kern.cam.tur_timeout=60 kern.cam.inquiry_timeout=60 kern.cam.modesense_timeout=60 and see if that works. You should see new errors on boot if his is the issue. Can you share a dmesg? I kinda doubt they'd cause the issues that you've had. If disks are gone, then there'd be different errors to what you are seeing, I'd think. To recover, your best bet is to use a USB stick from one of the release or snapshots. Warner > A very desastrous side effect of this crash was the inability to reboot > the box (CURRENT pre- > 16.0-CURRENT #11 master-n282659-7f39d05b67ae: Sat Dec 20 09:35:32 CET > 2025amd64, the runtime > system was from 16th or 17th of December). > After several tenth of minutes I had to hadr reboot the box - with obvious > data loss on the > system SSD. And here my problems start to turn into a mess. > > After the first initial reboot I performed a fsck -fy, rebootet and > whitnessed that > jails didn't come up anymore and SSHD didn't work. So I installed prior to > the crash already > compiled CURRENT from /usr/src which is "master-n282659-7f39d05b67ae" (as > the sibling box which > is runnig great by the way, but different CPU and smaller RAID, but also > system SSD based on > UFS filesystem, same HBA. So CURRENT seem to operate in general on similar > hardware. > > After the second reboot with the old kernel the box in question went into > debugger, rebooting > in single user mode and performing fsck -fy revealed a lot of repairs on > the first partitions, > /, /var, /usr. After a reboot I realized that most services now are broken > - jails do not > start, sshd doesn't start and the whole system is going into multiuser, > but seems to have > serious problems. > > uname -a remains empty > cd /usr/src; make buildworld returns immediately empty, no further action > service ldconfig start also returns complete empty on console > > Several onboard/base tools simply return nothing. > > trying "/resucue/sh" (install date indicates 20th of December, so it is > the latest ) seems to > give me the first indication of something has terribly gone wrong or even > /rescue/vi (to edit > loader to change to boot.old): > > elf_load_section: truncated ELF file > Abort trap > > Checking /boot/kernel, /lib, /usr/lib, /bin or /sbin seems to be intakt > (as far as I can > check, all timestamps are 20th Dec 2025, 9:48 UTC). > > Well, since this is not the first time I ran into some problems using > CURRENT, the outage due > to two lost ZFS drives after the recent chenges seems worthy to make some > note here. > Can you provide error messages at boot for this? You talk about fsck and about ZFS, so I'm a little confused as to your setup. Warner > The other question would be how to fix: one strategy would be to boot from > an official image > from flash drive and try to perform a "make installkernel installworld". > Maybe there is > another way idicativ to that what I described above ... > > Thanks in advance, > > oh > > > -- > > A FreeBSD user > [-- Attachment #2 --] <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Sat, Dec 20, 2025 at 6:12 AM A FreeBSD User <<a href="mailto:freebsd@walstatt-de.de">freebsd@walstatt-de.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br> <br> recently a small server running recent CURRENT with a UFS basesd system SSD (NVMe) and a data<br> graveyard based on RAID level 5 with ZFS (attached to a Fujitsu HBA controler) gets corrupted<br> because of "loosing" a driver - this time the system reported TWO drives a removed froma RAID<br> level 5 - which is like a death sentence.<br> <br> I guess this is a fallout of the recently changed timie parameters to the CAM infrastructure<br> (I can't find any notes on this in man cam, so I feel lost).<br></blockquote><div><br></div><div>Unlikely, but you can set this in the boot loader:</div><div>kern.cam.tur_timeout=60</div><div>kern.cam.inquiry_timeout=60</div><div>kern.cam.modesense_timeout=60</div><div><br></div><div>and see if that works. You should see new errors on boot if his is the issue. Can you share a dmesg?</div><div><br></div><div>I kinda doubt they'd cause the issues that you've had. If disks are gone, then there'd be different errors to what you are seeing, I'd think. </div><div><br></div><div>To recover, your best bet is to use a USB stick from one of the release or snapshots.</div><div><br></div><div>Warner</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> A very desastrous side effect of this crash was the inability to reboot the box (CURRENT pre-<br> 16.0-CURRENT #11 master-n282659-7f39d05b67ae: Sat Dec 20 09:35:32 CET 2025amd64, the runtime<br> system was from 16th or 17th of December). <br> After several tenth of minutes I had to hadr reboot the box - with obvious data loss on the<br> system SSD. And here my problems start to turn into a mess.<br> <br> After the first initial reboot I performed a fsck -fy, rebootet and whitnessed that<br> jails didn't come up anymore and SSHD didn't work. So I installed prior to the crash already<br> compiled CURRENT from /usr/src which is "master-n282659-7f39d05b67ae" (as the sibling box which<br> is runnig great by the way, but different CPU and smaller RAID, but also system SSD based on<br> UFS filesystem, same HBA. So CURRENT seem to operate in general on similar hardware.<br> <br> After the second reboot with the old kernel the box in question went into debugger, rebooting<br> in single user mode and performing fsck -fy revealed a lot of repairs on the first partitions,<br> /, /var, /usr. After a reboot I realized that most services now are broken - jails do not<br> start, sshd doesn't start and the whole system is going into multiuser, but seems to have<br> serious problems.<br> <br> uname -a remains empty<br> cd /usr/src; make buildworld returns immediately empty, no further action <br> service ldconfig start also returns complete empty on console<br> <br> Several onboard/base tools simply return nothing.<br> <br> trying "/resucue/sh" (install date indicates 20th of December, so it is the latest ) seems to<br> give me the first indication of something has terribly gone wrong or even /rescue/vi (to edit<br> loader to change to boot.old):<br> <br> elf_load_section: truncated ELF file<br> Abort trap<br> <br> Checking /boot/kernel, /lib, /usr/lib, /bin or /sbin seems to be intakt (as far as I can<br> check, all timestamps are 20th Dec 2025, 9:48 UTC).<br> <br> Well, since this is not the first time I ran into some problems using CURRENT, the outage due<br> to two lost ZFS drives after the recent chenges seems worthy to make some note here.<br></blockquote><div><br></div><div>Can you provide error messages at boot for this? You talk about fsck and about ZFS, so I'm a little confused as to your setup.</div><div><br></div><div>Warner</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> The other question would be how to fix: one strategy would be to boot from an official image<br> from flash drive and try to perform a "make installkernel installworld". Maybe there is<br> another way idicativ to that what I described above ...<br></blockquote><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Thanks in advance,<br> <br> oh<br> <br> <br> -- <br> <br> A FreeBSD user<br> </blockquote></div></div>help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfo7SeJkYOO9eun%2Bfz-0yY5MW0OJ1G%2B--Ysnb3H2dR8qAA>