From owner-freebsd-current@freebsd.org Tue Jul 30 12:43:59 2019 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0E8E6A261C; Tue, 30 Jul 2019 12:43:59 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FD60981B2; Tue, 30 Jul 2019 12:43:57 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1564490619; bh=z9jucbR+yOSfOujhO9PnQzhm31ZuTTsui9zqaGNJZMA=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=GfjSf89KnfLL1wdgNeE6SAImYbC+P2+1aYX9seIpuWUHlchsDPzpoqcjYZ3+smjrW w8OO8ebNmO0BETR5KK7lnwBm/z9+gx9IyJLUEAI52b3DE59fUUTPZoUhq+s4KLLu0o QllU/8ScQ0nUnYpgtobVr05cTST751ySbNYOcn9k= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from freyja ([46.88.81.15]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MPogF-1hg7Fg1BLP-00MwRs; Tue, 30 Jul 2019 14:43:39 +0200 Date: Tue, 30 Jul 2019 14:43:35 +0200 From: "O. Hartmann" To: Toomas Soome Cc: "O. Hartmann" , "freebsd-embedded@freebsd.org" , FreeBSD CURRENT Subject: Re: PCengines APU2C4, 12-STABLE: bootloader failure: Panic: free: guard2 fail @ 0x1000 + 2311663946 from Message-ID: <20190730144331.24f83e0c@freyja> In-Reply-To: <93EFC481-50A1-491C-87BB-12A35F3302EE@me.com> References: <20190724101132.5560377a@thor.intern.walstatt.dynvpn.de> <20190724120720.4382397c@thor.intern.walstatt.dynvpn.de> <20190724154901.0c114ebe@thor.intern.walstatt.dynvpn.de> <93EFC481-50A1-491C-87BB-12A35F3302EE@me.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:gH5J5OFgDIOn0c5ONXoL7YKMaNDuHHhZMdxkwGTwVLnwqSzVu++ no6PSYq1iuC8kQw6Zvauif8JrdlKIYBMj7wYBMvY8qQlg5eB7ViHBS5bIiDnfCJuZE6ei6T vriix+9jSsfyCdLwn1cv1krnSszEF5mR8E31oUjljvRtSaZdDTRP07HOj1BYZTk6au2ndCP 712v6ZueGWeKIyGTZtxjw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:3KwtfWap57g=:i0/0/JHAmHsIIEFH1ALhkA tJj536pKdrPp74aZmhsD5ojlcVLV9qXlQQyJbD5HQFdUDZbIanXaqQWj2Z7O5LjbDUB39Cn/l Ew+sJLg/+8p6b5ZhA2PHYGS+q1Ic5jCwJI0un6FqSfrBHH4okPfMcf8sQuytG36MPL4wTeWAy jOfkeeUTbzsVBvDXIMaXaF9thW6OkcwMPpN9IIg2WKZoZBaCYW6rtSLLMYGooChehFuYKsZQM NnEZYIlQ+A+0GMCbV5ZdBt5QVf5fpNbtOcoTCkIHlEwNWYACOTARcYGaRl5A14AcVRgfsINaF e3dgvAEBU8wsAgYdY8FdyqYp3+EXJd6DdxL+ArH5KTtn71rtGnhTooGOruX6FzPe0P+yI3C8s M4dBLO0Ah24bmqK4YNy2mbN9R7cOGXjcWaPPWsrR2vkTaKIOSvULyJJARd0my8cBDu23pQTUi WTSrzhNcRsWaUk3J0m5TMv5N5tfgXB3CRfPHMDCbCimvo8lReG9Mj05V4tWiCNGJWh+UuYBaS l46Th7X4HKYCgwodzPbwrjND8hkgpnveXF+Q8qGup/iRiQErJSIfD2D/19K/v9CrCV9OS1TgY rEWX5bvk1/0XbJh+XAFUJVxHbE83OmB+2mDiqbGPvzFewmq6GzzxzfM9/Zl9pfXpgb+YEItkk dpkYaQtY7HCzUcA9U64VbO8QxDdfEzbR22701fhbROizwruwe8SWMtgYVb7CZQ7HTKWw93y6i khbKPXPw9fJhk9qME/0xElAQMvqjp3fDLIO4SsxBny+bc+FzsXDfJPR8E96pezU9WcmmaejtL UNyvC+Zj+IhKJ84JVKfVwdAeWnSdk1fLgZgOUlNy988fHRoMOjPx9E9G45vnqfEmSfru4MrAD j/Paj4tIl4Tk3pe746tlThoW9zuWaDYko6kxOnLR05CDjcEmDa5RNj7Tr4tCjYtZ3+lwD0Uox kHEH36xWtJ8sfSNYTW758Vcx3EEXg4GVFVONm2tyspU1aDS8BQ3qMG1BelyIqcjCEJNuIpt5x KUJQN0Obzea3jcAQjSKkfp+w2Qbetq+SOfKTJSuX8qCYyxMxC8LUrbLfm1uGG4J4VPJXBklQ5 h2JLHwrdNn4tKfq9/O99/dfk259zgf7oQ+m X-Rspamd-Queue-Id: 2FD60981B2 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmx.net header.s=badeba3b8450 header.b=GfjSf89K X-Spamd-Result: default: False [-3.77 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmx.net:s=badeba3b8450]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-1.08)[ip: (-6.28), ipnet: 212.227.0.0/16(-1.45), asn: 8560(2.31), country: DE(-0.01)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[walstatt.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmx.net:+]; MX_GOOD(-0.01)[cached: mx01.gmx.net]; NEURAL_HAM_SHORT(-0.78)[-0.779,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[me.com]; RCVD_IN_DNSWL_LOW(-0.10)[20.17.227.212.list.dnswl.org : 127.0.3.1]; RECEIVED_SPAMHAUS_PBL(0.00)[15.81.88.46.zen.spamhaus.org : 127.0.0.10]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-Mailman-Approved-At: Tue, 30 Jul 2019 13:08:04 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Jul 2019 12:43:59 -0000 On Wed, 24 Jul 2019 18:07:22 +0300 Toomas Soome wrote: > > On 24 Jul 2019, at 16:48, O. Hartmann wrote: > >=20 > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA256 > >=20 > > Am Wed, 24 Jul 2019 12:06:53 +0200 > > "O. Hartmann" > > > schrieb:=20 > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA256 > >>=20 > >> Am Wed, 24 Jul 2019 12:09:16 +0300 > >> Toomas Soome schrieb: > >> =20 > >>>> On 24 Jul 2019, at 11:11, O. Hartmann wrote: > >>>>=20 > >>>> -----BEGIN PGP SIGNED MESSAGE----- > >>>> Hash: SHA256 > >>>>=20 > >>>> Hallo, > >>>>=20 > >>>> on APU2C4 from PCengines with latest firmware apu2_v4.9.0.7.rom, Sea= BIOS > >>>> rel-1.12.1.3-0-g300e8b7, booting via legacy MBR FreeBSD 12-STABLE > >>>> r350274 (the same with r350115) fails to boot with an immediate load= er > >>>> error: > >>>>=20 > >>>> [...] > >>>> SeaBIOS (version rel-1.12.1.3-0-g300e8b7) > >>>>=20 > >>>> Press F10 key now for boot menu > >>>>=20 > >>>> Booting from Hard Disk... > >>>> / > >>>>=20 > >>>> onsoles: internal video/keyboard =20 > >>>> IOS drive C: is disk0=20 > >>>> IOS drive D: is disk1=20 > >>>> IOS 639kB/3404444kB available memory=20 > >>>>=20 > >>>> reeBSD/x86 bootstrap loader, Revision 1.1 =20 > >>>> Mon Apr 15 21:28:11 CEST 2019 root@thor)=20 > >>>> anic: free: guard2 fail @ 0x1000 + 2311663946 from > >>>> X=C3=A7u0=C3=A7}4=C3=A7l$=E2=99=A6=E2=94=9C=C3=AD@=E2=94=A4=E2=99=A0= :2106163957 -> Press a key on the console to reboot > >>>> <-- [=E2=80=A6] =20 > >>>=20 > >>>=20 > >>> This is definitely something =E2=80=9Cfunny=E2=80=9D, we are apparent= ly attempting to > >>> free pointer 0x1000 which is definitely wrong because our heap should= be > >>> just below 4GB line. Since we do get list of disks printed, also memo= ry > >>> and version, it means we get error from interpretator - it is possible > >>> the stack did clash with bss and hence the corruption. =20 > >>=20 > >> I realized that I have defined=20 > >>=20 > >> WITH_KERNEL_RETPOLINE=3DYES > >>=20 > >> and since I use to build NanoBSD with -DNO_CLEAN, I'm just now compili= ng a > >> clean NanoBSD with RETPOLINE mitigations disabled so far - trying to c= heck > >> whether either of the ways to build causes the issue. > >> =20 > >>>=20 > >>> You can try to press space on first spinner and enter alternate loade= r on > >>> boot: prompt. (enter ?/boot on boot: prompt to see the file list). = =20 > >>=20 > >> I try a soon as the build process has finished and if the problem is t= hen > >> still present. =20 > >=20 > >=20 > > With a fresh build and no RETPOLINE mitigation (neither kernel nor worl= d) > > the phenomenon as described above is still the same. I tried an alterna= tive > > loader as requested, but without success. When choosing loader_4th, I g= et > > this error: > >=20 > > [...] > > FreeBSD/x86 boot > > Default: 0:ad(0p3)/boot/loader > > boot: /boot/loader_4th/ > >=20 > > onsoles: internal video/keyboard > > IOS drive C: is disk0 > > IOS drive D: is disk1 > > IOS 639kB/3404444kB available memory > >=20 > > reeBSD/x86 bootstrap loader, Revision 1.1 > > Wed Jul 24 12:51:12 CEST 2019 root@thor) > > anic: No heap setup =20 > > -> Press a key on the console to reboot <=E2=80=94 =20 > > =20 >=20 > Now this is bad. if my math is correct, this system is supposed to have 3= GB > of RAM, so are there specific build exceptions in place? see > stand/i386/loader/main.c, function main, after call to bios_getmem(). >=20 > rgds, > toomas Hello Toomas, the PCengine APU2C4 is supposed to have 4GB of RAM - wouldn't have a 64bit system seen the whole range? On 32bit systems, there was a memory hole I as= sume for memory mapped I/O of several PCI devices. This is the first time I hav= e a look at the memory reported by the kernel and its confusing me a bit. I do not have any kind of specific build exceptions in place targetting the memory. Of course, for memory usage and image size optimizations I defined several WITHOUT_ and WITH_ tags for build and install - but they never caus= ed any trouble and have not been changed so far. >=20 > >=20 > > Loader loader_simp ends up in stuck console with no output: > >=20 > > [...] > > FreeBSD/x86 boot > > Default: 0:ad(0p3)/boot/loader > > boot: /boot/loader_4th/ > >=20 > > onsoles: internal video/keyboard > > IOS drive C: is disk0 > > IOS drive D: is disk1 > > IOS 639kB/3404444kB available memory > >=20 > > reeBSD/x86 bootstrap loader, Revision 1.1 > > Wed Jul 24 12:59:23 CEST 2019 root@thor) > > [...] > >=20 > > regards > > oh =20 > >> =20 > >>>=20 > >>> =20 > >>>>=20 > >>>>=20 > >>>>=20 > >>>>=20 > >>>> Booting 12.0-STABLE #78 r349288: Sat Jun 22 09:10:25 CEST 2019 amd64 > >>>> works fine with nothing changed except the OS version. > >>>>=20 > >>>>=20 > >>>> Booting 2.0-STABLE #78 r349288 works fine: > >>>>=20 > >>>> [...] > >>>> SeaBIOS (version rel-1.12.1.3-0-g300e8b7) > >>>>=20 > >>>> Press F10 key now for boot menu > >>>>=20 > >>>> Booting from Hard Disk... > >>>> | > >>>>=20 > >>>> onsoles: internal video/keyboard =20 > >>>> IOS drive C: is disk0=20 > >>>> IOS drive D: is disk1=20 > >>>> IOS 639kB/3404444kB available memory=20 > >>>>=20 > >>>> reeBSD/x86 bootstrap loader, Revision 1.1=20 > >>>> Mon Apr 15 21:28:11 CEST 2019 root@thor)=20 > >>>> oading /boot/defaults/loader.conf=20 > >>>> oading /boot/device.hints=20 > >>>> oading /boot/loader.conf=20 > >>>> oading /boot/loader.conf.local=20 > >>>> Loading kernel... > >>>> /boot/kernel/kernel text=3D0xb005e8 \ > >>>> [...] > >>>>=20 > >>>> In the message taken from the serial console the first column of > >>>> characters is lost due to an error in the output which seems FreeBSD > >>>> related.=20 > >>>=20 > >>> It certainly does look weird - sio_putc() is used in boot2 and it=E2= =80=99s > >>> implementation is using same principe as comc_putchat() in comconsole= .c > >>> (even if it is asm versus c code). Since the serial data is interpret= ed > >>> by terminal, it feels more about terminal emulator issue (line > >>> discipline, cabling, usb to serial dongle?) =20 > >>=20 > >> We use here a null modem cabling with an integrated USB-to-UART/TTL > >> converter, which is attached to a FreeBSD CURRENT (most recent) box: > >>=20 > >> [...] > >> ugen2.3: at usbus2 > >> uftdi0 on uhub4 > >> uftdi0: on usbus2 > >> [...] > >>=20 > >> it is a=20 > >> StarTech.com 1 Port USB Nullmodem RS232 Adapter Kabel (USB 2.0 FTDI > >> chipset). > >>=20 > >> Regards, > >> oh > >>=20 > >> =20 > >>>=20 > >>> rgds, > >>> toomas > >>>=20 > >>> =20 > >>>>=20 > >>>> The file /boot/loader.conf.local contains these lines in both, worki= ng > >>>> and non-working, scenario: > >>>>=20 > >>>> [...] > >>>> boot_serial=3D"YES" > >>>> # serial speed in bits/s > >>>> comconsole_speed=3D"115200" > >>>> console=3D"comconsole" > >>>>=20 > >>>> autoboot_delay=3D"0" > >>>>=20 > >>>> verbose_loading=3D"YES" > >>>> loader_logo=3D"orb" > >>>> beastie_disable=3D"YES" > >>>>=20 > >>>> ### Microcode > >>>> #cpu_microcode_load=3D"YES" # Set this to YES to load= and > >>>> apply a #cpu_microcode_name=3D"/boot/firmware/intel-ucode.bin" # Set= this > >>>> to the microcode #cpu_microcode_type=3D"cpu_microcode" # Requir= ed for > >>>> the kernel to find # the microcode update file. > >>>>=20 > >>>>=20 > >>>> # disable Process Table Isolation > >>>> #vm.pmap.pti=3D0 > >>>>=20 > >>>> kern.geom.label.gptid.enable=3D0 > >>>>=20 > >>>> # Limit the phys. memory > >>>> #hw.physmem=3D1073741824 # 1 G > >>>> #hw.physmem=3D536870912 # 512 MB > >>>> #hw.physmem=3D268435456 # 256 MB > >>>>=20 > >>>> # Da mehr als 1 igb NIC an Bord! Siehe man igb(4) > >>>> kern.ipc.nmbclusters=3D757350 > >>>> #kern.ipc.nmbjumbo9k=3D8192 > >>>>=20 > >>>> # NIC > >>>> #hw.em.max_interrupt_rate=3D32000 > >>>> hw.em.max_interrupt_rate=3D16000 > >>>>=20 > >>>> #If non-zero, enable EXPERIMENTAL feature to improve concurrent Fort= una > >>>> performance kern.random.fortuna.concurrent_read=3D"1" > >>>>=20 > >>>> # Forward Information Bases (FIBs) > >>>> net.fibs=3D10 > >>>> net.add_addr_allfibs=3D0 > >>>>=20 > >>>> [...] > >>>>=20 > >>>>=20 > >>>> Again, with the exact same setting 12-STABLE r349288 boots fine, > >>>> rr350274 doesn't. FreeBSD 12-STABLE r > >>>>=20 > >>>> Can someone please help? > >>>>=20 > >>>> Thanks in advance, oh =20 [...]