Date: Sun, 17 May 2015 09:59:39 -0400 (EDT) From: Keith White <kwhite@site.uottawa.ca> To: Warner Losh <imp@bsdimp.com> Cc: Daisuke Aoyama <aoyama@peach.ne.jp>, freebsd-arm@freebsd.org, Ian Lepore <ian@freebsd.org> Subject: Re: Translation Fault panic when trying to use an mfs_root on BBB [solved?] [patch] Message-ID: <alpine.BSF.2.20.1505170953120.93686@localhost.my.domain> In-Reply-To: <B4D05B26-E699-4BC5-8B13-2FDCF6BDF75B@bsdimp.com> References: <alpine.BSF.2.20.1505131848310.98564@localhost.my.domain> <CAFHCsPXfj0Q9gYn-auK_7zQiA_HaijhzFddm1r2TJ1K7ftAebA@mail.gmail.com> <alpine.BSF.2.20.1505142004010.13692@localhost.my.domain> <CAFHCsPVF4dQryUb%2B3PmeZvpsQ8u2Xch2JirQ3uFcUhXn5fFkuQ@mail.gmail.com> <1431822588.91685.48.camel@freebsd.org> <F54C6DD3CE664644A5FA6CA32E989279@ad.peach.ne.jp> <alpine.BSF.2.20.1505162216230.46783@localhost.my.domain> <B4D05B26-E699-4BC5-8B13-2FDCF6BDF75B@bsdimp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 16 May 2015, Warner Losh wrote:
>
>> On May 16, 2015, at 8:40 PM, Keith White <kwhite@site.uottawa.ca> wrote:
>>
>> On Sun, 17 May 2015, Daisuke Aoyama wrote:
>>
>>> --------------------------------------------------
>>> From: "Ian Lepore" <ian@freebsd.org>
>>> Sent: Sunday, May 17, 2015 9:29 AM
>>> To: "Svatopluk Kraus" <onwahe@gmail.com>
>>> Cc: "Keith White" <kwhite@site.uottawa.ca>; <freebsd-arm@freebsd.org>
>>> Subject: Re: Translation Fault panic when trying to use an mfs_root on BBB [solved?] [patch]
>>>
>>>> On Fri, 2015-05-15 at 10:18 +0200, Svatopluk Kraus wrote:
>>>>> On Fri, May 15, 2015 at 2:13 AM, Keith White <kwhite@site.uottawa.ca> wrote:
>>>>>> On Thu, 14 May 2015, Svatopluk Kraus wrote:
>>>>>>
>>>>>>> On Thu, May 14, 2015 at 2:03 AM, Keith White <kwhite@site.uottawa.ca>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I get the following panic when trying to load an mfs_root on a
>>>>>>>> reasonably current BBB image:
>>>>>>>>
>>>>>>>> Fatal kernel mode data abort: 'Translation Fault (L1)' on read
>>>>>>>> trapframe: 0xdd43fd50
>>>>>>>> FSR=00000005, FAR=01211ef0, spsr=20000113
>>>>>>>> r0 =c35f4200, r1 =01211f00, r2 =000001e0, r3 =3dc1dd00
>>>>>>>> r4 =c3638930, r5 =c362d838, r6 =c362d810, r7 =c06037b8
>>>>>>>> r8 =0000002d, r9 =00000000, r10=c3638930, r11=dd43fdf0
>>>>>>>> r12=00000000, ssp=dd43fde0, slr=c0249918, pc =c05c6a68
>>>>>>>>
>>>>>>>> [ thread pid 4 tid 100054 ]
>>>>>>>> Stopped at memmove+0x29c: ldmdb r1!, {r3-r4, r12, r14}
>>>>>>>> db>
>>>>>>>>
>>>>>>>> Hints, suggestions?
>>>>>>>>
>>>>>>>> ...keith
>>>>>>>>
>>>>>>>> ---------------------------------
>>>>>>>> More (trimmed) details from boot:
>>>>>>>>
>>>>>>>> ...
>>>>>>>> U-Boot 2014.10 (Mar 19 2015 - 18:29:51)
>>>>>>>> ...
>>>>>>>> FreeBSD/armv6hf U-Boot loader, Revision 1.2
>>>>>>>> (kwhite@freebsd11, Tue Mar 17 22:23:25 EDT 2015)
>>>>>>>> ...
>>>>>>>> Found U-Boot device: disk
>>>>>>>> Checking unit=1 slice=<auto> partition=<auto>... good.
>>>>>>>> /boot/kernel/kernel data=0x505c2c+0x923d4 syms=[0x4+0x606a0+0x4+0x65ed4]
>>>>>>>> /boot/kernel/snd_uaudio.ko text=0xed3c data=0x620+0x10
>>>>>>>> syms=[0x4+0x1ec0+0x4+0x1a2f]
>>>>>>>> ...
>>>>>>>> Type '?' for a list of commands, 'help' for more detailed help.
>>>>>>>> loader> load -t mfs_root /rootfs
>>>>>>>> /rootfs size=0x858000
>>>>>>>> loader> boot -asv
>>>>>>>> Booting...
>>>>>>>> /boot/dtb/beaglebone-black.dtb size=0x24b4
>>>>>>>> Loaded DTB from file 'beaglebone-black.dtb'.
>>>>>>>> Kernel entry at 0x80200100...
>>>>>>>> Kernel args: -asv
>>>>>>>> KDB: debugger backends: ddb
>>>>>>>> KDB: current backend: ddb
>>>>>>>> Copyright (c) 1992-2015 The FreeBSD Project.
>>>>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>>>>>>> The Regents of the University of California. All rights reserved.
>>>>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>>>>>>>> FreeBSD 11.0-CURRENT #1 r282672M: Tue May 12 06:57:24 EDT 2015
>>>>>>>>
>>>>>>>> kwhite@freebsd11:/usr/obj/arm.armv6hf/tank/RPI/head/sys/BEAGLEBONE-LOCAL
>>>>>>>> arm
>>>>>>>> FreeBSD clang version 3.6.0 (tags/RELEASE_360/final 230434) 20150225
>>>>>>>> WARNING: WITNESS option enabled, expect reduced performance.
>>>>>>>> Preloaded elf kernel "/boot/kernel/kernel" at 0xc1219000.
>>>>>>>> ...
>>>>>>>> mmc0: Probing bus
>>>>>>>> usbus0: 480Mbps High Speed USB v2.0
>>>>>>>> usbus1: 480Mbps High Speed USB v2.0
>>>>>>>> md0: Preloaded image </rootfs> 8749056 bytes at 0x9b9f00
>>>>>>>> ugen1.1: <Mentor Graphics> at usbus1
>>>>>>>> uhub0: <Mentor Graphics OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1>
>>>>>>>> on
>>>>>>>> usbus1
>>>>>>>> ugen0.1: <Mentor Graphics> at usbus0
>>>>>>>> uhub1: <Mentor Graphics OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1>
>>>>>>>> on
>>>>>>>> usbus0
>>>>>>>> Fatal kernel mode data abort: 'Translation Fault (L1)' on read
>>>>>>>> trapframe: 0xdd43fd50
>>>>>>>> FSR=00000005, FAR=01211ef0, spsr=20000113
>>>>>>>> r0 =c35f4200, r1 =01211f00, r2 =000001e0, r3 =3dc1dd00
>>>>>>>> r4 =c3638930, r5 =c362d838, r6 =c362d810, r7 =c06037b8
>>>>>>>> r8 =0000002d, r9 =00000000, r10=c3638930, r11=dd43fdf0
>>>>>>>> r12=00000000, ssp=dd43fde0, slr=c0249918, pc =c05c6a68
>>>>>>>>
>>>>>>>> [ thread pid 4 tid 100054 ]
>>>>>>>> Stopped at memmove+0x29c: ldmdb r1!, {r3-r4, r12, r14}
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Well, FAR (fault address) points to user address space. System is
>>>>>>> still in boot process and no user address should be used. The first
>>>>>>> thing is to find out if arguments pushed to bcopy() in
>>>>>>> mdstart_preload() are correct. Can you print them out?
>>>>>>> ...
>>>>>>
>>>>>>
>>>>>> Thanks for the hint! After a little digging with ddb, it appears
>>>>>> that the fault address is missing an offset of KERNBASE (0xc0000000).
>>>>>> The mfs_root being found at 0x9b9f00 ("md0: preloaded image at") +
>>>>>> KERNBASE (0xc0000000). The following patch fixes the symptom, and
>>>>>> allows a boot to successfully complete. The resulting md is also
>>>>>> usable as /.
>>>>>>
>>>>>> I'm sure there's a more correct patch?
>>>>>>
>>>>>> Index: sys/dev/md/md.c
>>>>>> ===================================================================
>>>>>> --- sys/dev/md/md.c (revision 282672)
>>>>>> +++ sys/dev/md/md.c (working copy)
>>>>>> @@ -1590,7 +1590,11 @@
>>>>>> len = preload_fetch_size(mod);
>>>>>> if (ptr != NULL && len != 0) {
>>>>>> sx_xlock(&md_sx);
>>>>>> +#ifdef __arm__
>>>>>> + md_preloaded(KERNBASE + ptr, len, name);
>>>>>> +#else
>>>>>> md_preloaded(ptr, len, name);
>>>>>> +#endif
>>>>>> sx_xunlock(&md_sx);
>>>>>> }
>>>>>> }
>>>>> Thanks for debugging it. The preload_fetch_addr() already uses
>>>>> preload_addr_relocate to adjust returned address. IMO, a patch should
>>>>> either deal with preload_addr_relocate and init it correctly or fix
>>>>> MODINFO_ADDR info in kernel modules. Now, preload_addr_relocate is set
>>>>> only when FREEBSD_BOOT_LOADER is defined. In this moment, I know
>>>>> nothing about how MODINFO_ADDR info is generated.
>>>>> BTW, physical address space starts at 0x80000000 on BBB and KERNBASE
>>>>> is 0xC0000000, so I would be expecting that preload_addr_relocate
>>>>> should be set to 0x40000000. The FAR is 0x01211ef0, so it's not
>>>>> physical address. And as your patch helps, it's an offset to either
>>>>> physical or virtual address space. So it looks more like MODINFO_ADDR
>>>>> info problem.
>>>> I think the problem here is likely to be in ubldr. Today I tried to
>>>> recreate this and all I could get was complete system lockup right after
>>>> the kernel copyright displayed. Eventually I noticed that your root
>>>> image is less than 10MB, so I created a little image about the same size
>>>> as yours and put init(8) in it, and with that I get the same panic you
>>>> do (different physical address, I'm doing it on a wandboard).
>>>> I think there are at least two separate problems here. ubldr isn't
>>>> setting the right metadata for the load address, and then separately
>>>> from that, trying to load really big files corrupts memory somehow in a
>>>> way that locks up the kernel.
>>>> I'll keep poking at it.
>>>
>>> I posted about ubldr/mfsroot last year. no one interest it...
>>> http://lists.freebsd.org/pipermail/freebsd-arm/2014-December/009839.html
>>> I think ubldr should be fixed.
>>> --
>>> Daisuke Aoyama
>>>
>>>
>>
>> Thanks Ian for looking at this!
>>
>> Based upon the comments by Daisuke Aoyama, the patch above using
>> KERNBASE becomes this using preload_addr_relocate:
>>
>> -----------------
>> --- sys/dev/md/md.c (revision 282672)
>> +++ sys/dev/md/md.c (working copy)
>> @@ -1590,7 +1590,11 @@
>> len = preload_fetch_size(mod);
>> if (ptr != NULL && len != 0) {
>> sx_xlock(&md_sx);
>> +#ifdef __arm__
>> + md_preloaded(ptr - preload_addr_relocate, len, name);
>> +#else
>> md_preloaded(ptr, len, name);
>> +#endif
>> sx_xunlock(&md_sx);
>> }
>> }
>
> This is almost certainly the wrong place to fix this. The right place to fix it is in ubldr. It says that all the ptrs are AFU only on arm, and this one is likely the first of many.
>
> Warner
>
>
You're too generous, no "almost" about it.
ubldr is well beyond my capabilities to fix. Sticky tape for md
allows me to use an mfs_root in the meantime...
...keith
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.20.1505170953120.93686>