Date: Sat, 7 Jan 2017 09:51:26 +0100 From: Roman Divacky <rdivacky@vlakno.cz> To: Mark Millard <markmi@dsl-only.net> Cc: Justin Hibbits <chmeeedalf@gmail.com>, Nathan Whitehorn <nwhitehorn@freebsd.org>, FreeBSD Toolchain <freebsd-toolchain@freebsd.org>, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org> Subject: Re: clang 3.9.0 buildkernel on old powerpc64's vs. trying to build hwpmc_e500.o and the like. . . [Actually .__start code failure vs. working and R_PPC64_ADDR16_DS vs. R_PPC64_TOC16_DS] Message-ID: <20170107085126.GA82107@vlakno.cz> In-Reply-To: <613BB28B-46F1-4959-B576-C8AD42A21200@dsl-only.net> References: <D3DE2D12-9885-4154-B680-6DA5A8B62A56@dsl-only.net> <D9C54972-8D21-4D55-A707-4FFC2BDCD9FE@dsl-only.net> <20161207190057.GA58950@vlakno.cz> <E1376C20-C1BD-418D-81C6-CDDE479342CA@dsl-only.net> <CE88C1F4-B9BD-4D45-8DF0-F1079C3257A5@dsl-only.net> <20161208185541.GA33364@vlakno.cz> <E49F7EE4-7A62-4601-98DC-4C4791B7158D@dsl-only.net> <20161208221452.GA42380@vlakno.cz> <20161212210922.GA27403@vlakno.cz> <613BB28B-46F1-4959-B576-C8AD42A21200@dsl-only.net>
next in thread | previous in thread | raw e-mail | index | archive | help
That's a great progress. Can you produce minimal self contained test case that exhibits this bug? And submit it to llvm bugzilla? Also, clang3.9 defaults to using it's own internal asm, what happens if you add -no-integrated-as to CFLAGS and recompile the kernel? That should remove this llvm assembly problem. Does it boot? Thanks Mark, really great progress. Roman On Thu, Jan 05, 2017 at 09:39:31PM -0800, Mark Millard wrote: > [Summary: I tracked the boot code problem back to clang 3.9.x using R_PPC64_ADDR16_DS > with .toc in locore.o (that does not work) but xtoolchain using R_PPC64_TOC16_DS > with .toc in locore.o (that does work).] > > On 2016-Dec-12, at 1:09 PM, Roman Divacky <rdivacky at vlakno.cz> wrote: > > > Ping.... Can you take a look Nathan? > > > > Thanks! Roman > > > > On Thu, Dec 08, 2016 at 11:14:52PM +0100, Roman Divacky wrote: > >> I believe the code comes from sys/powerpc/aim/locore64.S and if you compare > >> the different values from the disssembly with the .S code you can see > >> it's __tocbase and TOC_REF(). > >> > >> I wonder if the assembly has the -mminimal-toc knowledge hardcoded in somehow. > >> I am not sure what expectations the locore64.S has about the kernel layout that > >> we're probably breaking. > >> > >> I've CCed Nathan Whitehorn. Nathan, can you take a look please? > >> > >> Thanks, Roman > >> > >> On Thu, Dec 08, 2016 at 02:03:58PM -0800, Mark Millard wrote: > >>> [I have dropped the patch related information and just have > >>> failing-boot related information here.] > >>> > >>> On 2016-Dec-8, at 10:55 AM, Roman Divacky <rdivacky at vlakno.cz> wrote: > >>> > >>>> Can you try to investigate why it dies? I am not convinced -mminimal-toc > >>>> would result in a boot failure. I think the kernel would fail to link. > >>> > >>> I give information for both devel/powerpc64-binutils based > >>> and for WITH_BINUTILS_BOOTSTRAP= based. They are different. > >>> > >>> For using 2.25.1 of devel/powerpc64-binutils (a cross build): > >>> (from camera image of screen) > >>> > >>> . . . (omitted material) . . . > >>> Type '?' for a list of commands, 'help' for more detailed help. > >>> OK unload > >>> OK boot ker390 > >>> /boot/ker390/kernel data=0xf851a8+0x42dd98 syms=[0x8+0xd6848+0x8+0xf1137] > >>> /boot/entropy size=0x1000 > >>> Booting. . . > >>> Kernel entry at 0x100160 > >>> > >>> Invalid memory access at %SSR0: 00000000.001001b0 %SRR1:90000000.00003030 > >>> > >>> Apple PowerMac11,2 5.2.7f1 BootROM builtin on 09/30/005 at 15:31:03 > >>> . . . (omitted material) . . . > >>> ok > >>> 0 > > >>> > >>> The only options at this point are: > >>> > >>> mac-boot > >>> shut-down > >>> > >>> > >>> From objdump for the above failing boot > >>> but with notes added: > >>> (Note: booting xtoolchain kernel starts at > >>> 0000000000100120 instead; relative > >>> offsets are unchanged and the code > >>> is mostly the same.) > >>> > >>> Disassembly of section .text: > >>> 0000000000100160 <.__start> mfmsr r20 > >>> 0000000000100164 <.__start+0x4> li r21,1 > >>> 0000000000100168 <.__start+0x8> rldimi r20,r21,63,0 > >>> 000000000010016c <.__start+0xc> mtmsrd r20 > >>> 0000000000100170 <.__start+0x10> isync > >>> 0000000000100174 <.__start+0x14> nop > >>> 0000000000100178 <.__start+0x18> b 0000000000100180 <.__start+0x20> > >>> 000000000010017c <.__start+0x1c> nop > >>> 0000000000100180 <.__start+0x20> nop > >>> 0000000000100184 <.__start+0x24> bl 0000000000100190 <.__start+0x30> > >>> 0000000000100188 <.__start+0x28> .long 0x0 > >>> 000000000010018c <.__start+0x2c> .long 0xf8ce78 > >>> booting xtoolchain based kernel has: 0xfebeb8 above <<<=== note > >>> 0000000000100190 <.__start+0x30> mflr r2 > >>> 0000000000100194 <.__start+0x34> ld r1,0(r2) > >>> 0000000000100198 <.__start+0x38> add r2,r1,r2 > >>> 000000000010019c <.__start+0x3c> ld r31,-32768(r2) > >>> 00000000001001a0 <.__start+0x40> subf r31,r31,r2 > >>> 00000000001001a4 <.__start+0x44> ld r1,0(r2) <<<=== !!!!! > >>> booting xtoolchain based kernel has: r1,-32760(r2) above <<<=== !!!!! > >>> 00000000001001a8 <.__start+0x48> addi r1,r1,16288 > >>> 00000000001001ac <.__start+0x4c> add r1,r1,r31 > >>> 00000000001001b0 <.__start+0x50> std r3,48(r1) > >>> SRR0 points at the above instruction > >>> (I stopped comparing to the booting xtoolchain > >>> based code after this.) > >>> 00000000001001b4 <.__start+0x54> std r4,56(r1) > >>> 00000000001001b8 <.__start+0x58> std r5,64(r1) > >>> 00000000001001bc <.__start+0x5c> std r6,72(r1) > >>> 00000000001001c0 <.__start+0x60> bl 00000000001001cc <.__start+0x6c> > >>> 00000000001001c4 <.__start+0x64> .long 0x0 > >>> 00000000001001c8 <.__start+0x68> .long 0xf84ed4 > >>> 00000000001001cc <.__start+0x6c> mflr r3 > >>> 00000000001001d0 <.__start+0x70> ld r4,0(r3) > >>> 00000000001001d4 <.__start+0x74> add r3,r4,r3 > >>> 00000000001001d8 <.__start+0x78> mr r4,r31 > >>> 00000000001001dc <.__start+0x7c> bl 0000000000b18ebc <.elf_reloc_self> > >>> 00000000001001e0 <.__start+0x80> nop > >>> 00000000001001e4 <.__start+0x84> ld r3,48(r1) > >>> 00000000001001e8 <.__start+0x88> ld r4,56(r1) > >>> 00000000001001ec <.__start+0x8c> ld r5,64(r1) > >>> 00000000001001f0 <.__start+0x90> ld r6,72(r1) > >>> 00000000001001f4 <.__start+0x94> mr r4,r2 > >>> 00000000001001f8 <.__start+0x98> bl 0000000000b1e980 <.powerpc_init> > >>> 00000000001001fc <.__start+0x9c> nop > >>> 0000000000100200 <.__start+0xa0> mr r1,r3 > >>> 0000000000100204 <.__start+0xa4> li r3,0 > >>> 0000000000100208 <.__start+0xa8> std r3,0(r1) > >>> 000000000010020c <.__start+0xac> bl 00000000005c4af8 <.mi_startup> > >>> 0000000000100210 <.__start+0xb0> nop > >>> 0000000000100214 <.__start+0xb4> b 0000000000100214 <.__start+0xb4> > >>> > >>> > >>> > >>> For using WITH_BINUTILS_BOOTSTRAP= based binutils (a cross build): > >>> (completes for buildkernel; fails for buildworld) > >>> > >>> . . . (omitted material) . . . > >>> Type '?' for a list of commands, 'help' for more detailed help. > >>> OK unload > >>> OK boot ker39a > >>> /boot/ker39a/kernel data=0xfd6318+0x42dda8 syms=[0x8+0xd6860+0x8+0xf1193] > >>> /boot/entropy size=0x1000 > >>> Booting. . . > >>> Kernel entry at 0x100160 > >>> > >>> Invalid memory access at %SSR0: 00000000.00000000 %SRR1:10000000.00081000 > >>> > >>> Apple PowerMac11,2 5.2.7f1 BootROM builtin on 09/30/005 at 15:31:03 > >>> . . . (omitted material) . . . > >>> ok > >>> 0 > > >>> > >>> The only options at this point are: > >>> > >>> mac-boot > >>> shut-down > >>> > >>> The problem here is a different code order and a matching > >>> wrong start address that does not track the difference. > >>> (From objdump.) Note: the same 0(r2) vs. -32760(r2) oddity > >>> exists in the start routine as well. > >>> > >>> Disassembly of section .text: > >>> 0000000000100160 <.__start-0x2030> std r2,40(r1) > >>> 0000000000100164 <.__start-0x202c> addis r2,r2,1 > >>> 0000000000100168 <.__start-0x2028> addi r2,r2,-8 > >>> 000000000010016c <.__start-0x2024> b 0000000000b2c8e0 <.cpu_switch> > >>> 0000000000100170 <.__start-0x2020> std r2,40(r1) > >>> 0000000000100174 <.__start-0x201c> addis r2,r2,1 > >>> 0000000000100178 <.__start-0x2018> addi r2,r2,-8 > >>> 000000000010017c <.__start-0x2014> b 0000000000ade6c8 <.sf_buf_alloc> > >>> 0000000000100180 <.__start-0x2010> std r2,40(r1) > >>> 0000000000100184 <.__start-0x200c> addis r2,r2,1 > >>> 0000000000100188 <.__start-0x2008> addi r2,r2,-8 > >>> 000000000010018c <.__start-0x2004> b 0000000000a83f78 <.vm_fault_hold> > >>> 0000000000100190 <.__start-0x2000> std r2,40(r1) > >>> 0000000000100194 <.__start-0x1ffc> addis r2,r2,1 > >>> 0000000000100198 <.__start-0x1ff8> addi r2,r2,-8 > >>> 000000000010019c <.__start-0x1ff4> b 0000000000b1f1e8 <.fill_regs32> > >>> 00000000001001a0 <.__start-0x1ff0> std r2,40(r1) > >>> 00000000001001a4 <.__start-0x1fec> addis r2,r2,1 > >>> 00000000001001a8 <.__start-0x1fe8> addi r2,r2,-8 > >>> 00000000001001ac <.__start-0x1fe4> b 0000000000b1a7e4 <.casueword32> > >>> 00000000001001b0 <.__start-0x1fe0> std r2,40(r1) > >>> 00000000001001b4 <.__start-0x1fdc> addis r2,r2,1 > >>> 00000000001001b8 <.__start-0x1fd8> addi r2,r2,-8 > >>> 00000000001001bc <.__start-0x1fd4> b 0000000000a8fa30 <.kmap_free_wakeup> > >>> . . . > >>> 0000000000102190 <.__start> mfmsr r20 > >>> 0000000000102194 <.__start+0x4> li r21,1 > >>> 0000000000102198 <.__start+0x8> rldimi r20,r21,63,0 > >>> 000000000010219c <.__start+0xc> mtmsrd r20 > >>> 00000000001021a0 <.__start+0x10> isync > >>> 00000000001021a4 <.__start+0x14> nop > >>> 00000000001021a8 <.__start+0x18> b 00000000001021b0 <.__start+0x20> > >>> 00000000001021ac <.__start+0x1c> nop > >>> 00000000001021b0 <.__start+0x20> nop > >>> 00000000001021b4 <.__start+0x24> bl 00000000001021c0 <.__start+0x30> > >>> 00000000001021b8 <.__start+0x28> .long 0x0 > >>> 00000000001021bc <.__start+0x2c> .long 0xfc8e48 > >>> 00000000001021c0 <.__start+0x30> mflr r2 > >>> 00000000001021c4 <.__start+0x34> ld r1,0(r2) > >>> 00000000001021c8 <.__start+0x38> add r2,r1,r2 > >>> 00000000001021cc <.__start+0x3c> ld r31,-32768(r2) > >>> 00000000001021d0 <.__start+0x40> subf r31,r31,r2 > >>> 00000000001021d4 <.__start+0x44> ld r1,0(r2) <<< same 0 vs. -32760 oddity!!! > >>> 00000000001021d8 <.__start+0x48> addi r1,r1,16288 > >>> 00000000001021dc <.__start+0x4c> add r1,r1,r31 > >>> 00000000001021e0 <.__start+0x50> std r3,48(r1) > >>> 00000000001021e4 <.__start+0x54> std r4,56(r1) > >>> 00000000001021e8 <.__start+0x58> std r5,64(r1) > >>> 00000000001021ec <.__start+0x5c> std r6,72(r1) > >>> 00000000001021f0 <.__start+0x60> bl 00000000001021fc <.__start+0x6c> > >>> 00000000001021f4 <.__start+0x64> .long 0x0 > >>> 00000000001021f8 <.__start+0x68> .long 0xfd4014 > >>> 00000000001021fc <.__start+0x6c> mflr r3 > >>> 0000000000102200 <.__start+0x70> ld r4,0(r3) > >>> 0000000000102204 <.__start+0x74> add r3,r4,r3 > >>> 0000000000102208 <.__start+0x78> mr r4,r31 > >>> 000000000010220c <.__start+0x7c> bl 0000000000101a20 <begin+0x1a20> > >>> 0000000000102210 <.__start+0x80> ld r2,40(r1) > >>> 0000000000102214 <.__start+0x84> ld r3,48(r1) > >>> 0000000000102218 <.__start+0x88> ld r4,56(r1) > >>> 000000000010221c <.__start+0x8c> ld r5,64(r1) > >>> 0000000000102220 <.__start+0x90> ld r6,72(r1) > >>> 0000000000102224 <.__start+0x94> mr r4,r2 > >>> 0000000000102228 <.__start+0x98> bl 00000000001019a0 <begin+0x19a0> > >>> 000000000010222c <.__start+0x9c> ld r2,40(r1) > >>> 0000000000102230 <.__start+0xa0> mr r1,r3 > >>> 0000000000102234 <.__start+0xa4> li r3,0 > >>> 0000000000102238 <.__start+0xa8> std r3,0(r1) > >>> 000000000010223c <.__start+0xac> bl 00000000005c6b28 <.mi_startup> > >>> 0000000000102240 <.__start+0xb0> nop > >>> 0000000000102244 <.__start+0xb4> b 0000000000102244 <.__start+0xb4> > >>> > >>> > >>> Who is most appropriate to send such information to for powerpc64? > >>> > >>> === > >>> Mark Millard > >>> markmi at dsl-only.net > > Note: > > I discovered with file that the bootstrap binutils and devel/binutils > (and devel/powerpc64-binutils) produce differing types of files: > > bootstrap binutils: sys/GENERIC64vtsc-NODBG/kernel: ELF 64-bit MSB shared object > (only tried with clang) > > devel/*binutils: sys/GENERIC64vtsc-NODBG/kernel: ELF 64-bit MSB executable > (both clang and xtoolchain tried) > > It is the devel/*binutils used with xtoolchain that produces what boots. > > > For the devel/*binutils (with clang vs. xtoolchain) . . . > > > Using objdump on locore.o I see variations based on clang vs. xtoolchain, > including the below relative to .toc handling: > (- -> clang , + -> xtoolchain) > > RELOCATION RECORDS FOR [.text]: > OFFSET TYPE VALUE > . . . > -0000000000000046 R_PPC64_ADDR16_DS .toc > +0000000000000046 R_PPC64_TOC16_DS .toc > . . . > -0000000000000182 R_PPC64_ADDR16_DS .toc > +0000000000000182 R_PPC64_TOC16_DS .toc > . . . > -0000000000000916 R_PPC64_ADDR16_DS .toc > . . . > +0000000000000916 R_PPC64_TOC16_DS .toc > . . . > > In the boot code (/boot/kernel/kernel) these match up with. . . > > Disassembly of section .text: > 0000000000100160 <.__start> mfmsr r20 # clang > vs. > Disassembly of section .text: > 0000000000100120 <.__start> mfmsr r20 # xtoolchain > > . . . > 00000000001001a4 <.__start+0x44> ld r1,0(r2) # 100160+46 clang > vs. > 0000000000100164 <.__start+0x44> ld r1,-32760(r2) # 100120+46 xtoolchain > > . . . > 00000000001002e0 <rstcodeend+0x8> ld r1,0(r2) # 100160+182 clang > vs. > 00000000001002a0 <rstcodeend+0x8> ld r1,-32760(r2) # 100120+182 xtoolchain > > . . . > 0000000000100a74 <dbtrap+0x10> ld r1,0(r1) # 100160+916 clang > vs. > 0000000000100a34 <dbtrap+0x10> ld r1,-32760(r1) # 100120+916 xtoolchain > > > > === > Mark Millard > markmi at dsl-only.net
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170107085126.GA82107>