From owner-freebsd-arm@freebsd.org Tue Mar 14 18:34:02 2017 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 543E5D0B3D6 for ; Tue, 14 Mar 2017 18:34:02 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-211-180.reflexion.net [208.70.211.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 131201C4D for ; Tue, 14 Mar 2017 18:34:01 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 6576 invoked from network); 14 Mar 2017 18:07:20 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 14 Mar 2017 18:07:20 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.30.2) with SMTP; Tue, 14 Mar 2017 14:07:20 -0400 (EDT) Received: (qmail 742 invoked from network); 14 Mar 2017 18:07:20 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 14 Mar 2017 18:07:20 -0000 Received: from [192.168.1.111] (c-67-170-167-181.hsd1.or.comcast.net [67.170.167.181]) by iron2.pdx.net (Postfix) with ESMTPSA id AEAB4EC8534; Tue, 14 Mar 2017 11:07:19 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: arm64 fork/swap data corruptions: A ~110 line C program demonstrating an example (Pine64+ 2GB context) [Corrected subject: arm64!] From: Mark Millard In-Reply-To: <16B3D614-62E1-4E58-B409-8DB9DBB35BCB@dsl-only.net> Date: Tue, 14 Mar 2017 11:07:19 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <5BEAFC6C-DA80-4D7B-AB55-977E585D1ACC@dsl-only.net> References: <01735A68-FED6-4E63-964F-0820FE5C446C@dsl-only.net> <16B3D614-62E1-4E58-B409-8DB9DBB35BCB@dsl-only.net> To: freebsd-arm , FreeBSD Current , FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3259) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Mar 2017 18:34:02 -0000 [This is just a correction to the subject-line text to say arm64 instead of amd64.] On 2017-Mar-14, at 12:58 AM, Mark Millard wrote: [Another correction I'm afraid --about alternative program variations this time.] On 2017-Mar-13, at 11:52 PM, Mark Millard wrote: > I'm still at a loss about how to figure out what stages are messed > up. (Memory coherency? Some memory not swapped out? Bad data swapped > out? Wrong data swapped in?) >=20 > But at least I've found a much smaller/simpler example to demonstrate > some problem with in my Pine64+_ 2GB context. >=20 > The Pine64+ 2GB is the only amd64 context that I have access to. Someday I'll learn to type arm64 the first time instead of amd64. > The following program fails its check for data > having its expected byte pattern in dynamically > allocated memory after a fork/swap-out/swap-in > sequence. >=20 > I'll note that the program sleeps for 60s after > forking to give time to do something else to > cause the parent and child processes to swap > out (RES=3D0 as seen in top). The following about the extra test_check() was wrong. > Note the source code line: >=20 > // test_check(); // Adding this line prevents failure. >=20 > It seem that accessing the region contents before forking > and swapping avoids the problem. But there is a problem > if the region was only written-to before the fork/swap. This was because I'd carelessly moved some loop variables to globals in a way that depended on the initialization of the globals and the extra call changed those values. I've noted code adjustments below (3 lines). I get the failures with them as well. > Another point is the size of the region matters: <=3D 14K Bytes > fails and > 14K Bytes works for as much has I have tested. >=20 >=20 > # more swap_testing.c > // swap_testing.c >=20 > // Built via (c++ was clang++ 4.0 in my case): > // > // cc -g -std=3Dc11 -Wpedantic swap_testing.c > // -O0 and -O2 also gets the problem. >=20 > #include // for fork(), sleep(.) > #include // for pid_t > #include // for wait(.) >=20 > extern void test_setup(void); // Sets up the memory byte pattern. > extern void test_check(void); // Tests the memory byte pattern. >=20 > int main(void) > { > test_setup(); test_check(); // This test passes. >=20 > pid_t pid =3D fork(); > int wait_status =3D 0;; >=20 > if (0=20 > if (-1!=3Dwait_status && 0<=3Dpid) > { > if (0=3D=3Dpid) > { > sleep(60); >=20 > // During this manually force this process to > // swap out. I use something like: >=20 > // stress -m 1 --vm-bytes 1800M >=20 > // in another shell and ^C'ing it after top > // shows the swapped status desired. 1800M > // just happened to work on the Pine64+ 2GB > // that I was using. > } >=20 > test_check(); > } > } >=20 > // The memory and test code follows. >=20 > #include // for bool, true, false > #include // for size_t, NULL > #include // for malloc(.), free(.) >=20 > #include // for raise(.), SIGABRT >=20 > #define region_size (14u*1024u) > // Bad dyn_region pattern, parent and child > // processes: > // 256u, 4u*1024u, 8u*1024u, 9u*1024u, > // 12u*1024u, 14u*1024u >=20 > // Works: > // 14u*1024u+1u, 15u*1024u, 16u*1024u, > // 32u*1024u, 256u*1024u*1024u >=20 > typedef volatile unsigned char value_type; >=20 > struct region_struct { value_type array[region_size]; }; > typedef struct region_struct region; >=20 > static region gbl_region; > static region * volatile dyn_region =3D NULL; >=20 > static value_type value(size_t v) { return (value_type)v; } >=20 > void test_setup(void) { > dyn_region =3D malloc(sizeof(region)); > if (!dyn_region) raise(SIGABRT); >=20 > for(size_t i=3D0u; i (*dyn_region).array[i] =3D gbl_region.array[i] =3D value(i); > } > } >=20 > static volatile bool gbl_failed =3D false; // Until potentially = disproved > static volatile size_t gbl_pos =3D 0u; >=20 > static volatile bool dyn_failed =3D false; // Until potentially = disproved > static volatile size_t dyn_pos =3D 0u; >=20 > void test_check(void) { gbl_pos =3D 0u; > while (!gbl_failed && gbl_pos gbl_failed =3D (value(gbl_pos) !=3D gbl_region.array[gbl_pos]); > gbl_pos++; > } >=20 dyn_pos =3D 0u; > while (!dyn_failed && dyn_pos dyn_failed =3D (value(dyn_pos) !=3D = (*dyn_region).array[dyn_pos]); > // Note: When the memory pattern fails this case is that > // records the failure. > dyn_pos++; > } >=20 > if (gbl_failed) raise(SIGABRT); > if (dyn_failed) raise(SIGABRT); // lldb reports this line for the = __raise call. > // when it fails (both parent and = child processes). > } I'm not bothering to redo the details below for the line number variations. > Other details from lldb (not using -O2 so things are > simpler, not presented in the order examined): >=20 > # lldb a.out -c /var/crash/a.out.11575.core > (lldb) target create "a.out" --core "/var/crash/a.out.11575.core" > Core file '/var/crash/a.out.11575.core' (aarch64) was loaded. > (lldb) bt > * thread #1, name =3D 'a.out', stop reason =3D signal SIGABRT > * frame #0: 0x0000000040113d38 libc.so.7`_thr_kill + 8 > frame #1: libc.so.7`__raise(s=3D6) at raise.c:52 > frame #2: a.out`test_check at swap_testing.c:103 > frame #3: a.out`main at swap_testing.c:42 > frame #4: 0x0000000000020184 a.out`__start + 364 > frame #5: ld-elf.so.1`.rtld_start at rtld_start.S:41 >=20 > (lldb) up 2 > frame #2: a.out`test_check at swap_testing.c:103 > 100 } > 101 =09 > 102 if (gbl_failed) raise(SIGABRT); > -> 103 if (dyn_failed) raise(SIGABRT); // lldb reports this = line for the __raise call. > 104 // when it fails (both = parent and child processes). > 105 } >=20 > (lldb) print dyn_pos > (size_t) $0 =3D 2 >=20 > (That is one after the failure position.) >=20 >=20 > (lldb) print dyn_region > (region *volatile) $3 =3D 0x0000000040616000 >=20 > (lldb) print *dyn_region > (region) $1 =3D { > array =3D { > [0] =3D '\0' > [1] =3D '\0' > [2] =3D '\0' > . . . (all '\0' bytes) . . . > [251] =3D '\0' > [252] =3D '\0' > [253] =3D '\0' > [254] =3D '\0' > [255] =3D '\0' > ... > } > } >=20 > (lldb) print gbl_region > (region) $2 =3D { > array =3D { > [0] =3D '\0' > [1] =3D '\x01' > [2] =3D '\x02' > . . . > [251] =3D '\xfb' > [252] =3D '\xfc' > [253] =3D '\xfd' > [254] =3D '\xfe' > [255] =3D '\xff' > ... > } > } >=20 > (lldb) disass -n main > a.out`main: > 0x2022c <+0>: sub sp, sp, #0x30 ; =3D0x30=20 > 0x20230 <+4>: stp x29, x30, [sp, #0x20] > 0x20234 <+8>: add x29, sp, #0x20 ; =3D0x20=20 > 0x20238 <+12>: stur wzr, [x29, #-0x4] > 0x2023c <+16>: bl 0x202b0 ; test_setup at = swap_testing.c:74 > 0x20240 <+20>: bl 0x20580 ; symbol stub for: = fork > 0x20244 <+24>: mov w8, wzr > 0x20248 <+28>: stur w0, [x29, #-0x8] > 0x2024c <+32>: stur wzr, [x29, #-0xc] > 0x20250 <+36>: ldur w0, [x29, #-0x8] > 0x20254 <+40>: cmp w8, w0 > 0x20258 <+44>: b.ge 0x20268 ; <+60> at = swap_testing.c > 0x2025c <+48>: sub x0, x29, #0xc ; =3D0xc=20 > 0x20260 <+52>: bl 0x20590 ; symbol stub for: = wait > 0x20264 <+56>: str w0, [sp, #0x10] > 0x20268 <+60>: mov w8, #-0x1 > 0x2026c <+64>: ldur w9, [x29, #-0xc] > 0x20270 <+68>: cmp w8, w9 > 0x20274 <+72>: b.eq 0x202a0 ; <+116> at = swap_testing.c:44 > 0x20278 <+76>: mov w8, wzr > 0x2027c <+80>: ldur w9, [x29, #-0x8] > 0x20280 <+84>: cmp w8, w9 > 0x20284 <+88>: b.gt 0x202a0 ; <+116> at = swap_testing.c:44 > 0x20288 <+92>: ldur w8, [x29, #-0x8] > 0x2028c <+96>: cbnz w8, 0x2029c ; <+112> at = swap_testing.c:42 > 0x20290 <+100>: orr w0, wzr, #0x3c > 0x20294 <+104>: bl 0x205a0 ; symbol stub for: = sleep > 0x20298 <+108>: str w0, [sp, #0xc] > 0x2029c <+112>: bl 0x20348 ; test_check at = swap_testing.c:89 > 0x202a0 <+116>: ldur w0, [x29, #-0x4] > 0x202a4 <+120>: ldp x29, x30, [sp, #0x20] > 0x202a8 <+124>: add sp, sp, #0x30 ; =3D0x30=20 > 0x202ac <+128>: ret =20 >=20 > (lldb) disass -n value > a.out`value: > 0x204cc <+0>: sub sp, sp, #0x10 ; =3D0x10=20 > 0x204d0 <+4>: str x0, [sp, #0x8] > 0x204d4 <+8>: ldrb w8, [sp, #0x8] > 0x204d8 <+12>: mov w1, w8 > 0x204dc <+16>: mov w0, w8 > 0x204e0 <+20>: str w1, [sp, #0x4] > 0x204e4 <+24>: add sp, sp, #0x10 ; =3D0x10=20 > 0x204e8 <+28>: ret =20 >=20 > (lldb) disass -n test_setup > a.out`test_setup: > 0x202b0 <+0>: sub sp, sp, #0x20 ; =3D0x20=20 > 0x202b4 <+4>: stp x29, x30, [sp, #0x10] > 0x202b8 <+8>: add x29, sp, #0x10 ; =3D0x10=20 > 0x202bc <+12>: orr x0, xzr, #0x3800 > 0x202c0 <+16>: bl 0x205b0 ; symbol stub for: = malloc > 0x202c4 <+20>: adrp x30, 48 > 0x202c8 <+24>: add x30, x30, #0x0 ; =3D0x0=20 > 0x202cc <+28>: str x0, [x30] > 0x202d0 <+32>: ldr x0, [x30] > 0x202d4 <+36>: cbnz x0, 0x202e4 ; <+52> at = swap_testing.c:78 > 0x202d8 <+40>: orr w0, wzr, #0x6 > 0x202dc <+44>: bl 0x205c0 ; symbol stub for: = raise > 0x202e0 <+48>: str w0, [sp, #0x4] > 0x202e4 <+52>: str xzr, [sp, #0x8] > 0x202e8 <+56>: orr x8, xzr, #0x3800 > 0x202ec <+60>: ldr x9, [sp, #0x8] > 0x202f0 <+64>: cmp x9, x8 > 0x202f4 <+68>: b.hs 0x2033c ; <+140> at = swap_testing.c:81 > 0x202f8 <+72>: ldr x0, [sp, #0x8] > 0x202fc <+76>: bl 0x204cc ; value at = swap_testing.c:72 > 0x20300 <+80>: adrp x30, 48 > 0x20304 <+84>: add x30, x30, #0x0 ; =3D0x0=20 > 0x20308 <+88>: adrp x8, 48 > 0x2030c <+92>: add x8, x8, #0x8 ; =3D0x8=20 > 0x20310 <+96>: ldr x9, [sp, #0x8] > 0x20314 <+100>: add x8, x8, x9 > 0x20318 <+104>: strb w0, [x8] > 0x2031c <+108>: ldr x8, [x30] > 0x20320 <+112>: ldr x9, [sp, #0x8] > 0x20324 <+116>: add x8, x8, x9 > 0x20328 <+120>: strb w0, [x8] > 0x2032c <+124>: ldr x8, [sp, #0x8] > 0x20330 <+128>: add x8, x8, #0x1 ; =3D0x1=20 > 0x20334 <+132>: str x8, [sp, #0x8] > 0x20338 <+136>: b 0x202e8 ; <+56> at = swap_testing.c > 0x2033c <+140>: ldp x29, x30, [sp, #0x10] > 0x20340 <+144>: add sp, sp, #0x20 ; =3D0x20=20 > 0x20344 <+148>: ret =20 >=20 > (lldb) disass -n test_check > a.out`test_check: > 0x20348 <+0>: sub sp, sp, #0x20 ; =3D0x20=20 > 0x2034c <+4>: stp x29, x30, [sp, #0x10] > 0x20350 <+8>: add x29, sp, #0x10 ; =3D0x10=20 > 0x20354 <+12>: b 0x20358 ; <+16> at = swap_testing.c > 0x20358 <+16>: mov w8, wzr > 0x2035c <+20>: adrp x9, 51 > 0x20360 <+24>: add x9, x9, #0x808 ; =3D0x808=20 > 0x20364 <+28>: ldrb w10, [x9] > 0x20368 <+32>: stur w8, [x29, #-0x4] > 0x2036c <+36>: tbnz w10, #0x0, 0x2038c ; <+68> at = swap_testing.c > 0x20370 <+40>: orr x8, xzr, #0x3800 > 0x20374 <+44>: adrp x9, 51 > 0x20378 <+48>: add x9, x9, #0x810 ; =3D0x810=20 > 0x2037c <+52>: ldr x9, [x9] > 0x20380 <+56>: cmp x9, x8 > 0x20384 <+60>: cset w10, lo > 0x20388 <+64>: stur w10, [x29, #-0x4] > 0x2038c <+68>: ldur w8, [x29, #-0x4] > 0x20390 <+72>: tbz w8, #0x0, 0x203ec ; <+164> at = swap_testing.c:95 > 0x20394 <+76>: adrp x8, 51 > 0x20398 <+80>: add x8, x8, #0x810 ; =3D0x810=20 > 0x2039c <+84>: ldr x0, [x8] > 0x203a0 <+88>: bl 0x204cc ; value at = swap_testing.c:72 > 0x203a4 <+92>: adrp x8, 51 > 0x203a8 <+96>: add x8, x8, #0x810 ; =3D0x810=20 > 0x203ac <+100>: adrp x30, 51 > 0x203b0 <+104>: add x30, x30, #0x808 ; =3D0x808=20 > 0x203b4 <+108>: adrp x9, 48 > 0x203b8 <+112>: add x9, x9, #0x8 ; =3D0x8=20 > 0x203bc <+116>: uxtb w0, w0 > 0x203c0 <+120>: ldr x10, [x8] > 0x203c4 <+124>: add x9, x9, x10 > 0x203c8 <+128>: ldrb w11, [x9] > 0x203cc <+132>: cmp w0, w11 > 0x203d0 <+136>: cset w11, ne > 0x203d4 <+140>: and w11, w11, #0x1 > 0x203d8 <+144>: strb w11, [x30] > 0x203dc <+148>: ldr x9, [x8] > 0x203e0 <+152>: add x9, x9, #0x1 ; =3D0x1=20 > 0x203e4 <+156>: str x9, [x8] > 0x203e8 <+160>: b 0x20358 ; <+16> at = swap_testing.c > 0x203ec <+164>: b 0x203f0 ; <+168> at = swap_testing.c > 0x203f0 <+168>: mov w8, wzr > 0x203f4 <+172>: adrp x9, 51 > 0x203f8 <+176>: add x9, x9, #0x818 ; =3D0x818=20 > 0x203fc <+180>: ldrb w10, [x9] > 0x20400 <+184>: str w8, [sp, #0x8] > 0x20404 <+188>: tbnz w10, #0x0, 0x20424 ; <+220> at = swap_testing.c > 0x20408 <+192>: orr x8, xzr, #0x3800 > 0x2040c <+196>: adrp x9, 51 > 0x20410 <+200>: add x9, x9, #0x820 ; =3D0x820=20 > 0x20414 <+204>: ldr x9, [x9] > 0x20418 <+208>: cmp x9, x8 > 0x2041c <+212>: cset w10, lo > 0x20420 <+216>: str w10, [sp, #0x8] > 0x20424 <+220>: ldr w8, [sp, #0x8] > 0x20428 <+224>: tbz w8, #0x0, 0x20488 ; <+320> at = swap_testing.c > 0x2042c <+228>: adrp x8, 51 > 0x20430 <+232>: add x8, x8, #0x820 ; =3D0x820=20 > 0x20434 <+236>: ldr x0, [x8] > 0x20438 <+240>: bl 0x204cc ; value at = swap_testing.c:72 > 0x2043c <+244>: adrp x8, 51 > 0x20440 <+248>: add x8, x8, #0x820 ; =3D0x820=20 > 0x20444 <+252>: adrp x30, 51 > 0x20448 <+256>: add x30, x30, #0x818 ; =3D0x818=20 > 0x2044c <+260>: adrp x9, 48 > 0x20450 <+264>: add x9, x9, #0x0 ; =3D0x0=20 > 0x20454 <+268>: uxtb w0, w0 > 0x20458 <+272>: ldr x9, [x9] > 0x2045c <+276>: ldr x10, [x8] > 0x20460 <+280>: add x9, x9, x10 > 0x20464 <+284>: ldrb w11, [x9] > 0x20468 <+288>: cmp w0, w11 > 0x2046c <+292>: cset w11, ne > 0x20470 <+296>: and w11, w11, #0x1 > 0x20474 <+300>: strb w11, [x30] > 0x20478 <+304>: ldr x9, [x8] > 0x2047c <+308>: add x9, x9, #0x1 ; =3D0x1=20 > 0x20480 <+312>: str x9, [x8] > 0x20484 <+316>: b 0x203f0 ; <+168> at = swap_testing.c > 0x20488 <+320>: adrp x8, 51 > 0x2048c <+324>: add x8, x8, #0x808 ; =3D0x808=20 > 0x20490 <+328>: ldrb w9, [x8] > 0x20494 <+332>: tbz w9, #0x0, 0x204a4 ; <+348> at = swap_testing.c > 0x20498 <+336>: orr w0, wzr, #0x6 > 0x2049c <+340>: bl 0x205c0 ; symbol stub for: = raise > 0x204a0 <+344>: str w0, [sp, #0x4] > 0x204a4 <+348>: adrp x8, 51 > 0x204a8 <+352>: add x8, x8, #0x818 ; =3D0x818=20 > 0x204ac <+356>: ldrb w9, [x8] > 0x204b0 <+360>: tbz w9, #0x0, 0x204c0 ; <+376> at = swap_testing.c:105 > 0x204b4 <+364>: orr w0, wzr, #0x6 > 0x204b8 <+368>: bl 0x205c0 ; symbol stub for: = raise > -> 0x204bc <+372>: str w0, [sp] > 0x204c0 <+376>: ldp x29, x30, [sp, #0x10] > 0x204c4 <+380>: add sp, sp, #0x20 ; =3D0x20=20 > 0x204c8 <+384>: ret =20 >=20 > # uname -apKU > FreeBSD pine64 12.0-CURRENT FreeBSD 12.0-CURRENT r314638M arm64 = aarch64 1200023 1200023 >=20 > buildworld buildlkernel did not have MALLOC_PRODUCTION=3D defined. The = kernel is a > non-debug kernel. (Previous to these experiments my other corruption = examples > were not caught by a debug kernel. I'm not hopeful that this simpler = context > would either.) =3D=3D=3D Mark Millard markmi at dsl-only.net _______________________________________________ freebsd-arm@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-arm To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org"