From owner-freebsd-arm@freebsd.org Fri Apr 7 08:22:55 2017 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD3CCD32B0A for ; Fri, 7 Apr 2017 08:22:55 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-4.reflexion.net [208.70.210.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 98882A8D for ; Fri, 7 Apr 2017 08:22:54 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 10850 invoked from network); 7 Apr 2017 08:16:13 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 7 Apr 2017 08:16:13 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Fri, 07 Apr 2017 04:16:13 -0400 (EDT) Received: (qmail 14788 invoked from network); 7 Apr 2017 08:16:13 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 7 Apr 2017 08:16:13 -0000 Received: from [192.168.1.106] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 7D2D1EC8937; Fri, 7 Apr 2017 01:16:12 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: The arm64 fork-then-swap-out-then-swap-in failures: a program source for exploring them From: Mark Millard In-Reply-To: <4DEA2D76-9F27-426D-A8D2-F07B16575FB9@dsl-only.net> Date: Fri, 7 Apr 2017 01:16:12 -0700 Cc: andrew@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <163B37B0-55D6-498E-8F52-9A95C036CDFA@dsl-only.net> References: <4DEA2D76-9F27-426D-A8D2-F07B16575FB9@dsl-only.net> To: freebsd-arm , freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Apr 2017 08:22:56 -0000 [I now can: (A) crudely control the number of allocated pages that get zeros (that should not). (B) Watch a "top -PCwaopid" display and predict if the test-architecture will fail or not before the fork() or swap-out happens.] On 2017-Apr-4, at 8:00 PM, Mark Millard wrote: > Uncommenting/commenting parts of the below program allows > exploring the problems with fork-then-swap-out-then-in on > arm64. >=20 > Note: By swap-out I mean that zero RES(ident memory) results, > for the process(s) of interest, as shown by > "top -PCwaopid" . >=20 > I discovered recently that swapping-out just before the > fork() prevents the failure from the swapping after the > fork(). >=20 > Note: > Without the fork() no problem happens. Without the later > swap-out no problem happens. Both are required. But some > activities before the fork() or between fork() and the > swap-out prevent the failures. >=20 > Some of the comments are based on a pine64+ 2GB context. > I use stress to force swap-outs during some sleeps in > the program. See also Buzilla 217239 and 217138. (I now > expect that they have the same cause.) >=20 > In my environment I've seen the fork-then-swap-out/swap-in > failures on a pine64+ 2GB and a rpi3. They are repeatable > on both. I do not have access to server-class machines, or > any other arm64 machines. >=20 >=20 > // swap_testing5.c >=20 > // Built via (cc was clang 4.0 in my case): > // > // cc -g -std=3Dc11 -Wpedantic -o swaptesting5 swap_testing5.c > // -O0 and -O2 also gets the problem. >=20 > // Note: jemalloc's tcache needs to be enabled to get the failure. > // But FreeBSD can get into a state were /etc/malloc.conf > // -> 'tcache:false' is ineffective. Also: the allocation > // size needs to by sufficiently small (<=3D SMALL_MAXCLASS) > // to see the problem. Other comments are based on a specific > // context (pine64+ 2GB). >=20 > #include // for raise(.), SIGABRT (induce core dump) > #include // for fork(), sleep(.) > #include // for pid_t > #include // for wait(.) >=20 > extern void test_setup(void); // Sets up the memory byte = patterns. > extern void test_check(void); // Tests the memory byte = patterns. > extern void memory_willneed(void); // For seeing if > // = posix_madvise(.,.,POSIX_MADV_WILLNEED) > // makes a difference. >=20 > int main(void) { > sleep(30); // Potentialy force swap-out here. > // [Swap-out here does not avoid later failures.] >=20 > test_setup(); > test_check(); // Before potential sleep(.)/swap-out or fork(.) = [passes] >=20 > sleep(30); // Potentialy force swap-out here. > // [Everything below passes if swapped-out here, > // no matter if there are later swap-outs > // or not.] >=20 > pid_t pid =3D fork(); // To test no-fork use: =3D 0; no-fork does = not fail. > int wait_status =3D 0; >=20 > // HERE: After fork; before sleep/swap-out/wait. >=20 > // if (0 < pid) memory_willneed(); // Does not prevent either = parent or > // child failure if enabled. >=20 > // if (0 =3D=3D pid) memory_willneed(); // Prevents both the parent = and the > // child failure. Disable to see > // failure of both parent and = child. > // [Presuming no prior swap-out: = that > // would make everything pass.] >=20 > // During sleep/wait: manually force this process to > // swap out. I use something like: > // stress -m 1 --vm-bytes 1800M > // in another shell and ^C'ing it after top shows the > // swapped status desired. 1800M just happened to work > // on the Pine64+ 2GB that I was using. I watch with > // top -PCwaopid [checking for zero RES(ident memory)]. >=20 > if (0 < pid) { > sleep(30); // Intend to swap-out during sleep. > // test_check(); // Test in parent before child runs (longer = sleep). > // This test fails if run for a failing = region_size > // unless earlier preventing-activity happened. > wait(&wait_status); // Only if test_check above passes or is > // disabled above. > } > if (-1 !=3D wait_status && 0 <=3D pid) { > if (0 =3D=3D pid) { sleep(90); } // Intend to swap-out during = sleep. > test_check(); // Fails for small-enough region_size, both > // parent and child processes, unless earlier > // preventing-activty happened. > } > } >=20 > // The memory and test code follows. >=20 > #include // for size_t, NULL > #include // for malloc(.), free(.) > #include // for POSIX_MADV_WILLNEED, = posix_madvise(.,.,.) >=20 > #define region_size (14u*1024u) > // Bad dyn_region pattern, parent and child processes examples: > // 256u, 2u*1024u, 4u*1024u, 8u*1024u, 9u*1024u, 12u*1024u, = 14u*1024u > // No failure examples: > // 14u*1024u+1u, 15u*1024u, 16u*1024u, 32u*1024u, = 256u*1024u*1024u > #define num_regions (256u*1024u*1024u/region_size) >=20 > typedef volatile unsigned char value_type; > struct region_struct { value_type array[region_size]; }; > typedef struct region_struct region; > static region * volatile dyn_regions[num_regions] =3D {NULL,}; >=20 > static value_type value(size_t v) { return = (value_type)((v&0xFEu)|0x1u); } > // value avoids zero values: the bad values are = zeros. >=20 > void test_setup(void) { > for(size_t i=3D0u; i dyn_regions[i] =3D malloc(sizeof(region)); > if (!dyn_regions[i]) raise(SIGABRT); >=20 > for(size_t j=3D0u; j (*dyn_regions[i]).array[j] =3D value(j); > } > } > } >=20 > void memory_willneed(void) { > for(size_t i=3D0u; i (void) posix_madvise(dyn_regions[i], region_size, = POSIX_MADV_WILLNEED); > } > } >=20 > static volatile size_t first_failure_idx =3D 0u; // dyn_regions index > static volatile size_t first_failure_pos =3D 0u; // sub-array index > static volatile size_t after_bad_idx =3D 0u; // dyn_regions index > static volatile size_t after_bad_pos =3D 0u; // sub-array index > static volatile size_t after_good_idx =3D 0u; // dyn_regions index > static volatile size_t after_good_pos =3D 0u; // sub-array index >=20 > // Note: Some failing cases get (conjunctive notation): > // > // 0 =3D=3D first_failure_idx < after_bad_idx < after_good_idx =3D=3D= num_regions > // && 0 =3D=3D first_failure_pos && 0<=3Dafter_bad_pos<=3Dregion_size = && after_good_idx=3D=3D0 > // && (after_bad_pos is a multiple of the page size in Bytes, here: > // after_bad_pos=3D=3DN*4096 for some non-negative integral value = N) > // > // other failing cases instead fail with: > // > // 0 =3D=3D first_failure && num_regions =3D=3D after_bad_idx =3D=3D = after_good_idx > // && 0 =3D=3D first_failure_pos =3D=3D after_bad_pos =3D=3D = after_good_idx > // > // after_bad_idx strongly tends to vary from failing run to failing = run > // as does after_bad_pos. >=20 > // Note: The working cases get: > // > // num_regions =3D=3D first_failure =3D=3D after_bad_idx =3D=3D = after_good_idx > // && 0 =3D=3D first_failure_pos =3D=3D after_bad_pos =3D=3D = after_good_idx >=20 > void test_check(void) { > first_failure_idx =3D first_failure_pos =3D 0u; >=20 > while (first_failure_idx < num_regions) { > while ( first_failure_pos < region_size > && ( value(first_failure_pos) > =3D=3D = (*dyn_regions[first_failure_idx]).array[first_failure_pos] > ) > ) { > first_failure_pos++; > } >=20 > if (region_size !=3D first_failure_pos) break; >=20 > first_failure_idx++; > first_failure_pos =3D 0u; > } >=20 > after_bad_idx =3D first_failure_idx; > after_bad_pos =3D first_failure_pos; >=20 > while (after_bad_idx < num_regions) { > while ( after_bad_pos < region_size > && ( value(after_bad_pos) > !=3D = (*dyn_regions[after_bad_idx]).array[after_bad_pos] > ) > ) { > after_bad_pos++; > } >=20 > if(region_size !=3D after_bad_pos) break; >=20 > after_bad_idx++; > after_bad_pos =3D 0u; > } >=20 > after_good_idx =3D after_bad_idx; > after_good_pos =3D after_bad_pos; >=20 > while (after_good_idx < num_regions) { > while ( after_good_pos < region_size > && ( value(after_good_pos) > =3D=3D = (*dyn_regions[after_good_idx]).array[after_good_pos] > ) > ) { > after_good_pos++; > } >=20 > if(region_size !=3D after_good_pos) break; >=20 > after_good_idx++; > after_good_pos =3D 0u; > } >=20 > if (num_regions !=3D first_failure_idx) raise(SIGABRT); > } I've found that for the above swap_testing5.c I can make variations that change how much of the allocated region prefix ends up zero vs. stays good. I vary the sleep time between testing the initialized allocations and doing the fork. The longer the sleep the more zero pages show up (be sure to read the comments): # diff swap_testing[56].c = = 1c1 < // swap_testing5.c --- > // swap_testing6.c 5c5 < // cc -g -std=3Dc11 -Wpedantic -o swaptesting5 swap_testing5.c --- > // cc -g -std=3Dc11 -Wpedantic -o swaptesting5 swap_testing6.c 33c33 < sleep(30); // Potentialy force swap-out here. --- > sleep(150); // Potentialy force swap-out here. 37a38,48 > // For no-swap-out here cases: > // > // The longer the sleep here the more allocations > // that end up as zero. > // > // top's Mem Active, Inact, Wired, Bug, Free and > // Swap Total, Used, and Free stay unchanged. > // What does change is the process RES decreases > // while the process SIZE and SWAP stay unchanged > // during this sleep. >=20 NOTE: On other architectures that I've tried (such as armv6/v7) RES does not decrease during the sleep --and the problem does not happen even for as long of sleeps as I've tried. (I use "stress -m 2 --vm-bytes 900M" on armv6/v7 instead of -m 1 --vm-bytes 1800M because that large in one process is not allowed.) So watching top's RES during the sleep (longer than a few seconds) just before the fork() predicts the later fails-vs.-not status: If RES decreases (while other things associated with the process status stay the same) then there will be a failure. At this point I've no clue why the sleeping process has a decreasing RES(ident memory) size. I infer that without the sleep there still is a small amount of loss of RES but on too short of a timescale to observe in a "top -PCwaopid" or other such: in other words that the same behavior is causing the failure then as well, possibly for a loss of only one page of RES. =3D=3D=3D Mark Millard markmi at dsl-only.net