From owner-freebsd-arm@freebsd.org Sun Apr 9 12:27:25 2017 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6CA0DD2AC9F; Sun, 9 Apr 2017 12:27:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D253DF49; Sun, 9 Apr 2017 12:27:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v39CRF34047280 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 9 Apr 2017 15:27:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v39CRF34047280 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v39CRFNJ047279; Sun, 9 Apr 2017 15:27:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 9 Apr 2017 15:27:15 +0300 From: Konstantin Belousov To: Mark Millard Cc: freebsd-arm , freebsd-hackers@freebsd.org, andrew@freebsd.org Subject: Re: The arm64 fork-then-swap-out-then-swap-in failures: a program source for exploring them Message-ID: <20170409122715.GF1788@kib.kiev.ua> References: <4DEA2D76-9F27-426D-A8D2-F07B16575FB9@dsl-only.net> <163B37B0-55D6-498E-8F52-9A95C036CDFA@dsl-only.net> <08E7A5B0-8707-4479-9D7A-272C427FF643@dsl-only.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <08E7A5B0-8707-4479-9D7A-272C427FF643@dsl-only.net> User-Agent: Mutt/1.8.0 (2017-02-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Apr 2017 12:27:25 -0000 On Sat, Apr 08, 2017 at 06:02:00PM -0700, Mark Millard wrote: > [I've identified the code path involved is the arm64 small allocations > turning into zeros for later fork-then-swapout-then-back-in, > specifically the ongoing RES(ident memory) size decrease that > "top -PCwaopid" shows before the fork/swap sequence. Hopefully > I've also exposed enough related information for someone that > knows what they are doing to get started with a specific > investigation, looking for a fix. I'd like for a pine64+ > 2GB to have buildworld complete despite the forking and > swapping involved (yep: for a time zero RES(ident memory) for > some processes involved in the build).] I was not able to follow the walls of text, but do not think that I pmap_ts_reference() is the real culprit there. Is my impression right that the issue occurs on fork, and looks as a memory corruption, where some page suddently becomes zero-filled ? And swapping seems to be involved ? It is somewhat interesting to see if the problem is reproducable on non-arm64 machines, e.g. armv7 or amd64. If answers to my two questions are yes, there is probably some bug with arm64 pmap handling of the dirty bit emulation. ARMv8.0 does not provide hardware dirty bit, and pmap interprets an accessed writeable page as unconditionally dirty. More, accessed bit is also not maintained by hardware, instead if should be set by pmap. And arm64 pmap sets the AF bit unconditionally when creating valid pte. Hmm, could you try the following patch, I did not even compiled it. diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c index 3d5756ba891..55aa402eb1c 100644 --- a/sys/arm64/arm64/pmap.c +++ b/sys/arm64/arm64/pmap.c @@ -2481,6 +2481,11 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot) sva += L3_SIZE) { l3 = pmap_load(l3p); if (pmap_l3_valid(l3)) { + if ((l3 & ATTR_SW_MANAGED) && + pmap_page_dirty(l3)) { + vm_page_dirty(PHYS_TO_VM_PAGE(l3 & + ~ATTR_MASK)); + } pmap_set(l3p, ATTR_AP(ATTR_AP_RO)); PTE_SYNC(l3p); /* XXX: Use pmap_invalidate_range */