From owner-freebsd-arm@freebsd.org Thu Jul 19 03:56:22 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E082104738F for ; Thu, 19 Jul 2018 03:56:22 +0000 (UTC) (envelope-from warlock@phouka1.phouka.net) Received: from phouka1.phouka.net (phouka1.phouka.net [107.170.196.116]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "phouka.net", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0C2667B7F0 for ; Thu, 19 Jul 2018 03:54:28 +0000 (UTC) (envelope-from warlock@phouka1.phouka.net) Received: from phouka1.phouka.net (localhost [127.0.0.1]) by phouka1.phouka.net (8.15.2/8.15.2) with ESMTPS id w6J3r5sX073551 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 18 Jul 2018 20:53:05 -0700 (PDT) (envelope-from warlock@phouka1.phouka.net) Received: (from warlock@localhost) by phouka1.phouka.net (8.15.2/8.15.2/Submit) id w6J3r5v3073550; Wed, 18 Jul 2018 20:53:05 -0700 (PDT) (envelope-from warlock) Date: Wed, 18 Jul 2018 20:53:05 -0700 From: John Kennedy To: bob prohaska Cc: freebsd-arm@freebsd.org Subject: Re: RPI3 swap experiments Message-ID: <20180719035305.GB92843@phouka1.phouka.net> References: <20180629233937.GC35717@www.zefox.net> <0f137e06-214a-3e8c-a216-f061ec04ac2c@sentry.org> <20180630005145.GA43801@www.zefox.net> <6f3406e2-71f3-d0c2-2b65-703e1a1d3c25@sentry.org> <8e92b2b7-da61-3efb-7231-9fac76b2c1d4@sentry.org> <2deaaec3-f78f-0b09-5ca7-27e14c6979f9@sentry.org> <20180704004554.GA61273@www.zefox.net> <20180718060650.GA24566@www.zefox.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180718060650.GA24566@www.zefox.net> User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jul 2018 03:56:22 -0000 On Tue, Jul 17, 2018 at 11:06:50PM -0700, bob prohaska wrote: > It appears that some progress has been made in getting swap working reasonably > on the RPI3. A -j4 buildworld attempt running r336356 to compile 336431 failed > with "out of swap" but the worst read and write delays were less than 5 seconds, > a marked improvement over previous examples. > > In this case swap was split, 2 each 1GB partitions on USB flash plus 1 GB on the > microSD card. Previous attempts using 3 each 1 GB partitions on USB flash have been > repeatedly successful, while a single attempt using 3 each 1GB partitions on microSD > failed. So I've got a RPI3B+. YMMV. I've recompiled 12 ~7 times since June 27th and run into a handful of problems but haven't had issues lately. I think I started out with the stock images (*-RPI3-20180618-r335317 and *-RPI3-20180709-r336134). I had OOM problems, so added swap (typical swap-on-ufs). Could only get ~3.6G of swap without triggering the kern.maxswzone warning, and then had the pauses (forget the exact text, but basically FreeBSD telling me that something disk related had timed out after ~20 secs). That stopped being a problem as I took the stock images and created a real freebsd-swap partition (and maybe a SDCARD upgrade). Just a gut feeling, but I figured that the kernel recompile was causing enough buffering (and perhaps double-buffering with swap-on-ufs) that it was getting itself soft-wedged (so timeouts, and OOM even though it didn't look like I was getting anywhere near maxing out ~3.6G of swap). OOM, kill ntpd, then eventually kill part of the build and eventually the build would fail. I didn't know if there might have been some memory spikes that blew it up that I wasn't seeing with top. I could restart and it would eventually finish, but building everything takes too long as it is (very spoiled by beefy AMD64 box, but RPI3B+ destroys my old RPI1). So yes, I've (re)compiled everything on just an SDCARD. I'll note that said SDCARD is a SanDisk Extreme Plus 64G (UHS3 / V30, so ~30MB/s). I think I was using a SanDisk Ultra Plus (UHS1, no V-speed, so maybe ~10MB/s) initially because it was laying around. That was an educated guess on my part that something that could sink 30MB/s of video would make a better sink that something that averaged out to ~10MB/s data, and that spending more than that was a waste based on iostat stats (I think topping out at ~6+MB/s with -w5). I think I managed to get up to ~12+MB/s with some dd-bs=1M chunking. I theory, I think the RPI3 should basically have a ~60MB/s USB2 "backplane", so not sure what the bottleneck is. At the moment, I have /usr/src and /usr/obj NFS-mounted, mostly to waste a lot of time copying that work around while I try to figure out what is wrong with my attempted ZFS setup.