From owner-freebsd-arm@freebsd.org Fri Jun 15 15:57:19 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 367A8101097C for ; Fri, 15 Jun 2018 15:57:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C77D674D26 for ; Fri, 15 Jun 2018 15:57:18 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x22e.google.com with SMTP id f1-v6so11134799ioh.6 for ; Fri, 15 Jun 2018 08:57:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=j+k1J4IIBUaMylVDMa6Id4RqPMv3cH7DQkDPpDne3eo=; b=jZdNPOKWOhS1mv5zUArefdL11ZQ7CtNxUNGcflcrylEDO/JLSo9T7UN+tRGQ/c4vag PVmkVSa0JiMCR6nGY2XtpdLhV9D6Il2H8VfKJyrKbfJW+lm5LKRVGJGEb1q0j5iRIcIm XYvBFJe1mCvzflcNvc4cEM54ULQETPgAdqmkTk6UwLjXDzvkbp6FWqOOlArRDtdbwvX9 4tB+mPtWAGlM/C67/msacs25d5KegvUnJq3ol913UQ8PTh3DiKxwVMuIHBl8crMxkLNo zKQqDt/lc7D4ZhN/XaRtp87F4+hRkta+/huHu6H9XEbBa1DUOnRw7rNeQhMYNsbKN5jo RNJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=j+k1J4IIBUaMylVDMa6Id4RqPMv3cH7DQkDPpDne3eo=; b=NxIMofVi2Hxv4RKOelGLmRp7jSN1MemioEpjzzawZn8BhqLfra4c6KtXjAy8GzdZp2 4VdrDqWIfVhqY36MQ8IrURiH+93V5X2MP2JwujSQOMKZiDjYOjGzne4+J5JB/Zauqvqw S65K4zxUUqHKcCNU7t1J3++KwoxjE33mpOUDvgadH4oOCpnKtNTH7UXHpjgxXMMh+L+c 7NuiFICPgBUrWQottMgNBiF6Gt4CWls3nmXKLg/3Dzoebagi4Q/5kdhqSHjvQfl9/QOR QEI+rf4EuD99YThLsrkVg8kR5gIqu7LIR7q40mDcDVg5w5ha/omLF+9nWXR2CHgQNilt /anQ== X-Gm-Message-State: APt69E2e2qes0bvcmsn6UkdgS0/gQBtqW7pIDaDctMmSeFdTPk2R3Acc RPO1QuAA8pQQNSMQwn4+iBIvxd/c2OFKhNJNrRmcBg== X-Google-Smtp-Source: ADUXVKJfaVH81UQ8qfidplqVf1q0TnAgExaJ4VrqblEKPlWKiGc8gV58r9JpFEktECnAgQC3Ue2R7lNs9jq1AdToDFY= X-Received: by 2002:a6b:d40c:: with SMTP id l12-v6mr2029062iog.37.1529078237844; Fri, 15 Jun 2018 08:57:17 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 2002:a4f:5945:0:0:0:0:0 with HTTP; Fri, 15 Jun 2018 08:57:17 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: <20180615154334.GA39777@www.zefox.net> References: <20180614175622.GC35161@www.zefox.net> <201806142110.w5ELAL0N046840@pdx.rh.CN85.dnsmgr.net> <20180615035225.GA37370@www.zefox.net> <20180615051527.GB37370@www.zefox.net> <834EA7A6-B567-436F-96B2-0C75FACA3FF9@yahoo.com> <20180615154334.GA39777@www.zefox.net> From: Warner Losh Date: Fri, 15 Jun 2018 09:57:17 -0600 X-Google-Sender-Auth: LAROaXZ-X6G4P2OsIxkqc7VlLV4 Message-ID: Subject: Re: GPT vs MBR for swap devices To: bob prohaska Cc: Mark Millard , "freebsd-arm@freebsd.org" , "Rodney W. Grimes" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jun 2018 15:57:19 -0000 On Fri, Jun 15, 2018 at 9:43 AM, bob prohaska wrote: > On Thu, Jun 14, 2018 at 11:37:48PM -0700, Mark Millard wrote: > > > > When I look at: > > > > # vmstat -c -w 5 > > procs memory page disks faults cpu > > r b w avm fre flt re pi po fr sr da0 ad0 in sy cs > us sy id > > 1 0 0 416M 224M 1647 1 0 0 1856 142 0 0 144 1791 1024 > 4 2 94 > > 0 0 0 416M 224M 9 0 0 0 0 1 0 0 4 85 116 > 0 0 100 > > 0 0 0 416M 224M 12 0 0 0 0 1 0 0 2 93 113 > 0 0 100 > > 0 0 0 416M 224M 9 0 0 0 2 1 1 0 4 64 121 > 0 0 100 > > . . . > > > > and "man vmstat" I do not see any column that is the swap space > > usage (nor any combination of columns to do such a calculation > > from). > > > > I do not expect that vmstat reports what you are likely/primarily > > looking for. > > > > An example is "avm" which for which the man page reports: > > > > . . . Note that the entire > > memory object's size is considered mapped even if only a > subset > > of the object's pages are currently mapped. This statistic > is > > not related to the active page queue which is used to track > real > > memory. > > > > The free list size ("fre") is not sufficient either. > > > > That seems astonishing. I imagined that among those columns _had_ to be > reads from and writes to the swap partitions. > > It looks as if > top -d 1000 | grep Swap > produces a running list of swap usage, but one must guess how many > times to iterate: > > bob@www:/usr/src % top -d 1000 | grep Swap > Swap: 3072M Total, 30M Used, 3041M Free > Swap: 3072M Total, 30M Used, 3041M Free > Swap: 3072M Total, 30M Used, 3041M Free > Swap: 3072M Total, 30M Used, 3041M Free > Swap: 3072M Total, 30M Used, 3041M Free > ....... > > Replacing the "1000" with "0" or "infinite" triggers > a syntax error. Is there a special parameter that makes top run till > it's killed, as in interactive mode? I didn't recognize any hint in the > man page. > > Thanks for reading! > Right, this is why I was suggesting gstat. It's a direct measure of the read/write performance of the device with some latency numbers. It will give the kind of data I'm looking for. vmstat won't, top won't. I don't care about used/free swap usage. I care about performance to the swap partition. That's what I'm suspecting in the USB thumb drive FTL. I don't care what the total swap usage is. I suspect that's irrelevant to the issue at hand since the OOM isn't triggering because we're filling swap, but more that it's due to not being able to get enough pages to the swap device fast enough to satisfy the memory shortages, triggering OOM. As for why it would affect the USB drive and not SD cards, I can only say that USB drives tend to be first to market with bigger capacities. This has traditionally made them less well tuned for anything other than large, long sequential reads or writes that aren't mixed. More so than even SD or uSD cards which tend to do better than USB drives at that workload. It's the FTL that's the issue, not the NAND itself. The FTL is the software that translates the log-style device you have to have for flash to work to the LBA style devices that people attach to systems. If it can't cope with a mixed workload, or needs to do too much garbage collection or read/modify/write operations due to it's poor quality / tuning, that will show up as long delays. USB flash also tends to suck more with BIO_DELETE than others, though the swapper doesn't do that, so that's one fewer wildcards we need to look at. gstat -Bd -I 10 -f > gstat-swap-data.dat would be how I'd recommend collecting it. This file may get kinda big depending how long it takes to trigger the weird state. I'm hoping that if you put this on a known good device, we'll power through the issues. We might not get perfect correlation with this, but the data should show all kinds of crazy before the system drives off the cliff if I'm right, so we don't need perfect data. There's some higher fidelity numbers we can get from the I/O scheduler with dynamic scheduling compiled in, but I don't think we'll need those. Warner