From owner-freebsd-hackers@freebsd.org Mon Oct 15 21:05:57 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD2A410C91B0 for ; Mon, 15 Oct 2018 21:05:57 +0000 (UTC) (envelope-from james.shank@gmail.com) Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B0EB84D20 for ; Mon, 15 Oct 2018 21:05:57 +0000 (UTC) (envelope-from james.shank@gmail.com) Received: by mail-qt1-x82f.google.com with SMTP id q40-v6so23243502qte.0 for ; Mon, 15 Oct 2018 14:05:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:date:subject:message-id :to; bh=GcYyukUk4QPwNM41ccv3hCTmeZ61rYMQIdky4tnI2wc=; b=fS++5FD6hUADGeiyPVigjxuGh5qm1XDBLpfq2WnbS2gXS9KWPvh4ZV221XnexzEMv4 xkY1mY7XGDgA50pE3v7J60dOjsmWKGkSYdaT3o8E1h4lTwvkbnSjNRDgLwFzhAK0vQ1J QL4PyXQRBczbq4RZrRTDuNpwC/RudPvFs526E3Z/+YcrqqNejrtYiUNYWjNlxzTwQ5bj s1EVY1UXwU0TfzLJUF7BNeGbUnEC05pdtLRC0jSoMVWcIFVWDv0gtV+0nVp1qk5DS9AB 3jUX3SIsOu7i5/RYf/a+KiesXBhJv1wNt61Xf1tvPiEAvCM9O/bA7KJhZ7uGObpZLmC9 ppSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version:date :subject:message-id:to; bh=GcYyukUk4QPwNM41ccv3hCTmeZ61rYMQIdky4tnI2wc=; b=SJeF1yf0GLIYinTdnlRQV7vgyoBX/+LAF7hQYHZTCrBvGIRySz7yMVj7IbqQot5TMy ta1tC6G/ap2GDOa8wr2FI58cP3Ds1nuSlPm5R72v8oGTgd6Uvz5FXauKo2gbsgBq5SbK FXbKSH/A1TIpKyj68UcIm8+BrqsGqSc1kk/R0+SCMOQqgKrTD0gFDcIezBYN+Gk5fJf7 gcM+0qQ61JttnmBiP0NQT3HAg6dLPwyxsT3mbzJ/Ts1yeKBlFoj+KGeo/fgbAWEon08z JUpq6vto6O4DP4AqR2WLp8lt28wv3MOdKaDqU2XrhXAaPLYG9DEiu2TbKxzdilTo72a1 OSxg== X-Gm-Message-State: ABuFfogbG+lZN+HO/2Ebx42K7yzfXuedqGujEHkQX1tj3WnR2dV23TZu 4KajlbZn0IgTmI5lIyhJ/MdsXsCf X-Google-Smtp-Source: ACcGV61HtiJJrCs/f2n9M92tjN5ctEQjbiDUyzd9dOnGUn75tDl9mhC6cuflIpQY0mB3MzI801eqwQ== X-Received: by 2002:ac8:2092:: with SMTP id 18-v6mr17573216qtd.192.1539637556674; Mon, 15 Oct 2018 14:05:56 -0700 (PDT) Received: from ?IPv6:2600:380:944e:d585:41a9:7ef7:e8ee:334c? ([2600:380:944e:d585:41a9:7ef7:e8ee:334c]) by smtp.gmail.com with ESMTPSA id r57-v6sm8569156qtc.36.2018.10.15.14.05.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Oct 2018 14:05:55 -0700 (PDT) From: James Shank Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Date: Mon, 15 Oct 2018 17:05:54 -0400 Subject: High I/O RAID on UFS2 Message-Id: <1BA62E58-93D7-4763-B27A-E21B7DA362B2@gmail.com> To: freebsd-hackers@freebsd.org X-Mailer: iPhone Mail (16A404) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2018 21:05:58 -0000 Hi FreeBSD Hackers, I am running FreeBSD 11.1-RELEASE-p15 with a UFS2 filesystem block size 64K,= with a RAID 6 of 16 SSD drives on a MRSAS compatible RAID card stripe size o= f 256K (and yes, I know about the misalignment) and 384GB memory on 48 real c= ores with hyper threading enabled. I invite critiques of this configuration= , but am more keen on getting help making it run efficiently. I am running 11.1 because 11.2-RELEASE repeatably panics after failing somew= here after allocbuf() tries to allocate a page (sorry, I don=E2=80=99t still= have the output from this state). I am doing very read heavy operations, via multithreaded readers. My writes= tend to be small, but come in every 15 seconds or so. I=E2=80=99d estimate= read to write ratio of about 100:1 or more. What I see happening is pagedaemon coming alive and bufferspacedaemon doing t= he same to clear the corresponding resources. bufferspacedaemon runs nearly= all the time at 100%. I haven=E2=80=99t yet had much success with tuning t= hings to keep enough resources free to allow my read rates to be sustained. My read rates can hit and exceed 6GBps when everything is humming along, but= only for short periods of time, then they drop off to much lower rates afte= r the kernel starts shuffling resources. For long periods of time, my best s= ustainable rate seems to be 3.5GBps, and I can=E2=80=99t seem to get the ful= l read rate for sustained periods. I have tried several things, but I believe now that I may be missing somethi= ng that the more experienced gurus will find obvious. Any pointers on where I should look next? Thanks! James