From owner-freebsd-questions@freebsd.org Fri Jul 5 04:32:24 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42A5615E49E2 for ; Fri, 5 Jul 2019 04:32:24 +0000 (UTC) (envelope-from greg.marsh@gmail.com) Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 279B484F45 for ; Fri, 5 Jul 2019 04:32:23 +0000 (UTC) (envelope-from greg.marsh@gmail.com) Received: by mail-wr1-x441.google.com with SMTP id p13so8414838wru.10 for ; Thu, 04 Jul 2019 21:32:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0/nX+63p8IL0x7mkRcWIFLRGwH6uAcRk5zcK+q/NRyE=; b=oj+Uah7g+uA0MZL4lo49KsIjCDAJ1cAbKogMzbrzryq5DUedqC9M17DBnnvzBTdTKk 3+lf4/Z8eyVEhpw8Ht2SsayuBlex7UyzI6D540/xTd9kqTx2mBPkYdIToyluLnBsY/VZ vRGEWl9riscMGloZ/UhHKpKSynYsqfJG3KeLjcBTamcxTCmCWaBxiqX8oefQASC9SnO/ t1qXbceYlGRK0R2e4y+klDmkHo0XnFCjAydYzQKajaWy3AfKoHPyvTyIvzrUtmKjV/eu +KQb6+das6EbNT4vlOnfFmFms2m9hXt4D5Gph829vv1vDafgh6eqaN/hvME+Sw9ExYKH yjyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0/nX+63p8IL0x7mkRcWIFLRGwH6uAcRk5zcK+q/NRyE=; b=SnMCQlOZfn8GgTajSmao4hERo+TsgvaMlyOwT8S1Qq0JLVZTOXSkiOxORFTOqn90Zy F9Q/IWhnyqrHDtkaV9pK302XhGXzO+Qy/yVaIECg0XSjvRt/mXIduxWaxXKtznXJOYq4 XP0ND4nikbBmZW23M8G1owJWYgWvRo+3ArP0R5ei5HkGE/L70jmQceocaSApY6QpU9BL IxvMY6wjtycuGxGpKNWT/676xpiVtnYv6RGej38XV/ldoKlb1SVw5yNptz5Gl3zD4b2W Kyt+1GXV5XIx3Yudio9a/7XwzGlBI8xq0xd9xAnuOx7M1I2MxzWJJ+hi3A7car0VAWVo aW5Q== X-Gm-Message-State: APjAAAV8EOz82gb93iFf6IN5+um9jb3sG3Ml3CJtq1GQym9S5wnVwJ4d 4hG4RFd8vi6ESFiGOmOoeZnmwlEvl6OdfNPpF9s= X-Google-Smtp-Source: APXvYqwx8VIL3iHRGT9tWQrVfyu3szn9csC3VQ+iUn+88QIMYeEn3/zSzBbZOpEleUHJj1Mr3KBV61HHap9fQa4eSO4= X-Received: by 2002:a5d:5303:: with SMTP id e3mr1556741wrv.239.1562301141342; Thu, 04 Jul 2019 21:32:21 -0700 (PDT) MIME-Version: 1.0 References: <5adc029e-198c-5189-cd6c-4fff1e4980f0@bytecamp.net> <3b067eb4-f0c7-4680-c3a9-6dac18f45952@bytecamp.net> In-Reply-To: <3b067eb4-f0c7-4680-c3a9-6dac18f45952@bytecamp.net> From: Greg Marsh Date: Fri, 5 Jul 2019 00:32:10 -0400 Message-ID: Subject: Re: extremely slow disk I/O after updating to 12.0 To: Robert Schulze Cc: freebsd-questions@freebsd.org X-Rspamd-Queue-Id: 279B484F45 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=oj+Uah7g; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of gregmarsh@gmail.com designates 2a00:1450:4864:20::441 as permitted sender) smtp.mailfrom=gregmarsh@gmail.com X-Spamd-Result: default: False [-4.13 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.45)[-0.453,0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.67)[ip: (1.86), ipnet: 2a00:1450::/32(-2.75), asn: 15169(-2.40), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jul 2019 04:32:24 -0000 I have been enjoying this conversation a lot. I deal with 10's of thousands of small files daily. Since doing the sysctl vfs.zfs.abd_scatter_enabled=3D0 I've noticed a massive uptick in the use of the L2ARC and a boost to my regular processing times (this is hearsay, I haven't done the numbers yet, but my day has been going by quicker) My storage system a is 5 disk raidz w/ SSD L2ARC and SLOG/ZIL. In reading the abd_scatter header file (I'm not a programmer), I'm making an educated guess the ABD information, no longer being able to be put wherever, needing a contiguous space of memory, has been using the L2ARC more frequently. Please correct me if I'm wrong with this interpretation. How file systems work is of deep interest to me and I <3 ZFS. In any event, I have seen an uptick in the processing of these files since doing the suggested modification. Thank you, Greg On Thu, 4 Jul 2019 at 03:52, Robert Schulze wrote: > Hi, > > Am 03.07.19 um 16:33 schrieb David Demelier: > > Le 03/07/2019 =C3=A0 13:43, Robert Schulze a =C3=A9crit : > >> we have seen a drop in I/O performance beginning with 11.2-RELEASE. > >> Try disabling scattered ABD: > >> > >> sysctl vfs.zfs.abd_scatter_enabled=3D0 > > > > Thanks for the answer. I'm trying this. > > > > Is this tunable available at boot (i.e. /boot/loader.conf) or do you pu= t > > it in /etc/sysctl.conf? > > I have set this tuneable in /boot/loader.conf. > > with kind regards, > Robert Schulze > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" >