From owner-dev-commits-src-main@freebsd.org Sat May 15 23:24:41 2021 Return-Path: Delivered-To: dev-commits-src-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8FDFC62A7CC for ; Sat, 15 May 2021 23:24:41 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FjM1P3NdZz4dtp for ; Sat, 15 May 2021 23:24:41 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk1-x733.google.com with SMTP id o27so2494514qkj.9 for ; Sat, 15 May 2021 16:24:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=kR5UUuS67BzBCIwANsMwLI5sCoL9aMgF930m9lLpzG4=; b=Ca8bC712qFl2S0t9HZMjBx5GogIlTeNecu9ZmGmtcJxUfbDHgBMnK4Y6kOkdttLRUq HM7BnXTSCNU5CIP6ZNZh7f3E6rX+C/2J8GHxklhWl92Zej+QWhvOJps/4x1h6F0dylbE HUJNTdOjKhExs91KHiX4rYm3wsL6TIjh8es/wC6RxKeMPUzVFtmHYAAINZb3vONNnNmR JQPl/5FILILO5xEIAiEeeOrjeEqdgc7fUU1mtYR9FJuS2Ckr0UgGXHZ6BYPPDMOuOCmk YnJOb0jmqbIRlySzhcDKV6ercTS2MhvYcpdEoBJ8A8a4H7H/k6JlgKXXcMRBeLDtoZmE m/Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=kR5UUuS67BzBCIwANsMwLI5sCoL9aMgF930m9lLpzG4=; b=WcRvAB5itkpwdmjBnzOvwW5TX2TgMEXqdnibWKMDO7ObNpvV644R5v1Dvx0QU00NS7 NzM86Ndfa6iEj+nbrCv+Bop/MCB/rv8kV+xK6lxSboBK/J1UIWJ5F9SlH1r79ug9fTlm HfUzRTHoPI+VCebcHwnGy+ullBctpxajD50SjWrjkh5Nn/TLkFffeMtwNZMXZwzHxXhe vYXsObgsaRtaZb1xQGvO/hkl2tfF3ylzKkjfci30f1aE38bpzxaOI6CnIVxvfGOGPp14 bSa382kb70pUbjaaXNAvYtynECZ6VeSPT56Y4kgus0ewxfsAdLe98O2inhCS5n4HXTeJ ajyg== X-Gm-Message-State: AOAM530pBhhjTf9LttXIwSlW/W10lwEZ8iY3ivOAxVlUP/QvCDcWwhLz EDlCcXUWR30Ob4eZ+Yawpv4qkQ== X-Google-Smtp-Source: ABdhPJz6KIup3BvglK/USF/2tVBV6PNnuNOx+5Hor9gHUD0EsjMTHiW3iM2KI0oV0FxkzAYoi10HrQ== X-Received: by 2002:a37:815:: with SMTP id 21mr48899075qki.498.1621121080238; Sat, 15 May 2021 16:24:40 -0700 (PDT) Received: from mutt-hbsd (pool-100-16-224-136.bltmmd.fios.verizon.net. [100.16.224.136]) by smtp.gmail.com with ESMTPSA id i9sm7940282qtg.18.2021.05.15.16.24.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 May 2021 16:24:39 -0700 (PDT) Date: Sat, 15 May 2021 19:24:38 -0400 From: Shawn Webb To: Edward Tomasz Napierala Cc: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: Re: git: 3394d4239b85 - main - cam: allocate CCBs from UMA for SCSI and ATA IO Message-ID: <20210515232438.7siob6di7bdvaefs@mutt-hbsd> X-Operating-System: FreeBSD mutt-hbsd 14.0-CURRENT-HBSD FreeBSD 14.0-CURRENT-HBSD X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc References: <202105151105.14FB5nv7005915@gitrepo.freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="6k5h3synwn52j6at" Content-Disposition: inline In-Reply-To: <202105151105.14FB5nv7005915@gitrepo.freebsd.org> X-Rspamd-Queue-Id: 4FjM1P3NdZz4dtp X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: dev-commits-src-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for the main branch of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 May 2021 23:24:41 -0000 --6k5h3synwn52j6at Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 15, 2021 at 11:05:49AM +0000, Edward Tomasz Napierala wrote: > The branch main has been updated by trasz: >=20 > URL: https://cgit.FreeBSD.org/src/commit/?id=3D3394d4239b85b5577845d9e6de= 4e97b18d3dba58 >=20 > commit 3394d4239b85b5577845d9e6de4e97b18d3dba58 > Author: Edward Tomasz Napierala > AuthorDate: 2021-05-15 10:17:22 +0000 > Commit: Edward Tomasz Napierala > CommitDate: 2021-05-15 11:03:49 +0000 >=20 > cam: allocate CCBs from UMA for SCSI and ATA IO > =20 > This patch makes it possible for CAM to use small CCBs allocated > from an periph-specific UMA zone instead of the usual, huge ones. > The end result is that CCBs issued via da(4) take 544B (size of > ccb_scsiio) instead of the usual 2kB (size of 'union ccb', ~1.5kB, > rounded up by malloc(9)). For ATA it's 272B. We waste less > memory, we avoid zeroing the unused 1kB, and it should be easier > to allocate those CCBs in low memory conditions. It should also > be possible to use uma_zone_reserve(9) to improve behaviour > in low memory conditions even further. > =20 > Note that this does not change the size, or the layout, of CCBs > as such. CCBs get allocated in various different ways, in particular > on the stack, and I don't want to redo all that. Instead, this > provides an opt-in mechanism for the periph to declare "my start() > callback is fine with receiving a CCB allocated from this UMA zone". > In other words, most of the code works exactly as it used to; the > change only happens to IOs issued by xpt_run_allockq(), which > is - conveniently - pretty much all that matters for performance. > =20 > The reason for doing it this way is that it's pretty small, localized > change, and can be implemented gradually and iteratively: take a > periph, make sure its start() callback only casts the CCBs it takes > to a particular type of CCB, for example ccb_scsiio, and that it only > casts CCBs returned by cam_periph_getccb() to that type, then add UMA > zone for that size, and declare it safe to XPT. > =20 > This is disabled by default. Set 'kern.cam.ada.enable_uma_ccbs=3D1' > and 'kern.cam.da.enable_uma_ccbs=3D1' tunables to enable it. Testing > is welcome; I will flip the default to enable in two weeks from now. > =20 > Reviewed By: imp > Sponsored by: NetApp, Inc. > Sponsored by: Klara, Inc. > Differential Revision: https://reviews.freebsd.org/D28674 > --- > sys/cam/ata/ata_da.c | 18 ++++++++++++++++++ > sys/cam/ata/ata_xpt.c | 7 +++++++ > sys/cam/cam_ccb.h | 14 +++++++++++++- > sys/cam/cam_periph.h | 3 +++ > sys/cam/cam_xpt.c | 32 +++++++++++++++++++++++++++++--- > sys/cam/scsi/scsi_da.c | 18 ++++++++++++++++++ > sys/cam/scsi/scsi_xpt.c | 7 +++++++ > 7 files changed, 95 insertions(+), 4 deletions(-) Hey Edward, I suspect this may be the cause of a kernel panic I just had when booting my Softiron Overdrive 1000 arm64 system: panic: ata_action: ccb 0xffff000040388828, func_code 0x15 should not be all= ocated from UMA zone cpuid =3D 1 time =3D 1 __HardenedBSD_version =3D 1400000 __FreeBSD_version =3D 1400013 version =3D FreeBSD 14.0-CURRENT-HBSD #1 hardened/current/master-n190374-a= 541810e5fa: Sat May 15 17:05:42 EDT 2021 shawn@arm64-02:/usr/obj/usr/src/arm64.aarch64/sys/HARDENEDBSD KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 vpanic() at vpanic+0x188 panic() at panic+0x44 ata_action() at ata_action+0x340 _ata_announce_periph() at _ata_announce_periph+0x74 ata_announce_periph_sbuf() at ata_announce_periph_sbuf+0x24 xpt_announce_periph_sbuf() at xpt_announce_periph_sbuf+0x1a8 adaregister() at adaregister+0x420 cam_periph_alloc() at cam_periph_alloc+0x528 adaasync() at adaasync+0xa8 xptsetasyncfunc() at xptsetasyncfunc+0x100 xptdevicetraverse() at xptdevicetraverse+0x9c xpttargettraverse() at xpttargettraverse+0x78 xptbustraverse() at xptbustraverse+0x80 xpt_register_async() at xpt_register_async+0x1fc adainit() at adainit+0x4c periphdriver_init() at periphdriver_init+0x54 xpt_finishconfig_task() at xpt_finishconfig_task+0x18 taskqueue_run_locked() at taskqueue_run_locked+0x188 taskqueue_thread_loop() at taskqueue_thread_loop+0x9c fork_exit() at fork_exit+0x74 fork_trampoline() at fork_trampoline+0x14 KDB: enter: panic [ thread pid 0 tid 100023 ] Stopped at kdb_enter+0x44: undefined f904411f Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --6k5h3synwn52j6at Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmCgWDMACgkQ/y5nonf4 4fruaQ/+PIkDG/eUlPCAdIrcETOpYDRNCaCK08kg1k6F4OE4H6PUeltr00DQhjH8 NdmIGPlne3Bs2y0qsZLGIMcD5s+NdFyj6bm2d6IX0UP3Si8zSiI8L7bum7jQlv2K Bbk7pAFKBHwYlNm5CotlwoIlDAWztr0N/YNIi4SWX+m2MStoVXZTOcO+3fRB/yFE buWVb1hZlPECmpVnhP/XQtMParxbUd1i9I00oxdYHSVux8CQq73D/Rkm4PT5pj4S iOWouMUGed6fDLS7o1M/wB9t9KOGpvnRqVaolvRfZglTI8ek5d6w9Po0fI4aSUNy LcWZKS0LsQH7zTlevpNvQNDBq1a7xm5UbhMOs3wNiBCgjsJ0anwV9j3/951x/jTS EjqDVfMsa1Vb7s+hXb4Gf9f1RF4HoUeB7P1nOjn+LjZv3kCT/b6mKEMYjrCjeOJz JChaBU2BBY8Q8hsIk4Ndx2Vn3XMLxDjsCsBPQ7f88LX+PZfUiIWtP5aHUuPbQw7l 9zDAkRV7Bb1SeIJeDPvfNbQSHq2waRvd+cZM278X7fM5GmvZG4SfUnEpGpiSA+aD /Eq74lJN35/SdVWH/oRvgpj1zKCwP62Pc112S2Tv6qog2BApWRlPdfMEk+FzMImm B92knlTQDqe1zf2/P5koVWLzLBXteQ9gqpmV30mP8Ki2FLNeoF8= =02Rf -----END PGP SIGNATURE----- --6k5h3synwn52j6at--