From nobody Wed Jul 7 08:38:05 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 82C5111E37B1 for ; Wed, 7 Jul 2021 08:38:10 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GKXr13cVpz3kgq for ; Wed, 7 Jul 2021 08:38:09 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by mail-wm1-x32f.google.com with SMTP id q18-20020a1ce9120000b02901f259f3a250so1121733wmc.2 for ; Wed, 07 Jul 2021 01:38:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-disposition:in-reply-to; bh=/WIiuc8jBC5ntDkEUyo5iooLoiF14KyBCBDIsTHFB+k=; b=AKGxxb+1NZLXLs6yQNEQArqXp9mD+6dq3Bu/AqlPyzSicQP0Y35QkLRx86SpHKsYQ/ 8m1qE0HHrYSxTABZBDN0ZXFrBleLP2U8iMLV8vXM61DL2d5mPM1M4szt/01wl+yjTmnZ 4k2g37EtUsczQ2hR1nywNWdA7diAjKV7AFtQrapSbL/1dgYwddtDdqTfFnp+BMcz4tZf E/6fhB/POE7Q2CfITqPgfFTpDucA/yHAZ7/FhX/gXQlNWToA+6FmmbtqqYSkwKdeydU7 59NzzkqKr14fJTM68coQaHF3WyyZkQR566Bg7SjoAqHLiXGV7ha2uwti4TgPDEogLZeX I+rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to; bh=/WIiuc8jBC5ntDkEUyo5iooLoiF14KyBCBDIsTHFB+k=; b=NrzvgwTdgLukwm0ugdwdzkpMMUoDiPUQA5vXclsm8n28+neaaDzUOZXnu1FaeX18IP 5vufyiYE/HnPzaZcFrkHcbrp0KWoZbSI2OcEb7oMZPQnmcV1wJc1pyxLnEWVprxJai9B dftL+5pDRp1nPCIr4D97hPm6a3+3TqoBy9TS7wlAo4xTG/tWR92v2Rnyr3V/qzO9FrL6 gE0uj74wCWGSFS2AyNyEHVASoqKLxHPQoBYkqXSTr9ekRZUdg+4W+PpgIVmzS9nXP2c5 2GWug9fjLxui7bx3+on4XO2wTHqAZA2Tlcv3RaldXPXSb+kfJ+ylQ0GSeXj5TSSKWYoo 2w/Q== X-Gm-Message-State: AOAM532hcMAm13Ad8XwsBfYPq1FuDD+F5l1q4SX/6idfeW43YFlAO06X oW5gpRIePp5p96UMwMhHqU0HNaak3BU= X-Google-Smtp-Source: ABdhPJyeeimyVTOWQYdBVg82Y32nlimb4+eAG/5JDImjNZhFZzfI3807+DaDhQUc7LXnrE4fwiyfug== X-Received: by 2002:a05:600c:19d2:: with SMTP id u18mr5467208wmq.21.1625647088127; Wed, 07 Jul 2021 01:38:08 -0700 (PDT) Received: from brick (cpc159423-cmbg20-2-0-cust338.5-4.cable.virginm.net. [86.7.147.83]) by smtp.gmail.com with ESMTPSA id l4sm18249942wme.26.2021.07.07.01.38.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Jul 2021 01:38:07 -0700 (PDT) Date: Wed, 7 Jul 2021 09:38:05 +0100 From: Edward Tomasz Napiera?a To: Gary Jennejohn Cc: Warner Losh , FreeBSD Current Subject: Re: panic: Unaligned free (was: kernel panic while copying files) Message-ID: Mail-Followup-To: Gary Jennejohn , Warner Losh , FreeBSD Current References: <20210608155405.5cf0e200@ernst.home> <20210610095041.38d7597c@ernst.home> <20210629094201.77ef5f22@ernst.home> <20210630125703.2b5544e7@ernst.home> <20210701035800.410d2376@ernst.home> <20210701113026.59f864e9@ernst.home> <20210705163324.29466849@ernst.home> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210705163324.29466849@ernst.home> X-Rspamd-Queue-Id: 4GKXr13cVpz3kgq X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=AKGxxb+1; dmarc=none; spf=pass (mx1.freebsd.org: domain of etnapierala@gmail.com designates 2a00:1450:4864:20::32f as permitted sender) smtp.mailfrom=etnapierala@gmail.com X-Spamd-Result: default: False [-0.04 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-0.34)[-0.341]; FREEMAIL_TO(0.00)[gmail.com]; FORGED_SENDER(0.30)[trasz@freebsd.org,etnapierala@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[86.7.147.83:received]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[trasz@freebsd.org,etnapierala@gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::32f:from]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-0.999]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[0.998]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::32f:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32f:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current] X-ThisMailContainsUnwantedMimeParts: N On 0705T1833, Gary Jennejohn wrote: > On Mon, 5 Jul 2021 15:04:48 +0100 > Edward Tomasz Napiera__a wrote: > > > On 0701T1330, Gary Jennejohn wrote: > > > Gary Jennejohn wrote: > > > > I noticed that the value of vm.debug.divisor affects what value is > > > > returned in uma_core.c:uma_dbg_kskip(), so I decided to try a few > > > > different values. > > > > > > > > The returned value is used to set skipdbg in uma_core.c:item_dtor(). > > > > > > > > The default is vm.debug.divisor=1. > > > > > > > > vm.debug.divisor is only present when INVARIANTS is defined. > > > > > > > > kskipdbg eventually affects the value of freei. > > > > > > > > With these values: > > > > vm.debug.divisor: 0 > > > > kern.cam.da.enable_uma_ccbs: 1 > > > > I can turn on the disk and it comes up without a panic! > > > > > > > > However, I didn't try to do any large data transfers to the disk. > > > > > > > > So, it appears that at least vm.debug.divisor is a big factor in > > > > whether or not a panic happens with INVARIANTS. > > > > > > > > > > I decided to do a real test. So I built a kernel w/o INVARIANTS and > > > installed it to /boot/test. > > > > > > Then I stuck a 160GB disk I had around into an external USB3 enclosure > > > and put a filesystem on it. > > > > > > The I booted the new kernel from /boot/test and set the sysctls so: > > > kern.cam.da.enable_uma_ccbs: 1 > > > kern.cam.ada.enable_uma_ccbs: 1 > > > > > > After that I plugged in the external USB3 enclosure and copied about > > > 114GiB of data from an internal SSD to it - without a kernel panic: > > > Filesystem Size Used Avail Capacity Mounted on > > > /dev/da0p1 144G 114G 18G 86% /mnt > > > > > > I'm pretty sure that's more than I could copy without a kernel panic > > > prior to the recent changes made in cam and umass. > > > > > > My test may not be real proof that all bugs have been squashed, but it > > > certainly seems to be a better situation than we had before. > > > > I think the vm.debug.divisor simply masks the problem; the underlying > > bug is still there. > > > > Could you go back to the setup which panics, and then test the patch > > at https://reviews.freebsd.org/D31054? It fixes the scenario described > > by Warner. > > > > It looks like this patch fixes things. > > I used the default value vm.debug.divisor=1 and both enable_uma_ccbs=1 > (which are now the default values on my system). > > I used the 8TiB disk, which spins up very slowly and usually resulted very > quickly in a panic - no panic with the patch. > > Then using dd to /dev/null (bs=1m) I transferred: > > 308755+0 records in > 308755+0 records out > 323753082880 bytes transferred in 1366.162410 secs (236979938 bytes/sec) > > from the disk, so about 324GiB without a panic. Perfect, I've committed the fix. Thank you!