From owner-freebsd-amd64@freebsd.org Sat Mar 31 23:06:02 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19917F63B7B for ; Sat, 31 Mar 2018 23:06:02 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A91637B34E for ; Sat, 31 Mar 2018 23:06:01 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 5D7E6F63B77; Sat, 31 Mar 2018 23:06:01 +0000 (UTC) Delivered-To: amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A9AAF63B74; Sat, 31 Mar 2018 23:06:01 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tensor.andric.com", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A6C87B34B; Sat, 31 Mar 2018 23:06:00 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from coleburn.home.andric.com (coleburn.home.andric.com [192.168.0.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 1A99636A4E; Sun, 1 Apr 2018 01:05:58 +0200 (CEST) From: Dimitry Andric Message-Id: <3FAD36FD-FA90-49F6-9141-B9CCCCA2BF00@FreeBSD.org> Content-Type: multipart/signed; boundary="Apple-Mail=_FEF4CFF3-A8E3-4C92-9CB5-9B04D7FD34B3"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: i386 4/4 change Date: Sun, 1 Apr 2018 01:05:57 +0200 In-Reply-To: <20180401004833.L3296@besplex.bde.org> Cc: Konstantin Belousov , current@freebsd.org, amd64@freebsd.org, bde@freebsd.org To: Bruce Evans References: <20180331102901.GN1774@kib.kiev.ua> <20180401004833.L3296@besplex.bde.org> X-Mailer: Apple Mail (2.3273) X-Mailman-Approved-At: Sun, 01 Apr 2018 00:38:17 +0000 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2018 23:06:02 -0000 --Apple-Mail=_FEF4CFF3-A8E3-4C92-9CB5-9B04D7FD34B3 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On 31 Mar 2018, at 17:57, Bruce Evans wrote: > > On Sat, 31 Mar 2018, Konstantin Belousov wrote: > >> the change to provide full 4G of address space for both kernel and >> user on i386 is ready to land. The motivation for the work was to both >> mitigate Meltdown on i386, and to give more breazing space for still >> used 32bit architecture. The patch was tested by Peter Holm, and I am >> satisfied with the code. >> >> If you use i386 with HEAD, I recommend you to apply the patch from >> https://reviews.freebsd.org/D14633 >> and report any regressions before the commit, not after. Unless >> a significant issue is reported, I plan to commit the change somewhere >> at Wed/Thu next week. >> >> Also I welcome patch comments and reviews. > > It crashes at boot time in getmemsize() unless booted with loader which > I don't want to use. For me, it at least compiles and boots OK, but I'm one of those crazy people who use the default boot loader. ;) I haven't yet run any performance tests, I'll try building world and a few large ports tomorrow. General operation from the command line does not feel "sluggish" in any way, however. -Dimitry --Apple-Mail=_FEF4CFF3-A8E3-4C92-9CB5-9B04D7FD34B3 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.2 iF0EARECAB0WIQR6tGLSzjX8bUI5T82wXqMKLiCWowUCWsAUVQAKCRCwXqMKLiCW o98sAKCDkLWdzL8NVaBN8mYfJAtBViO+lgCg/53fGIcQd2gJ7rA6b/pxf8X50K4= =P8fD -----END PGP SIGNATURE----- --Apple-Mail=_FEF4CFF3-A8E3-4C92-9CB5-9B04D7FD34B3-- From owner-freebsd-amd64@freebsd.org Sun Apr 1 07:05:23 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EB26F54CCC for ; Sun, 1 Apr 2018 07:05:23 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DE73B77D1D for ; Sun, 1 Apr 2018 07:05:22 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 9B057F54CCA; Sun, 1 Apr 2018 07:05:22 +0000 (UTC) Delivered-To: amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60BE8F54CC7; Sun, 1 Apr 2018 07:05:22 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail107.syd.optusnet.com.au (mail107.syd.optusnet.com.au [211.29.132.53]) by mx1.freebsd.org (Postfix) with ESMTP id A291F77CF6; Sun, 1 Apr 2018 07:05:14 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail107.syd.optusnet.com.au (Postfix) with ESMTPS id 6BE09D4ECD3; Sun, 1 Apr 2018 17:05:06 +1000 (AEST) Date: Sun, 1 Apr 2018 17:05:03 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Dimitry Andric cc: Konstantin Belousov , current@FreeBSD.org, amd64@FreeBSD.org Subject: Re: i386 4/4 change In-Reply-To: <3FAD36FD-FA90-49F6-9141-B9CCCCA2BF00@FreeBSD.org> Message-ID: <20180401151124.G893@besplex.bde.org> References: <20180331102901.GN1774@kib.kiev.ua> <20180401004833.L3296@besplex.bde.org> <3FAD36FD-FA90-49F6-9141-B9CCCCA2BF00@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=cIaQihWN c=1 sm=1 tr=0 a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=6I5d2MoRAAAA:8 a=RRKNnNHb0Om-JY6aZ2kA:9 a=HfVbGaZjtTc9tSMn:21 a=VRhFOslev4N6E7P5:21 a=CjuIK1q_8ugA:10 a=IjZwj45LgO3ly-622nXo:22 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Apr 2018 07:05:23 -0000 On Sun, 1 Apr 2018, Dimitry Andric wrote: > On 31 Mar 2018, at 17:57, Bruce Evans wrote: >> >> On Sat, 31 Mar 2018, Konstantin Belousov wrote: >> >>> the change to provide full 4G of address space for both kernel and >>> user on i386 is ready to land. The motivation for the work was to both >>> mitigate Meltdown on i386, and to give more breazing space for still >>> used 32bit architecture. The patch was tested by Peter Holm, and I am >>> satisfied with the code. >>> >>> If you use i386 with HEAD, I recommend you to apply the patch from >>> https://reviews.freebsd.org/D14633 >>> and report any regressions before the commit, not after. Unless >>> a significant issue is reported, I plan to commit the change somewhere >>> at Wed/Thu next week. >>> >>> Also I welcome patch comments and reviews. >> >> It crashes at boot time in getmemsize() unless booted with loader which >> I don't want to use. > For me, it at least compiles and boots OK, but I'm one of those crazy > people who use the default boot loader. ;) I found a quick fix and sent it to kib. (2 crashes in vm86 code for memory sizing. This is not called if loader is used && the system has smap. Old systems don't have smap, so they crash even if loader is used.) > I haven't yet run any performance tests, I'll try building world and a > few large ports tomorrow. General operation from the command line does > not feel "sluggish" in any way, however. Further performance tests: - reading /dev/zero using tinygrams is 6 times slower - read/write of a pipe using tinygrams is 25 times slower. It also gives unexpected values in wait statuses on exit, hopefully just because the bug is in the test program is exposed by the changed timing (but later it also gave SIGBUS errors). This does a context switch or 2 for every read/write. It now runs 7 times slower using 2 4.GHz CPUs than in FreeBSD-5 using 1 2.0 GHz CPU. The faster CPUs and 2 of them used to make it run 4 times faster. It shows another slowdown since FreeBSD-5, and much larger slowdowns since FreeBSD-1: 1996 FreeBSD on P1 133MHz: 72k/s 1997 FreeBSD on P1 133MHz: 44k/s (after dyson's opts for large sizes) 1997 Linux on P1 133MHz: 93k/s (simpler is faster for small sizes) 1999 FreeBSD on K6 266MHz: 129k/s 2018 FBSD-~5 on AthXP 2GHz: 696k/s 2018 FreeBSD on i7 2x4GHz: 2900k/s 2018 FBSD4+4 on i7 2x4GHz: 113k/s (faster than Linux on a P1 133MHz!!) Netblast to localhost has much the same 6 times slowness as reading /dev/zero using tinygrams. This is the slowdown for syscalls. Tinygrams are hard to avoid for UDP. Even 1500 bytes is a tinygram for /dev/zero. Without 4+4, localhost is very slow because it does a context switch or 2 for every packet (even with 2 CPUs when there is no need to switch). Without 4+4 this used to cost much the same as the context switches for the pipe benchmark. Now it costs relatively much less since (for netblast to localhost) all of the context switches are between kernel threads. The pipe benchmark uses select() to avoid busy-waiting. That was good for UP. But for SMP with just 2 CPUs, it is better to busy-wait and poll in the reader and writer. netblast already uses busy-waiting. It used to be a bug that select() doesn't work on sockets, at least for UDP, so blasting using busy-waiting is the only possible method (timeouts are usually too coarse-grained to go as fast as blasting, and if they are fine-grained enough to go fast then they are not much better than busy-waiting with time wasted for setting up timeouts). SMP makes this a feature. It forces use of busy- waiting, which is best if you have a CPU free to run it and this method doesn't take to much power. Context switches to task queues give similar slowness. This won't be affected by 4+4 since task queues are in the kernel. I don't like networking in userland since it has large syscall and context switch costs. Increasing these by factors of 6 and 25 doesn't help. It can only be better by combining i/o in a way that the kernel neglects to do or which is imposed by per-packet APIs. Slowdown factors of 6 or 25 require the combined i/o to be 6 or 25 larger to amortise the costs. Bruce From owner-freebsd-amd64@freebsd.org Sun Apr 1 21:33:55 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52194F7D6BE for ; Sun, 1 Apr 2018 21:33:55 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) Received: from sonic311-22.consmr.mail.gq1.yahoo.com (sonic311-22.consmr.mail.gq1.yahoo.com [98.137.65.203]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CB82D7EB62 for ; Sun, 1 Apr 2018 21:33:54 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) X-YMail-OSG: LJWjjPkVM1lBbkCAqLEhXZwCmfXs3q0ANb39gxtBjcY41nSItRNBGQAacqdb9JE QTVa715ZN4xVJyj0eGgUoF_tRcAs.0GgFCF0rgdUiADH4rfB9GQW3odgZnXYPHzvI3AEXwfM1JCJ YpNmFplqapjRCE.rFAHl1Zm.K2zHQjrRjPT_t4AXIMC772NhrT88bZZuu_G5K0i81Q.0edMLIUhI XoDIU9PcrzHZ_g8I60APqQ4lFUA901OL7HPFxDlO7figczbeckcvCtegXjpvFVWrR1BxVLZpd6V9 .2qxTHr76FvoqXNcVdUZO1y8i8dRl19ZcP3BSD_zGoGXXlr_LUBmwCFdlkVd8rroyqAYPh1y4E.8 yzvEp40AK9Tmz0pLiGUraQTkbepD5.J7lAoBEM_c6ZUBMr8MgJP_tVFyw1PQ2_sG0XoR9wHgFPID sH5NbDiTYd1Gzi.5rY5ZHesTK9AdxHnwAXqrJVUti9EYddC1FD5JM1tx3zKH4DpgDYxZtZ83WrEw zYQ6RincT1zDohvaBwiDeRddvghbEsg-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Sun, 1 Apr 2018 21:33:47 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp420.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID c89bb7fd000d589b52f8f7b1312f48ea; Sun, 01 Apr 2018 21:23:37 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) Subject: "Could not allocate I/O space" and "intsmb0 attach returned 6" in a under-Hyper-V context on Ryzen Threadripper: Is this expected? Message-Id: <621E84E6-D42C-4778-B028-AF3E1042CE7D@yahoo.com> Date: Sun, 1 Apr 2018 14:23:36 -0700 To: FreeBSD Current , FreeBSD Hackers , freebsd-amd64@freebsd.org X-Mailer: Apple Mail (2.3445.6.18) X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Apr 2018 21:33:55 -0000 For: # uname -apKU FreeBSD FBSDHUGE 12.0-CURRENT FreeBSD 12.0-CURRENT r331831M amd64 = amd64 1200060 1200060 I get: . . . pci0: at device 7.3 (no driver attached) . . . intsmb0: at device 7.3 on pci0 intsmb0: Could not allocate I/O space device_attach: intsmb0 attach returned 6 on a Ryzen Threadripper 1950X where FreeBSD is being run under Hyper-V (on a Windows 10 Pro machine). Is this expected? Did I misconfigure something in Hyper-V? This may have been true for a long time and I just had not noticed until now. For reference: # pciconf -l hostb0@pci0:0:0:0: class=3D0x060000 card=3D0x00000000 = chip=3D0x71928086 rev=3D0x03 hdr=3D0x00 isab0@pci0:0:7:0: class=3D0x060100 card=3D0x00001414 = chip=3D0x71108086 rev=3D0x01 hdr=3D0x00 atapci0@pci0:0:7:1: class=3D0x010180 card=3D0x00000000 = chip=3D0x71118086 rev=3D0x01 hdr=3D0x00 none0@pci0:0:7:3: class=3D0x068000 card=3D0x00000000 = chip=3D0x71138086 rev=3D0x02 hdr=3D0x00 vgapci0@pci0:0:8:0: class=3D0x030000 card=3D0x00000000 = chip=3D0x53531414 rev=3D0x00 hdr=3D0x00 # pciconf -l -v 0:0:7:3 none0@pci0:0:7:3: class=3D0x068000 card=3D0x00000000 = chip=3D0x71138086 rev=3D0x02 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82371AB/EB/MB PIIX4 ACPI' class =3D bridge And . . . Hyper-V Version: 10.0.16299 [SP0] = Features=3D0x2e7f PM Features=3D0x0 [C2] Features3=3D0xbed7b2 Timecounter "Hyper-V" frequency 10000000 Hz quality 2000 CPU: AMD Ryzen Threadripper 1950X 16-Core Processor (3393.73-MHz = K8-class CPU) Origin=3D"AuthenticAMD" Id=3D0x800f11 Family=3D0x17 Model=3D0x1 = Stepping=3D1 = Features=3D0x1783fbff = Features2=3D0xfed83203 AMD Features=3D0x2e500800 AMD Features2=3D0x3f3 Structured Extended = Features=3D0x201c01a9 XSAVE Features=3D0xf AMD Extended Feature Extensions ID EBX=3D0x4 Hypervisor: Origin =3D "Microsoft Hv" real memory =3D 53687091200 (51200 MB) avail memory =3D 52206305280 (49787 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 29 CPUs FreeBSD/SMP: 1 package(s) x 29 core(s) The local changes to /usr/src/ are mostly tied to powerpc64 and powerpc experimental activity, but there is some arm64 and arm material: # svnlite status /usr/src/ | sort ? /usr/src/nohup.out ? /usr/src/sys/amd64/conf/GENERIC-DBG ? /usr/src/sys/amd64/conf/GENERIC-NODBG ? /usr/src/sys/arm/conf/GENERIC-DBG ? /usr/src/sys/arm/conf/GENERIC-NODBG ? /usr/src/sys/arm64/conf/GENERIC-DBG ? /usr/src/sys/arm64/conf/GENERIC-NODBG ? /usr/src/sys/dts/arm/a83t.dtsi ? /usr/src/sys/dts/arm/sinovoip-bpi-m3.dts ? /usr/src/sys/dts/arm/sun8i-a83t-sinovoip-bpi-m3.dts ? /usr/src/sys/dts/arm/sun8i-a83t.dtsi ? /usr/src/sys/powerpc/conf/GENERIC64vtsc-DBG ? /usr/src/sys/powerpc/conf/GENERIC64vtsc-NODBG ? /usr/src/sys/powerpc/conf/GENERICvtsc-DBG ? /usr/src/sys/powerpc/conf/GENERICvtsc-NODBG M /usr/src/contrib/llvm/lib/Target/PowerPC/PPCFrameLowering.cpp M /usr/src/contrib/llvm/tools/lld/ELF/Arch/PPC64.cpp M /usr/src/crypto/openssl/crypto/armcap.c M /usr/src/lib/libkvm/kvm_powerpc.c M /usr/src/lib/libkvm/kvm_private.c M /usr/src/stand/defs.mk M /usr/src/stand/powerpc/boot1.chrp/Makefile M /usr/src/stand/powerpc/kboot/Makefile M /usr/src/sys/arm64/arm64/identcpu.c M /usr/src/sys/conf/kmod.mk M /usr/src/sys/conf/ldscript.powerpc M /usr/src/sys/kern/subr_pcpu.c M /usr/src/sys/modules/dtb/allwinner/Makefile M /usr/src/sys/powerpc/aim/mmu_oea64.c M /usr/src/sys/powerpc/ofw/ofw_machdep.c M /usr/src/sys/powerpc/powerpc/interrupt.c M /usr/src/sys/powerpc/powerpc/mp_machdep.c M /usr/src/sys/powerpc/powerpc/trap.c M /usr/src/usr.bin/top/machine.c I've modified top to show "MaxObsUsed" (Maximum Observed used) for Swap when it is positive: Swap: 194G Total, 4235M Used, 4235M MaxObsUsed, 190G Free, 2% Inuse, = 416K In =3D=3D=3D Mark Millard marklmi26-fbsd at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-amd64@freebsd.org Mon Apr 2 02:53:37 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 986CAF6BEB5 for ; Mon, 2 Apr 2018 02:53:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3082269850 for ; Mon, 2 Apr 2018 02:53:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 700482FDA3 for ; Mon, 2 Apr 2018 02:53:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w322raRd053341 for ; Mon, 2 Apr 2018 02:53:36 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w322raF0053340 for freebsd-amd64@FreeBSD.org; Mon, 2 Apr 2018 02:53:36 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-amd64@FreeBSD.org Subject: [Bug 210671] 'du' may report smaller than expected size(s) when using zfs Date: Mon, 02 Apr 2018 02:53:36 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 10.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: curtis.dunham@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 02:53:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D210671 Curtis Dunham changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |curtis.dunham@gmail.com --- Comment #1 from Curtis Dunham --- Can confirm this exact issue on 11.1-RELEASE. ZFS filesystem (not UFS) does not show the correct disk usage of a file unt= il after (presumably, per [1]) the transaction group commits. My timings agree with the 6 second sync interval. I do not agree that this is a feature. Causes bup [2]'s t/test-sparse-files.sh testcase to fail as it checks disk usage immediately after (test-) restoring a backup. Therefore it appears the failure is benign. [1] https://forums.freebsd.org/threads/du-behavior.56763/#post-323135 [2] https://github.com/bup/bup --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-amd64@freebsd.org Mon Apr 2 13:46:07 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E567DF63DD5 for ; Mon, 2 Apr 2018 13:46:06 +0000 (UTC) (envelope-from tdteoenming@gmail.com) Received: from mail-yw0-x244.google.com (mail-yw0-x244.google.com [IPv6:2607:f8b0:4002:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 73B847F6C6 for ; Mon, 2 Apr 2018 13:46:06 +0000 (UTC) (envelope-from tdteoenming@gmail.com) Received: by mail-yw0-x244.google.com with SMTP id v130so4994301ywa.0 for ; Mon, 02 Apr 2018 06:46:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=pi1/LsyRFc6Q9Y97LAMd5GGZIytuaDq3mgsTsres+kc=; b=LriSQpIZ435CoidWgiBnumO4tOddVOjoGF9qI/8J+InZoxTeVTSBr8AeGKc/+kO/Pp 0PU1HDSBH8dQ9Drx89269UDAXWCDM/c53Ur4tAbkmRudeHtf6J+OoSWsSO+QXqZxTslI vGFNMZKsFt2xCux7yAAQr36zBdG5TMScQc61YjU7Z76v1EnasNNOWyh2ttJ1i0hDuGY7 4d67VlYwZKlbtbP/ahY5690tmnJVfQJ0e4Wglu/64IgYba8kVJgpcWR4nTSAMtRH+gev zW3Di0pi71YIg40T9TtZFhbpdytWV4p5b7xY46CwuH0+6HzDLQ+sz00vn+NQW6RuFEmZ eRQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=pi1/LsyRFc6Q9Y97LAMd5GGZIytuaDq3mgsTsres+kc=; b=H1r39kf8SGfSMPtEfcsQn2PV+6RhF4TQMFJXlS2/hbTZB2USf8KC40AWBezzPv9ulQ k41EKpRWsHSjRHZkBaa6t8s+m05vQrhi5Q5hdp/mkuj8Yqi44XGjDMnMH05wtSDYJ6RE X1qEZ5enfH8s3DUpDmOBR9InMRELPDMltu5w7qPayV07T10H1T4gGXdQI5eUQYZVg81m z6SkS3jaKHYD1OQwZMnuYNxXDFOOpxtNtTIId2FIZ0w7PMXZbBEcWEv6oExhPxkAzw51 mc9IU2KFMcNBgqXRaGRTPnMWXn9/6YJug59OfJfm885BhzE/NEvr+8EfWCgBYn7SvER2 QHaA== X-Gm-Message-State: AElRT7E5sWF0SsGa1sFhmFzWA/8Et4F4RpccTkfK7UZknFqrKbA5Cfnk NbANyp6TU0K7o+vwUC0GpafhRFOmoQY8fLyKgqzayqWEPQ== X-Google-Smtp-Source: AIpwx49Ol4NaLoq6/L42eXHcJsE9bZ6T3vYkEpQ3wnHj0iHckIzSK6a3Na02QowSLKUeXZRnmkty4zBML0Ws1G0M5uE= X-Received: by 10.13.203.85 with SMTP id n82mr5450908ywd.417.1522676765717; Mon, 02 Apr 2018 06:46:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.13.219.150 with HTTP; Mon, 2 Apr 2018 06:46:05 -0700 (PDT) From: Turritopsis Dohrnii Teo En Ming Date: Mon, 2 Apr 2018 21:46:05 +0800 Message-ID: Subject: What is the universal (world wide) understanding behind degaussing harddisks? To: freebsd-amd64@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 13:46:07 -0000 Good evening from Singapore! The foremost question which I want to ask is, what is the universal (world wide) understanding behind degaussing hard drives? I work for No Secrets Agency (NSA) Pte Ltd (fictitious company name used). My sales manager Edward Joseph Snowden (fictitious individual name used) had *promised* our customer Leave Me in the Lurch (S) Pte Ltd (fictitious company name used) that we would "DEGAUSS" their hard disks after the PC replacement and data migration exercise for 15 trillion PCs (fictitious number used). PC = Personal Computer, which includes desktops and laptops Last Friday, I had already reflected to my sales manager Edward Snowden that since we are definitely NOT going to wipe our customer's data by using strong and powerful magnets (physical means), should I send an email to the IT Administrator of our customer Lady Gaga (fictitious individual name used) asking her which data sanitization method (by software means) I should use? My sales manager Edward Snowden had quickly deflected my concerns (that is, wanting to send an email to our customer Lady Gaga asking her which data sanitization method I should use). I had brought up to the sales manager Edward Snowden a number of data wiping methods by software means last Friday. (1) Very very simple 1-pass data wiping, quickest a. Using "sudo dd if=/dev/zero of=/dev/sda", overwriting harddisks from beginning to end with zeroes, where /dev/sda refers to the 500 GB harddisk, not /dev/loop0 and not /dev/sdb which refers to the bootable live operating system on thumb drive b. Using "sudo dd if=/dev/urandom of=/dev/sda", overwriting harddisks from beginning to end with random data, where /dev/sda refers to the 500 GB harddisk, not /dev/loop0 and not /dev/sdb which refers to the bootable live operating system on thumb drive Any bootable Live CD/DVD/flash media could do it. (2) 3-pass U.S. Government/Department of Defense (DoD) standard (DoD 5220.22-M) Certified commercial software required (3) 7-pass U.S. Government/Department of Defense (DoD) standard (DoD 5220.22-M) Certified commercial software required (4) 35-pass Gutmann method, slowest Certified commercial software required All these was last Friday. In the midst of our argument over the cellular network "just now", my sales manager Edward Snowden tried to cover up himself by suddenly and unexpectedly making an excuse that he had told me last Friday I was supposed to wipe user data only, not the operating system! If he had wanted me to wipe user data and retain/keep the Windows operating system, he should simply have told me to Reset the PC (for Windows 10 only) or use a =secure File Shredder=! For Windows 7, you can still wipe user data and preserve the operating system by using the Recovery Partition. On Lenovo desktops, press and hold F11 when Windows 7 is starting and reset to factory defaults. I had advised my sales manager Edward Snowden not to use BIG WORDS like "degaussing the harddisk" and market to the customer using these big words. Any person who sees the word "degauss" would have understood it to mean sanitize *ALL* data on the harddisk straight away and without question. Please refer to Figure 1. Exhibit A below for the "degaussing" instructions communicated to me by my sales manager Edward Snowden. Please click the link below. URI: https://i.imgur.com/bGOMyVs.png Please advise. Thank you very much. Regards, Mr. Turritopsis Dohrnii Teo En Ming Systems and Network Engineer Republic of Singapore 2nd April 2018 Monday 9:35 PM Singapore Time GMT+8 From owner-freebsd-amd64@freebsd.org Mon Apr 2 15:59:25 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1205EF73DC4 for ; Mon, 2 Apr 2018 15:59:25 +0000 (UTC) (envelope-from db@db.net) Received: from diana.db.net (unknown [IPv6:2620:64:0:1:223:7dff:fea2:c8f2]) by mx1.freebsd.org (Postfix) with ESMTP id 9A6A784A6D for ; Mon, 2 Apr 2018 15:59:24 +0000 (UTC) (envelope-from db@db.net) Received: from night.db.net (localhost [127.0.0.1]) by diana.db.net (Postfix) with ESMTP id 038C82AA371; Mon, 2 Apr 2018 09:49:44 -0600 (MDT) Received: by night.db.net (Postfix, from userid 1000) id D6F0F39813; Mon, 2 Apr 2018 11:59:20 -0400 (EDT) Date: Mon, 2 Apr 2018 11:59:20 -0400 From: Diane Bruce To: Turritopsis Dohrnii Teo En Ming Cc: freebsd-amd64@freebsd.org Subject: Re: What is the universal (world wide) understanding behind degaussing harddisks? Message-ID: <20180402155920.GB75445@night.db.net> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 15:59:25 -0000 On Mon, Apr 02, 2018 at 09:46:05PM +0800, Turritopsis Dohrnii Teo En Ming wrote: > Good evening from Singapore! > > The foremost question which I want to ask is, what is the universal > (world wide) understanding behind degaussing hard drives? If you degauss a modern drive you will make it totally useless to use. You might as well quarter the drive with a bandsaw and incinerate. (This was the recommended procedure for security disks from a (fictitious) agency I worked (indirectly) for years ago.) The problem is modern drives lay down servo tracks on the platters which can only be done at the factory. > > (1) Very very simple 1-pass data wiping, quickest > > a. Using "sudo dd if=/dev/zero of=/dev/sda", overwriting harddisks ... > > b. Using "sudo dd if=/dev/urandom of=/dev/sda", overwriting harddisks Any method you use will *not* remove all data due to the slight wobble of the track due to temperature changes in the disk, vibration all sorts of problems. However at that point if done properly it would take specialized gear only (fictitious) security agencies such as the NSA (fictitious company) would be likely to bother with. The TL;DR answer is. If you want to use the drives afterwards don't simply demagnetize them; A triple write is probably sufficient if it is merely company data. if the data is drug dealings or state secrets then destroy the drives. ;) > Mr. Turritopsis Dohrnii Teo En Ming > Systems and Network Engineer > Republic of Singapore > 2nd April 2018 Monday 9:35 PM Singapore Time GMT+8 -- - db@FreeBSD.org db@db.net http://www.db.net/~db From owner-freebsd-amd64@freebsd.org Mon Apr 2 16:47:16 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 208E7F76C4A for ; Mon, 2 Apr 2018 16:47:16 +0000 (UTC) (envelope-from cfaber@gmail.com) Received: from mail-io0-x241.google.com (mail-io0-x241.google.com [IPv6:2607:f8b0:4001:c06::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 979FA86D27 for ; Mon, 2 Apr 2018 16:47:15 +0000 (UTC) (envelope-from cfaber@gmail.com) Received: by mail-io0-x241.google.com with SMTP id q80so18554842ioi.13 for ; Mon, 02 Apr 2018 09:47:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=g2dXCSGfvMZEsXL3u9lEXNBErdyttrf/eu0+/uQTlTg=; b=fMYQ/uqe6hwntbH839d4mYtSZPDwyyMBGDBFQmWv1DAdQPyETRplqVYa9RfdXcGrwi wY7R5mo1NGHmdKq/Jo8I9QDur3F04DGxDPpj1g2MpoqWQapD6lIPeLymFCRxrwE7TDJc ttXoYRn/QfD5n26d5V6sJWaT+IAa00gsHfUDcw60p+49g/vXKjL7/hlnndyS4xqMsfm9 no7PI0UEkGo8qDqQP2nwUuF1iJ6nmnE6yiCM1tlTznCeJ22Dq+9/+WTf/6h6Rj+JasBp oI6ptH0rGk6MIa9nTHfaZHY4ydkt6mOwXSHL2Hofej9Vvf13nNNUd+p+0f5DIPJ93Gd2 C/yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=g2dXCSGfvMZEsXL3u9lEXNBErdyttrf/eu0+/uQTlTg=; b=DGHoQdwddabd7Vi7Nz8oo5s0uJJjX3DneUR/xKMS67/hl7D5oiDKzHAR/vmXLbpu6Z sbbh0O77wUXDO/usr8H9ZYXGrpRSWlhMAanm3Q6satUctdYPIAnlOUVNOeDlcGpdsclK PjaTzcDEmicDOvSpbOfSlHNsmB+HIJUfJQxuYzB2xncCBex1DXqR+TfNUywQI2C8474L Qtl1rttnPFh8WhqCQtylZ0ngjxuvCpgjEJ6pfVsW5MS3xSOqev86YlRUC4peDhu5Hv07 6Jj5vdliv/nZgr0XNvQ8TbrYJX50Kr1OiZx8Kcgz9j0AfIuv7m85hrfX+MVlnDdI9leL 2CiA== X-Gm-Message-State: AElRT7G44DckhJZ+Qbiclx4RV4HhsssfwbcfYuWAjK/LRMlPJ0BPyp1w WoDvBkWu4hV3VQsOBmg2t9q1/ga5mRx5nugSNl4= X-Google-Smtp-Source: AIpwx49tAb6TEAqI7dLKBwG2EkIpr4/3Ji5rE/G5cff+dDSjeVfoyJxiWLtX6fsiflNt8KIfbw9rrQoIRKW72RdaFwo= X-Received: by 10.107.199.195 with SMTP id x186mr9346567iof.176.1522687635003; Mon, 02 Apr 2018 09:47:15 -0700 (PDT) MIME-Version: 1.0 References: <20180402155920.GB75445@night.db.net> In-Reply-To: <20180402155920.GB75445@night.db.net> From: Colin Faber Date: Mon, 02 Apr 2018 16:47:04 +0000 Message-ID: Subject: Re: What is the universal (world wide) understanding behind degaussing harddisks? To: Diane Bruce Cc: Turritopsis Dohrnii Teo En Ming , freebsd-amd64@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 16:47:16 -0000 2nd, shred with random write 3 pass and zero'd forth pass should be enough for most data recovery efforts. SSDs however are another problem, as they're copy on write devices and are over provisioned to accommodate for failure. In those cases I would consider 9 to 16 full disk random writes with at least 4 zeroing passes, but that still doesn't guarantee the data will not be recoverable. On Mon, Apr 2, 2018, 10:00 AM Diane Bruce wrote: > On Mon, Apr 02, 2018 at 09:46:05PM +0800, Turritopsis Dohrnii Teo En Ming > wrote: > > Good evening from Singapore! > > > > The foremost question which I want to ask is, what is the universal > > (world wide) understanding behind degaussing hard drives? > > If you degauss a modern drive you will make it totally useless to > use. You might as well quarter the drive with a bandsaw and incinerate. > (This was the recommended procedure for security disks from a > (fictitious) agency I worked (indirectly) for years ago.) > > The problem is modern drives lay down servo tracks on the platters > which can only be done at the factory. > > > > > (1) Very very simple 1-pass data wiping, quickest > > > > a. Using "sudo dd if=/dev/zero of=/dev/sda", overwriting harddisks > ... > > > > b. Using "sudo dd if=/dev/urandom of=/dev/sda", overwriting harddisks > > Any method you use will *not* remove all data due to the slight wobble > of the track due to temperature changes in the disk, vibration all sorts > of problems. However at that point if done properly it would take > specialized gear only (fictitious) security agencies such as the NSA > (fictitious company) would be likely to bother with. > > The TL;DR answer is. If you want to use the drives afterwards don't > simply demagnetize them; A triple write is probably sufficient if it is > merely company data. if the data is drug dealings or state secrets > then destroy the drives. ;) > > > Mr. Turritopsis Dohrnii Teo En Ming > > Systems and Network Engineer > > Republic of Singapore > > 2nd April 2018 Monday 9:35 PM Singapore Time GMT+8 > > -- > - db@FreeBSD.org db@db.net http://www.db.net/~db > _______________________________________________ > freebsd-amd64@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-amd64 > To unsubscribe, send any mail to "freebsd-amd64-unsubscribe@freebsd.org" >