From owner-freebsd-hackers@freebsd.org Sun Jul 12 03:22:13 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 120D396DC for ; Sun, 12 Jul 2015 03:22:13 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 793D29D4 for ; Sun, 12 Jul 2015 03:22:12 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from [192.168.1.115] ([67.212.197.98]) by mail.gmx.com (mrgmx002) with ESMTPSA (Nemesis) id 0LaXEN-1YZaYQ1hnj-00mIjE for ; Sun, 12 Jul 2015 05:22:09 +0200 Message-ID: <55A1DD4F.1050507@gmx.com> Date: Sat, 11 Jul 2015 20:21:51 -0700 From: Don whY User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: Xorg for client-only support References: <559EDB56.70808@gmx.com> <2863155.VvqRCPmh7x@desk8.phess.net> In-Reply-To: <2863155.VvqRCPmh7x@desk8.phess.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:jDuISo+e4zQ81x0YpGlCto0Ak5pqBJZIIQhUJNXPKU8L5IZolHX zEkAsoH5mWmw9xGBsmYddW55NEIAl7cfi0+0jLGKk7SWA9UNM263jfVWfQXSIE0lvJnT9KV I7XXNn9EtzW692xDni6AlsgD7iPKXxE6oVT6fQb5cubl4DqXH+BOOXBYbCqJEGzKyCuhEfA 1VcHvkG7IZVbuFB41o2Gg== X-UI-Out-Filterresults: notjunk:1;V01:K0:ptYKIYrCp50=:BmNVjwzHUQGKTz+SIRuSX5 ccoswbQBJs8Yy6337Vvvsqzo7vJqrQrkGZ74FWQEtLJe37MzEi8znFfqCPd2JfVlHk8faXPDZ e4LHAvheFEh+gfJfVd2DVJ/65JlQNVB7dfJP1QvYAIY8IRDj9YMFBGg2S9YJr75jVcnwmuRoG z3Hctu7rGnMwHo2EYHlXVnKOV7scjnXxIzRZZlLifXbHlerRa1buedF8pnILLCDu7fIl6fO/8 IFbYHdX5QBXynAdmB9u8K6wvg2zbCFSHW01RQAFceS93rb6GuuMCftWjFlSt8RT+87ZDsMHxm p3QAW3rub9o4GZvdqGmrfH8s+iDUDz0s5XbeRfjkIyWTQ988Rtw2J6YDke7Rqb9vsdjO3gbEw MIVSsBYZnxEZkAbbT4sd8WK1cD/SuJLm2IqgZEaWqKfV+h3J8R06LfVFybm3/PSCMcHD+7NUt LRWNfZkcxFRg8iDy4Q9lb10NezAuR4fPJZnUA3WzuDQ3SElDkv2CIJjZY3Ogqr+79dsP3Zv1C 5x9/Nj7zO6K/OqMEZ13MMb3Y044fB3H+al3HvqnhNYCkj3ha162Wsznc43RThFG2oaRkBy6JL ZnB2/9KmQ14FA0NEIhTACL6h8sjh2fcihkzuq+ofbUzRHrpURVlVmuXRAGW4Yl034J2QizBD6 6Wth+TKySukCjm8WyrNC8iMHd1MkNativJf9jVkKjcccaK1zzG48WNDJcuPzKYIe8OdU= X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2015 03:22:13 -0000 On 7/10/2015 9:41 AM, Patrick Hess wrote: > Don whY wrote: >> For *headless* devices (i.e., no point in having a real *server*!), >> are there significant portions of xorg that I can omit? Or, is it >> easier to build and install it all and remove the unnecessary cruft, >> later? > > Since you don't need the server portions, I don't really see a reason > why you would want to worry about installing any X.org ports by hand. Not sure what you mean "by hand"; if "build from scratch", that's just the way I've done things for the past 20+ years... > Just installing the applications you want to use should automatically > pull in the X.org dependencies that are actually required by these > applications. Are the client dependencies that fine-grained? I.e., they don't just drag The Kitchen Sink in? > This will prevent any unnecessary parts, like server > components and video drivers, from being installed on your system > in the first place. I was hoping for a port akin to "xorg-clients". But, i should be able to hack one together based on your above comment. From owner-freebsd-hackers@freebsd.org Sun Jul 12 03:25:19 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83F81975D for ; Sun, 12 Jul 2015 03:25:19 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 15A10AF5 for ; Sun, 12 Jul 2015 03:25:19 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from [192.168.1.115] ([67.212.197.98]) by mail.gmx.com (mrgmx002) with ESMTPSA (Nemesis) id 0MdKgV-1ZVvkq0ICJ-00IUuN; Sun, 12 Jul 2015 05:25:11 +0200 Message-ID: <55A1DE05.4010304@gmx.com> Date: Sat, 11 Jul 2015 20:24:53 -0700 From: Don whY User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: Wojciech Puchar CC: FreeBSD-Hackers Mailing List Subject: Re: format/newfs larger external consumer drives References: <559EDAB8.9080804@gmx.com> <55A00743.4080609@gmx.com> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:tir9jWC8NMOusT1exvLcvFn7L6l/niVIMHZZ4hByR0+GjYlC/dR 9yfHtRpmannVTbdDxU7v5kCaajoXbrPqqoIg11bJr3gm2lVytcrq15mfqANQQ5oA+SPTP3z Pvx+2EdU4uxu4Ja1nll12nK63PDvC72nbFDOwp71dwvcebNj9zMNcBTkmw/03VgdNQnq71W TSf8+UvSypNWeFwMFUN1A== X-UI-Out-Filterresults: notjunk:1;V01:K0:9pOuhxTPXyQ=:34gisSu4Ps45Xy1aLaLqqg Ijx1hPEN8jpNNoN8VI7Q/NH+vypUH8F9uMC9gEtvogmpYh06ODXh5MaiyVqJWaZ76famdTqVm 8uRo/pUPKxMWJj0Xym+yS/KkR9cuxeEaDFgPAAQU6XQPDZLNTctC4yk00fWTkV3/P9/zjb91A ttdRT6hHPGaRTYQk/1KbIsTWy0csulaDQQnlZdXXH/Zirgr55VE2/1N3+vW4QPq6o3a2xExzf YT0VsPpIAMtAnHbhfveu5v39rnjrSYUPTXgOGDGpaHuRcVdRTqjBCDAOQmiHZKFqdX6vdod7y DyFVGGwmw9QdQhn+/Cv0xXt/4eteXfKm9j/YOiqfkrEDzkzdcxX9fV5z8QsGw90jFGbbWV9V5 xVTWxNoqWns9M6H+kvcJR9rWdzmxaOJ8EgbnZmWeMjNDwGYjSNGjGd6ZFJjAGlZHJj9XZgnH/ piqxf1DARA8H/NBBpGQP7GsyGlSSfuLJywCtTMtTZDgUwM4QDU5YvGFzJibv3yK9WLz/xHTEh VIo7x0rDo2agQcagI1cRwIQ4G4nOrz4po8r5JZHffJ+xs8CYU2zue5X1KAQyudC69qhG4KI8f BQr3+w7i+LFw4K426XOTr7a9G8hcXnAWJp71xp+zvSBklQnAlsvhqvJ0wn3ROGm3bmaqHKPlX 7F0zOK8vGPVKbSduSygwhhy+2OCKhAZXBeO++CDyubxGmf88oLNwK1pqrqnOtOOWYHkQ= X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2015 03:25:19 -0000 On 7/10/2015 1:20 PM, Wojciech Puchar wrote: >>> >>> i would assume you will most likely store large files. >> >> For the demo application I'm writing up (to illustrate the issues >> that appliance developers might face), I would be storing large files >> (e.g., ISO's). But, some other developer/application might choose >> to use the medium for smaller files -- or even smaller media capacity. >> > so right options. > > for smaller files > > newfs -m 0 -i -b 32768 -f 4096 -U > > this will mean longer fsck > > you may add -j if you like - soft updates journalling. > >> means little "mismatches" in configuration can have noticeable >> impact on the end user (e.g., he opts for finer-grained management >> and pays the price when a volume isn't properly dismounted, power >> fail, etc.) > > depends on I/O style. On random I/O it will not have big impact. I meant he pays the price when something goes *wrong*. With an appliance, you're not typically dealing with console access. Rather, expecting the device to just "Do The Right Thing". >>> newfs -m 0 -i 262144 -b 65536 -f 8192 -U /dev/yourdisk >>> >>> and it will be fast to fsck. From owner-freebsd-hackers@freebsd.org Sun Jul 12 06:22:04 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C190499933E for ; Sun, 12 Jul 2015 06:22:04 +0000 (UTC) (envelope-from patrickhess@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 469B7190C for ; Sun, 12 Jul 2015 06:22:03 +0000 (UTC) (envelope-from patrickhess@gmx.net) Received: from desk8.phess.net ([95.88.11.237]) by mail.gmx.com (mrgmx103) with ESMTPSA (Nemesis) id 0MA9FV-1Z7t861jEP-00BJ8l for ; Sun, 12 Jul 2015 08:21:55 +0200 From: Patrick Hess To: freebsd-hackers@freebsd.org Subject: Re: Xorg for client-only support Date: Sun, 12 Jul 2015 08:21:53 +0200 Message-ID: <4777726.x7gaKDivgh@desk8.phess.net> User-Agent: KMail/4.14.3 (FreeBSD/10.1-RELEASE-p10; KDE/4.14.3; i386; ; ) In-Reply-To: <55A1DD4F.1050507@gmx.com> References: <559EDB56.70808@gmx.com> <2863155.VvqRCPmh7x@desk8.phess.net> <55A1DD4F.1050507@gmx.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Provags-ID: V03:K0:T5QUL2UW+qbDQCgUMVCW4M3cvW2uM50+N/Uct2eZpbSCBYByRWh RTWEUFHBvQEin2tiPiAt4IC7zkwZ3WOAOpE3bMzM92lo15jK8mAQv2bPbWbnOzmyRgC+wjc 4xUlPOB3skw7zbEJQrpm9IIR/thWKn1CX1DD352FqkRPrgVmyUz6RmXwCHSwA3iOx5mySk9 O+z0DchiyWOJqRtZWHE2w== X-UI-Out-Filterresults: notjunk:1;V01:K0:FEX7/bGQ4SA=:zwBqPekVx1iCyLsXWWoh07 kL1EZm+Z2BQeSpAoe6dvGQhWKPmXGrtay5Yq/WCWSqsV0/UPwi6ZDXix7BttdCsTiQS1hqx1I XnnustdbZ6h2BC0Q349AgpGLjh26H0SiwgABaM6NcuwmLpHojtQh4pM2ufog6lWQN8PpXHVQL SLITPT2dEOjiHGzD0cPQja2M5QmwTxQUYKerCxAUYh76zcOSzmc8CSBeDU80x9r8YqKisUDIa iyQDE1AI0qD+2GTYmVzdmXXg7yD4iAw7GwrkFZj8FRQgb2SqawF/262almCiAvNEXjumA07ma QlkeKlABgXyc9Dk14bIeQZSpfErziTBlSPEuKQXynr5MeStKuQLr/9VtF0A/Ee2bzW3tZ2dUP 5iPoZnB9yN11eoRtosZRlc7KTHpbozrLNcifOpM3Bd6zcCXJtelpstCPt2eABNRzDbHj3vLxO uyPRC8yaTAsSWXAzxLST1RMue1UeuFWV2IzSIlOKlU9b09SyQBQnOmqBFGP+TrjXWQNY+5Ul0 amAIx7lwmp7VBDtOuLiJyUj7VwR72QR1onyO5gezVZLxj0WcrLx+MJUDvVqAFGa4pmrVNGBvW Y7p7ZfLeo/xLMFaOuQquWR55vWEMKOsEz7aksqv6TqxbbG1uTQeZKyLopgaBaxL1hqkAI92O8 frpcaRlsZPpZRD8hQ3F1GqOOayyKqlZW/pcCeLGtWyFXraG6E8WNgyrzzfHdWe7aze5M= X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2015 06:22:05 -0000 Don whY wrote: > On 7/10/2015 9:41 AM, Patrick Hess wrote: > > Don whY wrote: > >> For *headless* devices (i.e., no point in having a real *server*!), > >> are there significant portions of xorg that I can omit? Or, is it > >> easier to build and install it all and remove the unnecessary cruft, > >> later? > > > > Since you don't need the server portions, I don't really see a reason > > why you would want to worry about installing any X.org ports by hand. > > Not sure what you mean "by hand"; if "build from scratch", that's > just the way I've done things for the past 20+ years... Now I'm not so sure any more what you mean by "build from scratch". I was assuming that you were using the FreeBSD ports system, in which case I wouldn't bother to explicitly install any X libraries, like so: make -C /usr/ports/x11/libX11 install && make -C /usr/ports/x11/libXext install && make -C /usr/ports/x11-toolkits/libXt install && [...] Instead, I would just install the actual applications I wanted to run on that machine and have the above libraries be installed by the ports system automatically should one of those applications need them. Of course, if you were building from source without using the ports system, you'd have to resolve all of these dependency issues yourself. I don't see a compelling reason why you would want to take that route, though. > > Just installing the applications you want to use should automatically > > pull in the X.org dependencies that are actually required by these > > applications. > > Are the client dependencies that fine-grained? I.e., they don't just drag > The Kitchen Sink in? I'd say that most ports are pretty good at specifying only the minimum set of dependencies that are actually required by the port. Just off the top of my head, the one counter example that comes to my mind is x11/lumina, which has a dependency on x11/xorg, a metaport for the *entire* X.org distribution. I didn't have the time to take a closer look at that port just yet, but that feels like overkill to me. > > This will prevent any unnecessary parts, like server > > components and video drivers, from being installed on your system > > in the first place. > > I was hoping for a port akin to "xorg-clients". But, i should be able > to hack one together based on your above comment. x11/xorg-libraries seems to bundle a fair amount of client libraries. That being said, I wouldn't bother about it either. Should one of the applications you are to install require x11/xorg-libraries, the ports system will automatically install it for you anyway. Patrick From owner-freebsd-hackers@freebsd.org Sun Jul 12 18:18:55 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5089999576 for ; Sun, 12 Jul 2015 18:18:55 +0000 (UTC) (envelope-from lobo@bsd.com.br) Received: from mail-qk0-x22c.google.com (mail-qk0-x22c.google.com [IPv6:2607:f8b0:400d:c09::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 72D88192D for ; Sun, 12 Jul 2015 18:18:55 +0000 (UTC) (envelope-from lobo@bsd.com.br) Received: by qkbp125 with SMTP id p125so241214081qkb.2 for ; Sun, 12 Jul 2015 11:18:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsd.com.br; s=capeta; h=date:from:to:cc:subject:message-id:in-reply-to:references :organization:mime-version:content-type:content-transfer-encoding; bh=K5ng5x+DA45feloHtNSouwuy3m/J1t+pqiATla6a39I=; b=ReTEmmOZP1n5o95xNfAFDJqZ+mrC6OFa6631T3xuN00u9Zg7gmrwNvJiRZgw+ACmYo 2krFw7ZxRwh3+s39oxHUcCiovAHIecL1Rw2dJjS2fzeHafK1+7lEr+lz7BafGl42WXov sMZRLUYZQShL1YRWgDJPavcQ9/OsifXVYb914= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:organization:mime-version:content-type :content-transfer-encoding; bh=K5ng5x+DA45feloHtNSouwuy3m/J1t+pqiATla6a39I=; b=C0QdHCzf4QNmstY1SsXB+nMMFnEwuNTcz68N2F6bn+eRujM+wrBY8QfgZ4uxxrzx5f X/My+alO/+SP4xa10SxsGuRyfzjZ+0GMplAdU9KxIE/aed6BG32g1ImUc3ldMBk3TKfw cMuP0wOsLx9YGvtX1TcvpNfiQ5ASdElw2vLtt8vBit7HYXftXWQNtJ83EwdXARb5cziv mS9HUYEb/gmb5Eyvh4gApk6Xwam1Xucw2eBj6ecD+wMR63Zu8JgDRvqCKGlZKmN5KN+P utBOwaGZktmuMzEKT8Kx+fk8w9AKhP7gyafVusWdAm2C8HOiJECQL93jcKmyU+8kGJgN WeKA== X-Gm-Message-State: ALoCoQk8uBNUJpg81pDIuqAeIcWn456vminwOVzoXFs/TBQfS+bzTrMAPrCbFU2semG77c24n1CS X-Received: by 10.140.238.22 with SMTP id j22mr49895524qhc.98.1436725134374; Sun, 12 Jul 2015 11:18:54 -0700 (PDT) Received: from Papi ([186.212.119.146]) by smtp.gmail.com with ESMTPSA id f207sm9500691qhc.41.2015.07.12.11.18.52 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 12 Jul 2015 11:18:54 -0700 (PDT) Date: Sun, 12 Jul 2015 15:23:21 -0300 From: Mario Lobo To: "Herbert J. Skuhra" Cc: freebsd-hackers@freebsd.org Subject: Re: Gigabyte 970A-UD3P and hwpstate problem Message-ID: <20150712152321.51b8407c@Papi> In-Reply-To: <20150711135006.GB41680@oslo.ath.cx> References: <20150710213752.473cb831@Papi> <20150711135006.GB41680@oslo.ath.cx> Organization: BSD X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.25; amd64-portbld-freebsd10.1) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2015 18:18:55 -0000 On Sat, 11 Jul 2015 15:50:06 +0200 "Herbert J. Skuhra" wrote: > On Fri, Jul 10, 2015 at 09:37:52PM -0300, Mario Lobo wrote: > > Hi; > >=20 > > I just installed a Gigabyte 970A-UD3P mobo and updated BIOS to the > > latest version but the problem also showed with the previous > > version. > >=20 > > Here is my amd64 10-STABLE setup: > >=20 > > FreeBSD 10.2-PRERELEASE #0 r285207M: Tue Jul 7 00:11:01 BRT 2015 > > amd64 > >=20 > > CPU: AMD FX-8320E Eight-Core Processor (3214.93-MHz K8-class CPU) > > Origin=3D"AuthenticAMD" Id=3D0x600f20 Family=3D0x15 Model=3D0x2 > > Stepping=3D0 > > Features=3D0x178bfbff > > Features2=3D0x3e98320b > > AMD Features=3D0x2e500800 AMD > > Features2=3D0x1ebbfff > > Structured Extended Features=3D0x8 > > SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=3D65536 > > TSC: P-state invariant, performance statistics > > real memory =3D 17179869184 (16384 MB) > > [snip] > > ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has > > zero address or length: 0x0000000000000000/0x1 (20150515/tbfadt-673) > > [snip] > > hpet0: iomem 0xfed00000-0xfed003ff on > > acpi0 > >=20 > > The problem: > >=20 > > powerd: no cpufreq(4) support -- aborting: No such file or directory > > /etc/rc: WARNING: failed to start powerd >=20 > Could this be your problem: >=20 > r276986 | nwhitehorn | 2015-01-11 18:10:07 +0100 (s=F8n, 11 jan 2015) | > 8 lines >=20 > MFC r265329: > Disable ACPI and P4TCC throttling by default, following discussion on > freebsd-current. These CPU speed control techniques are usually > unhelpful at best. For now, continue building the relevant code into > GENERIC so that it can trivially be re-enabled at runtime if anyone > wants it. >=20 > Relnotes: yes >=20 > % svnlite diff -c 276986 sys/amd64/conf/GENERIC.hints=20 > Index: sys/amd64/conf/GENERIC.hints > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- sys/amd64/conf/GENERIC.hints (revision 276985) > +++ sys/amd64/conf/GENERIC.hints (revision 276986) > @@ -31,3 +31,5 @@ > hint.attimer.0.port=3D"0x40" > hint.attimer.0.irq=3D"0" > hint.wbwd.0.at=3D"isa" > +hint.acpi_throttle.0.disabled=3D"1" > +hint.p4tcc.0.disabled=3D"1" >=20 > Does it work if your unset or remove those lines > from /boot/device.hints? >=20 YEEESSS ! Thanks for pointing that out! I have these commented on my loader.conf but I didn't know they were active somewhere else. As far as cpufreq/powerd goes, it's working now and frequencies are being throttled but still no cool'n'quiet/hwpstate0 showing on dmesg. acpi_throttle attaches to cpu0 but not to the others. acpi_throttle0: on cpu0 acpi_throttle1: on cpu1 acpi_throttle1: failed to attach P_CNT device_attach: acpi_throttle1 attach returned 6 acpi_throttle2: on cpu2 acpi_throttle2: failed to attach P_CNT device_attach: acpi_throttle2 attach returned 6 acpi_throttle3: on cpu3 acpi_throttle3: failed to attach P_CNT device_attach: acpi_throttle3 attach returned 6 acpi_throttle4: on cpu4 acpi_throttle4: failed to attach P_CNT device_attach: acpi_throttle4 attach returned 6 acpi_throttle5: on cpu5 acpi_throttle5: failed to attach P_CNT device_attach: acpi_throttle5 attach returned 6 acpi_throttle6: on cpu6 acpi_throttle6: failed to attach P_CNT device_attach: acpi_throttle6 attach returned 6 acpi_throttle7: on cpu7 acpi_throttle7: failed to attach P_CNT device_attach: acpi_throttle7 attach returned 6 sysctl -a | grep freq dev.cpufreq.0.%parent: cpu0 dev.cpufreq.0.%pnpinfo:=20 dev.cpufreq.0.%location:=20 dev.cpufreq.0.%driver: cpufreq dev.cpufreq.0.%desc:=20 dev.cpufreq.%parent:=20 dev.acpi_throttle.0.freq_settings: 10000/-1 8750/-1 7500/-1 6250/-1 5000/-1 3750/-1 2500/-1 1250/-1 dev.cpu.0.freq_levels: 3214/-1 2812/-1 2410/-1 2008/-1 1607/-1 1205/-1 803/-1=20 dev.cpu.0.freq: 803 Thanks again! --=20 Mario Lobo http://www.mallavoodoo.com.br FreeBSD since 2.2.8 [not Pro-Audio.... YET!!] (99% winblows FREE) =20 "UNIX was not designed to stop you from doing stupid things,=20 because that would also stop you from doing clever things." From owner-freebsd-hackers@freebsd.org Sun Jul 12 20:02:44 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC49B99B95A for ; Sun, 12 Jul 2015 20:02:44 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 53B851E21 for ; Sun, 12 Jul 2015 20:02:44 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from [192.168.1.115] ([67.212.197.98]) by mail.gmx.com (mrgmx103) with ESMTPSA (Nemesis) id 0LgqEs-1YbEkY3yK6-00oE0G for ; Sun, 12 Jul 2015 22:02:41 +0200 Message-ID: <55A2C7CD.7080005@gmx.com> Date: Sun, 12 Jul 2015 13:02:21 -0700 From: Don whY User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: format/newfs larger external consumer drives References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:MUFfqIwY3/WdUDs4CUiu+7e8zrBompkgvpjb7qFZbKSJ4bkgTCE sTPm8oLm9/zPfxR94lNR14jCifXjmrUwAf26gaVe/PhJ1BCGXBuKGh7S3hCvBn6d2hFeh+E gXuIie4zB1MqGfhKTzuJvQmdelmeB9+QBQyeqagN/waFvS8qwZ4CRehnlRBIVw/INaMXDo3 QTzJgha6cJGN4oVD3mJdQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:eCo7smMUI9Q=:Kdqkei2Bdrtz0w/ziOtCVg E6O3abU5R3YIMT/eLpi4OETPj5oBYmnXXWpmoc4Ftv2ar6qVV024x1QsagoDiyrZDU5bT2esM EkOZiDuR01n0/jz1e7/PKn+5f4PJCW3n9zEUBJUWkDvwLuxevS9a4z48VWCXfERy/Qx4i3u23 vPVGEfhk69pmuQ6+B08s2Z+S89bSqaqkzYdNntP+SyVhKjNWjwIwKQKuz3xV+23CRP8yIXDIT 8UqxZgQipD1OQfnfnqS5ESr/tn0PraAm7kyLejlHjNwIlhkWXRPHwBU+kpvE96dqBJlrwuiZ7 iYDXGJq0Z9UxvCaxtckP2NEd7EkM/rOpfDBTGp6+hhRRNHx9cUOTriOqGYp2xuzFoactXAi0w clvJJVZ103Kj+hAnzTToEp7EK3sJQJtHL/0izWFqNohRcB26ogeUVQUnzPRwPLzh262uoGJzU /mQPEbOM4EYqh/e9VXYvJTPqNr7/AJt14s+lXF0ZvMKcvbsKlj31bzhKSlPS7n2dnnNAsJ9xQ tZ4eD0WTh9D5UrCOy7tRqakD6Cl/SbyOFO9MWoVdUshMK8DcgcLEct+wtwh9otzvZ2WsizsOb duXbsThAKSYIsbZIM+prTO7DdkxVzyWLWtpTV7HUzx4tGyt+DFGINARd9VZSnT2xyjLd1UE7R if5kfRbDCJ6Y4c0K2yZwsdNSkHUtTmurH71mHxVwJkABBgmObtEY1w3NfMINdMYcXyOw= X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2015 20:02:44 -0000 On 7/11/2015 10:53 AM, Dieter BSD wrote: >> When building a filesystem (FFS) on these 1/2/3/4TB external USB >> drives, any suggestions? E.g., changing block sizes? Cut into >> multiple slices to make fsck's job easier (as well as *recovery*)? > > If the average filesize will be large, use large block/frag sizes. > I use 64 KiB / 8 KiB. And reduce the number of inodes. I reduce > inodes as much as newfs allows and there are still way too many. Can you think of an algorithmic way to express this? I.e., you don't want blocks to get *too* large as you risk greater losses in "partial fragments", etc. Likewise, you don't want to run out of inodes. I'm trying to come up with a relatively simple set of guidelines so folks can look at volume size, expected number of files, typical file sizes and "risk" of loss/cost of recovery (when the medium *is* unceremoniously dismounted) > These 2 changes waste less space to filesystem overhead, and fsck > runs a lot faster. If the disk has 4 KiB sectors, have the frag > size be at least 4 KiB, and have partitions start at a multiple > of 4 KiB. In light of your below comment, that may be problematic. > I recommend soft updates. When I started using su I magically stopped > losing files. I can't say I've ever "lost files" on a *BSD box (though there was an issue on a real early FBSD release with one of the SCSI drives that I was using -- mounting it would promptly scramble the superblock?) I am concerned with the fact that users can so easily/carelessly "unplug" a USB device without the proper incantations beforehand. of course, *their* mistake is seen as a "product design flaw"! :-/ > Have newfs label the filesystem(s). Then use /dev/ufs/mylabel > in /etc/fstab and the filesystems will get mounted on the right > places regardless of the disk being da0 or da1 or whatever. Ha! That's worth knowing! I was simply going to actively probe volume labels and add a second level of indirection to the mounts as a "fixup". > Swap can be labeled with glabel(8). > > Multiple slices? Consider the size of whatever media you'll > be using for backups. Consider making a read-only partition, > as it will not require fscking after a crash. Consider gpt. > If you decide on one big filesystem you don't need any partitioning, > just newfs the raw drive. I don't think backups are an issue. When was the last time you "backed up" your TiVo? STB? etc. The trick is to avoid the *need* for a backup -- despite the users' tendencies to forget to Do The Right Thing. The "demo app" that I'm working on is a sort of (low performance) NAS built on SBC's and external drives. So, the "backup" is already an inherent part of the design. > USB specific stuff: There is an off by 1 sector problem, which will > bite you if you switch a drive between using the sata-usb bridge > and connecting the drive directly to a sata controller. I had to Ouch! I've not seen that with PATA-USB bridges. OTOH, I tend not to pull a drive *from* an external enclosure but, rather, rely on the external enclosures to provide portability. E.g., easier to move 500G of files from machineA to machineB by physically moving the volume containing them! [Imagine machineA dies or is taken out of service] Having said that, though, I have an "internal" volume with FFS from a NetBSD box that I will be examining with an external USB bridge (I dislike having to open machine cases as access is usually not very convenient, here). I will note if I encounter any problems in doing so. > do a fair bit of hacking on the kernel to deal with this. (No, > I don't have a nice clean patch to offer, sorry. Besides, I just > plastered over the problem rather than hunting down and fixing the > root cause, which is what really needs to be done.) I don't recall > if it is off by one physical sector, or one logical sector. On a > 4 KiB drive it *might* mess up having things be on 4K multiples. > Word is that the disk manufacturers have changed the interface > between the bridge and the drive on recent external drives, making it > difficult-to-impossible to use the drive connected directly to a sata > controller. :-( > Some people believe that drives are binned, and the less-wonderful > drives go into the externals. This could explain externals being > less expensive, despite having more stuff (enclosure, bridge, > wall-wart, cables). Given how high the failure rate of internal > drives is, I hate to think how bad the externals must be. I've actually only had bad experiences with laptop drives. And, typically in cases where I was trying to use the laptop as a sort of desktop -- on 24/7/365 and the poor little drives just get tired! :> My external USB drives have typically had their original drives removed prior to my acquiring them (rescues). So, I can stuff a drive of my choosing inside. The "demo app" will try to use the large multi-TB drives of which I have little long-term experience. OTOH, the usage model is "fire it up, pull off whichever files you need, then spin everything down"... until the next time you might need to retrieve an ISO (a week later?) > I have yet to see a [ps]ata-to-usb bridge that allows turning > the disk's write cache off. Having the write cache turned off > is kinda important. If anyone knows of a bridge that does allow > turning the write cache off I'd love to hear about it, > >> Of course, anything external means the user could insert/remove it at >> will -- which complicates how the system will deal with the resource. > > If the drive disappears with filesystem(s) mounted. the kernel might > very well panic. There was a discussion of this problem recently. > I thought that FUSE was suggested as a possible solution, but I > can't find the discussion. This problem is not limited to users > disconnecting usb drives without unmounting them. The problem > happens all by itself with internal drives, as the drive, port > multiplier, controller, or device driver decides to go out to lunch, > and the kernel panics. This happens *far* too often, and *kills* > reliability. We really need a solution for this. I think its hard to back-port these sorts of things. Much easier to consider the possibility of failure when initially designing the system, interfaces, etc. Of course, for systems that are reasonably open-ended, predicting the future is a skill better applied to financial futures!! :> From owner-freebsd-hackers@freebsd.org Sun Jul 12 20:08:59 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D106799B9DF for ; Sun, 12 Jul 2015 20:08:59 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C29A1F38 for ; Sun, 12 Jul 2015 20:08:59 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from [192.168.1.115] ([67.212.197.98]) by mail.gmx.com (mrgmx102) with ESMTPSA (Nemesis) id 0LgeFd-1Yc4fU24Vp-00nvp8 for ; Sun, 12 Jul 2015 22:08:57 +0200 Message-ID: <55A2C946.9010708@gmx.com> Date: Sun, 12 Jul 2015 13:08:38 -0700 From: Don whY User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: Xorg for client-only support References: <559EDB56.70808@gmx.com> <2863155.VvqRCPmh7x@desk8.phess.net> <55A1DD4F.1050507@gmx.com> <4777726.x7gaKDivgh@desk8.phess.net> In-Reply-To: <4777726.x7gaKDivgh@desk8.phess.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:/6neFrmR+CiOLWgAGYaNi2clUHQv4YOlNLyGigtISXdP4L0rPuc g1RS2pZ9r9dTsr2zDLSed4nT/mMLHClMfORgk7Bc26mfSiFxYvetU+2R12eXUs8V+VoriDz /t+3svXT9LtjRkaqE+US7jiZ29TL9sBwWNunwUI3D7Si37CrSuYm4sKKmRzJcGLCel6I3HR MRbTg5HJIO8pFkbA08S/g== X-UI-Out-Filterresults: notjunk:1;V01:K0:+4yjcZyNNN8=:d3EJlBhbuq8+3N5xmcPlSb rxR8aGMtWJlWqd4UFfzsMxNGOqjRvlW8nSvjcq5+lIJ3buklU1oSJpAxAtklM80w490vWBmJp ECA5qgHTfapbY7JGiiZPMGHg7EiTHuT4gV6lHIZzMV4tBj/R2vaqZb78zRtC8ucbt3kDSuk63 rkhA1krJneGsiWOx+ZTNXnlRQ6+FBHlIQVqhCeBANTJsgEIw0QYcqqkuE4xFJFE4iC79Y7KKQ yYoPXgutg2KRgf5tCtJEA3w207iyBhstr0jFARq7X0qm5RH8MRx8U7DYtzcgRNP8r+NOCL7Q8 JtTRnj4wtowR+frGXtEVHhfd3Lza97IydZDr5e0clygGpxmtEHX6+ZJA17tT2Zv1oTFJcZ37w UhFeq+Sp1Y/Dt99eb2mog2ZGBK/8PepuF2h9HlbgsNXGAJFF+mXbCLiNsxpLCtWlTA+Vzh5Ke Jsd0g/8kDV6mn8JIbS8b4pC1hVtD4fi85Y3UFR43n43Uuu9sfMudIUOchfEma5XizT7vixKlX pR/n7m0kmh0osEmpQGM2n2a1FfxswdFKwKWm7JsK0hZorrc8tAfPHrbrHOoSNpWVjStDLGxMD AZxnNNaEQSzLLaeCCifHA7e3Q7tpPjTf30nCkZybO2sbIN9Dq52Vdz+GzVKaIq9UKw8q1rgGW 2vUJylqztCOza44JgRQStoemrmXOX4OzJ8hZ/GHTdBvZmDORVHy1OjeXIjgl+7AuU9Z8= X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2015 20:08:59 -0000 On 7/11/2015 11:21 PM, Patrick Hess wrote: > Don whY wrote: >> On 7/10/2015 9:41 AM, Patrick Hess wrote: >>> Don whY wrote: >>>> For *headless* devices (i.e., no point in having a real *server*!), >>>> are there significant portions of xorg that I can omit? Or, is it >>>> easier to build and install it all and remove the unnecessary cruft, >>>> later? >>> >>> Since you don't need the server portions, I don't really see a reason >>> why you would want to worry about installing any X.org ports by hand. >> >> Not sure what you mean "by hand"; if "build from scratch", that's >> just the way I've done things for the past 20+ years... > > Now I'm not so sure any more what you mean by "build from scratch". > I was assuming that you were using the FreeBSD ports system, in which > case I wouldn't bother to explicitly install any X libraries, like so: > > make -C /usr/ports/x11/libX11 install && > make -C /usr/ports/x11/libXext install && > make -C /usr/ports/x11-toolkits/libXt install && > [...] Yes. Though if any of the above had dependencies, I would first "manually" make those. I.e., I don't like relying on too much magic as too many details get hidden that I might want to examine and opt for a different approach. E.g., something starts to drag in a new version of perl and I give serious thought to whether I really *want* that thing! > Instead, I would just install the actual applications I wanted to run > on that machine and have the above libraries be installed by the ports > system automatically should one of those applications need them. > > Of course, if you were building from source without using the ports system, > you'd have to resolve all of these dependency issues yourself. I don't see > a compelling reason why you would want to take that route, though. > >>> Just installing the applications you want to use should automatically >>> pull in the X.org dependencies that are actually required by these >>> applications. >> >> Are the client dependencies that fine-grained? I.e., they don't just drag >> The Kitchen Sink in? > > I'd say that most ports are pretty good at specifying only the minimum > set of dependencies that are actually required by the port. > > Just off the top of my head, the one counter example that comes to my > mind is x11/lumina, which has a dependency on x11/xorg, a metaport for > the *entire* X.org distribution. I didn't have the time to take a closer > look at that port just yet, but that feels like overkill to me. It's only natural to think "Oh, this relies on X..." and casually (unfortunately) drag in more dependencies than may be required. Likewise, default configuration choices may drag in other ports that *I* might not be interested in. >>> This will prevent any unnecessary parts, like server >>> components and video drivers, from being installed on your system >>> in the first place. >> >> I was hoping for a port akin to "xorg-clients". But, i should be able >> to hack one together based on your above comment. > > x11/xorg-libraries seems to bundle a fair amount of client libraries. > That being said, I wouldn't bother about it either. Should one of the > applications you are to install require x11/xorg-libraries, the ports > system will automatically install it for you anyway. I'll have to start looking at each client in detail and see what the "cost" (dependencies) of each will be. Some may prove not to be worth the extra effort/resources. From owner-freebsd-hackers@freebsd.org Sun Jul 12 23:42:41 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5996599BD0A for ; Sun, 12 Jul 2015 23:42:41 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from shell1.rawbw.com (shell1.rawbw.com [198.144.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3B78F1885 for ; Sun, 12 Jul 2015 23:42:41 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from yuri.doctorlan.com (c-50-184-63-128.hsd1.ca.comcast.net [50.184.63.128]) (authenticated bits=0) by shell1.rawbw.com (8.15.1/8.15.1) with ESMTPSA id t6CNgXPZ026874 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 12 Jul 2015 16:42:34 -0700 (PDT) (envelope-from yuri@rawbw.com) X-Authentication-Warning: shell1.rawbw.com: Host c-50-184-63-128.hsd1.ca.comcast.net [50.184.63.128] claimed to be yuri.doctorlan.com Message-ID: <55A2FB68.3070006@rawbw.com> Date: Sun, 12 Jul 2015 16:42:32 -0700 From: Yuri User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: freebsd-hackers@FreeBSD.org Subject: Does /dev/random in virtual guests provide good random data? Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jul 2015 23:42:41 -0000 'cat /dev/random' in Linux VM (tried Ubuntu and Arch) is extremely slow, supposedly because VM runs out of entropy. This cat sometimes stops for minutes, and usually produces very few bytes per minute. Randomly clicking on the window helps speed it up a bit. Same in FreeBSD VM produces steady ~28MB/s stream. Does FreeBSD VM do something special for entropy, or the resulting stream actually lacks entropy, or maybe Linux does something wrong? I ran VMs on the same FreeBSD VirtualBox host. Somewhat relevant link: https://stackoverflow.com/questions/26021181/not-enough-entropy-to-support-dev-random-in-docker-containers-running-in-boot2d Yuri From owner-freebsd-hackers@freebsd.org Mon Jul 13 01:15:08 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB77C999FE3 for ; Mon, 13 Jul 2015 01:15:08 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (kientzle.com [142.254.26.11]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 53AB51EFC for ; Mon, 13 Jul 2015 01:15:07 +0000 (UTC) (envelope-from tim@kientzle.com) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id t6D1FELx086137; Mon, 13 Jul 2015 01:15:14 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.100] (192.168.1.101 [192.168.1.101]) by kientzle.com with SMTP id b4i6i3j37rx7r7vqyew9ey34rw; Mon, 13 Jul 2015 01:15:13 +0000 (UTC) (envelope-from tim@kientzle.com) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Does /dev/random in virtual guests provide good random data? From: Tim Kientzle In-Reply-To: <55A2FB68.3070006@rawbw.com> Date: Sun, 12 Jul 2015 18:14:59 -0700 Cc: Freebsd hackers list Content-Transfer-Encoding: quoted-printable Message-Id: References: <55A2FB68.3070006@rawbw.com> To: Yuri X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 01:15:09 -0000 > On Jul 12, 2015, at 4:42 PM, Yuri wrote: >=20 > 'cat /dev/random' in Linux VM (tried Ubuntu and Arch) is extremely = slow, supposedly because VM runs out of entropy. This cat sometimes = stops for minutes, and usually produces very few bytes per minute. = Randomly clicking on the window helps speed it up a bit. >=20 > Same in FreeBSD VM produces steady ~28MB/s stream. >=20 > Does FreeBSD VM do something special for entropy, or the resulting = stream actually lacks entropy, or maybe Linux does something wrong? Here=E2=80=99s a good discussion of the difference between /dev/random = and /dev/urandom on Linux: http://www.2uo.de/myths-about-urandom/ In particular, it has this interesting comment: FreeBSD does the right thing: they don't have the distinction between /dev/random and /dev/urandom, both are the same device. At startup /dev/random blocks once until enough starting entropy has been gathered. Then it won't block ever again. From owner-freebsd-hackers@freebsd.org Mon Jul 13 04:21:46 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B13E399BE3D for ; Mon, 13 Jul 2015 04:21:46 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 93D5B1EEF; Mon, 13 Jul 2015 04:21:45 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-240-47.lns20.per4.internode.on.net [121.45.240.47]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t6D4BhPi068399 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sun, 12 Jul 2015 21:11:47 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <55A33A8A.6070505@freebsd.org> Date: Mon, 13 Jul 2015 12:11:54 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd , Kirk McKusick Subject: Re: Fix MNAMELEN or reimplement struct statfs References: <20140415233133.GA14686@ambrisko.com> <5452600C.5030003@omnilan.de> <20141101154004.GA40398@ambrisko.com> <559FD426.3000108@omnilan.de> <20150710154654.GA71708@ambrisko.com> In-Reply-To: <20150710154654.GA71708@ambrisko.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 04:21:46 -0000 this hits us occasionally too Kirk, you are mentioned here as having a finger in this pie. do we have a suggested way forward? On 7/10/15 11:46 PM, Doug Ambrisko wrote: > On Fri, Jul 10, 2015 at 04:18:14PM +0200, Harald Schmalzbauer wrote: > | > | > | Hello, > | > | > | > | first sorry for the missing thread references in the header, I'm not > | > | subscribed to hackers@. > | > | > | > | bdrewery@ pointed me to this discussion in response to my question to > | > | stable@ > | > | (http://lists.freebsd.org/pipermail/freebsd-fs/2014-August/019949.html) > | > | > | > | Last promising post I found: > | > | > | > | > |/ > I have a new patch at: > | > | > /|/ > http://people.freebsd.org/~ambrisko/mount_bigger_2.patch > | > | > /|/ > that I tested against head. This should be pretty close to commiting > | > | > /|/ > unless people find some issues with it. > | > | > /|/ > | > | > /|/ In sys/kern/vfs_mount.c: > | > | > /|/ + mp->mnt_path = malloc(strlen(fspath), M_MOUNT, M_WAITOK); > | > | > /|/ + strlcpy((char *)mp->mnt_path, fspath, strlen(fspath)); > | > | > /|/ > | > | > /|/ This always strips the last byte off the fspath. > | > | > /|/ > | > | > /|/ I like that this only touches the kernel, so it does not break anything > | > | > /|/ regarding mount/umount of filesystems with short paths, including > | > | > /|/ (NFS) filesystems that do not respond. > | > | > /|/ > | > | > /|/ The patch does not enlarge f_mntfromname which may be a problem for > | > | > /|/ nullfs. It is certainly a step forwards for poudriere but [ENAMETOOLONG] > | > | > /|/ errors could still occur in more extreme situations. > | > | > / > | > | > Good point on nullfs. I'll look at fixing that. To do that I'm > | > | > changing mnt_path to mnt_topath so then I can have a mnt_frompath. > | > | > I'll add nullfs to my test cases. I'll need to run through the uses > | > | > of f_mntfromname. It was pretty easy with f_mntonname since it was > | > | > only allocated in one place just used a bunch of other place. I assume > | > | > that mount root would be short. > | > | > | > | Thanks a lot so far for working hard on that problem! > | > | Is there anything newer than "mount_bigger_2.patch", which considers > | > | potential nullfs problems? > | > | I'm heavily using nullfs (without poudriere), but I'd give it a try on > | > | my rather lightly loaded local 10.1 storage box ??? almost all snapshots > | > | are useless, can't access them in case of the case; which happens > | > | frequently :-( > | > | Would I have to expect any nullfs regressions with the april > | > | (mount_bigger_2) patch?? > | > | Bez?glich Doug Ambrisko's Nachricht vom 01.11.2014 16:40 (localtime): > | > I should be able to resume working on this since things are starting to > | > slow down. It shouldn't be much more work to get it finished off to > | > put up for review. > | > | Hello Doug, > | > | I've been using your mount_bigger_2.path for some months without > | problems, but haven't done any kind of stress test. > | It just saves my soul in case I have to recover files from > | (zfs-)snapshots from time to time :-) > | > | Since releng/10.2 is to be created soon, I'm testing RELENG_10 on some > | of my production machines, Therefore I cosmetically altered your > | patchset to make it work with -stable: > | ftp://ftp.omnilan.de/pub/FreeBSD/OmniLAN/misc/local-patches/RELENG_10/mount_bigger_2_1.patch > | > | Have you made any progress in this area, e.g. is there anything > | different I can test, which might help in any way? > > It's been working fine for me. Glad to hear it is working good with > ZFS. Kirk asked me not to continue with this since it would make > the 64 bit inode work harder and that they were going to bump up > the max of the mount point. He also mentioned that it couldn't be > merged back since it changes the kernel API. So I'm not sure > where that leaves us for now except that this works for us. I use > it a lot at work since we mount things in chroot's of which the > path is pretty long especially when we mount stuff in a chroot of > chroot for our build process. It's better then my first attempt > since the user space ABI didn't change. So it mostly just works > except for listing the mount points get truncated. > > Thanks, > > Doug A. > > Thanks, > > Doug A. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Mon Jul 13 08:26:39 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDEC099BA94 for ; Mon, 13 Jul 2015 08:26:39 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from shell1.rawbw.com (shell1.rawbw.com [198.144.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id D7C381479 for ; Mon, 13 Jul 2015 08:26:39 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from yuri.doctorlan.com (c-50-184-63-128.hsd1.ca.comcast.net [50.184.63.128]) (authenticated bits=0) by shell1.rawbw.com (8.15.1/8.15.1) with ESMTPSA id t6D8Qake057908 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 13 Jul 2015 01:26:37 -0700 (PDT) (envelope-from yuri@rawbw.com) X-Authentication-Warning: shell1.rawbw.com: Host c-50-184-63-128.hsd1.ca.comcast.net [50.184.63.128] claimed to be yuri.doctorlan.com Message-ID: <55A3763B.7010303@rawbw.com> Date: Mon, 13 Jul 2015 01:26:35 -0700 From: Yuri User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Freebsd hackers list Subject: Re: Does /dev/random in virtual guests provide good random data? References: <55A2FB68.3070006@rawbw.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 08:26:40 -0000 On 07/12/2015 18:14, Tim Kientzle wrote: > http://www.2uo.de/myths-about-urandom/ > > In particular, it has this interesting comment: > > FreeBSD does the right thing: they don't have the distinction There are two approaches in random stream generation. One is to have the sufficient random seed, and keep generating the following pseudo-random numbers only from this seed. The second approach is to also continuously feed the stream from some external source of entropy. The fact that the long running linux VM still blocks on /dev/random indicates that linux tries to collect more entropy on the go, following the latter approach (intuitively I would also agree this is better for randomness). So it isn't clear why FreeBSD random stream would be of the same quality, if it doesn't collect entropy on the go. Because both Linux and BSD have exactly the same entropy sources in VM. Yuri From owner-freebsd-hackers@freebsd.org Mon Jul 13 09:13:43 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74DD33B59 for ; Mon, 13 Jul 2015 09:13:43 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-qk0-x231.google.com (mail-qk0-x231.google.com [IPv6:2607:f8b0:400d:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 339F91ABD; Mon, 13 Jul 2015 09:13:43 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by qkdv3 with SMTP id v3so54573850qkd.3; Mon, 13 Jul 2015 02:13:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7IEA//zyfbMMipD9/BtZaDoyiZ8uJYRcURf4936jFQ4=; b=IoSDeGTWRkt/uoMuOtbQHAtn2T9UKpG5DPVQ45b8qu4uY+jRFRyBLv1YPV9szQ4Gb4 bC1McEVce/HlIROBB+iSh8HFQ7cunt8UUaS1RUwMzZaImWyFp/dRMyMHbg01LStuCc5Q B6lrpGlMhz0uXFGWXJ5tNdr1cGUzupvCA5rQvAee5G4Jfk6JEOcpqdTQWzAVUKrix7yl iTD7lBQl43MwH9TJ0jxoqi7WQP4AAZzJFakbA1Mne350+BrUr/Q8eArr2LsmvK9Aw0bV zH7mduzoiDMZUzvjXyx0e2hI9AdaTtVXEn5uUUuBpMoJWwYvANWNbjevYbO/+qOk7hKQ Ej7Q== MIME-Version: 1.0 X-Received: by 10.140.151.203 with SMTP id 194mr30567742qhx.80.1436778822257; Mon, 13 Jul 2015 02:13:42 -0700 (PDT) Received: by 10.140.98.73 with HTTP; Mon, 13 Jul 2015 02:13:42 -0700 (PDT) In-Reply-To: <55A33A8A.6070505@freebsd.org> References: <20140415233133.GA14686@ambrisko.com> <5452600C.5030003@omnilan.de> <20141101154004.GA40398@ambrisko.com> <559FD426.3000108@omnilan.de> <20150710154654.GA71708@ambrisko.com> <55A33A8A.6070505@freebsd.org> Date: Mon, 13 Jul 2015 02:13:42 -0700 Message-ID: Subject: Re: Fix MNAMELEN or reimplement struct statfs From: NGie Cooper To: Julian Elischer Cc: freebsd , Kirk McKusick Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 09:13:43 -0000 On Sun, Jul 12, 2015 at 9:11 PM, Julian Elischer wrote: > this hits us occasionally too > Kirk, you are mentioned here as having a finger in this pie. > > do we have a suggested way forward? Alternatively, is the ino64_t work slated for CURRENT anytime soon (if it was before 11.0-RELEASE that would be best probably...). Thanks! From owner-freebsd-hackers@freebsd.org Mon Jul 13 13:46:41 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84BD999BA82 for ; Mon, 13 Jul 2015 13:46:41 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 14599E6 for ; Mon, 13 Jul 2015 13:46:40 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: by wibud3 with SMTP id ud3so30269938wib.1 for ; Mon, 13 Jul 2015 06:46:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=date:from:to:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; bh=QR4tZ72bX64cZEiJKqUzptFiHjB5hOMyT2u9VP+XBSs=; b=bidIy60pKoITlgxS50isaU4s2pOsmjHI1hatEsiWQyLgiamWLoCgsa19D6b4z1Vv4W aKPdPpH98WScrVr9XiSDr15Nel9jwFD9aws2MVvhMBZsqdzQYtRVAwW/Q7bLyfh2zHkD /7NYdFZSSN6rh1nnsxdSI5AlR0WovUdmzgD5fW12n3QOe3uht7ADW1rHIu1w2DUL4K/w ak21K+GjyT7kLtor89R3subcu9YSTOVqmOg70G4XlIMA6b9KxEHKiu68++sDlcf7hSne hQgx4AV/AKb2QQnEuFQz5XBLytCu3FEPxurITxvT3wpJkldu5OeYHi/tbfZDqRF6PvWH Mfxg== X-Received: by 10.180.182.33 with SMTP id eb1mr22583484wic.8.1436795198990; Mon, 13 Jul 2015 06:46:38 -0700 (PDT) Received: from gumby.homeunix.com (5ec1f6f9.skybroadband.com. [94.193.246.249]) by smtp.gmail.com with ESMTPSA id ez4sm14735282wid.14.2015.07.13.06.46.33 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 Jul 2015 06:46:38 -0700 (PDT) Date: Mon, 13 Jul 2015 14:46:30 +0100 From: RW To: freebsd-hackers@freebsd.org Subject: Re: Does /dev/random in virtual guests provide good random data? Message-ID: <20150713144630.32cd851a@gumby.homeunix.com> In-Reply-To: <55A3763B.7010303@rawbw.com> References: <55A2FB68.3070006@rawbw.com> <55A3763B.7010303@rawbw.com> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; amd64-portbld-freebsd10.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 13:46:41 -0000 On Mon, 13 Jul 2015 01:26:35 -0700 Yuri wrote: > On 07/12/2015 18:14, Tim Kientzle wrote: > > http://www.2uo.de/myths-about-urandom/ > > > > In particular, it has this interesting comment: > > > > FreeBSD does the right thing: they don't have the distinction > > There are two approaches in random stream generation. One is to have > the sufficient random seed, and keep generating the following > pseudo-random numbers only from this seed. The second approach is to > also continuously feed the stream from some external source of > entropy. The real point of adding entropy after a PRNG is secure is to recover from compromise and break state extension attacks. FreeBSD does that, it just doesn't block. Actually no competently designed PRNG feeds in entropy continuously because it's more secure to add it in batches. Linux uses a single batch size (IIRC 128 bits) but FreeBSD reseeds on two separate cycles, a fast one designed to secure (or resecure) the device as quickly as possible, and an ultra-conservative slow one. FreeBSD also has the advantage of not splitting the entropy between two separate devices. > The fact that the long running linux VM still blocks on /dev/random > indicates that linux tries to collect more entropy on the go, > following the latter approach (intuitively I would also agree this is > better for randomness). > > So it isn't clear why FreeBSD random stream would be of the same > quality, if it doesn't collect entropy on the go. Because both Linux > and BSD have exactly the same entropy sources in VM. FreeBSD uses Yarrow, which was designed by Bruce Schneier, a professional cryptographer who created the Blowfish cipher, the AES candidate Twofish and PGP. Linux's /dev/random was designed by programmers; actually a lot of them, its greatest problem is that it's a mess of patches from amateurs. From owner-freebsd-hackers@freebsd.org Mon Jul 13 15:53:56 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4536599BDDF for ; Mon, 13 Jul 2015 15:53:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id F0D161DA8; Mon, 13 Jul 2015 15:53:54 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id SAA24949; Mon, 13 Jul 2015 18:51:31 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1ZEg0x-000Fh7-CD; Mon, 13 Jul 2015 18:51:31 +0300 Message-ID: <55A3DE4A.4050102@FreeBSD.org> Date: Mon, 13 Jul 2015 18:50:34 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: NGie Cooper , Kirk McKusick CC: Julian Elischer , freebsd Subject: Re: Fix MNAMELEN or reimplement struct statfs References: <20140415233133.GA14686@ambrisko.com> <5452600C.5030003@omnilan.de> <20141101154004.GA40398@ambrisko.com> <559FD426.3000108@omnilan.de> <20150710154654.GA71708@ambrisko.com> <55A33A8A.6070505@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2015 15:53:56 -0000 On 13/07/2015 12:13, NGie Cooper wrote: > On Sun, Jul 12, 2015 at 9:11 PM, Julian Elischer wrote: >> this hits us occasionally too >> Kirk, you are mentioned here as having a finger in this pie. >> >> do we have a suggested way forward? > > Alternatively, is the ino64_t work slated for CURRENT anytime soon (if > it was before 11.0-RELEASE that would be best probably...). > Thanks! I also would love to see MNAMELEN work in head as soon as possible. I am personally not that much interested in ino64_t, but it would be great as well. -- Andriy Gapon From owner-freebsd-hackers@freebsd.org Tue Jul 14 01:33:49 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 512069989B6 for ; Tue, 14 Jul 2015 01:33:49 +0000 (UTC) (envelope-from marcus@odin.blazingdot.com) Received: from odin.blazingdot.com (odin.blazingdot.com [204.109.60.170]) by mx1.freebsd.org (Postfix) with ESMTP id 399291633 for ; Tue, 14 Jul 2015 01:33:48 +0000 (UTC) (envelope-from marcus@odin.blazingdot.com) Received: by odin.blazingdot.com (Postfix, from userid 1001) id C76351320ED; Mon, 13 Jul 2015 18:33:42 -0700 (PDT) Date: Mon, 13 Jul 2015 18:33:42 -0700 From: Marcus Reid To: RW Cc: freebsd-hackers@freebsd.org Subject: Re: Does /dev/random in virtual guests provide good random data? Message-ID: <20150714013342.GA79791@blazingdot.com> References: <55A2FB68.3070006@rawbw.com> <55A3763B.7010303@rawbw.com> <20150713144630.32cd851a@gumby.homeunix.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150713144630.32cd851a@gumby.homeunix.com> X-Coffee-Level: nearly-fatal User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 01:33:49 -0000 On Mon, Jul 13, 2015 at 02:46:30PM +0100, RW via freebsd-hackers wrote: > FreeBSD uses Yarrow As of recently, -CURRENT uses Fortuna by default, which is the successor to Yarrow. It was also devised by Bruce Schneier (with Neils Ferguson). > , which was designed by Bruce Schneier, a professional cryptographer > who created the Blowfish cipher, the AES candidate Twofish and PGP. PGP was created by Phil Zimmermann, not Schneier. Marcus > Linux's /dev/random was designed by programmers; actually a lot of > them, its greatest problem is that it's a mess of patches from > amateurs. From owner-freebsd-hackers@freebsd.org Tue Jul 14 04:42:37 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53F4099CFF8; Tue, 14 Jul 2015 04:42:37 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "wonkity.com", Issuer "wonkity.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E76E22FDE; Tue, 14 Jul 2015 04:42:36 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.9/8.14.9) with ESMTP id t6E4gYKu055263 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 13 Jul 2015 22:42:34 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.9/8.14.9/Submit) with ESMTP id t6E4gYPc055260; Mon, 13 Jul 2015 22:42:34 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Mon, 13 Jul 2015 22:42:34 -0600 (MDT) From: Warren Block To: Benjamin Kaduk cc: wblock@FreeBSD.org, freebsd-hackers@freebsd.org, freebsd-current@freebsd.org Subject: Re: Call for FreeBSD 2015Q2 (April-June) Status Reports In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Mon, 13 Jul 2015 22:42:34 -0600 (MDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 04:42:37 -0000 Have you submitted your quarterly report yet? People need to know about the cool things being done with, or to, or by FreeBSD. In an effort that will surely never come back to haunt us all, I have committed all the reports I saw in monthly@ to the upcoming status report. However, I am also easily distracted by shiny objects and might have missed some. If you have not yet received a passive-aggressive "thanks" mail, please make sure your report is included here: https://svnweb.freebsd.org/doc/head/en_US.ISO8859-1/htdocs/news/status/report-2015-04-2015-06.xml?view=markup If your report is missing or not present, please resubmit to bjk@FreeBSD.org, monthly@FreeBSD.org, and cc me at wblock@FreeBSD.org. The deadline is July 14 (yes, today), but of course some of us would rather wait than miss interesting reports. Or dull ones, to be honest. But there is an interesting report lurking inside even the dull ones. Particularly if some words are replaced with others. Assistance in polishing a bare list of feats into a truly Herculean accomplishment can be provided on a limited-time, first-come first-serve, contents may have settled during shipment basis. Grammer can be goodered and Englished up, too. On Wed, 8 Jul 2015, Benjamin Kaduk wrote: > Dear FreeBSD Community, > > Only one week remains before the deadline to submit entries for the next > FreeBSD Quarterly Status update -- the deadline is July 14, 2015, for work > done in April through June. > > Status report submissions do not have to be very long. They may be > about anything happening in the FreeBSD project and community, and > provide a great way to inform FreeBSD users and developers about what > you're working on. Submission of reports is not restricted to > committers. Anyone doing anything interesting and FreeBSD-related > can -- and should -- write one! > > The preferred and easiest submission method is to use the XML > generator [1] with the results emailed to the status report team at > monthly at freebsd.org . There is also an XML template [2] which can be > filled out manually and attached if preferred. For the expected > content and style, please study our guidelines on how to write a good > status report [3]. You can also review previous issues [4][5] for > ideas on the style and format. > > We are looking forward to all of your 2015Q2 reports! > > Thanks, > Ben (on behalf of monthly@) > > > [1] http://www.freebsd.org/cgi/monthly.cgi > [2] http://www.freebsd.org/news/status/report-sample.xml > [3] http://www.freebsd.org/news/status/howto.html > [4] http://www.freebsd.org/news/status/report-2014-10-2014-12.html > [5] http://www.freebsd.org/news/status/report-2015-01-2015-03.html > > From owner-freebsd-hackers@freebsd.org Tue Jul 14 12:45:40 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06D1B999726 for ; Tue, 14 Jul 2015 12:45:40 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [IPv6:2001:418:3fd::f7]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B9D1BDED for ; Tue, 14 Jul 2015 12:45:39 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from [IPv6:2001:418:3fd::1f] (haymarket.m5p.com [IPv6:2001:418:3fd::1f]) by mailhost.m5p.com (8.14.5/8.14.5) with ESMTP id t6ECjVSs083379 for ; Tue, 14 Jul 2015 08:45:37 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: <55A5046B.4090000@m5p.com> Date: Tue, 14 Jul 2015 08:45:31 -0400 From: George Mitchell User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: Does /dev/random in virtual guests provide good random data? References: <55A2FB68.3070006@rawbw.com> <55A3763B.7010303@rawbw.com> <20150713144630.32cd851a@gumby.homeunix.com> <20150714013342.GA79791@blazingdot.com> In-Reply-To: <20150714013342.GA79791@blazingdot.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.73 X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (mailhost.m5p.com [IPv6:2001:418:3fd::f7]); Tue, 14 Jul 2015 08:45:37 -0400 (EDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 12:45:40 -0000 On 07/13/15 21:33, Marcus Reid wrote: > On Mon, Jul 13, 2015 at 02:46:30PM +0100, RW via freebsd-hackers wrote: >> FreeBSD uses Yarrow > > As of recently, -CURRENT uses Fortuna by default, which is the successor > to Yarrow. It was also devised by Bruce Schneier (with Neils Ferguson). > >> , which was designed by Bruce Schneier, a professional cryptographer >> who created the Blowfish cipher, the AES candidate Twofish and PGP. > > PGP was created by Phil Zimmermann, not Schneier. > > Marcus > >> Linux's /dev/random was designed by programmers; actually a lot of >> them, its greatest problem is that it's a mess of patches from >> amateurs. > [...] Donald Knuth's excellent books on the Art of Computer Programming give an example of the pitfalls of programmers designing random number generators at the beginning of Chapter 3, "Random Numbers," with "Algorithm K" devised by Knuth himself in his youth. It converged almost immediately. "The moral to this story," writes Knuth, "is that /random numbers should not be generated with a method chosen at random/." (c) 1969 Addison-Wesley -- George From owner-freebsd-hackers@freebsd.org Tue Jul 14 13:30:27 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0087D99B0D9 for ; Tue, 14 Jul 2015 13:30:27 +0000 (UTC) (envelope-from vsevolod@FreeBSD.org) Received: from mail.highsecure.ru (mail6.highsecure.ru [IPv6:2a01:4f8:190:43b5::99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B821737E for ; Tue, 14 Jul 2015 13:30:26 +0000 (UTC) (envelope-from vsevolod@FreeBSD.org) Received: from secret-bunker.localdomain (unknown [81.145.134.172]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: vsevolod@highsecure.ru) by mail.highsecure.ru (Postfix) with ESMTPSA id 5706D300129; Tue, 14 Jul 2015 15:30:16 +0200 (CEST) Received: from [127.0.0.1] (localhost [127.0.0.1]) by secret-bunker.localdomain (Postfix) with ESMTP id A32E2626D1E; Tue, 14 Jul 2015 14:30:17 +0100 (BST) Message-ID: <55A50EE9.1020900@FreeBSD.org> Date: Tue, 14 Jul 2015 14:30:17 +0100 From: Vsevolod Stakhov User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Yuri , Freebsd hackers list Subject: Re: Does /dev/random in virtual guests provide good random data? References: <55A2FB68.3070006@rawbw.com> <55A3763B.7010303@rawbw.com> In-Reply-To: <55A3763B.7010303@rawbw.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 13:30:27 -0000 On 13/07/2015 09:26, Yuri wrote: > On 07/12/2015 18:14, Tim Kientzle wrote: >> http://www.2uo.de/myths-about-urandom/ >> >> In particular, it has this interesting comment: >> >> FreeBSD does the right thing: they don't have the distinction > > There are two approaches in random stream generation. One is to have the > sufficient random seed, and keep generating the following pseudo-random > numbers only from this seed. The second approach is to also continuously > feed the stream from some external source of entropy. > > The fact that the long running linux VM still blocks on /dev/random > indicates that linux tries to collect more entropy on the go, following > the latter approach (intuitively I would also agree this is better for > randomness). > > So it isn't clear why FreeBSD random stream would be of the same > quality, if it doesn't collect entropy on the go. Because both Linux and > BSD have exactly the same entropy sources in VM. That's *not* the correct definition of how the modern PRNG work. In both Linux and FreeBSD there are a single or multiple pools of entropy that are seeded from many entropy sources (the algorithm of entropy distribution among pools can vary). These pools are used to seed generator which subsequently generates blocks of pseudo-random data. A cryptographic PRF (such as AES-CTR) is used for generator, hence, by definition, there are no *efficient* ways to distinguish its output from purely random. Moreover, since it is seeded with true random data, an attacker cannot predict the subsequent data without controlling *all* entropy sources in system. The key difference between FreeBSD and Linux is that in Linux, /dev/urandom *never* blocks which is bad: on boot, when there is no entropy gained, it is really a bad idea to generate something like SSH keys. On the contrary, FreeBSD /dev/urandom will *block* if there is no entropy in the pools. In conclusion, it is always safe to use both /dev/random and /dev/urandom in FreeBSD and it is safe to use /dev/urandom in Linux almost all the time but not before the initial entropy harvesting that occurs on boot (but you can check the amount of entropy in the pools prior to generating something sensible, e.g. keys generation or DSA/ECDSA). There are some improvements in the Fortuna algorithm that's going to replace Yarrow in the HEAD. They are negligible for the most systems but are quite useful for low entropy systems (but there are no practical attacks on yarrow as well AFAIK). -- Vsevolod Stakhov From owner-freebsd-hackers@freebsd.org Tue Jul 14 13:35:21 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF26099B3C3 for ; Tue, 14 Jul 2015 13:35:21 +0000 (UTC) (envelope-from vsevolod@FreeBSD.org) Received: from mail.highsecure.ru (l.highsecure.ru [5.9.155.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 82362B7F for ; Tue, 14 Jul 2015 13:35:21 +0000 (UTC) (envelope-from vsevolod@FreeBSD.org) Received: from secret-bunker.localdomain (unknown [81.145.134.172]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: vsevolod@highsecure.ru) by mail.highsecure.ru (Postfix) with ESMTPSA id B8F1A300129; Tue, 14 Jul 2015 15:35:18 +0200 (CEST) Received: from [127.0.0.1] (localhost [127.0.0.1]) by secret-bunker.localdomain (Postfix) with ESMTP id 0C5B2626D6A; Tue, 14 Jul 2015 14:35:20 +0100 (BST) Message-ID: <55A51017.9080202@FreeBSD.org> Date: Tue, 14 Jul 2015 14:35:19 +0100 From: Vsevolod Stakhov User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Yuri , Freebsd hackers list Subject: Re: Does /dev/random in virtual guests provide good random data? References: <55A2FB68.3070006@rawbw.com> <55A3763B.7010303@rawbw.com> <55A50EE9.1020900@FreeBSD.org> In-Reply-To: <55A50EE9.1020900@FreeBSD.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 13:35:21 -0000 On 14/07/2015 14:30, Vsevolod Stakhov wrote: > On 13/07/2015 09:26, Yuri wrote: >> On 07/12/2015 18:14, Tim Kientzle wrote: >>> http://www.2uo.de/myths-about-urandom/ >>> >>> In particular, it has this interesting comment: >>> >>> FreeBSD does the right thing: they don't have the distinction >> >> There are two approaches in random stream generation. One is to have the >> sufficient random seed, and keep generating the following pseudo-random >> numbers only from this seed. The second approach is to also continuously >> feed the stream from some external source of entropy. >> >> The fact that the long running linux VM still blocks on /dev/random >> indicates that linux tries to collect more entropy on the go, following >> the latter approach (intuitively I would also agree this is better for >> randomness). >> >> So it isn't clear why FreeBSD random stream would be of the same >> quality, if it doesn't collect entropy on the go. Because both Linux and >> BSD have exactly the same entropy sources in VM. > > That's *not* the correct definition of how the modern PRNG work. And I forgot to mention that in Linux, both /dev/random and /dev/urandom are using pseudo-random generator seeded by the entropy pool(s). So you would never ever access these pools directly. The key difference is that /dev/random blocks unless there is 'enough' entropy in those pools. But it makes a system even *less* secure if an attacker can force you to use /dev/random, as at least it would give her information about the amount of entropy available in your system which is quite dangerous for Yarrow (but not for Fortuna). -- Vsevolod Stakhov From owner-freebsd-hackers@freebsd.org Tue Jul 14 20:32:49 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A73179A1262 for ; Tue, 14 Jul 2015 20:32:49 +0000 (UTC) (envelope-from rdarbha@juniper.net) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bn0109.outbound.protection.outlook.com [157.56.110.109]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D92F1EFE for ; Tue, 14 Jul 2015 20:32:49 +0000 (UTC) (envelope-from rdarbha@juniper.net) Received: from DM2PR0501MB1150.namprd05.prod.outlook.com (10.160.245.152) by DM2PR0501MB1151.namprd05.prod.outlook.com (10.160.245.153) with Microsoft SMTP Server (TLS) id 15.1.213.14; Tue, 14 Jul 2015 19:59:06 +0000 Received: from DM2PR0501MB1150.namprd05.prod.outlook.com ([10.160.245.152]) by DM2PR0501MB1150.namprd05.prod.outlook.com ([10.160.245.152]) with mapi id 15.01.0213.000; Tue, 14 Jul 2015 19:59:06 +0000 From: Raviprakash Darbha To: freebsd-hackers , "freebsd-hackers@freebsd.org" Subject: vm.kmem_size setting for kstack allocation failure Thread-Topic: vm.kmem_size setting for kstack allocation failure Thread-Index: AQHQvm+JmQ631KgkqE6uQLJvRGMtLA== Date: Tue, 14 Jul 2015 19:59:05 +0000 Message-ID: <9956D08B-2609-4AA2-AD62-721B753F6BFA@juniper.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: juniper.net; dkim=none (message not signed) header.d=none; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [66.129.239.13] x-microsoft-exchange-diagnostics: 1; DM2PR0501MB1151; 5:111cYEpuy105HPv568rQ6z3o1GS3u7XwvPXO9vQtGFZxh/EFram0EbEITq19kZ/7BX2QwErh/IMD8RVcEiWmnu7pYG1bcdSu+mYUFzAhzfDXbIY7LIXKCZmR60+fjy5ok4fH+JfurHdYYe2l0yuULQ==; 24:QaqtB50e8QvQJM2nBA/aHeg+OWzRbW9dC0oO9jSumwZGMoLn+91S9rMHDhv6aOpS0C9GoqOQW2NBqwiiFBGAlS98kxgCSc9j4e/yFeCoSfM=; 20:lLXBEvItH56iZvWdvwvEP4W8MgJayqIzGwWWXRGcKlKSGj7+An0UsbtvCagLVoYHnnYKvB1ZH9p02llKdxhhkA== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DM2PR0501MB1151; dm2pr0501mb1151: X-MS-Exchange-Organization-RulesExecuted x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(5005006)(3002001); SRVR:DM2PR0501MB1151; BCL:0; PCL:0; RULEID:; SRVR:DM2PR0501MB1151; x-forefront-prvs: 0637FCE711 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(53754006)(107886002)(77096005)(5001960100002)(33656002)(2656002)(5001770100001)(5002640100001)(558084003)(87936001)(99286002)(46102003)(189998001)(92566002)(102836002)(36756003)(40100003)(2900100001)(83716003)(2501003)(66066001)(122556002)(82746002)(450100001)(50986999)(106116001)(54356999)(229853001)(77156002)(16236675004)(62966003)(86362001)(104396002); DIR:OUT; SFP:1102; SCL:1; SRVR:DM2PR0501MB1151; H:DM2PR0501MB1150.namprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; MIME-Version: 1.0 X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Jul 2015 19:59:05.8404 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR0501MB1151 X-Mailman-Approved-At: Tue, 14 Jul 2015 20:43:20 +0000 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 20:32:49 -0000 Hello All I have been working on a panic due to kernel stack allocation failure. vm_t= hread_new: kstack allocation failed I understand that this could be a kmem_size setting problem where too less = space is left for the stack. Any thoughts on setting vm.kmem_size to a lesser value ? Thanks Ravi From owner-freebsd-hackers@freebsd.org Tue Jul 14 21:22:10 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D6EE9A10D4 for ; Tue, 14 Jul 2015 21:22:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 080FD19F6 for ; Tue, 14 Jul 2015 21:22:09 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id t6ELM407050660 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 15 Jul 2015 00:22:05 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t6ELM407050660 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id t6ELM4Ej050659; Wed, 15 Jul 2015 00:22:04 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 15 Jul 2015 00:22:04 +0300 From: Konstantin Belousov To: Raviprakash Darbha Cc: freebsd-hackers , "freebsd-hackers@freebsd.org" Subject: Re: vm.kmem_size setting for kstack allocation failure Message-ID: <20150714212204.GA2404@kib.kiev.ua> References: <9956D08B-2609-4AA2-AD62-721B753F6BFA@juniper.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9956D08B-2609-4AA2-AD62-721B753F6BFA@juniper.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 21:22:10 -0000 On Tue, Jul 14, 2015 at 07:59:05PM +0000, Raviprakash Darbha wrote: > Hello All > > I have been working on a panic due to kernel stack allocation failure. vm_thread_new: kstack allocation failed > I understand that this could be a kmem_size setting problem where too less space is left for the stack. > > Any thoughts on setting vm.kmem_size to a lesser value ? > There is no such panic in the FreeBSD, since r173361, done in 2007. From owner-freebsd-hackers@freebsd.org Wed Jul 15 05:35:40 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8FB399CA3D for ; Wed, 15 Jul 2015 05:35:40 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C235B1D1A for ; Wed, 15 Jul 2015 05:35:40 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-240-47.lns20.per4.internode.on.net [121.45.240.47]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t6F5ZZvM078627 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 14 Jul 2015 22:35:38 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <55A5F121.2070702@freebsd.org> Date: Wed, 15 Jul 2015 13:35:29 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Raviprakash Darbha , freebsd-hackers , "freebsd-hackers@freebsd.org" Subject: Re: vm.kmem_size setting for kstack allocation failure References: <9956D08B-2609-4AA2-AD62-721B753F6BFA@juniper.net> In-Reply-To: <9956D08B-2609-4AA2-AD62-721B753F6BFA@juniper.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 05:35:41 -0000 On 7/15/15 3:59 AM, Raviprakash Darbha wrote: > Hello All > > I have been working on a panic due to kernel stack allocation failure. vm_thread_new: kstack allocation failed > I understand that this could be a kmem_size setting problem where too less space is left for the stack. > > Any thoughts on setting vm.kmem_size to a lesser value ? more information is needed. what OS rev, hardware, etc. > > Thanks > Ravi > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@freebsd.org Wed Jul 15 15:21:46 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7917799BDAC for ; Wed, 15 Jul 2015 15:21:46 +0000 (UTC) (envelope-from romapera15@gmail.com) Received: from mail-lb0-x231.google.com (mail-lb0-x231.google.com [IPv6:2a00:1450:4010:c04::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 030391C35 for ; Wed, 15 Jul 2015 15:21:46 +0000 (UTC) (envelope-from romapera15@gmail.com) Received: by lblf12 with SMTP id f12so27033148lbl.2 for ; Wed, 15 Jul 2015 08:21:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Qyz1klZASr1YXwlsnhvGxFj9a4PZ8IXadXbOobm2Rm0=; b=X9dH3+3E4iDGTc8ZUqHS0rOlwLbL8VnZFk3zaL99zbJ8HiyB7B1waR5Pcqji1ilDtc TNkdy9ag9EZJTB8vOmKxwtnyDZShIGmKYMORhZ7e264gTBdz+nbI3nUSy+GoFIwtnRU5 4Qp5rDGbTCv8/5stb1uU3LgaaJrT8I9jyqu+BxHbJNBJs3clz1NzmrH7wey0cxLXgBXs l56pbo4XF3poH4DXulGFJZUAG2tQgAQRp2CJxVmHrr5Nr5QDoL2O+8PwLse6PWilZszj NjJvfdT9drlvjaywTU2CCEymERquEqSdGAME99+fHsQBLcKts8J9DqvTr5ua1vTrmyaC clGw== MIME-Version: 1.0 X-Received: by 10.152.10.72 with SMTP id g8mr4605096lab.97.1436973703430; Wed, 15 Jul 2015 08:21:43 -0700 (PDT) Received: by 10.25.205.209 with HTTP; Wed, 15 Jul 2015 08:21:43 -0700 (PDT) Date: Wed, 15 Jul 2015 12:21:43 -0300 Message-ID: Subject: The IXP of Ashburn in Commonwealth of Virginia, LAX or WU Seattle work using what BSD operating system and what are its importance? From: =?UTF-8?Q?fran=C3=A7ai_s?= To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 15:21:46 -0000 The IXP of Ashburn in Commonwealth of Virginia, LAX or WU Seattle work using what BSD operating system and what are its importance? From owner-freebsd-hackers@freebsd.org Wed Jul 15 17:37:20 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E1359A29F9; Wed, 15 Jul 2015 17:37:20 +0000 (UTC) (envelope-from dieterbsd@gmail.com) Received: from mail-ig0-x236.google.com (mail-ig0-x236.google.com [IPv6:2607:f8b0:4001:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E3F61C92; Wed, 15 Jul 2015 17:37:20 +0000 (UTC) (envelope-from dieterbsd@gmail.com) Received: by igvi1 with SMTP id i1so82616523igv.1; Wed, 15 Jul 2015 10:37:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Gy3H2s4CSpQWi0u5RhQBajRjbP49PON3N+867t4x0Ak=; b=LbUPURhYDF/GMwYWtX8neQ+4FwW4ecO+95c0v7fQdXeg1MPwlWuiohYNjXBLgFHdI2 XthF1RFHvkHPHYhDhXYpC4eCR8/Wew1GB8G4vTQjRrDTDfsT/cX7SU0GjNaWK4/VYBRJ q4sMqQq+tgvvKPKPlT9P4hS9LXHmcV0Eaeewkm/g1C8mpolDrfeU4OcT9kNSJ9EvX1Hk P/3zYd+t6dtfiK3bijFlxdh3Kr2noxJdxS6wjI3o76t/r8yRHnJWI4CM6jqGqzAxDD1e 4h3uJ3tP1UXChTlubOqsPXKsbUR844g1KRoh9Qh3YyrnKFKHLvia5HOZqdAZ8XuSKm0f fmvg== MIME-Version: 1.0 X-Received: by 10.107.6.194 with SMTP id f63mr5901717ioi.61.1436981839598; Wed, 15 Jul 2015 10:37:19 -0700 (PDT) Received: by 10.64.2.132 with HTTP; Wed, 15 Jul 2015 10:37:19 -0700 (PDT) Date: Wed, 15 Jul 2015 10:37:19 -0700 Message-ID: Subject: Re: format/newfs larger external consumer drives From: Dieter BSD To: freebsd-hackers@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 17:37:20 -0000 [ freebsd-fs@ added ] >> If the average filesize will be large, use large block/frag sizes. >> I use 64 KiB / 8 KiB. And reduce the number of inodes. I reduce >> inodes as much as newfs allows and there are still way too many. > > Can you think of an algorithmic way to express this? I.e., you don't > want blocks to get *too* large as you risk greater losses in "partial > fragments", etc. Likewise, you don't want to run out of inodes. I look at df -i for existing filesystems with similar filesizes. My data filesystems usually get an entire disk (..., 2TB, 3TB, recently 5TB) and with 64/8 block/frag and as few inodes as newfs will allow df still reports numbers like 97% full but only using 0% or 1% of inodes. density reduced from 67108864 to 14860288 /dev/ada1: 4769307.0MB (9767541168 sectors) block size 65536, fragment size 8192 using 1315 cylinder groups of 3628.00MB, 58048 blks, 256 inodes. with soft updates I should take another look at increasing the size of cylinder groups. Newfs likes very small cylinder groups, which made sense 30 years when disks were like 40 MB and file sizes were a lot smaller. IIRC, each cylinder group gets at least one block of inodes, and with file sizes of 1-20 GB I get way too many inodes. Yes, a larger frag size will waste some space in the last frag of a file, but having smaller block&frag sizes uses a lot of space to keep track of all those blocks and frags. And makes more work for fsck. > "risk" of loss/cost of recovery (when the medium > *is* unceremoniously dismounted Some panics don't sync the disks. Sometimes disks just go into a coma. Soft updates is supposed to limit problems to those that fsck -p will automagicly fix. (assuming the disk's write cache is turned off) There is at least one case where it does not. See PR 166499 (from 2012, still not fixed). As long as I'm whining about unfixed filesystem PRs, see also bin/170676: Newfs creates a filesystem that does not pass fsck. (also from 2012) > I am concerned with the fact that users can so easily/carelessly "unplug" > a USB device without the proper incantations beforehand. of course, *their* > mistake is seen as a "product design flaw"! :-/ Superglue the cable in place? :-) Perhaps print up something like "Unmount filesystem(s) before unplugging or powering off external disk, or you might lose your data.", laminate it and attach it to the cables? > The "demo app" that I'm working on is a sort of (low performance) NAS > built on SBC's and external drives. I assume that the drives *have* to be external? Do they have to be usb? Could they be e-sata? E-sata is faster and avoids the various usb problems. They used to sell external drives where the sata-to-usb bridge was in a separate little module box. They had alternate modules with e-sata, firewire, etc. The disk box had a standard internal ('L') sata connector, except a standard sata connector was too large to fit. So I took out my Swiss Army Knife and carved off some plastic from the connector on a standard sata cable so that it would fit. You could also put a standard sata drive into an enclosure (with a small fan) and use your choice of connection to the computer. >> USB specific stuff: There is an off by 1 sector problem, which will >> bite you if you switch a drive between using the sata-usb bridge >> and connecting the drive directly to a sata controller. I had to > > Ouch! I've not seen that with PATA-USB bridges. OTOH, I tend not > to pull a drive *from* an external enclosure but, rather, rely on > the external enclosures to provide portability. E.g., easier to > move 500G of files from machineA to machineB by physically moving > the volume containing them! Apparently they vary, see the message from Warren. Mine was missing the first sector, so I had to have the kernel hunt for the partitioning info. The external drives I've seen do not have fans, and have little or no ventilation. If the drive will be spinning for awhile I worry about it overheating. > The "demo app" will try to use the large multi-TB drives of which I > have little long-term experience. OTOH, the usage model is "fire it > up, pull off whichever files you need, then spin everything down"... > until the next time you might need to retrieve an ISO (a week later?) With this usage model it sounds like you could use a read-only mount. Would an optical drive work for this application? >> If the drive disappears with filesystem(s) mounted. the kernel might >> very well panic. There was a discussion of this problem recently. >> I thought that FUSE was suggested as a possible solution, but I >> can't find the discussion. This problem is not limited to users >> disconnecting usb drives without unmounting them. The problem >> happens all by itself with internal drives, as the drive, port >> multiplier, controller, or device driver decides to go out to lunch, >> and the kernel panics. This happens *far* too often, and *kills* >> reliability. We really need a solution for this. > > I think its hard to back-port these sorts of things. Much easier > to consider the possibility of failure when initially designing the > system, interfaces, etc. I wonder how hard it would be to create a FUSE version of FFS? Any thoughts from the filesystem wizards? Alternately, instead of panicing, could the filesystem just umount -f the offending filesystem? (And whine to log(9).) I am very tired of having an entire machine panic just because one disk decided to take a nap. This is not how you get 5 9s. :-( From owner-freebsd-hackers@freebsd.org Wed Jul 15 20:42:26 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37F179A2585; Wed, 15 Jul 2015 20:42:26 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "wonkity.com", Issuer "wonkity.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BF71C1FDD; Wed, 15 Jul 2015 20:42:25 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.9/8.14.9) with ESMTP id t6FKgNp9051714 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 15 Jul 2015 14:42:23 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.9/8.14.9/Submit) with ESMTP id t6FKgNv8051711; Wed, 15 Jul 2015 14:42:23 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Wed, 15 Jul 2015 14:42:23 -0600 (MDT) From: Warren Block To: Dieter BSD cc: freebsd-hackers@freebsd.org, freebsd-fs@freebsd.org Subject: Re: format/newfs larger external consumer drives In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Wed, 15 Jul 2015 14:42:23 -0600 (MDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 20:42:26 -0000 On Wed, 15 Jul 2015, Dieter BSD wrote: > I wonder how hard it would be to create a FUSE version of FFS? > Any thoughts from the filesystem wizards? http://sourceforge.net/projects/fuse-ufs2/ Untested by me, though. From owner-freebsd-hackers@freebsd.org Wed Jul 15 21:15:05 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0480C9A2DE1 for ; Wed, 15 Jul 2015 21:15:05 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 87F101028 for ; Wed, 15 Jul 2015 21:15:04 +0000 (UTC) (envelope-from Don.whY@gmx.com) Received: from [192.168.1.115] ([67.212.197.98]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0MF9WR-1ZD3Jc3RSV-00GGAG for ; Wed, 15 Jul 2015 23:14:56 +0200 Message-ID: <55A6CD65.6060102@gmx.com> Date: Wed, 15 Jul 2015 14:15:17 -0700 From: Don whY User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: format/newfs larger external consumer drives References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:1Gxu87ZjUlXDlpGYHdCRyV8bAImILNpeqCyNRub7ss3qsNpjQUk jfBcWTLXjqqqQW5IZb2yz0qlB0pPtWpGxvZTL79anNMsZEofcx5iL28lql37J8EpbTgFxQm t7J7V0exKrUHwUqyxlKgK+xgpf8dZmH9BcP1sXlfsG/WJyW7935SNjwHMSVRRVQt/DOSXeb WUB+JhPphVJd4jf6C4GDg== X-UI-Out-Filterresults: notjunk:1;V01:K0:PQHlHUYaKXk=:ntIZdnnJwnQ1fVy1bTmx62 hhKYQ56RiJkJ7iqVegu5BwQOyi1mK1lFwg9ewfF7FI5t4saBmOLDpdXAuXNxdRgufD9KZ2N7s E9eZET6rgd7EiAIRl4vy7nRwbZy718qziojOz2DydNbnOLuGDtaFLQBP1yNybOHMb9j+ZBWf5 100UUFvFjcVieEl6eSivtNiCNQ4CeSoy+2e0TuyOKTrxMtthOyKI1jlHxNxBG2TRaZTRUwivl l2qBJs9ktkysDJupeYj//kj74bcWu4GgtwFDhwKUM48FTt7IX0DRgZRm8DWTI81rYCFOsvUq8 Ndb3xl2VvokiT2iT4XDRJN/hcfh+TwVR7ry0mZRhreW6yz0iVxyNVzb97haODXPt17+Ektq2J QWr/ZEY+NbZ5/sEcYfxG0qzd6ddxfvydFO/GF3FimrIr+eN0jSDY2T+kEe1o9dgfrwweNpY5A 0U4U11zbM6Vukoz9zIRM1vc5+Y9C6dRpbQxVakb/CrTI1JA9/iqGdtPZne6mZPiIr8cAUVCBB lV9IKR9iPedRhNJm8fPsndkZrKRs8XFVgvucQL/2dF1V+dYmkP4WCvRSZN9RO7WwrHnPow4+H rb1XIQ8hGnYPxwvhE38loVIk/0sW32PPQJQO9A0y5NksHfU9eNP/4bzcPCVT/Fyp3oDN/ixU3 KTIQ2qWVDGgk0YMQKrokJJRBpJpWyzihLNW1oxNZuUBSyt7kxQcbtC99lHfuG97nFCEI= X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 21:15:05 -0000 On 7/15/2015 10:37 AM, Dieter BSD wrote: > [ freebsd-fs@ added ] > >>> If the average filesize will be large, use large block/frag sizes. >>> I use 64 KiB / 8 KiB. And reduce the number of inodes. I reduce >>> inodes as much as newfs allows and there are still way too many. >> >> Can you think of an algorithmic way to express this? I.e., you don't >> want blocks to get *too* large as you risk greater losses in "partial >> fragments", etc. Likewise, you don't want to run out of inodes. > > I look at df -i for existing filesystems with similar filesizes. OK, that makes sense. A developer could build an artificial dataset and then see what it looks like; tweek the filesystem structure, rebuild and see what *that* looks like, etc. I guess with experience, it would be relatively easy to select a good starting point for such iterations... > My data filesystems usually get an entire disk (..., 2TB, 3TB, recently 5TB) > and with 64/8 block/frag and as few inodes as newfs will allow > df still reports numbers like 97% full but only using 0% or 1% > of inodes. > > density reduced from 67108864 to 14860288 > /dev/ada1: 4769307.0MB (9767541168 sectors) block size 65536, fragment size 8192 > using 1315 cylinder groups of 3628.00MB, 58048 blks, 256 inodes. > with soft updates > > I should take another look at increasing the size of cylinder groups. > Newfs likes very small cylinder groups, which made sense 30 years when > disks were like 40 MB and file sizes were a lot smaller. IIRC, each > cylinder group gets at least one block of inodes, and with file sizes > of 1-20 GB I get way too many inodes. > > Yes, a larger frag size will waste some space in the last frag of a file, > but having smaller block&frag sizes uses a lot of space to keep track of > all those blocks and frags. And makes more work for fsck. So, fsck's effort (and execution *time*) is based *mostly* on inodes? The "volume" of data is largely unimportant? >> "risk" of loss/cost of recovery (when the medium >> *is* unceremoniously dismounted > > Some panics don't sync the disks. Sometimes disks just go into a coma. > Soft updates is supposed to limit problems to those that fsck -p will > automagicly fix. (assuming the disk's write cache is turned off) There > is at least one case where it does not. See PR 166499 (from 2012, > still not fixed). > > As long as I'm whining about unfixed filesystem PRs, see also > bin/170676: Newfs creates a filesystem that does not pass fsck. > (also from 2012) > >> I am concerned with the fact that users can so easily/carelessly "unplug" >> a USB device without the proper incantations beforehand. of course, *their* >> mistake is seen as a "product design flaw"! :-/ > > Superglue the cable in place? :-) You know *that* will go over like a lead balloon! > Perhaps print up something like "Unmount filesystem(s) before unplugging > or powering off external disk, or you might lose your data.", > laminate it and attach it to the cables? You'll still get folks who realize their mistake an ohnosecond too late. And, of course, want to push the blame onto the device: "Why can't it make sure I don't do this? (well, that's obvious!) Or, at least give me an option whereby I can plug it in within N seconds and everything behaves as if I hadn't done that??" (that, of course, opens even more cans of worms) [The intended "user" isn't expected to be a "hacker" -- or even computer. Hence "appliance" and not "computer system"] >> The "demo app" that I'm working on is a sort of (low performance) NAS >> built on SBC's and external drives. > > I assume that the drives *have* to be external? Do they have to be > usb? Could they be e-sata? E-sata is faster and avoids the various usb > problems. They used to sell external drives where the sata-to-usb bridge > was in a separate little module box. They had alternate modules with > e-sata, firewire, etc. The disk box had a standard internal ('L') > sata connector, except a standard sata connector was too large to fit. > So I took out my Swiss Army Knife and carved off some plastic from > the connector on a standard sata cable so that it would fit. > You could also put a standard sata drive into an enclosure (with > a small fan) and use your choice of connection to the computer. Two separate issues, here. My "demo" -- and the sorts of apps that other developers are likely to pursue. In my case, there isn't any real physical room for an internal disk. I'm using Dell FX160's -- they'll support a SATA laptop drive and a SATA "memory module" (i.e., this connector is mounted *on* the SBC and "points straight up"). Power supply is severely limited, etc. OTOH, lots of USB ports and the throughput needs are minimal (I'm only using this to fetch ISO's, etc. when I need to reinstall some software, drivers/documentation for odd bits of hardware, etc. The point is to get rid of the piles of CD/DVD media that I've accumulated over the years) In the "developer" case, the hardware that they will typically have available will be more along the lines of SoC's. Early devices had "slave/peripheral mode" USB controllers but the more recent offerings include host support. So, a developer wishing to integrate something like a disk drive (magnetic/optical/SSD) into a product offering can usually just hang it off a USB "connection" -- instead of having to include a disk controller in his/her design. This also frees the developer from getting dragged into the "commodity markup" mess -- folks know what disks are, what they cost, etc. So, if you offer a disk of a particular capacity in your product, they immediately equate that to the commodity pricing they've seen for similar "components": "Wow, that's a helluva lot to pay for a disk of that size!" or "Well, the disk is worth $X so he's charging an awful lot for the rest of the device!" >>> USB specific stuff: There is an off by 1 sector problem, which will >>> bite you if you switch a drive between using the sata-usb bridge >>> and connecting the drive directly to a sata controller. I had to >> >> Ouch! I've not seen that with PATA-USB bridges. OTOH, I tend not >> to pull a drive *from* an external enclosure but, rather, rely on >> the external enclosures to provide portability. E.g., easier to >> move 500G of files from machineA to machineB by physically moving >> the volume containing them! > > Apparently they vary, see the message from Warren. Mine was missing > the first sector, so I had to have the kernel hunt for the partitioning > info. That must have been a painful discovery! ("WTF???") > The external drives I've seen do not have fans, and have little or > no ventilation. If the drive will be spinning for awhile I worry > about it overheating. Correct. I only use external drives for sporadic service. E.g., spin up drive, get/put what I want, then spin it down. E.g., most of my largish external drives are 5 or 6 years old and have very few total hours... >> The "demo app" will try to use the large multi-TB drives of which I >> have little long-term experience. OTOH, the usage model is "fire it >> up, pull off whichever files you need, then spin everything down"... >> until the next time you might need to retrieve an ISO (a week later?) > > With this usage model it sounds like you could use a read-only mount. > Would an optical drive work for this application? No. Optical media are too small. The whole point was to get rid of the spindles of CD/DVD media and put things in a more accessible form. I have a job that runs on each box that methodically walks through the contents of the attached media (which might vary from boot to boot) verifying checksums (which are stored in a RDBMS on another box). So, in theory, I have a bit of assurance as to whether or not a particular file is still "intact" (like running an audit on a RAID array) as well as the integrity of the medium as a whole. At the same time, another job rsync's changes detected on the monitored media to any "mirrors" that are also detected as being on-line (perhaps on a different network node). The RDBMS's role lets me query it to identify what I want and where it is likely to be located -- before spinning up any media. A simple schema lets me store (ID, name, containerID, size, MD5, etc.) tuples (so I can even find specific files inside tarballs, ISO's, etc.). Lastly, the fact that the disks are external, "simple" filesystems (e.g., not RAID/ZFS) means I can physically move a drive to another machine and treat it like a "normal" drive -- without having to worry about rebuilding arrays, etc. [I initially tried doing this with NTFS volumes. This would allow the volumes to be conveniently mounted on Windows machines without relying on the appliance's role in providing access. But, "checking" an NTFS volume of that size tethered via a USB interface takes AGES!! (And, I'm not sure how comfortable I am with the current NTFS support)] >>> If the drive disappears with filesystem(s) mounted. the kernel might >>> very well panic. There was a discussion of this problem recently. >>> I thought that FUSE was suggested as a possible solution, but I >>> can't find the discussion. This problem is not limited to users >>> disconnecting usb drives without unmounting them. The problem >>> happens all by itself with internal drives, as the drive, port >>> multiplier, controller, or device driver decides to go out to lunch, >>> and the kernel panics. This happens *far* too often, and *kills* >>> reliability. We really need a solution for this. >> >> I think its hard to back-port these sorts of things. Much easier >> to consider the possibility of failure when initially designing the >> system, interfaces, etc. > > I wonder how hard it would be to create a FUSE version of FFS? > Any thoughts from the filesystem wizards? > > Alternately, instead of panicing, could the filesystem just > umount -f the offending filesystem? (And whine to log(9).) > > I am very tired of having an entire machine panic just because > one disk decided to take a nap. This is not how you get 5 9s. :-( Or, power glitches, firmware bugs, etc. From owner-freebsd-hackers@freebsd.org Wed Jul 15 22:06:22 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC4A79A3613; Wed, 15 Jul 2015 22:06:22 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 847721B28; Wed, 15 Jul 2015 22:06:22 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t6FM6LpY072052 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 15 Jul 2015 15:06:21 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t6FM6LBS072051; Wed, 15 Jul 2015 15:06:21 -0700 (PDT) (envelope-from jmg) Date: Wed, 15 Jul 2015 15:06:21 -0700 From: John-Mark Gurney To: Dieter BSD Cc: freebsd-hackers@freebsd.org, freebsd-fs@freebsd.org Subject: Re: format/newfs larger external consumer drives Message-ID: <20150715220621.GP8523@funkthat.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Wed, 15 Jul 2015 15:06:21 -0700 (PDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2015 22:06:22 -0000 Dieter BSD wrote this message on Wed, Jul 15, 2015 at 10:37 -0700: > [ freebsd-fs@ added ] > > >> If the average filesize will be large, use large block/frag sizes. > >> I use 64 KiB / 8 KiB. And reduce the number of inodes. I reduce > >> inodes as much as newfs allows and there are still way too many. > > > > Can you think of an algorithmic way to express this? I.e., you don't > > want blocks to get *too* large as you risk greater losses in "partial > > fragments", etc. Likewise, you don't want to run out of inodes. > > I look at df -i for existing filesystems with similar filesizes. > My data filesystems usually get an entire disk (..., 2TB, 3TB, recently 5TB) > and with 64/8 block/frag and as few inodes as newfs will allow > df still reports numbers like 97% full but only using 0% or 1% > of inodes. > > density reduced from 67108864 to 14860288 > /dev/ada1: 4769307.0MB (9767541168 sectors) block size 65536, fragment size 8192 > using 1315 cylinder groups of 3628.00MB, 58048 blks, 256 inodes. > with soft updates > > I should take another look at increasing the size of cylinder groups. Right now the cg by default is made to fill a block... I don't believe it can be made larger without a major overhaul of the code... The default used to be even smaller than a full block causing even more cg's to be created and you had to do trial and error to figure out how to make a cg a full block... > Newfs likes very small cylinder groups, which made sense 30 years when > disks were like 40 MB and file sizes were a lot smaller. IIRC, each > cylinder group gets at least one block of inodes, and with file sizes > of 1-20 GB I get way too many inodes. This is partly the default number of inodes are too large... The current documented default is an inode for every 4 * frag_size bytes of data space, which isn't correct!!! This was changed to 2 in r228794 to keep the number of inodes the same when the transition from 16k/2k to 32k/4k happened, but the documentation was not updated... It has now been updated in r285615 and will be MFC'd... On my dev server where I have a few source trees checked out: Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/ada0s2d 185G 122G 48G 72% 2.8M 9.5M 23% /a This fs has non-standard config in that my frag size is 8k... If it was standard, I'd have twice as many inodes... Increaseing the frag size both cuts the # of inodes in half, but also increases the cg size... Standard: /dev/ada0s2d: 192068.0MB (393355264 sectors) block size 32768, fragment size 4096 using 307 cylinder groups of 626.09MB, 20035 blks, 80256 inodes. Non-standard: /dev/ada0s2d: 192068.0MB (393355264 sectors) block size 32768, fragment size 8192 using 166 cylinder groups of 1162.97MB, 37215 blks, 74496 inodes. The other thing I didn't realize (and would be useful for someone to benchmark) is that many SSD's now use 8k page size instead of the previous 4k.. Maybe this needs to be more of a sliding scale based upon disk size? Maybe go from 2 * frag to 4 * frag at fs's larger than 1TB? Though this is still something that a system admin needs to address, it's impossible to make the defaults sane for all use cases... There are some people that will only keep multi GB files on their 5 TB fs, and so only need a few thousand inodes, but others may keep more smaller files... It'd be nice to put together a fs survey to see what sizes of filesystems people have, and the distribution of files sizes... I'll try to do that... > Yes, a larger frag size will waste some space in the last frag of a file, > but having smaller block&frag sizes uses a lot of space to keep track of > all those blocks and frags. And makes more work for fsck. Yep... > > "risk" of loss/cost of recovery (when the medium > > *is* unceremoniously dismounted > > Some panics don't sync the disks. Sometimes disks just go into a coma. > Soft updates is supposed to limit problems to those that fsck -p will > automagicly fix. (assuming the disk's write cache is turned off) There > is at least one case where it does not. See PR 166499 (from 2012, > still not fixed). > > As long as I'm whining about unfixed filesystem PRs, see also > bin/170676: Newfs creates a filesystem that does not pass fsck. > (also from 2012) > > > I am concerned with the fact that users can so easily/carelessly "unplug" > > a USB device without the proper incantations beforehand. of course, *their* > > mistake is seen as a "product design flaw"! :-/ > > Superglue the cable in place? :-) > > Perhaps print up something like "Unmount filesystem(s) before unplugging > or powering off external disk, or you might lose your data.", > laminate it and attach it to the cables? Same problem goes for Windows.. They have a policy of turning of write buffering on pluggable thumb drives to help eliminate this.. For UFS, the sync flag should be provided to mount... [...] > Alternately, instead of panicing, could the filesystem just > umount -f the offending filesystem? (And whine to log(9).) > > I am very tired of having an entire machine panic just because > one disk decided to take a nap. This is not how you get 5 9s. :-( There has been lots of work to try to make file systems not panic when the underlying drives disappear, though clearly more work is needed... Patches welcome! :) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@freebsd.org Thu Jul 16 14:13:29 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 624EA9A1390 for ; Thu, 16 Jul 2015 14:13:29 +0000 (UTC) (envelope-from pjalaber@gmail.com) Received: from mail-qg0-x22d.google.com (mail-qg0-x22d.google.com [IPv6:2607:f8b0:400d:c04::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 20C261E61; Thu, 16 Jul 2015 14:13:29 +0000 (UTC) (envelope-from pjalaber@gmail.com) Received: by qgy5 with SMTP id 5so33052206qgy.3; Thu, 16 Jul 2015 07:13:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=EZHR0JTRGeDg1NKa02WC+HlGCLrpwKh1hD4hqIYfHHc=; b=P8ImltVXsHLM8BG5+Iwltim7TL70PnlYdX1L+DIpnW6wK5fwaKRy5/+CZHm/rBdfy0 MQpQyrKbFA9/nJzgslSNNUx0FU9m1ZZuS3fImi6bNBUoeL1i9Tmy8weNQXa9KxHxuVkG nx6ips1RewuuIIkAbNKqIkDXnBmE5FJ9iqSRrYzV9DA/DNBEL+LN8Qs5wqm3tsPyBYtb mtj/Ewczkl23vsWYlVByAkeVjwWQf3pXXE45PhQH45Kk9Dllp7kRPa/SdzIbo+qei/PW SdajBcg3/Uu/9jdKojIznJb55Isf5aV7S7OD/NIJKE/UXzOM80Inv7Oucf/nGBU8hKuA aMmA== MIME-Version: 1.0 X-Received: by 10.140.146.83 with SMTP id 80mr11067267qhs.76.1437056007987; Thu, 16 Jul 2015 07:13:27 -0700 (PDT) Received: by 10.140.92.247 with HTTP; Thu, 16 Jul 2015 07:13:27 -0700 (PDT) In-Reply-To: References: Date: Thu, 16 Jul 2015 16:13:27 +0200 Message-ID: Subject: Re: adaptive rwlock deadlock From: Philippe Jalaber To: freebsd-hackers@freebsd.org Cc: jhb@freebsd.org, attilio@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2015 14:13:29 -0000 2015-07-07 12:10 GMT+02:00 Philippe Jalaber : > Hi, > > I am facing a strange problem using the network stack and adaptive rwlocks > running Freebsd 9.3. > Basically I can reproduce the problem with 3 threads: > > 1) thread 1 has taken the rwlock of structure inpcb in exclusive mode in > tcp_input.c. This thread also runs my own code and repeatedly takes a > rwlock (called g_rwlock) in shared mode and releases it, until a shared > object is marked not "busy" any more: > > rwlock(inp_lock); > .... > do { // thread is active waiting in the loop > rlock(g_rwlock); > o = find(); > if ( o == NULL ) > break; > busy = o.busy; > if (o != NULL && busy) > runlock(g_rwlock); > } while ( busy ); > > if ( o != NULL ) > { > // do something with o > .... > } > runlock(g_rwlock); > .... > > 2) thread 2 wants to set the shared object as "ready". So it tries to take > g_rwlock in exclusive mode and is blocked in _rw_wlock_hard@kern_rwlock.c:815 > "turnstile_wait(ts, rw_owner(rw), TS_EXCLUSIVE_QUEUE)" because thread 1 has > already taken it in shared mode: > > wlock(g_rwlock); > o = find(); > if ( o != NULL ) > o.busy = 1; > wunlock(g_rwlock); > > // o is busy so work on it without any lock > .... > > wlock(g_rwlock); // thread is blocked here > o.busy = 0; > maybe_delete(o); > wunlock(g_rwlock); > > 3) thread 3 spins on the same inpcb rwlock than thread 1 in > _rw_wlock_hard@kern_rwlock.c:721 "while ((struct > thread*)RW_OWNER(rw->rw_lock) == owner && TD_IS_RUNNING(owner)) " > > > My target machine has two cpus. > Thread 1 is pinned to cpu 0. > Thread 2 and Thread 3 are pinned to cpu 1. > Thread 1 and Thread 2 have a priority of 28. > Thread 3 has a priority of 127 > > Now what seems to happen is that when thread 1 calls runlock(g_rwlock), it > calls turnstile_broadcast@kern_rwlock.c:650, but thread 2 never regains > control because thread 3 is spinning on the inpcb rwlock. Also the > condition TD_IS_RUNNING(owner) is always true because thread 1 is active > waiting in a loop. So the 3 threads deadlock. > Note that if I compile the kernel without adaptive rwlocks it works > without any problem. > A workaround is to add a call to "sched_relinquish(curthread)" in thread 1 > in the loop just after the call to runlock. > > I am also wondering about the code in _rw_runlock after > "turnstile_broadcast(ts, queue)". Isn't the flag RW_LOCK_WRITE_WAITERS > definitely lost if the other thread which is blocked in turnstile_wait > never regains control ? > > Thank you for your time, > Regards, > Philippe > > the sched_relinquish workaround does not seem to work every time. one possible solution (which seems to work) is to rlock/runlock in thread 1, and if the busy flag is set, then take the lock in exclusive mode, like this: shared_count = 0; rwlock(inp_lock); .... do { // thread is active waiting in the loop if ( shared_count == 0 ) rlock(g_rwlock); else wlock(g_rwlock); o = find(); if ( o == NULL ) break; busy = o.busy; if (o != NULL && busy) { if ( shared_count == 0 ) runlock(g_rwlock); else wunlock(g_rwlock); shared_count++; } } while ( busy ); if ( o != NULL ) { // do something with o .... } if ( shared_count == 0 ) runlock(g_rwlock); else wunlock(g_rwlock); with this code, deadlock does not happen anymore but I don't really see why. Any idea ? Thanks, Philippe From owner-freebsd-hackers@freebsd.org Thu Jul 16 15:42:31 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48CA39A38DC for ; Thu, 16 Jul 2015 15:42:31 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2740E1EB9 for ; Thu, 16 Jul 2015 15:42:31 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 263979A38DB; Thu, 16 Jul 2015 15:42:31 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 25C279A38DA for ; Thu, 16 Jul 2015 15:42:31 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mail-la0-x233.google.com (mail-la0-x233.google.com [IPv6:2a00:1450:4010:c03::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B28891EB8 for ; Thu, 16 Jul 2015 15:42:30 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by lagw2 with SMTP id w2so45893832lag.3 for ; Thu, 16 Jul 2015 08:42:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:subject:message-id:date:to:mime-version; bh=PS4rGTqKTbv8Q0sYDu1Mf34GMzJtitHJE/eC9Y/bFdU=; b=Y6sJxz3vQZZA9tKR6dqsvXiKNjh1WEffeVg0ulty2MJp92JaA6B8bW3j+2T6E2A5cq jjUKTGsvfsapr54z296dJcYqOFHd1tK4I2CHVTb2a5v15LCRmbadwLZbl/Dk2yF8JHaS hlDMnHcMTkch4umBv2SjtIbSld3YGf42p8ApYuKriXNYoR0CJMCjYK6+cN1vb+GNgCz+ gS22K7d0/2wTLx35P5zVnSOC71c4hv115XQ46WJ0E3vK7+8ZO06ZwvVbUiSb4W6+ok6j VgPLPHrCzdFtjphIYEmQ4buI+/z4Sl1vhrDDZd79FODzEmojKrxjb+KZFcHE4tcSfWJu xVLQ== X-Received: by 10.112.27.138 with SMTP id t10mr9963934lbg.42.1437061348692; Thu, 16 Jul 2015 08:42:28 -0700 (PDT) Received: from 95.108.174.185-red.dhcp.yndx.net (95.108.174.185-red.dhcp.yndx.net. [95.108.174.185]) by smtp.gmail.com with ESMTPSA id si3sm2131847lbb.32.2015.07.16.08.42.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 16 Jul 2015 08:42:27 -0700 (PDT) From: Dmitry Sivachenko Content-Type: multipart/mixed; boundary="Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA" Subject: Strange memory management with mmap() Message-Id: Date: Thu, 16 Jul 2015 18:42:26 +0300 To: hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2015 15:42:31 -0000 --Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hello! I am using FreeBSD-10-stable and writing a program that uses large data = file via mmap() in read only mode. To be specific, I have 256GB RAM machine and typical size of data file = is ~160GB (more than 1/2 of RAM and less that the whole RAM). There is no other programs running during the test. Consider the following use case: I have two files on disk. I mmap() the = first one and prefetch data to RAM (touch every page of the file). After that I expect all data to be cached in RAM and subsequent access = will be fast. Next I do munmap() on the first file, mmap() the second one and do the = same test: prefetch data and expect it to be cached in RAM (and some of = the pages belonging to the first file to be purged out, because = size_of(file1)+size_of(file2) > size_of(RAM). Please find my test program attached. I run the program with 2 files provided via command line (both about = 160GB). What I observe in real is: -- before I run the program all RAM is in FREE state as reported by = top(1). -- after first prefetch() of the first file, all it's data goes to = "Cache" state, RES column of the process remains the same (small) -- second prefetch() works fast as expected, memory goes from Cache to = Active state, RES column of the process grows up to match file size = (SIZE=3D=3DRES now) -- now first prefetch() for second file starts: the remaining Free = memory goes to Cache state, Active size still equals to first file size. -- second prefetch() for second file works as slow as first one, like if = nothing was cached in memory during the first prefetch() run, RES column = does not change. Here is the output: % /tmp/a.out file1.dat file2.dat file1.dat... First prefault time: 1235.747351 seconds Second prefault time: 74.893323 seconds Ok. file2.dat... First prefault time: 1316.405527 seconds Second prefault time: 1311.491842 seconds Ok. I treat this like the bug somewhere in virtual memory management. Am I = right or my expectations about how that test program should work are = false? Thanks in advance. --Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA Content-Disposition: attachment; filename=mmap_test.c Content-Type: application/octet-stream; name="mmap_test.c" Content-Transfer-Encoding: 7bit #include #include #include #include #include #include #include #include static void prefault(const char *buf, size_t s) { volatile const char *p; size_t page_size; if (s == 0) return; page_size = sysconf(_SC_PAGESIZE); for (p = buf; p < buf + s; p += page_size) *p; *(volatile const char *)(buf + s - 1); } int main(int argc, char* argv[]) { if (argc < 2) { fprintf(stderr, "Usage: %s ...\n", argv[0]); exit(0); } int i, fd; struct stat st; void *p; struct timeval tv1, tv2; for (i=1; i < argc; i++) { printf("%s... ", argv[i]); if ((fd = open(argv[i], O_RDONLY)) < 0) err(1, "open"); if (fstat(fd, &st) != 0) err(1, "fstat"); if (st.st_size > 0) { if ((p = mmap(NULL, st.st_size, PROT_READ, MAP_NOCORE, fd, 0)) == MAP_FAILED) err(1, "mmap"); gettimeofday(&tv1, NULL); prefault(p, st.st_size); gettimeofday(&tv2, NULL); printf("First prefault time: %f seconds\n", (double) (tv2.tv_usec - tv1.tv_usec) / 1000000 + (double) (tv2.tv_sec - tv1.tv_sec)); gettimeofday(&tv1, NULL); prefault(p, st.st_size); gettimeofday(&tv2, NULL); printf("Second prefault time: %f seconds\n", (double) (tv2.tv_usec - tv1.tv_usec) / 1000000 + (double) (tv2.tv_sec - tv1.tv_sec)); if (munmap(p, st.st_size) != 0) err(1, "munmap"); } close(fd); printf("Ok.\n"); } return 0; } --Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA-- From owner-freebsd-hackers@freebsd.org Thu Jul 16 16:16:27 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0E0F9A307D for ; Thu, 16 Jul 2015 16:16:27 +0000 (UTC) (envelope-from bsdcon@bsdcon.com.br) Received: from leviatan.freebsdbrasil.com.br (leviatan.freebsdbrasil.com.br [177.10.156.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 15EE012FC for ; Thu, 16 Jul 2015 16:16:26 +0000 (UTC) (envelope-from bsdcon@bsdcon.com.br) Received: (qmail 34245 invoked from network); 16 Jul 2015 13:09:32 -0300 Received: from unknown (HELO [200.129.11.217]) (contato@bsdcon.com.br@[200.129.11.217]) (envelope-sender ) by capeta.freebsdbrasil.com.br (qmail-ldap-1.03) with SMTP for ; 16 Jul 2015 13:09:32 -0300 Message-ID: <55A7D73B.2050206@bsdcon.com.br> Date: Thu, 16 Jul 2015 13:09:31 -0300 From: BSDCon Brasil 2015 Organization: BSDCon Brasil 2015 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: BSDCon Brasil 2015: =?UTF-8?B?QXByZXNlbnRhw6fDtWVzIC8gVGFsa3M=?= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2015 16:16:27 -0000 https://twitter.com/bsdcon_br/status/620727031630798852 ===== pt Acompanhe as atualizações da conferência através do Twitter ou Facebook (fb.com/bsdconbrasil2015). Gostaria de patrocinar a BSDCon Brasil 2015? Entre em contato conosco através das redes sociais ou mande-nos um email. Escreva para patrocinio@bsdcon.com.br Conheça nossos patrocinadores: http://2015.bsdcon.com.br/patrocinio ===== en Follow conference updates on Twitter or Facebook (fb.com/bsdconbrasil2015). Would you like to become a sponsor? Please write to sponsorship@bsdcon.com.br Meet our sponsors: http://2015.bsdcon.com.br/sponsorship -- BSDCon Brasil 2015 http://2015.bsdcon.com.br http://2015.bsdcon.com.br/start http://twitter.com/bsdcon_br From owner-freebsd-hackers@freebsd.org Thu Jul 16 18:19:18 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 020B69A3B68 for ; Thu, 16 Jul 2015 18:19:18 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D514C1170 for ; Thu, 16 Jul 2015 18:19:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id D23799A3B67; Thu, 16 Jul 2015 18:19:17 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7E429A3B66 for ; Thu, 16 Jul 2015 18:19:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mail-la0-x231.google.com (mail-la0-x231.google.com [IPv6:2a00:1450:4010:c03::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F90A116F for ; Thu, 16 Jul 2015 18:19:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by laem6 with SMTP id m6so48405921lae.0 for ; Thu, 16 Jul 2015 11:19:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=YGesokFBKxDmqEMCGoX+OyJyeyZiqj3iSmphAsJjD6E=; b=fkyMHdK6oSVFW11l013d8kHfBNUvPD5RzhP05xeTvNfRUFzZFj9yKoVbRCZDA1nuJg vaqhSZKOwWJvcWtfKm1n6tYkxQqGO2wmAXnqmcRfaDYzf2V7v9ddebvH37t6ZIhV7v8u gU+OioLGS8KLu2oPvb2pxp3TF2xgqFvedbxv0UNBZj05ZR6uk5W7N5kOFjUvBJ7fK42c +D5Phqxl/9Vb3ziyiuUN4tewItXr6IxUYVXnolkgjuaHDVSSddxu/a73eoD2RHKAaseV 6tUe2hdUsj5XeuVY52cN0jXBEMkJ8LX/086PevTVyNkti65u5gJEUjFsM9ZtHrLuTi7Q 4xBg== X-Received: by 10.112.124.164 with SMTP id mj4mr10476884lbb.3.1437070755357; Thu, 16 Jul 2015 11:19:15 -0700 (PDT) Received: from [10.0.1.6] (broadband-5-228-251-172.nationalcablenetworks.ru. [5.228.251.172]) by smtp.gmail.com with ESMTPSA id ky9sm304694lab.49.2015.07.16.11.19.14 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 16 Jul 2015 11:19:14 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Strange memory management with mmap() From: Dmitry Sivachenko In-Reply-To: Date: Thu, 16 Jul 2015 21:19:13 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <3434ED75-7994-4E9E-9B06-FACCD7DC90FF@gmail.com> References: To: hackers@freebsd.org X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2015 18:19:18 -0000 > On 16 =D0=B8=D1=8E=D0=BB=D1=8F 2015 =D0=B3., at 18:42, Dmitry = Sivachenko wrote: >=20 > Hello! >=20 > I am using FreeBSD-10-stable and writing a program that uses large = data file via mmap() in read only mode. > To be specific, I have 256GB RAM machine and typical size of data file = is ~160GB (more than 1/2 of RAM and less that the whole RAM). > There is no other programs running during the test. >=20 > Consider the following use case: I have two files on disk. I mmap() = the first one and prefetch data to RAM (touch every page of the file). > After that I expect all data to be cached in RAM and subsequent access = will be fast. >=20 > Next I do munmap() on the first file, mmap() the second one and do the = same test: prefetch data and expect it to be cached in RAM (and some of = the pages belonging to the first file to be purged out, because = size_of(file1)+size_of(file2) > size_of(RAM). >=20 > Please find my test program attached. >=20 > I run the program with 2 files provided via command line (both about = 160GB). > What I observe in real is: > -- before I run the program all RAM is in FREE state as reported by = top(1). > -- after first prefetch() of the first file, all it's data goes to = "Cache" state, RES column of the process remains the same (small) > -- second prefetch() works fast as expected, memory goes from Cache to = Active state, RES column of the process grows up to match file size = (SIZE=3D=3DRES now) > -- now first prefetch() for second file starts: the remaining Free = memory goes to Cache state, Active size still equals to first file size. > -- second prefetch() for second file works as slow as first one, like = if nothing was cached in memory during the first prefetch() run, RES = column does not change. >=20 >=20 > Here is the output: > % /tmp/a.out file1.dat file2.dat > file1.dat... First prefault time: 1235.747351 seconds > Second prefault time: 74.893323 seconds > Ok. > file2.dat... First prefault time: 1316.405527 seconds > Second prefault time: 1311.491842 seconds > Ok. >=20 And one more strange observation I made:=20 if I add=20 if (madvise(p, st.st_size, MADV_RANDOM) !=3D 0) err(1, "madvise"); if (madvise(p, st.st_size, MADV_WILLNEED) !=3D 0) err(1, "madvise"); after mmap() call in my test program, these times change as follows: % /tmp/a.out file1.dat file2.dat file1.dat... First prefault time: 2706.270454 seconds Second prefault time: 3.812012 seconds Ok. file2.dat... First prefault time: 4089.096647 seconds Second prefault time: 1132.210888 seconds Ok. From owner-freebsd-hackers@freebsd.org Fri Jul 17 15:32:27 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A0859A474C; Fri, 17 Jul 2015 15:32:27 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from mail.metricspace.net (207-172-209-89.c3-0.arl-ubr1.sbo-arl.ma.static.cable.rcn.com [207.172.209.89]) by mx1.freebsd.org (Postfix) with ESMTP id 749241B1F; Fri, 17 Jul 2015 15:32:26 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from [172.20.3.28] (unknown [208.71.37.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: eric) by mail.metricspace.net (Postfix) with ESMTPSA id B84BF7B3; Fri, 17 Jul 2015 15:32:19 +0000 (UTC) Message-ID: <55A92003.8090504@metricspace.net> Date: Fri, 17 Jul 2015 11:32:19 -0400 From: Eric McCorkle User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "freebsd-mobile@freebsd.org" , "freebsd-hackers@freebsd.org" Subject: Interesting open laptop project Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="vfE4BklrfrbdRLGRWLxeXtxaJoPqrnvVu" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jul 2015 15:32:27 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --vfE4BklrfrbdRLGRWLxeXtxaJoPqrnvVu Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi folks, I ran across this project last week: https://www.crowdsupply.com/purism/librem-15 While a bit pricey, it seems like the best shot at having a laptop that actually really works. My only concern lies with the graphics, as I'm not sure what's going to be supported by the i915 driver updates. Are we likely to get Iris 6100 (Broadwell, if I'm not mistaken) support in the near future? --vfE4BklrfrbdRLGRWLxeXtxaJoPqrnvVu Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJVqSAHAAoJECuREQPg1IcE/loP/0A/rNn2tc9SLf1hC5qXCop/ UcM7NBiEr2beSwR65y0O2oK1kB9TmX9tU0KgNRnv9dDxRssy15WRiLOxmyI9uQWW UdztKGnLi3lr/ZkaK5JZ025IF3KGmutnvVDT8w3pVMqLbHNJxbwmG0MhfYjfWVNI dMfjmhDwHWrRc6/kkmAYhXOeYAc86eqcUkmyqPtHoaffO9kxZ01jWzO76B8RSSIl LHimAOGIlGAodp0b5qQvFE+kHj1ux5ftbjmSQ3TRxQWkQ7a3T/lsKdGJYOJ3ikoF unM2aaNZxhrjNnSjD6SpHt4uDg6KvTJg1zC+yR59o35+Pz9n2/Z4dUvo3O9iW0KH cqmhuPGVHEYeBT2ihzlUuHYhh0kLUYQPY2MqNSBN1nSE8PsTiHy/UkTHKCboh8HO T71S8O8TVBln6DCgCQXEjkdB4eR8oW50dfhYFT8tXFGeTrRSlEpOHnBubtKskKh9 kCrsmDHccyqByXFpIWr5tL59Ok5/ee6P4MKOMkb9zeE7Rko0LlIvcaAcDgqtTm5t HyfbDo/L9BfuXaObW2boW0Cptcv7NlwzJmm1slBTe7hyfVcTroTZYlpWr8TbFNYV akEOtYGTYpzN1Am7RM/nFefiWRGyH8Feh2pLdS06VnIu0fGk9vptQ3pmmQHEb+jp sksQQksjlqKhj2DhBBm0 =sk7M -----END PGP SIGNATURE----- --vfE4BklrfrbdRLGRWLxeXtxaJoPqrnvVu-- From owner-freebsd-hackers@freebsd.org Sat Jul 18 07:12:34 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64A759A437E; Sat, 18 Jul 2015 07:12:34 +0000 (UTC) (envelope-from erichsfreebsdlist@alogt.com) Received: from alogt.com (alogt.com [69.36.191.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CD25D1DDA; Sat, 18 Jul 2015 07:12:33 +0000 (UTC) (envelope-from erichsfreebsdlist@alogt.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=alogt.com; s=default; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=3Osa7liwCRv43PHkP4Mxwc5fKTc4Sqv7dnlwqu3ge5c=; b=ZgN9pGLvwtJaLO1gMSq8pKii/QixLABZav6BJY6fVoFipWEdKb+BXN6kKXKk2z2CvtmtbXhZ3ewS5CpEiFu/9hJDKjmRJzm4DXpyTwgi5tIJoJ084d5hwEgIhu/p/D2V4oXKquY+9WXw52vvvSSR+uIsmgBM/YpWjYD2o9y7xcM=; Received: from [114.121.160.184] (port=56126 helo=X220.alogt.com) by sl-508-2.slc.westdc.net with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.85) (envelope-from ) id 1ZGMIR-001kJx-PR; Sat, 18 Jul 2015 01:12:32 -0600 Date: Sat, 18 Jul 2015 15:12:25 +0800 From: Erich Dollansky To: Eric McCorkle Cc: "freebsd-mobile@freebsd.org" , "freebsd-hackers@freebsd.org" Subject: Re: Interesting open laptop project Message-ID: <20150718151225.522fe671@X220.alogt.com> In-Reply-To: <55A92003.8090504@metricspace.net> References: <55A92003.8090504@metricspace.net> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - sl-508-2.slc.westdc.net X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - alogt.com X-Get-Message-Sender-Via: sl-508-2.slc.westdc.net: authenticated_id: erichsfreebsdlist@alogt.com X-Source: X-Source-Args: X-Source-Dir: X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jul 2015 07:12:34 -0000 Hi, On Fri, 17 Jul 2015 11:32:19 -0400 Eric McCorkle wrote: > Hi folks, > > I ran across this project last week: > > https://www.crowdsupply.com/purism/librem-15 > > > While a bit pricey, it seems like the best shot at having a laptop > that actually really works. My only concern lies with the graphics, > as I'm not sure what's going to be supported by the i915 driver > updates. Are we likely to get Iris 6100 (Broadwell, if I'm not > mistaken) support in the near future? > the price seems ok. It is the size which bothers me. And that it does not have the stick. I hate touchpads and never use them. At least, it seems to go to the right direction. Erich From owner-freebsd-hackers@freebsd.org Sat Jul 18 12:56:05 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF9669A3033; Sat, 18 Jul 2015 12:56:05 +0000 (UTC) (envelope-from tijl@freebsd.org) Received: from mailrelay117.isp.belgacom.be (mailrelay117.isp.belgacom.be [195.238.20.144]) by mx1.freebsd.org (Postfix) with ESMTP id 56F771E27; Sat, 18 Jul 2015 12:56:04 +0000 (UTC) (envelope-from tijl@freebsd.org) X-Belgacom-Dynamic: yes X-Cloudmark-SP-Filtered: true X-Cloudmark-SP-Result: v=1.1 cv=i+EX5EtM4g4P/CrLY+sx2nOWHsqv9AlVwqc0uudqMuE= c=1 sm=2 a=9CNj8_f3AAAA:8 a=s1Xs-aBeAAAA:8 a=Pw90fszaAAAA:8 a=UrY1a8OmyAULDwDnpH4A:9 a=CjuIK1q_8ugA:10 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BUBgB/S6pV/++YsFtbgxNUabtpgXWFdwKBJjsSAQEBAQEBAYEKhCQBAQQ6HCMQCxgJJQ8qHgYTiDIBCMw1AQEBAQEBBAEBAQEBGQSKSoEChQYHhCsBBJRSjCCZCCaDfjwxgksBAQE Received: from 239.152-176-91.adsl-dyn.isp.belgacom.be (HELO kalimero.tijl.coosemans.org) ([91.176.152.239]) by relay.skynet.be with ESMTP; 18 Jul 2015 14:54:56 +0200 Received: from kalimero.tijl.coosemans.org (kalimero.tijl.coosemans.org [127.0.0.1]) by kalimero.tijl.coosemans.org (8.15.2/8.15.2) with ESMTP id t6ICsqDi001798; Sat, 18 Jul 2015 14:54:53 +0200 (CEST) (envelope-from tijl@FreeBSD.org) Date: Sat, 18 Jul 2015 14:54:52 +0200 From: Tijl Coosemans To: Erich Dollansky Cc: Eric McCorkle , "freebsd-hackers@freebsd.org" , "freebsd-mobile@freebsd.org" Subject: Re: Interesting open laptop project Message-ID: <20150718145452.265cc270@kalimero.tijl.coosemans.org> In-Reply-To: <20150718151225.522fe671@X220.alogt.com> References: <55A92003.8090504@metricspace.net> <20150718151225.522fe671@X220.alogt.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jul 2015 12:56:06 -0000 On Sat, 18 Jul 2015 15:12:25 +0800 Erich Dollansky wrote: > On Fri, 17 Jul 2015 11:32:19 -0400 > Eric McCorkle wrote: >> I ran across this project last week: >> >> https://www.crowdsupply.com/purism/librem-15 >> >> >> While a bit pricey, it seems like the best shot at having a laptop >> that actually really works. My only concern lies with the graphics, >> as I'm not sure what's going to be supported by the i915 driver >> updates. Are we likely to get Iris 6100 (Broadwell, if I'm not >> mistaken) support in the near future? > > the price seems ok. It is the size which bothers me. And that it does > not have the stick. I hate touchpads and never use them. There's also a 13" version: https://www.crowdsupply.com/purism/librem-13 From owner-freebsd-hackers@freebsd.org Sat Jul 18 14:23:52 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E8D79A5085; Sat, 18 Jul 2015 14:23:52 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ED1C8109F; Sat, 18 Jul 2015 14:23:51 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: by igbpg9 with SMTP id pg9so54918380igb.0; Sat, 18 Jul 2015 07:23:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=fah4uM0EvYLe7+jmEqoarxSZrjWu0u40d5jWWIGeLyA=; b=RXCvPCGWp4kt+ZfUkhZhsxwOu8X89xTlVmyQE7h+ln6yDZDS1kBLpLTn5Mk+tVgjeW 8yYsmY+JtGAEEPAHl1+fzX8l4k7UEHqYY2fjX12lxW5wk00P9YMqh4DbC0GtBeH0Uo97 bbilNawIcFuRnYP/pxMW+LNlsauttJ/8zEfOzyn9CG2EoF30cYoGcxf3QChfKtZLh5+y MF8aE4fM29Whm1uUzzObq6iYVUxQw4nKGQWxYZOqxblRKq36SwT6YBeB1pG6P2tqM1zu gP+lgfcjGzgNuplefNrcYAfIZOOEVPU/TW2Lh0TycmQJywN5mQ5aWHZ33u8CBYqOsK14 H79A== MIME-Version: 1.0 X-Received: by 10.50.4.102 with SMTP id j6mr3432879igj.40.1437229431232; Sat, 18 Jul 2015 07:23:51 -0700 (PDT) Received: by 10.107.157.201 with HTTP; Sat, 18 Jul 2015 07:23:51 -0700 (PDT) In-Reply-To: <20150718145452.265cc270@kalimero.tijl.coosemans.org> References: <55A92003.8090504@metricspace.net> <20150718151225.522fe671@X220.alogt.com> <20150718145452.265cc270@kalimero.tijl.coosemans.org> Date: Sun, 19 Jul 2015 00:23:51 +1000 Message-ID: Subject: Re: Interesting open laptop project From: Outback Dingo To: Tijl Coosemans Cc: Erich Dollansky , Eric McCorkle , "freebsd-hackers@freebsd.org" , "freebsd-mobile@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jul 2015 14:23:52 -0000 On Sat, Jul 18, 2015 at 10:54 PM, Tijl Coosemans wrote: > On Sat, 18 Jul 2015 15:12:25 +0800 Erich Dollansky < > erichsfreebsdlist@alogt.com> wrote: > > On Fri, 17 Jul 2015 11:32:19 -0400 > > Eric McCorkle wrote: > >> I ran across this project last week: > >> > >> https://www.crowdsupply.com/purism/librem-15 > >> > >> > >> While a bit pricey, it seems like the best shot at having a laptop > >> that actually really works. My only concern lies with the graphics, > >> as I'm not sure what's going to be supported by the i915 driver > >> updates. Are we likely to get Iris 6100 (Broadwell, if I'm not > >> mistaken) support in the near future? > > > > the price seems ok. It is the size which bothers me. And that it does > > not have the stick. I hate touchpads and never use them. > > There's also a 13" version: > https://www.crowdsupply.com/purism/librem-13 seems lik e a decent unit, n bot a bad price, however every intel chipset as of late ive tried to get FreeBSD to run X on has failed miserably, like my Lenovo W540, and an MSI board with built in graphics. So id be inclined to buy one if they would actually run FreeBSD, linux and windows... ( i know, sorry guys) hate to say have always worked fine on them. > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Sat Jul 18 14:10:44 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E8269A3DBE for ; Sat, 18 Jul 2015 14:10:44 +0000 (UTC) (envelope-from holger@layer-acht.org) Received: from alpha.holgerlevsen.de (mail.holgerlevsen.de [62.201.164.66]) by mx1.freebsd.org (Postfix) with ESMTP id 806661BA8; Sat, 18 Jul 2015 14:10:42 +0000 (UTC) (envelope-from holger@layer-acht.org) Received: from localhost (alpha.holgerlevsen.de [62.201.164.66]) by alpha.holgerlevsen.de (Postfix) with ESMTP id 7489ACAD650; Sat, 18 Jul 2015 16:10:34 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at alpha.holgerlevsen.de Received: from alpha.holgerlevsen.de ([62.201.164.66]) by localhost (mail.holgerlevsen.de [62.201.164.66]) (amavisd-new, port 10024) with ESMTP id o8sdsVQHbsG2; Sat, 18 Jul 2015 16:10:24 +0200 (CEST) Received: from matrix.localnet (epsilon.holgerlevsen.de [62.201.164.82]) by alpha.holgerlevsen.de (Postfix) with ESMTP id D49AACAD093; Sat, 18 Jul 2015 16:10:23 +0200 (CEST) From: Holger Levsen To: "freebsd-hackers@freebsd.org" , reproducible-builds@lists.alioth.debian.org Subject: Re: reproducible builds of FreeBSD in a chroot on Linux Date: Sat, 18 Jul 2015 16:09:45 +0200 User-Agent: KMail/1.13.7 (Linux/3.16.0-0.bpo.4-amd64; KDE/4.8.4; x86_64; ; ) References: <201505071122.36037.holger@layer-acht.org> <201506162350.11646.holger@layer-acht.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart4827853.JqjbN4J8qL"; protocol="application/pgp-signature"; micalg=pgp-sha512 Content-Transfer-Encoding: 7bit Message-Id: <201507181609.49815.holger@layer-acht.org> X-Mailman-Approved-At: Sat, 18 Jul 2015 14:45:17 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jul 2015 14:10:44 -0000 --nextPart4827853.JqjbN4J8qL Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, so I made some progress on this: a.) there is a build host running freebsd= =20 10.1 (called freebsd-jenkins.debian.net) now, on which the jenkins user fro= m=20 jenkins.debian.net can login via ssh as jenkins user b.) besides the base=20 system it has "screen git vim sudo denyhosts" installed and c.) the=20 directories /srv/workspace/chroots/ and /srv/reproducible-results have been= =20 created (and are owned by the jenkins user) and d.) /usr/obj/srv is a link = to=20 /srv. With this,=20 http://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/tree/bin/reproduci= ble_freebsd.sh=20 gets as far as=20 https://jenkins.debian.net/view/reproducible/job/reproducible_freebsd/7/con= sole=20 where "stage 2.1: cleaning up the object tree" fails on "make buildworld",= =20 because /srv/workspace/chroots/freebsd- XXXXXXXX.v1adN6Qo/freebsd/lib/libc/tests does not exist. And at this point I'm stuck as to why this happens. Any hint much welcome! (Please note that reproducible_freebsd.sh is just a work-in-progress now an= d=20 there are still some bits from it's source, reproducible_netbsd.sh visible.= =20 This need to be cleaned up, but shouldn't be too confusing know that this i= s=20 clear.) On Mittwoch, 17. Juni 2015, Ed Maste wrote: > > https://wiki.freebsd.org/ReproducibleBuilds claims there are 3 known > > issues (for "make world" AIUI) for HEAD, I would like to build twice and > > verify myself. > I'm interested in fixing the remaining kernel / world issues, with the > kernel being my higher priority. cool! =20 > For the kernel we have the username, hostname, and build timestamp. > The path is included too, but I don't anticipate trying to address it > at first; release builds are done in a consistent location anyhow > (/usr/src). /me nods - that's what we are doing in (reproducible builds for) Debian too= ,=20 the path has to be the same on rebuilds (as it is included in too many buil= d=20 artifacts to deeply.) > These are used only as user-facing strings for the kern.version sysctl > and reported by uname. An example kern.version string: > FreeBSD 10.1-STABLE #28 r280427+86df2de(stable-10): Thu Mar 26 16:07:47 E= DT > 2015 > emaste@feynman:/tank/emaste/obj/tank/emaste/src/git-stable-10/sys/GENERIC >=20 > From a technical perspective they're trivially eliminated. There may > be some 3rd party ports expect the precise format, but probably not > very many (and they should be fixed, anyhow). There's a much larger > social issue in convincing the FreeBSD developer community to accept > their removal, though :-) If any build (of the same sources) results in the exact same bits, the buil= d=20 time becomes meaningless and thus a.) can be dropped or b.) replaced with t= he=20 date of the last modification of the sources - which is meaningful informat= ion=20 again! While this is/was a new thought for most everyone (me included...) in my=20 experience it also has been convincing logic for most everyone. The technic= al=20 details to achieve this are sometimes a bit harder to achieve, but not=20 impossible. (eg they differ whether git, svn or tarballs are the means to g= et=20 access to sources.) In Debian we want 100% bit identical packages (=3D.deb files) as this allow= s us=20 to only require a checksum comparison to see whether two builds created=20 reproducible results. > > https://wiki.freebsd.org/PortsReproducibleBuilds says "Of the 23599 > > packages which were built in both runs, 15164 have the same checksum > > when using the previously mentioned patch, giving 64.25% reproducible > > packages." - I'm also curious to re-confirm this - and set up a test > > bed, which can be triggered regularily and easily. Our jenkins set up > > allows this and I'm interested to do this. >=20 > I'm pleasantly surprised by the ports results -- 64.25% seems quite > good for such a straightforward change. The test there is on the same > host though, and so avoids any non-reproducibility from host/user/path > leaks. ah > > My interest is to help FreeBSD with reproducible builds as I want to see > > reproducible builds become the norm in the free software world and as I > > believe FreeBSD is an important part of this world. And also because I'm > > curious. :) >=20 > Great! Hopefully we can help lend some weight in convincing upstream > projects to accept reproducibility patches (once we get further along > in our ports effort). I'm looking forward to see this happen! ;-) cheers, Holger --nextPart4827853.JqjbN4J8qL Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIVAwUAVapeLQkauFYGmqocAQoKbA/+L//t613SZ3MeYETSLn9fz9jRllyaFLQq USIkeRfahmsGLeRbTqwwosvnAPhUv4xcKAeeC4ofLzo4pb8ySNhpl+JyZ90vMCGj +tiyZCaPwBp9EHiHIup78rsFJZOaBP1CfhnrWOIn1uwQJ7puopQ2pnYfXdydy+uQ grWxY3YzkXTDFLr+biDeaLyg2Qi0MTexvf5udShoEpZdI8AS2+AIVpzwwaCa+2t9 V8VS6x23o4uB65TP+DqKKb+u7Lg43d0/lc+ZdAEHxMFPWSXO2BAVTXfIPW2Nnw1B PbAJJF3jEsZyFVFLrkgKTkQyVH0yK1wFfiQqq8TqWZRca/tL3gi5gcqFMuiD+x9N Cmste0dcUnNv7a4YrcWQB7wGIzlVhQZehkCDoNBdtcJSPOOtCOMBj5Mh2GoSiaMQ dUgWxqy0scWv4tSTCIO8R/J5wa+2hURS2iDc+hMajXSjWgYLlWixMC/uNCDubghA OWVHGoWV1yDAdBkyKMSe2/yysPUP4xmKqCf97fQcyjXHNDbsrsHLabEH30YEn1ML S7mtFBNeSP2Ia6suvgzt9Ugp+7UkwPSYpiVIrRw4Jf+QzZ073BVoto9aq9wc5f8E tt63Yd1jRasmTuB+ZyT0IUO92Sexm9vo7SKl+NAQRWuiUuVxhr99RFxe0vDiudso Mfn7W96ZgOs= =mucW -----END PGP SIGNATURE----- --nextPart4827853.JqjbN4J8qL-- From owner-freebsd-hackers@freebsd.org Sat Jul 18 18:09:35 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 442949A5750 for ; Sat, 18 Jul 2015 18:09:35 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 05F921BCD; Sat, 18 Jul 2015 18:09:34 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t6II9Swb061385 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 18 Jul 2015 11:09:28 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t6II9SMi061384; Sat, 18 Jul 2015 11:09:28 -0700 (PDT) (envelope-from jmg) Date: Sat, 18 Jul 2015 11:09:28 -0700 From: John-Mark Gurney To: Ed Maste Cc: Holger Levsen , "freebsd-hackers@freebsd.org" , reproducible-builds@lists.alioth.debian.org Subject: Re: reproducible builds of FreeBSD in a chroot on Linux Message-ID: <20150718180928.GE8523@funkthat.com> References: <201505071122.36037.holger@layer-acht.org> <554B509B.8020608@fuckner.net> <201506162350.11646.holger@layer-acht.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Sat, 18 Jul 2015 11:09:28 -0700 (PDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jul 2015 18:09:35 -0000 Ed Maste wrote this message on Wed, Jun 17, 2015 at 16:48 -0400: > These are used only as user-facing strings for the kern.version sysctl > and reported by uname. An example kern.version string: > FreeBSD 10.1-STABLE #28 r280427+86df2de(stable-10): Thu Mar 26 16:07:47 EDT 2015 > emaste@feynman:/tank/emaste/obj/tank/emaste/src/git-stable-10/sys/GENERIC > > >From a technical perspective they're trivially eliminated. There may > be some 3rd party ports expect the precise format, but probably not > very many (and they should be fixed, anyhow). There's a much larger > social issue in convincing the FreeBSD developer community to accept > their removal, though :-) I don't know about others, but IMO, the only useful information there is the path it was built from... The machine isn't too useful and even less useful is probably the build user... Maybe on larger installs, the user/machine makes a difference, but that could be a config option to include those... So my vote is to eliminate user/machine and just leave the path... And we could just use user@machine to keep the format compatible, but constant... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."