From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 07:58:36 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7FEF81065687; Sun, 5 Feb 2012 07:58:36 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 53BF38FC1A; Sun, 5 Feb 2012 07:58:36 +0000 (UTC) Received: from julian-mac.elischer.org (c-67-180-24-15.hsd1.ca.comcast.net [67.180.24.15]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id q157Pg9o032775 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 4 Feb 2012 23:25:43 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4F2E2F44.6040007@freebsd.org> Date: Sat, 04 Feb 2012 23:27:00 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.25) Gecko/20111213 Thunderbird/3.1.17 MIME-Version: 1.0 To: Poul-Henning Kamp References: <12192.1328375145@critter.freebsd.dk> In-Reply-To: <12192.1328375145@critter.freebsd.dk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: hackers@freebsd.org Subject: Re: A dual-ISP hack with jail/vnet and ipfw X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 07:58:36 -0000 On 2/4/12 9:05 AM, Poul-Henning Kamp wrote: > Natd(8) knows how to deal with multiple NAT instances for different > interfaces, which is useful when you have multiple ISPs. > > The problem with it, is that it becomes incredibly hairy to configure > your IPFW rules, in particular if you have other policy to implement > too. this is sort of what I did when I switched ISPs recently, and had a transition period.. I had a jail/vnet for each ISP. and just switched at the top level an unexpected advantage was that sessions from the main machine were 'one hop' away from the disruption when I screwed things so instead of getting terminated when teh rules/routes were screwed, they just 'hung' until I fixed things. Much like they do when there is internet disruption between sites. I've meant to do something cleaner like this for a while.. good move. > I spent some quality time with a 9.0-Stable nanobsd image today, > and the script below is my proof of concept of a simpler way to > do that. > > The idea is to let a jail deal with the two ISPs and use an epair > to deliver a "normal default route interface" to the rest of the > firewall, making its configuration simpler and easier to understand. > [...] From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 09:02:51 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92C0C106566C for ; Sun, 5 Feb 2012 09:02:51 +0000 (UTC) (envelope-from rank1seeker@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2C7438FC08 for ; Sun, 5 Feb 2012 09:02:50 +0000 (UTC) Received: by eekb47 with SMTP id b47so2102987eek.13 for ; Sun, 05 Feb 2012 01:02:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:from:to:subject:date:content-type :content-transfer-encoding:x-mailer; bh=3oVQqCOu5UT+tCiRzcek9awhA8IfjQ3Vfzw6NAt7TqE=; b=RVY5ZfWmjRbrXP4ZAX6afs4aQ5/VHV8HCOO2GdqGXoy45nocmACMmKhIDlfX8zfIy7 kY07W66b8/BpexE9cxIySaSKuHLMWt5Sy6u4PCNjI0S0L91NA9KMlCAf670G/U6Dg/nB Gk3qIXd7sbpck+tn/7lKyvosa79kaPfMsVZlQ= Received: by 10.14.131.13 with SMTP id l13mr4173505eei.45.1328432569933; Sun, 05 Feb 2012 01:02:49 -0800 (PST) Received: from DOMYPC ([82.193.208.173]) by mx.google.com with ESMTPS id x4sm45578707eeb.4.2012.02.05.01.02.47 (version=SSLv3 cipher=OTHER); Sun, 05 Feb 2012 01:02:48 -0800 (PST) Message-ID: <20120205.090250.960.1@DOMY-PC> From: rank1seeker@gmail.com To: hackers@freebsd.org Date: Sun, 05 Feb 2012 10:02:50 +0100 Content-Type: text/plain; charset="Windows-1250" Content-Transfer-Encoding: quoted-printable X-Mailer: POP Peeper (3.8.1.0) Cc: Subject: 9.0 doesn't crosscompile X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 09:02:51 -0000 On i386, kernel build fails, with = msg:=0D=0A--=0D=0A/usr/src/sys/compat/ia32/ia32_genassym.c:1: error: code = model 'kernel' not supported in the 32 bit = mode=0D=0A--=0D=0A=0D=0A=0D=0ADomagoj Smol=E8i=E6 From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 10:41:40 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3C221065670 for ; Sun, 5 Feb 2012 10:41:40 +0000 (UTC) (envelope-from janm-freebsd-hackers@transactionware.com) Received: from midgard.transactionware.com (mail2.transactionware.com [203.14.245.36]) by mx1.freebsd.org (Postfix) with SMTP id EFA2A8FC0A for ; Sun, 5 Feb 2012 10:41:39 +0000 (UTC) Received: (qmail 92974 invoked by uid 907); 5 Feb 2012 10:41:37 -0000 Received: from b13FC.static.pacific.net.au (HELO [192.168.1.154]) (202.7.88.252) (smtp-auth username janm, mechanism plain) by midgard.transactionware.com (qpsmtpd/0.84) with (AES128-SHA encrypted) ESMTPSA; Sun, 05 Feb 2012 21:41:37 +1100 Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=windows-1252 From: Jan Mikkelsen In-Reply-To: Date: Sun, 5 Feb 2012 21:41:57 +1100 Content-Transfer-Encoding: quoted-printable Message-Id: <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> References: <201201110806.30620.jhb@freebsd.org> To: freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.1251.1) Cc: Garrett Cooper , Xin LI , Ivan Voras , davidxu@freebsd.org Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 10:41:40 -0000 On 12/01/2012, at 3:47 AM, Garrett Cooper wrote: > [ builds hanging in python with waf =85 ] > Glad to see that iXsystems isn't the only one ([1] -- please add a "me > too" to the PR). The problem is that we do FreeNAS nightlies and they > frequently get stuck building tdb (10%~20% of the time) and it sticks > when doing interactive builds as well. The issue appears to be > exacerbated when we have more builds running in parallel on the same > machine. I've also run into the same issue compiling talloc because it > uses the same waf infrastructure as tdb, which was designed to "speed > things up by forcing builds to be parallelized" (It builds > kern.smp.ncpus jobs instead of -j 1). Furthermore, it seems to occur > regardless of whether or not we have the WITH_SEM enabled in python or > not (build.ix's copy of python doesn't have it enabled, but > streetfighter.ix, my system bayonetta, etc do). Any progress on this? Or a workaround? Just had another build stuck on this =85 Jan. From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 10:44:10 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 955D7106564A; Sun, 5 Feb 2012 10:44:10 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 40D7D8FC0A; Sun, 5 Feb 2012 10:44:09 +0000 (UTC) Received: by obcwo16 with SMTP id wo16so8320735obc.13 for ; Sun, 05 Feb 2012 02:44:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=27lq39PhnuUq3bj2iFIYRgmo5GP+JmXZdrD7Rhqhybo=; b=g5MEiDhaThXWKxUsFkpT1pFKW+arRqG+ToMZhbsORwDn3yvJjXUna17wpWrxe8eHWQ jvCaPlj3IBjkLv8hO/L3KGSl+k5bh8Zn1YLvaFS9zCoQkA8P2yxkK4rRDBIcZVqIliYa lJ6FVUpse26a7fQzFfqB0gNT9kCF9TlKYedf0= MIME-Version: 1.0 Received: by 10.182.49.1 with SMTP id q1mr12854023obn.48.1328438649574; Sun, 05 Feb 2012 02:44:09 -0800 (PST) Received: by 10.182.46.163 with HTTP; Sun, 5 Feb 2012 02:44:09 -0800 (PST) In-Reply-To: <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> Date: Sun, 5 Feb 2012 02:44:09 -0800 Message-ID: From: Garrett Cooper To: Jan Mikkelsen Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Xin LI , Ivan Voras , davidxu@freebsd.org Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 10:44:10 -0000 On Sun, Feb 5, 2012 at 2:41 AM, Jan Mikkelsen wrote: > > On 12/01/2012, at 3:47 AM, Garrett Cooper wrote: > >> [ builds hanging in python with waf =85 ] > >> Glad to see that iXsystems isn't the only one ([1] -- please add a "me >> too" to the PR). The problem is that we do FreeNAS nightlies and they >> frequently get stuck building tdb (10%~20% of the time) and it sticks >> when doing interactive builds as well. The issue appears to be >> exacerbated when we have more builds running in parallel on the same >> machine. I've also run into the same issue compiling talloc because it >> uses the same waf infrastructure as tdb, which was designed to "speed >> things up by forcing builds to be parallelized" (It builds >> kern.smp.ncpus jobs instead of -j 1). Furthermore, it seems to occur >> regardless of whether or not we have the WITH_SEM enabled in python or >> not (build.ix's copy of python doesn't have it enabled, but >> streetfighter.ix, my system bayonetta, etc do). > > Any progress on this? Or a workaround? > > Just had another build stuck on this =85 'make MAKE_JOBS_NUMBER=3D1' is the workground used right now.. Thanks, -Garrett From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 10:49:50 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C29C91065676 for ; Sun, 5 Feb 2012 10:49:50 +0000 (UTC) (envelope-from janm-freebsd-hackers@transactionware.com) Received: from midgard.transactionware.com (mail2.transactionware.com [203.14.245.36]) by mx1.freebsd.org (Postfix) with SMTP id 0A75E8FC0C for ; Sun, 5 Feb 2012 10:49:49 +0000 (UTC) Received: (qmail 93077 invoked by uid 907); 5 Feb 2012 10:49:49 -0000 Received: from b13FC.static.pacific.net.au (HELO [192.168.1.154]) (202.7.88.252) (smtp-auth username janm, mechanism plain) by midgard.transactionware.com (qpsmtpd/0.84) with (AES128-SHA encrypted) ESMTPSA; Sun, 05 Feb 2012 21:49:49 +1100 Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=windows-1252 From: Jan Mikkelsen In-Reply-To: Date: Sun, 5 Feb 2012 21:50:09 +1100 Content-Transfer-Encoding: quoted-printable Message-Id: <322DFFC9-5ABC-4B12-B593-F9383B732767@transactionware.com> References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> To: Garrett Cooper X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-hackers@freebsd.org, Xin LI , Ivan Voras , davidxu@freebsd.org Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 10:49:50 -0000 On 05/02/2012, at 9:44 PM, Garrett Cooper wrote: > On Sun, Feb 5, 2012 at 2:41 AM, Jan Mikkelsen > wrote: >>=20 >> On 12/01/2012, at 3:47 AM, Garrett Cooper wrote: >>=20 >>> [ builds hanging in python with waf =85 ] >>=20 >> Any progress on this? Or a workaround? >>=20 >> Just had another build stuck on this =85 >=20 > 'make MAKE_JOBS_NUMBER=3D1' is the workground used right now.. Great, thanks. Jan. From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 12:03:06 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 277061065672; Sun, 5 Feb 2012 12:03:06 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id CE8A58FC13; Sun, 5 Feb 2012 12:03:05 +0000 (UTC) Received: by obcwo16 with SMTP id wo16so8357863obc.13 for ; Sun, 05 Feb 2012 04:03:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; bh=0k2a+ByiI2e1RH6ywR0wa7kewUN/z6lL0QUyY26IwqI=; b=aueavf0N5Cg4ph8UEv7dEf2Saok+Kb8k3jz4D+PV757BAQc8J8aSLwv8By5kj7O32g l5iweg7IRKEe7dBdbxaV1tIU3Hk1tAtUYBDcA2G3MqPlAaapAur3hA8LTrs0MeyESdFQ rsPvtfpcSAmQQH/gWIO6e1MTvYlAX+g0juJo8= Received: by 10.182.132.105 with SMTP id ot9mr13158584obb.34.1328443385220; Sun, 05 Feb 2012 04:03:05 -0800 (PST) MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.182.1.66 with HTTP; Sun, 5 Feb 2012 04:02:25 -0800 (PST) In-Reply-To: References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> From: Ivan Voras Date: Sun, 5 Feb 2012 13:02:25 +0100 X-Google-Sender-Auth: 58t-trazw2H4r5essy1gE812ez8 Message-ID: To: Garrett Cooper Content-Type: multipart/mixed; boundary=14dae93b58c8148ec504b83654c2 Cc: freebsd-hackers@freebsd.org, Xin LI , davidxu@freebsd.org, Jan Mikkelsen Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 12:03:06 -0000 --14dae93b58c8148ec504b83654c2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 5 February 2012 11:44, Garrett Cooper wrote: > > =C2=A0 =C2=A0'make MAKE_JOBS_NUMBER=3D1' is the workground used right now= .. David Xu suggested that it is a bug in Python - it doesn't set process-shared attribute when it calls sem_init(), but i've tried patching it (replacing the port patchfile file the one I've attached) and I still get the hang. --14dae93b58c8148ec504b83654c2 Content-Type: application/octet-stream; name="patch-Python_thread__pthread.h.new" Content-Disposition: attachment; filename="patch-Python_thread__pthread.h.new" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gya0jz5g0 LS0tIFB5dGhvbi90aHJlYWRfcHRocmVhZC5oLm9yaWcJMjAxMi0wMi0wNSAxMjo0NjowNy4wMDAw MDAwMDAgKzAxMDAKKysrIFB5dGhvbi90aHJlYWRfcHRocmVhZC5oCTIwMTItMDItMDUgMTI6NDQ6 NTguMDAwMDAwMDAwICswMTAwCkBAIC0zOCwxMyArMzgsMTggQEAKICNlbmRpZgogI2VuZGlmCiAK KyNpZmRlZiBfX0ZyZWVCU0RfXworI2luY2x1ZGUgPG9zcmVsZGF0ZS5oPgorI2VuZGlmCisKIC8q IFRoZSBQT1NJWCBzcGVjIHNheXMgdGhhdCBpbXBsZW1lbnRhdGlvbnMgc3VwcG9ydGluZyB0aGUg c2VtXyoKICAgIGZhbWlseSBvZiBmdW5jdGlvbnMgbXVzdCBpbmRpY2F0ZSB0aGlzIGJ5IGRlZmlu aW5nCiAgICBfUE9TSVhfU0VNQVBIT1JFUy4gKi8KICNpZmRlZiBfUE9TSVhfU0VNQVBIT1JFUwog LyogT24gRnJlZUJTRCA0LngsIF9QT1NJWF9TRU1BUEhPUkVTIGlzIGRlZmluZWQgZW1wdHksIHNv CiAgICB3ZSBuZWVkIHRvIGFkZCAwIHRvIG1ha2UgaXQgd29yayB0aGVyZSBhcyB3ZWxsLiAqLwot I2lmIChfUE9TSVhfU0VNQVBIT1JFUyswKSA9PSAtMQorI2lmIGRlZmluZWQoX19GcmVlQlNEX18p ICYmIF9fRnJlZUJTRF92ZXJzaW9uIDwgNzAxMTA0ICYmIFwKKyAgICAoX1BPU0lYX1NFTUFQSE9S RVMrMCkgPT0gLTEKICNkZWZpbmUgSEFWRV9CUk9LRU5fUE9TSVhfU0VNQVBIT1JFUwogI2Vsc2UK ICNpbmNsdWRlIDxzZW1hcGhvcmUuaD4KQEAgLTU2LDcgKzYxLDYgQEAKICAgIGluIGRlZmF1bHQg c2V0dGluZy4gIFNvIHRoZSBwcm9jZXNzIHNjb3BlIGlzIHByZWZlcnJlZCB0byBnZXQKICAgIGVu b3VnaCBudW1iZXIgb2YgdGhyZWFkcyB0byB3b3JrLiAqLwogI2lmZGVmIF9fRnJlZUJTRF9fCi0j aW5jbHVkZSA8b3NyZWxkYXRlLmg+CiAjaWYgX19GcmVlQlNEX3ZlcnNpb24gPj0gNTAwMDAwICYm IF9fRnJlZUJTRF92ZXJzaW9uIDwgNTA0MTAxCiAjdW5kZWYgUFRIUkVBRF9TWVNURU1fU0NIRURf U1VQUE9SVEVECiAjZW5kaWYKQEAgLTE2MSw2ICsxNjUsNyBAQAogewogICAgIHB0aHJlYWRfdCB0 aDsKICAgICBpbnQgc3RhdHVzOworICAgIHNpZ3NldF90IHNldCwgb3NldDsKICNpZiBkZWZpbmVk KFRIUkVBRF9TVEFDS19TSVpFKSB8fCBkZWZpbmVkKFBUSFJFQURfU1lTVEVNX1NDSEVEX1NVUFBP UlRFRCkKICAgICBwdGhyZWFkX2F0dHJfdCBhdHRyczsKICNlbmRpZgpAQCAtMTg5LDYgKzE5NCw4 IEBACiAjaWYgZGVmaW5lZChQVEhSRUFEX1NZU1RFTV9TQ0hFRF9TVVBQT1JURUQpCiAgICAgcHRo cmVhZF9hdHRyX3NldHNjb3BlKCZhdHRycywgUFRIUkVBRF9TQ09QRV9TWVNURU0pOwogI2VuZGlm CisgICAgc2lnZmlsbHNldCgmc2V0KTsKKyAgICBTRVRfVEhSRUFEX1NJR01BU0soU0lHX0JMT0NL LCAmc2V0LCAmb3NldCk7CiAKICAgICBzdGF0dXMgPSBwdGhyZWFkX2NyZWF0ZSgmdGgsCiAjaWYg ZGVmaW5lZChUSFJFQURfU1RBQ0tfU0laRSkgfHwgZGVmaW5lZChQVEhSRUFEX1NZU1RFTV9TQ0hF RF9TVVBQT1JURUQpCkBAIC0yMDAsNiArMjA3LDcgQEAKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgKHZvaWQgKilhcmcKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKTsKIAorICAg IFNFVF9USFJFQURfU0lHTUFTSyhTSUdfU0VUTUFTSywgJm9zZXQsIE5VTEwpOwogI2lmIGRlZmlu ZWQoVEhSRUFEX1NUQUNLX1NJWkUpIHx8IGRlZmluZWQoUFRIUkVBRF9TWVNURU1fU0NIRURfU1VQ UE9SVEVEKQogICAgIHB0aHJlYWRfYXR0cl9kZXN0cm95KCZhdHRycyk7CiAjZW5kaWYKQEAgLTI2 NSw3ICsyNzMsNyBAQAogICAgIGxvY2sgPSAoc2VtX3QgKiltYWxsb2Moc2l6ZW9mKHNlbV90KSk7 CiAKICAgICBpZiAobG9jaykgewotICAgICAgICBzdGF0dXMgPSBzZW1faW5pdChsb2NrLDAsMSk7 CisgICAgICAgIHN0YXR1cyA9IHNlbV9pbml0KGxvY2ssMSwxKTsKICAgICAgICAgQ0hFQ0tfU1RB VFVTKCJzZW1faW5pdCIpOwogCiAgICAgICAgIGlmIChlcnJvcikgewo= --14dae93b58c8148ec504b83654c2-- From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 12:41:20 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20FB2106566B; Sun, 5 Feb 2012 12:41:20 +0000 (UTC) (envelope-from listlog2011@gmail.com) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 071438FC1A; Sun, 5 Feb 2012 12:41:20 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q15CfHIV083142; Sun, 5 Feb 2012 12:41:17 GMT (envelope-from listlog2011@gmail.com) Message-ID: <4F2E78EB.3060007@gmail.com> Date: Sun, 05 Feb 2012 20:41:15 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: Ivan Voras References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Garrett Cooper , freebsd-hackers@freebsd.org, Xin LI , davidxu@freebsd.org, Jan Mikkelsen Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: davidxu@freebsd.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 12:41:20 -0000 On 2012/2/5 20:02, Ivan Voras wrote: > On 5 February 2012 11:44, Garrett Cooper wrote: > >> 'make MAKE_JOBS_NUMBER=1' is the workground used right now.. > David Xu suggested that it is a bug in Python - it doesn't set > process-shared attribute when it calls sem_init(), but i've tried > patching it (replacing the port patchfile file the one I've attached) > and I still get the hang. Although I don't know where is the python bug, since I don't know the piece of source code. But general rule to use anonymous shared semaphore between forked processes is the semaphore should be initialized in shared memory page and sem_init() with pshared set 1, such as: sem_ptr = mmap(MMAP_SHARED); sem_init(sem_ptr, pshared=1, init_value); Regards, David Xu From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 16:49:25 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CB341065678 for ; Sun, 5 Feb 2012 16:49:25 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7FBAB8FC1E for ; Sun, 5 Feb 2012 16:49:23 +0000 (UTC) Received: by werm13 with SMTP id m13so6072282wer.13 for ; Sun, 05 Feb 2012 08:49:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=ZjSyFtEQurqxppA0JOEca+njceqKpO5ZKd9teLjXhlg=; b=PFp6ifV8nKDPIBf1+9eWGvCMN3mwcFDdK/zjtfra/hdTGr2SqYjZTD71Q/4acuf283 oYKjrx2OIxZV5tRQGaE3+WQWYhUAIGVCwqHrU8VAvgdzcele/Tms20a+OyjqDh/ST/IA 3tdU1e6N6v5lGU4aco04QKhVRJKZ2Xlh+35ho= MIME-Version: 1.0 Received: by 10.216.137.219 with SMTP id y69mr5648560wei.51.1328460563222; Sun, 05 Feb 2012 08:49:23 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.177.73 with HTTP; Sun, 5 Feb 2012 08:49:23 -0800 (PST) In-Reply-To: References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> Date: Sun, 5 Feb 2012 17:49:23 +0100 X-Google-Sender-Auth: ZyyhwTeudYSAbHSs8t66zxiiI6E Message-ID: From: Attilio Rao To: Ivan Voras Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Garrett Cooper , freebsd-hackers@freebsd.org, Xin LI , davidxu@freebsd.org, Jan Mikkelsen Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 16:49:25 -0000 2012/2/5 Ivan Voras : > On 5 February 2012 11:44, Garrett Cooper wrote: > >> >> =C2=A0 =C2=A0'make MAKE_JOBS_NUMBER=3D1' is the workground used right no= w.. > > David Xu suggested that it is a bug in Python - it doesn't set > process-shared attribute when it calls sem_init(), but i've tried > patching it (replacing the port patchfile file the one I've attached) > and I still get the hang. Guys, it would be valuable if you do the following: 1) recompile your kernel with INVARIANTS, WITNESS and without WITNESS_SKIPS= PIN 2a) If you have a serial console, please run the DDB stuff through it (go to point 3) 2b) If you don't have a serial console please run the DDB stuff in textdump (go to point 3) 3) Collect the following informations: - show allpcpu - show alllocks - ps - alltrace 3a) If you had the serial console (thus not textdump) please collect the coredump with: call doadump 4) reset your machine You will end up with the textdump or coredump + all the serial logs necessary to debug this. If you cannot reproduce your issue with WITNESS enabled, please remove from your kernel config and avoid to call 'show alllocks' when in DDB. But try to leave INVARIANTS on. Hope this helps, Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 18:32:53 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 626CC106564A; Sun, 5 Feb 2012 18:32:53 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id 101A68FC13; Sun, 5 Feb 2012 18:32:52 +0000 (UTC) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.4/8.14.4/NETPLEX) with ESMTP id q15IWgR5014601; Sun, 5 Feb 2012 13:32:42 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.2.6 (mail.netplex.net [204.213.176.10]); Sun, 05 Feb 2012 13:32:43 -0500 (EST) Date: Sun, 5 Feb 2012 13:32:42 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Ivan Voras In-Reply-To: Message-ID: References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> MIME-Version: 1.0 Content-Type: MULTIPART/Mixed; boundary=14dae93b58c8148ec504b83654c2 Content-ID: Cc: Garrett Cooper , freebsd-hackers@freebsd.org, Xin LI , davidxu@freebsd.org, Jan Mikkelsen Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 18:32:53 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --14dae93b58c8148ec504b83654c2 Content-Type: TEXT/PLAIN; charset=UTF-8; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Content-ID: On Sun, 5 Feb 2012, Ivan Voras wrote: > On 5 February 2012 11:44, Garrett Cooper wrote: > >> >> =C2=A0 =C2=A0'make MAKE_JOBS_NUMBER=3D1' is the workground used right no= w.. > > David Xu suggested that it is a bug in Python - it doesn't set > process-shared attribute when it calls sem_init(), but i've tried > patching it (replacing the port patchfile file the one I've attached) > and I still get the hang. I don't understand how process shared semaphores can work. Perhaps I'm dumb and ignorant, but a sem_id_t is an allocated struct. The actual kernel sem_id is inside the struct. Isn't this the same reason pthread_mutex_t and pthread_cond_t cannot be process-shared? --=20 DE --14dae93b58c8148ec504b83654c2-- From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 19:00:31 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAC4F1065720; Sun, 5 Feb 2012 19:00:31 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) by mx1.freebsd.org (Postfix) with ESMTP id CC4108FC33; Sun, 5 Feb 2012 19:00:23 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 62BE91DEDDB; Sun, 5 Feb 2012 20:00:22 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 49E9428468; Sun, 5 Feb 2012 20:00:22 +0100 (CET) Date: Sun, 5 Feb 2012 20:00:22 +0100 From: Jilles Tjoelker To: Daniel Eischen Message-ID: <20120205190022.GC57421@stack.nl> References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Garrett Cooper , freebsd-hackers@freebsd.org, Ivan Voras , Xin LI , Jan Mikkelsen , davidxu@freebsd.org Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 19:00:32 -0000 On Sun, Feb 05, 2012 at 01:32:42PM -0500, Daniel Eischen wrote: > On Sun, 5 Feb 2012, Ivan Voras wrote: > > On 5 February 2012 11:44, Garrett Cooper wrote: > >>    'make MAKE_JOBS_NUMBER=1' is the workground used right now.. > > David Xu suggested that it is a bug in Python - it doesn't set > > process-shared attribute when it calls sem_init(), but i've tried > > patching it (replacing the port patchfile file the one I've attached) > > and I still get the hang. > I don't understand how process shared semaphores can work. Perhaps > I'm dumb and ignorant, but a sem_id_t is an allocated struct. The > actual kernel sem_id is inside the struct. Isn't this the same > reason pthread_mutex_t and pthread_cond_t cannot be process-shared? That's how the old implementation works. It does not support process-shared semaphores although they may happen to work in some specific cases. However, in 9.0, sem_t works differently and contains the actual lock word directly, so that process-shared semaphores work. The implementation is in lib/libc/gen/sem_new.c. The pshared flag to sem_init() is not a no-op because it tells the kernel to allow for use from multiple processes. Note that the old implementation is still present as well, for compatibility with old binaries. -- Jilles Tjoelker From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 5 19:29:50 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49DAB1065677; Sun, 5 Feb 2012 19:29:50 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id D67A48FC18; Sun, 5 Feb 2012 19:29:49 +0000 (UTC) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.4/8.14.4/NETPLEX) with ESMTP id q15JThjX036757; Sun, 5 Feb 2012 14:29:43 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.2.6 (mail.netplex.net [204.213.176.10]); Sun, 05 Feb 2012 14:29:44 -0500 (EST) Date: Sun, 5 Feb 2012 14:29:43 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Jilles Tjoelker In-Reply-To: <20120205190022.GC57421@stack.nl> Message-ID: References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> <20120205190022.GC57421@stack.nl> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-851401618-1328470183=:28760" Cc: Garrett Cooper , freebsd-hackers@freebsd.org, Ivan Voras , Xin LI , Jan Mikkelsen , davidxu@freebsd.org Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Feb 2012 19:29:50 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---559023410-851401618-1328470183=:28760 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Sun, 5 Feb 2012, Jilles Tjoelker wrote: > On Sun, Feb 05, 2012 at 01:32:42PM -0500, Daniel Eischen wrote: >> On Sun, 5 Feb 2012, Ivan Voras wrote: >>> On 5 February 2012 11:44, Garrett Cooper wrote: > >>>> =A0 =A0'make MAKE_JOBS_NUMBER=3D1' is the workground used right now.. > >>> David Xu suggested that it is a bug in Python - it doesn't set >>> process-shared attribute when it calls sem_init(), but i've tried >>> patching it (replacing the port patchfile file the one I've attached) >>> and I still get the hang. > >> I don't understand how process shared semaphores can work. Perhaps >> I'm dumb and ignorant, but a sem_id_t is an allocated struct. The >> actual kernel sem_id is inside the struct. Isn't this the same >> reason pthread_mutex_t and pthread_cond_t cannot be process-shared? > > That's how the old implementation works. It does not support > process-shared semaphores although they may happen to work in some > specific cases. > > However, in 9.0, sem_t works differently and contains the actual lock > word directly, so that process-shared semaphores work. The > implementation is in lib/libc/gen/sem_new.c. The pshared flag to > sem_init() is not a no-op because it tells the kernel to allow for use > from multiple processes. > > Note that the old implementation is still present as well, for > compatibility with old binaries. Ahh, okay, I was looking at the old implementation in libc. --=20 DE ---559023410-851401618-1328470183=:28760-- From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 07:04:37 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 88C1B106564A; Mon, 6 Feb 2012 07:04:37 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id E24628FC18; Mon, 6 Feb 2012 07:04:36 +0000 (UTC) Received: by eekb47 with SMTP id b47so2443995eek.13 for ; Sun, 05 Feb 2012 23:04:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=SC4GdLGrv3QFmxwdGo3U5/7IxlNvMjqlzX93WPwOXWE=; b=n9SE1eBBxDbrwiORaoKOCbXWUeX/Nz+3cRdCPAlDYSwir2jKfo++DJdEz7PgfFJSaD qLgBTyG94TpefrYdjmYJg2OaqCebdIRkURMbqPwaJiqdRoe7f2SvZnb4FrZSnA+Urn9m 6MyyV9gATQa7+6z5KoFpSMRk32vBQLBIEQ5oc= Received: by 10.14.48.8 with SMTP id u8mr5383188eeb.37.1328511874339; Sun, 05 Feb 2012 23:04:34 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id n17sm57847046eei.3.2012.02.05.23.04.32 (version=SSLv3 cipher=OTHER); Sun, 05 Feb 2012 23:04:33 -0800 (PST) Sender: Alexander Motin Message-ID: <4F2F7B7F.40508@FreeBSD.org> Date: Mon, 06 Feb 2012 09:04:31 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 07:04:37 -0000 Hi. I've analyzed scheduler behavior and think found the problem with HTT. SCHED_ULE knows about HTT and when doing load balancing once a second, it does right things. Unluckily, if some other thread gets in the way, process can be easily pushed out to another CPU, where it will stay for another second because of CPU affinity, possibly sharing physical core with something else without need. I've made a patch, reworking SCHED_ULE affinity code, to fix that: http://people.freebsd.org/~mav/sched.htt.patch This patch does three things: - Disables strict affinity optimization when HTT detected to let more sophisticated code to take into account load of other logical core(s). - Adds affinity support to the sched_lowest() function to prefer specified (last used) CPU (and CPU groups it belongs to) in case of equal load. Previous code always selected first valid CPU of evens. It caused threads migration to lower CPUs without need. - If current CPU group has no CPU where the process with its priority can run now, sequentially check parent CPU groups before doing global search. That should improve affinity for the next cache levels. I've made several different benchmarks to test it, and so far results look promising: - On Atom D525 (2 physical cores + HTT) I've tested HTTP receive with fetch and FTP transmit with ftpd. On receive I've got 103MB/s on interface; on transmit somewhat less -- about 85MB/s. In both cases scheduler kept interrupt thread and application on different physical cores. Without patch speed fluctuating about 103-80MB/s on receive and is about 85MB/s on transmit. - On the same Atom I've tested TCP speed with iperf and got mostly the same results: - receive to Atom with patch -- 755-765Mbit/s, without patch -- 531-765Mbit/s. - transmit from Atom in both cases 679Mbit/s. Fluctuating receive behavior in both tests I think can be explained by some heavy callout handled by the swi4:clock process, called on receive (seen in top and schedgraph), but not on transmit. May be it is specifics of the Realtek NIC driver. - On the same Atom tested number of 512 byte reads from SSD with dd in 1 and 32 streams. Found no regressions, but no benefits also as with one stream there is no congestion and with multiple streams all cores congested. - On Core i7-2600K (4 physical cores + HTT) I've run more then 20 `make buildworld`s with different -j values (1,2,4,6,8,12,16) for both original and patched kernel. I've found no performance regressions, while for -j4 I've got 10% improvement: # ministat -w 65 res4A res4B x res4A + res4B +-----------------------------------------------------------------+ |+ | |++ x x x| |A| |______M__A__________| | +-----------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 1554.86 1617.43 1571.62 1581.3033 32.389449 + 3 1420.69 1423.1 1421.36 1421.7167 1.2439587 Difference at 95.0% confidence -159.587 ± 51.9496 -10.0921% ± 3.28524% (Student's t, pooled s = 22.9197) , and for -j6 -- 3.6% improvement: # ministat -w 65 res6A res6B x res6A + res6B +-----------------------------------------------------------------+ | + | | + + x x x | ||_M__A___| |__________A____M_____|| +-----------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 1381.17 1402.94 1400.3 1394.8033 11.880372 + 3 1340.4 1349.34 1341.23 1343.6567 4.9393758 Difference at 95.0% confidence -51.1467 ± 20.6211 -3.66694% ± 1.47842% (Student's t, pooled s = 9.09782) Who wants to do independent testing to verify my results or do some more interesting benchmarks? :) PS: Sponsored by iXsystems, Inc. -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 07:59:49 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 818CA106564A; Mon, 6 Feb 2012 07:59:49 +0000 (UTC) (envelope-from listlog2011@gmail.com) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6E3DB8FC08; Mon, 6 Feb 2012 07:59:49 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q167xinb024264; Mon, 6 Feb 2012 07:59:46 GMT (envelope-from listlog2011@gmail.com) Message-ID: <4F2F886F.1070706@gmail.com> Date: Mon, 06 Feb 2012 15:59:43 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: Alexander Motin References: <4F2F7B7F.40508@FreeBSD.org> <4F2F8405.2040103@gmail.com> <4F2F84E3.60809@FreeBSD.org> In-Reply-To: <4F2F84E3.60809@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@FreeBSD.org, davidxu@FreeBSD.org Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: davidxu@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 07:59:49 -0000 On 2012/2/6 15:44, Alexander Motin wrote: > On 06.02.2012 09:40, David Xu wrote: >> On 2012/2/6 15:04, Alexander Motin wrote: >>> Hi. >>> >>> I've analyzed scheduler behavior and think found the problem with HTT. >>> SCHED_ULE knows about HTT and when doing load balancing once a second, >>> it does right things. Unluckily, if some other thread gets in the way, >>> process can be easily pushed out to another CPU, where it will stay >>> for another second because of CPU affinity, possibly sharing physical >>> core with something else without need. >>> >>> I've made a patch, reworking SCHED_ULE affinity code, to fix that: >>> http://people.freebsd.org/~mav/sched.htt.patch >>> >>> This patch does three things: >>> - Disables strict affinity optimization when HTT detected to let more >>> sophisticated code to take into account load of other logical core(s). >> Yes, the HTT should first be skipped, looking up in upper layer to find >> a more idling physical core. At least, if system is a dual-core, >> 4-thread CPU, >> and if there are two busy threads, they should be run on different >> physical cores. >> >>> - Adds affinity support to the sched_lowest() function to prefer >>> specified (last used) CPU (and CPU groups it belongs to) in case of >>> equal load. Previous code always selected first valid CPU of evens. It >>> caused threads migration to lower CPUs without need. >> >> Even some level of imbalance can be borne, until it exceeds a threshold, >> this at least does not trash other cpu's cache, pushing a new thread >> to another cpu trashes its cache. The cpus and groups can be arranged in >> a circle list, so searching a lowest load cpu always starts from right >> neighborhood to tail, then circles from head to left neighborhood. >> >>> - If current CPU group has no CPU where the process with its priority >>> can run now, sequentially check parent CPU groups before doing global >>> search. That should improve affinity for the next cache levels. >>> >>> I've made several different benchmarks to test it, and so far results >>> look promising: >>> - On Atom D525 (2 physical cores + HTT) I've tested HTTP receive with >>> fetch and FTP transmit with ftpd. On receive I've got 103MB/s on >>> interface; on transmit somewhat less -- about 85MB/s. In both cases >>> scheduler kept interrupt thread and application on different physical >>> cores. Without patch speed fluctuating about 103-80MB/s on receive and >>> is about 85MB/s on transmit. >>> - On the same Atom I've tested TCP speed with iperf and got mostly the >>> same results: >>> - receive to Atom with patch -- 755-765Mbit/s, without patch -- >>> 531-765Mbit/s. >>> - transmit from Atom in both cases 679Mbit/s. >>> Fluctuating receive behavior in both tests I think can be explained by >>> some heavy callout handled by the swi4:clock process, called on >>> receive (seen in top and schedgraph), but not on transmit. May be it >>> is specifics of the Realtek NIC driver. >>> >>> - On the same Atom tested number of 512 byte reads from SSD with dd in >>> 1 and 32 streams. Found no regressions, but no benefits also as with >>> one stream there is no congestion and with multiple streams all cores >>> congested. >>> >>> - On Core i7-2600K (4 physical cores + HTT) I've run more then 20 >>> `make buildworld`s with different -j values (1,2,4,6,8,12,16) for both >>> original and patched kernel. I've found no performance regressions, >>> while for -j4 I've got 10% improvement: >>> # ministat -w 65 res4A res4B >>> x res4A >>> + res4B >>> +-----------------------------------------------------------------+ >>> |+ | >>> |++ x x x| >>> |A| |______M__A__________| | >>> +-----------------------------------------------------------------+ >>> N Min Max Median Avg Stddev >>> x 3 1554.86 1617.43 1571.62 1581.3033 32.389449 >>> + 3 1420.69 1423.1 1421.36 1421.7167 1.2439587 >>> Difference at 95.0% confidence >>> -159.587 ± 51.9496 >>> -10.0921% ± 3.28524% >>> (Student's t, pooled s = 22.9197) >>> , and for -j6 -- 3.6% improvement: >>> # ministat -w 65 res6A res6B >>> x res6A >>> + res6B >>> +-----------------------------------------------------------------+ >>> | + | >>> | + + x x x | >>> ||_M__A___| |__________A____M_____|| >>> +-----------------------------------------------------------------+ >>> N Min Max Median Avg Stddev >>> x 3 1381.17 1402.94 1400.3 1394.8033 11.880372 >>> + 3 1340.4 1349.34 1341.23 1343.6567 4.9393758 >>> Difference at 95.0% confidence >>> -51.1467 ± 20.6211 >>> -3.66694% ± 1.47842% >>> (Student's t, pooled s = 9.09782) >>> >>> Who wants to do independent testing to verify my results or do some >>> more interesting benchmarks? :) >>> >>> PS: Sponsored by iXsystems, Inc. >>> >> The benchmark is incomplete, a complete benchmark should at lease >> includes cpu intensive applications. >> Testing for release world databases and web servers and other importance >> applications is needed. > > I plan to do this, but you may help. ;) > Thanks, I need to find time. I have cc'ed hackers@, my first mail seems forgot to include it. I think designing a SMP scheduler is a dirty work, many test and refining and still, you may get imperfect result. ;-) Regards, David Xu From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 10:39:37 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD9A7106564A for ; Mon, 6 Feb 2012 10:39:37 +0000 (UTC) (envelope-from gljennjohn@googlemail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6D1A08FC0A for ; Mon, 6 Feb 2012 10:39:37 +0000 (UTC) Received: by eekb47 with SMTP id b47so2525188eek.13 for ; Mon, 06 Feb 2012 02:39:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :x-mailer:mime-version:content-type:content-transfer-encoding; bh=HT29Xf+NUhLXXLfzt47PFWUZnMYHeUPM5y3C7KePW9I=; b=AGTXQP0Ukzr8ZTIF2ilEpp8oLOSMoSlpjHFW+mdDIlWtSV89pp5BBMJsiKJHcGu4Cp V7Qtx/kGa6HvdHdVCercsP1VaQNcYSEeqreoDrG2zmsmSCBovJ+h1xYmxJq3Qq1AneWm obnVYFRWKAVkJdAmRW3FWeTlj3jMOH9J9IqVk= Received: by 10.14.194.134 with SMTP id m6mr5440681een.4.1328522964971; Mon, 06 Feb 2012 02:09:24 -0800 (PST) Received: from ernst.jennejohn.org (p578E2230.dip.t-dialin.net. [87.142.34.48]) by mx.google.com with ESMTPS id n17sm59407912eei.3.2012.02.06.02.09.23 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 06 Feb 2012 02:09:24 -0800 (PST) Date: Mon, 6 Feb 2012 11:09:21 +0100 From: Gary Jennejohn To: Alexander Motin Message-ID: <20120206110921.23741ee5@ernst.jennejohn.org> In-Reply-To: <4F2F7B7F.40508@FreeBSD.org> References: <4F2F7B7F.40508@FreeBSD.org> X-Mailer: Claws Mail 3.8.0 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gljennjohn@googlemail.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 10:39:37 -0000 On Mon, 06 Feb 2012 09:04:31 +0200 Alexander Motin wrote: > I've analyzed scheduler behavior and think found the problem with HTT. > SCHED_ULE knows about HTT and when doing load balancing once a second, > it does right things. Unluckily, if some other thread gets in the way, > process can be easily pushed out to another CPU, where it will stay for > another second because of CPU affinity, possibly sharing physical core > with something else without need. > > I've made a patch, reworking SCHED_ULE affinity code, to fix that: > http://people.freebsd.org/~mav/sched.htt.patch > 10.0-CURRENT FreeBSD r231026M: Mon Feb 6 10:40:10 CET 2012 amd64 CPU: AMD Phenom(tm) II X6 1090T Processor (3214.71-MHz K8-class CPU) This patch seems pretty good. Previously I was using SCHED_4BSD. My simple test was "make -j6 buildworld" while loading web pages at the same time and observing the CPU core loading using gkrellm. In general the loads seemed to be more evenly distributed than without the patch. With the old ULE CPU0 seemed to be underutilized, now it's an equal partner. Web pages also loaded quite quickly despite the high load on the cores. I did notice some mouse pointer lag, but it was quite minor. The buildworld time also seemed to be pretty much the same as with 4BSD. Seems like a good start on improving ULE. -- Gary Jennejohn From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 13:14:40 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B6B21065677 for ; Mon, 6 Feb 2012 13:14:40 +0000 (UTC) (envelope-from olivier@gid0.org) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1B1628FC08 for ; Mon, 6 Feb 2012 13:14:39 +0000 (UTC) Received: by lagz14 with SMTP id z14so4064779lag.13 for ; Mon, 06 Feb 2012 05:14:38 -0800 (PST) MIME-Version: 1.0 Received: by 10.152.147.202 with SMTP id tm10mr10200826lab.49.1328532610816; Mon, 06 Feb 2012 04:50:10 -0800 (PST) Received: by 10.152.29.234 with HTTP; Mon, 6 Feb 2012 04:50:10 -0800 (PST) In-Reply-To: <20120123215503.GA64787@geosci> References: <20120123215503.GA64787@geosci> Date: Mon, 6 Feb 2012 13:50:10 +0100 Message-ID: From: Olivier Smedts To: =?ISO-8859-2?Q?Edward_Tomasz_Napiera=B3a?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Speeding up the loader(8). X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 13:14:40 -0000 2012/1/23 Edward Tomasz Napiera=C5=82a : > Some time ago I've spent some time on trying to speed up loading > modules by the loader(8). =C2=A0Result can be found at: > > http://people.freebsd.org/~trasz/fast-loader-3.diff I use it successfully on my FreeBSD 9.0-STABLE amd64 with ZFS on root. It sped up the loading time significantly, now booting on ZFS feels nearly as speedy as on UFS. > In my tests under VMWare Fusion, this cut the modules loading time > by half. =C2=A0I don't intend to commit the patch as-is - the first part > looks dangerous (the splitting was probably done for a reason), > the second is hackish, and the third doesn't improve anything by itself. > I'm working on something else at a moment; feel free to pick this up. Thanks --=20 Olivier Smedts=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0 _ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ASCII ri= bbon campaign ( ) e-mail: olivier@gid0.org=C2=A0 =C2=A0 =C2=A0 =C2=A0 - against HTML email & = vCards=C2=A0 X www: http://www.gid0.org=C2=A0 =C2=A0 - against proprietary attachments / \ =C2=A0 "Il y a seulement 10 sortes de gens dans le monde : =C2=A0 ceux qui comprennent le binaire, =C2=A0 et ceux qui ne le comprennent pas." From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 16:01:36 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 1233) id A4749106564A; Mon, 6 Feb 2012 16:01:36 +0000 (UTC) Date: Mon, 6 Feb 2012 16:01:36 +0000 From: Alexander Best To: Alexander Motin Message-ID: <20120206160136.GA35918@freebsd.org> References: <4F2F7B7F.40508@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <4F2F7B7F.40508@FreeBSD.org> Cc: freebsd-hackers@freebsd.org Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 16:01:36 -0000 On Mon Feb 6 12, Alexander Motin wrote: > Hi. > > I've analyzed scheduler behavior and think found the problem with HTT. > SCHED_ULE knows about HTT and when doing load balancing once a second, > it does right things. Unluckily, if some other thread gets in the way, > process can be easily pushed out to another CPU, where it will stay for > another second because of CPU affinity, possibly sharing physical core > with something else without need. > > I've made a patch, reworking SCHED_ULE affinity code, to fix that: > http://people.freebsd.org/~mav/sched.htt.patch > > This patch does three things: > - Disables strict affinity optimization when HTT detected to let more > sophisticated code to take into account load of other logical core(s). > - Adds affinity support to the sched_lowest() function to prefer > specified (last used) CPU (and CPU groups it belongs to) in case of > equal load. Previous code always selected first valid CPU of evens. It > caused threads migration to lower CPUs without need. > - If current CPU group has no CPU where the process with its priority > can run now, sequentially check parent CPU groups before doing global > search. That should improve affinity for the next cache levels. > > I've made several different benchmarks to test it, and so far results > look promising: > - On Atom D525 (2 physical cores + HTT) I've tested HTTP receive with > fetch and FTP transmit with ftpd. On receive I've got 103MB/s on > interface; on transmit somewhat less -- about 85MB/s. In both cases > scheduler kept interrupt thread and application on different physical > cores. Without patch speed fluctuating about 103-80MB/s on receive and > is about 85MB/s on transmit. > - On the same Atom I've tested TCP speed with iperf and got mostly the > same results: > - receive to Atom with patch -- 755-765Mbit/s, without patch -- > 531-765Mbit/s. > - transmit from Atom in both cases 679Mbit/s. > Fluctuating receive behavior in both tests I think can be explained by > some heavy callout handled by the swi4:clock process, called on receive > (seen in top and schedgraph), but not on transmit. May be it is > specifics of the Realtek NIC driver. > > - On the same Atom tested number of 512 byte reads from SSD with dd in > 1 and 32 streams. Found no regressions, but no benefits also as with one > stream there is no congestion and with multiple streams all cores congested. > > - On Core i7-2600K (4 physical cores + HTT) I've run more then 20 > `make buildworld`s with different -j values (1,2,4,6,8,12,16) for both > original and patched kernel. I've found no performance regressions, > while for -j4 I've got 10% improvement: > # ministat -w 65 res4A res4B > x res4A > + res4B > +-----------------------------------------------------------------+ > |+ | > |++ x x x| > |A| |______M__A__________| | > +-----------------------------------------------------------------+ > N Min Max Median Avg Stddev > x 3 1554.86 1617.43 1571.62 1581.3033 32.389449 > + 3 1420.69 1423.1 1421.36 1421.7167 1.2439587 > Difference at 95.0% confidence > -159.587 ± 51.9496 > -10.0921% ± 3.28524% > (Student's t, pooled s = 22.9197) > , and for -j6 -- 3.6% improvement: > # ministat -w 65 res6A res6B > x res6A > + res6B > +-----------------------------------------------------------------+ > | + | > | + + x x x | > ||_M__A___| |__________A____M_____|| > +-----------------------------------------------------------------+ > N Min Max Median Avg Stddev > x 3 1381.17 1402.94 1400.3 1394.8033 11.880372 > + 3 1340.4 1349.34 1341.23 1343.6567 4.9393758 > Difference at 95.0% confidence > -51.1467 ± 20.6211 > -3.66694% ± 1.47842% > (Student's t, pooled s = 9.09782) > > Who wants to do independent testing to verify my results or do some more > interesting benchmarks? :) i don't have any benchmarks to offer, but i'm seeing a massive increase in responsiveness with your patch. with an unpatched kernel, opening xterm while unrar'ing some huge archive could take up to 3 minutes!!! with your patch the time it takes for xterm to start is never > 10 seconds!!! well done. :) really looking forward to seeing this commited. cheers. alex btw: i couldn't verify a decrease in my mouses input rate. nothing was lagging! however i'm not running moused(8). i can only advise anyone to turn it off in connection with usb mice. i was having massive problems with moused(8) and hald(8) (i.e. input rates < 1 Hz during heavy disk i/o). disabling moused(8) and relying on hald(8) completely (removing any mouse specific entry from my xorg.conf and disabling moused(8) in my rc.conf) solved the issue entirely. > > PS: Sponsored by iXsystems, Inc. > > -- > Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 16:29:20 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3CDA5106566B; Mon, 6 Feb 2012 16:29:20 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id 901AD8FC18; Mon, 6 Feb 2012 16:29:19 +0000 (UTC) Received: by wibhn14 with SMTP id hn14so6911416wib.13 for ; Mon, 06 Feb 2012 08:29:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=qBdmdL7spYKtiw1wyq7zA0WFkhXArs5difn2OiKhUuo=; b=mf1jXtpJdDADINaKHYrc8CxE6FHzk0toI7y72ratGa6/QjwJloZs0i3E0C3/C7G0rC Ner+rVdV7mmCt6tRymjlcnRnXLaN6KC9OqKWZnYO9L3onJRycFW0NypSPatzFPTc9CpK G5VZMk1yRTN46KUacxZJuky67l4W0SPJfCIZE= Received: by 10.180.83.97 with SMTP id p1mr17583038wiy.19.1328545758354; Mon, 06 Feb 2012 08:29:18 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id ex2sm47889613wib.1.2012.02.06.08.29.16 (version=SSLv3 cipher=OTHER); Mon, 06 Feb 2012 08:29:17 -0800 (PST) Sender: Alexander Motin Message-ID: <4F2FFFDA.2080608@FreeBSD.org> Date: Mon, 06 Feb 2012 18:29:14 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: Alexander Best References: <4F2F7B7F.40508@FreeBSD.org> <20120206160136.GA35918@freebsd.org> In-Reply-To: <20120206160136.GA35918@freebsd.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 16:29:20 -0000 On 02/06/12 18:01, Alexander Best wrote: > On Mon Feb 6 12, Alexander Motin wrote: >> I've analyzed scheduler behavior and think found the problem with HTT. >> SCHED_ULE knows about HTT and when doing load balancing once a second, >> it does right things. Unluckily, if some other thread gets in the way, >> process can be easily pushed out to another CPU, where it will stay for >> another second because of CPU affinity, possibly sharing physical core >> with something else without need. >> >> I've made a patch, reworking SCHED_ULE affinity code, to fix that: >> http://people.freebsd.org/~mav/sched.htt.patch >> >> This patch does three things: >> - Disables strict affinity optimization when HTT detected to let more >> sophisticated code to take into account load of other logical core(s). >> - Adds affinity support to the sched_lowest() function to prefer >> specified (last used) CPU (and CPU groups it belongs to) in case of >> equal load. Previous code always selected first valid CPU of evens. It >> caused threads migration to lower CPUs without need. >> - If current CPU group has no CPU where the process with its priority >> can run now, sequentially check parent CPU groups before doing global >> search. That should improve affinity for the next cache levels. >> Who wants to do independent testing to verify my results or do some more >> interesting benchmarks? :) > > i don't have any benchmarks to offer, but i'm seeing a massive increase in > responsiveness with your patch. with an unpatched kernel, opening xterm while > unrar'ing some huge archive could take up to 3 minutes!!! with your patch the > time it takes for xterm to start is never> 10 seconds!!! Thank you for the report. I can suggest explanation for this. Original code does only one pass looking for CPU where the thread can run immediately. That pass limited to the first level of CPU topology (for HTT systems it is one physical core). If it sees no good candidate, it just looks for the CPU with minimal load, ignoring thread priority. I suppose that may lead to priority violation, scheduling thread to CPU where higher-priority thread is running, where it may wait for a very long time, while there is some other CPU with minimal priority thread. My patch does more searches, that allows to handle priorities better. Unluckily on my newer tests of context-switch-intensive workloads (like doing 40K MySQL requests per second) I've found about 3% slowdown because of these additional searches. I'll finish some more tests and try to find some compromise solution. -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 17:55:23 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8EDF1106566C; Mon, 6 Feb 2012 17:55:23 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id EFF728FC0A; Mon, 6 Feb 2012 17:55:22 +0000 (UTC) Received: by wibhn14 with SMTP id hn14so7008982wib.13 for ; Mon, 06 Feb 2012 09:55:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=09Pd/ykkyaqEtP6pALNPVPhVKd+X6XAu4HVnK+jtiZQ=; b=G2i/CQQff5621NzZRZT4q12B+0T1N3VziZ3nX1Dm+jlppnkqoAtUswe8+WehJWkbr3 4jlr8Q/UChfPpDnpkvUIhVc0Ffp0IbHrzDWNLss7+TaDxWH/K+TmAtqJD1/bIXw9znJx ENcZA7L2GHt14WzotW6qbknXc6eD55jT9IIBk= Received: by 10.180.86.9 with SMTP id l9mr29012861wiz.15.1328550922029; Mon, 06 Feb 2012 09:55:22 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id l6sm26748847wiv.11.2012.02.06.09.55.19 (version=SSLv3 cipher=OTHER); Mon, 06 Feb 2012 09:55:20 -0800 (PST) Sender: Alexander Motin Message-ID: <4F301406.7080906@FreeBSD.org> Date: Mon, 06 Feb 2012 19:55:18 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: Tijl Coosemans References: <4F2F7B7F.40508@FreeBSD.org> <20120206160136.GA35918@freebsd.org> <4F2FFFDA.2080608@FreeBSD.org> <201202061837.48946.tijl@coosemans.org> In-Reply-To: <201202061837.48946.tijl@coosemans.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Alexander Best Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:55:23 -0000 On 02/06/12 19:37, Tijl Coosemans wrote: > On Monday 06 February 2012 17:29:14 Alexander Motin wrote: >> On 02/06/12 18:01, Alexander Best wrote: >>> On Mon Feb 6 12, Alexander Motin wrote: >>>> I've analyzed scheduler behavior and think found the problem with HTT. >>>> SCHED_ULE knows about HTT and when doing load balancing once a second, >>>> it does right things. Unluckily, if some other thread gets in the way, >>>> process can be easily pushed out to another CPU, where it will stay for >>>> another second because of CPU affinity, possibly sharing physical core >>>> with something else without need. >>>> >>>> I've made a patch, reworking SCHED_ULE affinity code, to fix that: >>>> http://people.freebsd.org/~mav/sched.htt.patch >>>> >>>> This patch does three things: >>>> - Disables strict affinity optimization when HTT detected to let more >>>> sophisticated code to take into account load of other logical core(s). >>>> - Adds affinity support to the sched_lowest() function to prefer >>>> specified (last used) CPU (and CPU groups it belongs to) in case of >>>> equal load. Previous code always selected first valid CPU of evens. It >>>> caused threads migration to lower CPUs without need. >>>> - If current CPU group has no CPU where the process with its priority >>>> can run now, sequentially check parent CPU groups before doing global >>>> search. That should improve affinity for the next cache levels. >>>> >>>> Who wants to do independent testing to verify my results or do some more >>>> interesting benchmarks? :) >>> >>> i don't have any benchmarks to offer, but i'm seeing a massive increase in >>> responsiveness with your patch. with an unpatched kernel, opening xterm while >>> unrar'ing some huge archive could take up to 3 minutes!!! with your patch the >>> time it takes for xterm to start is never> 10 seconds!!! >> >> Thank you for the report. I can suggest explanation for this. Original >> code does only one pass looking for CPU where the thread can run >> immediately. That pass limited to the first level of CPU topology (for >> HTT systems it is one physical core). If it sees no good candidate, it >> just looks for the CPU with minimal load, ignoring thread priority. I >> suppose that may lead to priority violation, scheduling thread to CPU >> where higher-priority thread is running, where it may wait for a very >> long time, while there is some other CPU with minimal priority thread. >> My patch does more searches, that allows to handle priorities better. > > But why would unrar have a higher priority? I am not good with ULE priority calculations yet, but I think there could be many factors. Both GUI and unrar probably running with the same nice level of 0, so initially they are equal. After they started, their priorities depend on spent CPU time, calculated "interactivity" and who knows what else. So possibly at some moments unrar may get priority higher then GUI. Also there can participate several kernel threads, that have higher priority by definition. -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 18:07:45 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54F4F1065675; Mon, 6 Feb 2012 18:07:45 +0000 (UTC) (envelope-from tijl@coosemans.org) Received: from mailrelay006.isp.belgacom.be (mailrelay006.isp.belgacom.be [195.238.6.172]) by mx1.freebsd.org (Postfix) with ESMTP id 9FC5F8FC15; Mon, 6 Feb 2012 18:07:44 +0000 (UTC) X-Belgacom-Dynamic: yes X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjwGAFkPME9bsaFX/2dsb2JhbABCFq1xgUCBBoFyAQEFViMQCxgVGTkeiBgGsFCLLgEqBgELAQgFAwMJCIMRDixtBA4CCgIVDIMcBKgE Received: from 87.161-177-91.adsl-dyn.isp.belgacom.be (HELO kalimero.tijl.coosemans.org) ([91.177.161.87]) by relay.skynet.be with ESMTP; 06 Feb 2012 18:37:52 +0100 Received: from kalimero.tijl.coosemans.org (kalimero.tijl.coosemans.org [127.0.0.1]) by kalimero.tijl.coosemans.org (8.14.5/8.14.5) with ESMTP id q16Hbpu9004031; Mon, 6 Feb 2012 18:37:51 +0100 (CET) (envelope-from tijl@coosemans.org) From: Tijl Coosemans To: freebsd-hackers@freebsd.org Date: Mon, 6 Feb 2012 18:37:42 +0100 User-Agent: KMail/1.13.7 (FreeBSD/10.0-CURRENT; KDE/4.7.3; i386; ; ) References: <4F2F7B7F.40508@FreeBSD.org> <20120206160136.GA35918@freebsd.org> <4F2FFFDA.2080608@FreeBSD.org> In-Reply-To: <4F2FFFDA.2080608@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart3534682.XvlJkZjjs3"; protocol="application/pgp-signature"; micalg=pgp-sha256 Content-Transfer-Encoding: 7bit Message-Id: <201202061837.48946.tijl@coosemans.org> Cc: Alexander Best , Alexander Motin Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 18:07:45 -0000 --nextPart3534682.XvlJkZjjs3 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: quoted-printable On Monday 06 February 2012 17:29:14 Alexander Motin wrote: > On 02/06/12 18:01, Alexander Best wrote: >> On Mon Feb 6 12, Alexander Motin wrote: >>> I've analyzed scheduler behavior and think found the problem with HTT. >>> SCHED_ULE knows about HTT and when doing load balancing once a second, >>> it does right things. Unluckily, if some other thread gets in the way, >>> process can be easily pushed out to another CPU, where it will stay for >>> another second because of CPU affinity, possibly sharing physical core >>> with something else without need. >>> >>> I've made a patch, reworking SCHED_ULE affinity code, to fix that: >>> http://people.freebsd.org/~mav/sched.htt.patch >>> >>> This patch does three things: >>> - Disables strict affinity optimization when HTT detected to let more >>> sophisticated code to take into account load of other logical core(s). >>> - Adds affinity support to the sched_lowest() function to prefer >>> specified (last used) CPU (and CPU groups it belongs to) in case of >>> equal load. Previous code always selected first valid CPU of evens. It >>> caused threads migration to lower CPUs without need. >>> - If current CPU group has no CPU where the process with its priority >>> can run now, sequentially check parent CPU groups before doing global >>> search. That should improve affinity for the next cache levels. >>> >>> Who wants to do independent testing to verify my results or do some more >>> interesting benchmarks? :) >> >> i don't have any benchmarks to offer, but i'm seeing a massive increase = in >> responsiveness with your patch. with an unpatched kernel, opening xterm = while >> unrar'ing some huge archive could take up to 3 minutes!!! with your patc= h the >> time it takes for xterm to start is never> 10 seconds!!! >=20 > Thank you for the report. I can suggest explanation for this. Original=20 > code does only one pass looking for CPU where the thread can run=20 > immediately. That pass limited to the first level of CPU topology (for=20 > HTT systems it is one physical core). If it sees no good candidate, it=20 > just looks for the CPU with minimal load, ignoring thread priority. I=20 > suppose that may lead to priority violation, scheduling thread to CPU=20 > where higher-priority thread is running, where it may wait for a very=20 > long time, while there is some other CPU with minimal priority thread.=20 > My patch does more searches, that allows to handle priorities better. But why would unrar have a higher priority? --nextPart3534682.XvlJkZjjs3 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iF4EABEIAAYFAk8wD+wACgkQfoCS2CCgtitlmgD/S3GpDkuBSUaNHdLgEEWFSowd KmHf7B6TAS3U3SVx6A0A/36+J5BUzFG/S/1eG6LNknsf/VeCzV96nrza27YLmi9c =gx99 -----END PGP SIGNATURE----- --nextPart3534682.XvlJkZjjs3-- From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 19:08:02 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A15E7106566C; Mon, 6 Feb 2012 19:08:02 +0000 (UTC) (envelope-from flo@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 87D508FC08; Mon, 6 Feb 2012 19:08:02 +0000 (UTC) Received: from nibbler-osx.local (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q16J80xx054812; Mon, 6 Feb 2012 19:08:00 GMT (envelope-from flo@FreeBSD.org) Message-ID: <4F302510.70106@FreeBSD.org> Date: Mon, 06 Feb 2012 20:08:00 +0100 From: Florian Smeets User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:11.0) Gecko/20120202 Thunderbird/11.0 MIME-Version: 1.0 To: davidxu@FreeBSD.org References: <4F2F7B7F.40508@FreeBSD.org> <4F2F8405.2040103@gmail.com> <4F2F84E3.60809@FreeBSD.org> <4F2F886F.1070706@gmail.com> In-Reply-To: <4F2F886F.1070706@gmail.com> X-Enigmail-Version: 1.4a1pre Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigCFA22324C169EDAF5C2AFA44" Cc: freebsd-hackers@FreeBSD.org, Alexander Motin , David Xu Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 19:08:02 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigCFA22324C169EDAF5C2AFA44 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 06.02.12 08:59, David Xu wrote: > On 2012/2/6 15:44, Alexander Motin wrote: >> On 06.02.2012 09:40, David Xu wrote: >>> On 2012/2/6 15:04, Alexander Motin wrote: >>>> Hi. >>>> >>>> I've analyzed scheduler behavior and think found the problem with HT= T. >>>> SCHED_ULE knows about HTT and when doing load balancing once a secon= d, >>>> it does right things. Unluckily, if some other thread gets in the wa= y, >>>> process can be easily pushed out to another CPU, where it will stay >>>> for another second because of CPU affinity, possibly sharing physica= l >>>> core with something else without need. >>>> >>>> I've made a patch, reworking SCHED_ULE affinity code, to fix that: >>>> http://people.freebsd.org/~mav/sched.htt.patch >>>> >>>> This patch does three things: >>>> - Disables strict affinity optimization when HTT detected to let mor= e >>>> sophisticated code to take into account load of other logical core(s= ). >>> Yes, the HTT should first be skipped, looking up in upper layer to fi= nd >>> a more idling physical core. At least, if system is a dual-core, >>> 4-thread CPU, >>> and if there are two busy threads, they should be run on different >>> physical cores. >>> >>>> - Adds affinity support to the sched_lowest() function to prefer >>>> specified (last used) CPU (and CPU groups it belongs to) in case of >>>> equal load. Previous code always selected first valid CPU of evens. = It >>>> caused threads migration to lower CPUs without need. >>> >>> Even some level of imbalance can be borne, until it exceeds a thresho= ld, >>> this at least does not trash other cpu's cache, pushing a new thread >>> to another cpu trashes its cache. The cpus and groups can be arranged= in >>> a circle list, so searching a lowest load cpu always starts from righ= t >>> neighborhood to tail, then circles from head to left neighborhood. >>> >>>> - If current CPU group has no CPU where the process with its priorit= y >>>> can run now, sequentially check parent CPU groups before doing globa= l >>>> search. That should improve affinity for the next cache levels. >>>> >>>> I've made several different benchmarks to test it, and so far result= s >>>> look promising: >>>> - On Atom D525 (2 physical cores + HTT) I've tested HTTP receive wit= h >>>> fetch and FTP transmit with ftpd. On receive I've got 103MB/s on >>>> interface; on transmit somewhat less -- about 85MB/s. In both cases >>>> scheduler kept interrupt thread and application on different physica= l >>>> cores. Without patch speed fluctuating about 103-80MB/s on receive a= nd >>>> is about 85MB/s on transmit. >>>> - On the same Atom I've tested TCP speed with iperf and got mostly t= he >>>> same results: >>>> - receive to Atom with patch -- 755-765Mbit/s, without patch -- >>>> 531-765Mbit/s. >>>> - transmit from Atom in both cases 679Mbit/s. >>>> Fluctuating receive behavior in both tests I think can be explained = by >>>> some heavy callout handled by the swi4:clock process, called on >>>> receive (seen in top and schedgraph), but not on transmit. May be it= >>>> is specifics of the Realtek NIC driver. >>>> >>>> - On the same Atom tested number of 512 byte reads from SSD with dd = in >>>> 1 and 32 streams. Found no regressions, but no benefits also as with= >>>> one stream there is no congestion and with multiple streams all core= s >>>> congested. >>>> >>>> - On Core i7-2600K (4 physical cores + HTT) I've run more then 20 >>>> `make buildworld`s with different -j values (1,2,4,6,8,12,16) for bo= th >>>> original and patched kernel. I've found no performance regressions, >>>> while for -j4 I've got 10% improvement: >>>> # ministat -w 65 res4A res4B >>>> x res4A >>>> + res4B >>>> +-----------------------------------------------------------------+ >>>> |+ | >>>> |++ x x x| >>>> |A| |______M__A__________| | >>>> +-----------------------------------------------------------------+ >>>> N Min Max Median Avg Stddev >>>> x 3 1554.86 1617.43 1571.62 1581.3033 32.389449 >>>> + 3 1420.69 1423.1 1421.36 1421.7167 1.2439587 >>>> Difference at 95.0% confidence >>>> -159.587 =C2=B1 51.9496 >>>> -10.0921% =C2=B1 3.28524% >>>> (Student's t, pooled s =3D 22.9197) >>>> , and for -j6 -- 3.6% improvement: >>>> # ministat -w 65 res6A res6B >>>> x res6A >>>> + res6B >>>> +-----------------------------------------------------------------+ >>>> | + | >>>> | + + x x x | >>>> ||_M__A___| |__________A____M_____|| >>>> +-----------------------------------------------------------------+ >>>> N Min Max Median Avg Stddev >>>> x 3 1381.17 1402.94 1400.3 1394.8033 11.880372 >>>> + 3 1340.4 1349.34 1341.23 1343.6567 4.9393758 >>>> Difference at 95.0% confidence >>>> -51.1467 =C2=B1 20.6211 >>>> -3.66694% =C2=B1 1.47842% >>>> (Student's t, pooled s =3D 9.09782) >>>> >>>> Who wants to do independent testing to verify my results or do some >>>> more interesting benchmarks? :) >>>> >>>> PS: Sponsored by iXsystems, Inc. >>>> >>> The benchmark is incomplete, a complete benchmark should at lease >>> includes cpu intensive applications. >>> Testing for release world databases and web servers and other importa= nce >>> applications is needed. >> >> I plan to do this, but you may help. ;) >> > Thanks, I need to find time. I have cc'ed hackers@, my first mail seems= > forgot to include it. I think designing a SMP scheduler is a dirty work= , > many test and refining and still, you may get imperfect result. ;-) >=20 Here are my tests for PostgreSQL (i still use r229659 as the baseline was taken with that revision) This is on a 2x4 core, no HTT box. Max throughput is at 10 threads, so that is what i used for ministat. x 229659 + 229659+mav-ule +---------------------------------------------------------------------+ | + x | |+ + + * x+xx x + x + +x x +x| | |__________________|______A__________A____M__M_____|____| | +---------------------------------------------------------------------+ N Min Max Median Avg Stdd= ev x 10 49647.932 50376.405 50194.668 50093.065 240.472= 36 + 10 49482.234 50359.181 50159.422 49936.298 341.255= 92 No difference proven at 95.0% confidence All the numbers are here https://docs.google.com/spreadsheet/ccc?key=3D0Ai0N1xDe3uNAdDRxcVFiYjNMSn= JWOTZhUWVWWlBlemc&hl=3Den_US#gid=3D4 I'll update the pbzip2 tab in the document later today. Florian --------------enigCFA22324C169EDAF5C2AFA44 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAk8wJRAACgkQapo8P8lCvwmzSwCg4+M+ApTZXYeQ7+YWcxwVzcKK At0AoNkfPcjB7wR5WuNvnfXJuHN7Yqcy =QR1N -----END PGP SIGNATURE----- --------------enigCFA22324C169EDAF5C2AFA44-- From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 19:10:32 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 1233) id 0C96F1065676; Mon, 6 Feb 2012 19:10:32 +0000 (UTC) Date: Mon, 6 Feb 2012 19:10:32 +0000 From: Alexander Best To: Alexander Motin Message-ID: <20120206191031.GA76990@freebsd.org> References: <4F2F7B7F.40508@FreeBSD.org> <20120206160136.GA35918@freebsd.org> <4F2FFFDA.2080608@FreeBSD.org> <201202061837.48946.tijl@coosemans.org> <4F301406.7080906@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4F301406.7080906@FreeBSD.org> Cc: freebsd-hackers@freebsd.org, Tijl Coosemans Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 19:10:32 -0000 On Mon Feb 6 12, Alexander Motin wrote: > On 02/06/12 19:37, Tijl Coosemans wrote: > >On Monday 06 February 2012 17:29:14 Alexander Motin wrote: > >>On 02/06/12 18:01, Alexander Best wrote: > >>>On Mon Feb 6 12, Alexander Motin wrote: > >>>>I've analyzed scheduler behavior and think found the problem with HTT. > >>>>SCHED_ULE knows about HTT and when doing load balancing once a second, > >>>>it does right things. Unluckily, if some other thread gets in the way, > >>>>process can be easily pushed out to another CPU, where it will stay for > >>>>another second because of CPU affinity, possibly sharing physical core > >>>>with something else without need. > >>>> > >>>>I've made a patch, reworking SCHED_ULE affinity code, to fix that: > >>>>http://people.freebsd.org/~mav/sched.htt.patch > >>>> > >>>>This patch does three things: > >>>> - Disables strict affinity optimization when HTT detected to let more > >>>>sophisticated code to take into account load of other logical core(s). > >>>> - Adds affinity support to the sched_lowest() function to prefer > >>>>specified (last used) CPU (and CPU groups it belongs to) in case of > >>>>equal load. Previous code always selected first valid CPU of evens. It > >>>>caused threads migration to lower CPUs without need. > >>>> - If current CPU group has no CPU where the process with its priority > >>>>can run now, sequentially check parent CPU groups before doing global > >>>>search. That should improve affinity for the next cache levels. > >>>> > >>>>Who wants to do independent testing to verify my results or do some more > >>>>interesting benchmarks? :) > >>> > >>>i don't have any benchmarks to offer, but i'm seeing a massive increase > >>>in > >>>responsiveness with your patch. with an unpatched kernel, opening xterm > >>>while > >>>unrar'ing some huge archive could take up to 3 minutes!!! with your > >>>patch the > >>>time it takes for xterm to start is never> 10 seconds!!! > >> > >>Thank you for the report. I can suggest explanation for this. Original > >>code does only one pass looking for CPU where the thread can run > >>immediately. That pass limited to the first level of CPU topology (for > >>HTT systems it is one physical core). If it sees no good candidate, it > >>just looks for the CPU with minimal load, ignoring thread priority. I > >>suppose that may lead to priority violation, scheduling thread to CPU > >>where higher-priority thread is running, where it may wait for a very > >>long time, while there is some other CPU with minimal priority thread. > >>My patch does more searches, that allows to handle priorities better. > > > >But why would unrar have a higher priority? > > I am not good with ULE priority calculations yet, but I think there > could be many factors. Both GUI and unrar probably running with the same > nice level of 0, so initially they are equal. After they started, their > priorities depend on spent CPU time, calculated "interactivity" and who > knows what else. So possibly at some moments unrar may get priority > higher then GUI. Also there can participate several kernel threads, that > have higher priority by definition. one factor might also be that by doing i/o, a process can gather a higher priority compared to a task that's doing cpu intensive stuff. statistics have shown (at least historically) that i/o intensive tasks tend to trigger i/o bursts and then either quit or stay idle. so schedulers usually prefer those processes, since there's quite a big chance their request can be delivered in one time slice. cpu intensive tasks on the other hand usually require more than one time slice. this might cause a situation, where the 'unrar'-process is being handled at a very high priority in order to finish its i/o burst. however since unrar'ing a big file isn't really an i/o burst, the cpu intensive process (or any other process in general) will suffer from starvation. this is just a wild guess though. it might be complete nonsense, since i haven't got a clue what kind of scheduling (rr, fcfs, ...) ULE is doing. btw: does anybody know, if there are plans to commit the BFS scheduler to HEAD at some point? personally i think what be even better than a new schuduling algorithm would be a scheduling infrastructure similar to GEOM. that way it would be much easier to implement new algorithms (maybe in XML). cheers. alex > > -- > Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 19:18:33 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41EBC106564A; Mon, 6 Feb 2012 19:18:33 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id A20EB8FC17; Mon, 6 Feb 2012 19:18:31 +0000 (UTC) Received: by eaan10 with SMTP id n10so3074695eaa.13 for ; Mon, 06 Feb 2012 11:18:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=Pl49uzHMKUiFK0WUMRrVco4Q2PibPDnfyhxTQovTE1Y=; b=dA7stBhmOQalYvwZHWwZPY8VfoQyDssxQzB2nwh1gkqFnmN2Va1kffjf5cYmXC9FAC tXBP8X6MIp/68WDNKLTYXT4cUIpWWM8+zglUhUvSXDc171IAJHjXzhqJvcuyx7fxF/X4 v/I5WdKEcazdfklvA2S1mPbj73QP2sFxdxFyE= Received: by 10.213.112.200 with SMTP id x8mr1304509ebp.37.1328555911288; Mon, 06 Feb 2012 11:18:31 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id o49sm64052276eeb.7.2012.02.06.11.18.29 (version=SSLv3 cipher=OTHER); Mon, 06 Feb 2012 11:18:30 -0800 (PST) Sender: Alexander Motin Message-ID: <4F302784.3090607@FreeBSD.org> Date: Mon, 06 Feb 2012 21:18:28 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: Florian Smeets References: <4F2F7B7F.40508@FreeBSD.org> <4F2F8405.2040103@gmail.com> <4F2F84E3.60809@FreeBSD.org> <4F2F886F.1070706@gmail.com> <4F302510.70106@FreeBSD.org> In-Reply-To: <4F302510.70106@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@FreeBSD.org, davidxu@FreeBSD.org Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 19:18:33 -0000 On 02/06/12 21:08, Florian Smeets wrote: > On 06.02.12 08:59, David Xu wrote: >> On 2012/2/6 15:44, Alexander Motin wrote: >>> On 06.02.2012 09:40, David Xu wrote: >>>> On 2012/2/6 15:04, Alexander Motin wrote: >>>>> Hi. >>>>> >>>>> I've analyzed scheduler behavior and think found the problem with HTT. >>>>> SCHED_ULE knows about HTT and when doing load balancing once a second, >>>>> it does right things. Unluckily, if some other thread gets in the way, >>>>> process can be easily pushed out to another CPU, where it will stay >>>>> for another second because of CPU affinity, possibly sharing physical >>>>> core with something else without need. >>>>> >>>>> I've made a patch, reworking SCHED_ULE affinity code, to fix that: >>>>> http://people.freebsd.org/~mav/sched.htt.patch >>>>> >>>>> This patch does three things: >>>>> - Disables strict affinity optimization when HTT detected to let more >>>>> sophisticated code to take into account load of other logical core(s). >>>> Yes, the HTT should first be skipped, looking up in upper layer to find >>>> a more idling physical core. At least, if system is a dual-core, >>>> 4-thread CPU, >>>> and if there are two busy threads, they should be run on different >>>> physical cores. >>>> >>>>> - Adds affinity support to the sched_lowest() function to prefer >>>>> specified (last used) CPU (and CPU groups it belongs to) in case of >>>>> equal load. Previous code always selected first valid CPU of evens. It >>>>> caused threads migration to lower CPUs without need. >>>> >>>> Even some level of imbalance can be borne, until it exceeds a threshold, >>>> this at least does not trash other cpu's cache, pushing a new thread >>>> to another cpu trashes its cache. The cpus and groups can be arranged in >>>> a circle list, so searching a lowest load cpu always starts from right >>>> neighborhood to tail, then circles from head to left neighborhood. >>>> >>>>> - If current CPU group has no CPU where the process with its priority >>>>> can run now, sequentially check parent CPU groups before doing global >>>>> search. That should improve affinity for the next cache levels. >>>>> >>>>> I've made several different benchmarks to test it, and so far results >>>>> look promising: >>>>> - On Atom D525 (2 physical cores + HTT) I've tested HTTP receive with >>>>> fetch and FTP transmit with ftpd. On receive I've got 103MB/s on >>>>> interface; on transmit somewhat less -- about 85MB/s. In both cases >>>>> scheduler kept interrupt thread and application on different physical >>>>> cores. Without patch speed fluctuating about 103-80MB/s on receive and >>>>> is about 85MB/s on transmit. >>>>> - On the same Atom I've tested TCP speed with iperf and got mostly the >>>>> same results: >>>>> - receive to Atom with patch -- 755-765Mbit/s, without patch -- >>>>> 531-765Mbit/s. >>>>> - transmit from Atom in both cases 679Mbit/s. >>>>> Fluctuating receive behavior in both tests I think can be explained by >>>>> some heavy callout handled by the swi4:clock process, called on >>>>> receive (seen in top and schedgraph), but not on transmit. May be it >>>>> is specifics of the Realtek NIC driver. >>>>> >>>>> - On the same Atom tested number of 512 byte reads from SSD with dd in >>>>> 1 and 32 streams. Found no regressions, but no benefits also as with >>>>> one stream there is no congestion and with multiple streams all cores >>>>> congested. >>>>> >>>>> - On Core i7-2600K (4 physical cores + HTT) I've run more then 20 >>>>> `make buildworld`s with different -j values (1,2,4,6,8,12,16) for both >>>>> original and patched kernel. I've found no performance regressions, >>>>> while for -j4 I've got 10% improvement: >>>>> # ministat -w 65 res4A res4B >>>>> x res4A >>>>> + res4B >>>>> +-----------------------------------------------------------------+ >>>>> |+ | >>>>> |++ x x x| >>>>> |A| |______M__A__________| | >>>>> +-----------------------------------------------------------------+ >>>>> N Min Max Median Avg Stddev >>>>> x 3 1554.86 1617.43 1571.62 1581.3033 32.389449 >>>>> + 3 1420.69 1423.1 1421.36 1421.7167 1.2439587 >>>>> Difference at 95.0% confidence >>>>> -159.587 ± 51.9496 >>>>> -10.0921% ± 3.28524% >>>>> (Student's t, pooled s = 22.9197) >>>>> , and for -j6 -- 3.6% improvement: >>>>> # ministat -w 65 res6A res6B >>>>> x res6A >>>>> + res6B >>>>> +-----------------------------------------------------------------+ >>>>> | + | >>>>> | + + x x x | >>>>> ||_M__A___| |__________A____M_____|| >>>>> +-----------------------------------------------------------------+ >>>>> N Min Max Median Avg Stddev >>>>> x 3 1381.17 1402.94 1400.3 1394.8033 11.880372 >>>>> + 3 1340.4 1349.34 1341.23 1343.6567 4.9393758 >>>>> Difference at 95.0% confidence >>>>> -51.1467 ± 20.6211 >>>>> -3.66694% ± 1.47842% >>>>> (Student's t, pooled s = 9.09782) >>>>> >>>>> Who wants to do independent testing to verify my results or do some >>>>> more interesting benchmarks? :) >>>>> >>>>> PS: Sponsored by iXsystems, Inc. >>>>> >>>> The benchmark is incomplete, a complete benchmark should at lease >>>> includes cpu intensive applications. >>>> Testing for release world databases and web servers and other importance >>>> applications is needed. >>> >>> I plan to do this, but you may help. ;) >>> >> Thanks, I need to find time. I have cc'ed hackers@, my first mail seems >> forgot to include it. I think designing a SMP scheduler is a dirty work, >> many test and refining and still, you may get imperfect result. ;-) >> > > Here are my tests for PostgreSQL (i still use r229659 as the baseline > was taken with that revision) This is on a 2x4 core, no HTT box. Max > throughput is at 10 threads, so that is what i used for ministat. > > x 229659 > + 229659+mav-ule > +---------------------------------------------------------------------+ > | + x | > |+ + + * x+xx x + x + +x x +x| > | |__________________|______A__________A____M__M_____|____| | > +---------------------------------------------------------------------+ > N Min Max Median Avg Stddev > x 10 49647.932 50376.405 50194.668 50093.065 240.47236 > + 10 49482.234 50359.181 50159.422 49936.298 341.25592 > No difference proven at 95.0% confidence > > All the numbers are here > https://docs.google.com/spreadsheet/ccc?key=0Ai0N1xDe3uNAdDRxcVFiYjNMSnJWOTZhUWVWWlBlemc&hl=en_US#gid=4 > > I'll update the pbzip2 tab in the document later today. I'm sorry, but I think you can put this on pause for a moment. After some tests with MySQL (where I've found 3% regression), new feedback and more thinking I have a wish to try rewrite the patch. I'll probably send new one to test next days. Thank you for your help. -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 20:09:27 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 637981065672 for ; Mon, 6 Feb 2012 20:09:27 +0000 (UTC) (envelope-from pmohanty.cdac@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2323D8FC1E for ; Mon, 6 Feb 2012 20:09:26 +0000 (UTC) Received: by vbbfa15 with SMTP id fa15so6388181vbb.13 for ; Mon, 06 Feb 2012 12:09:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=kkHBM59YvzSTiMNe3j9uFkBFHAclGc2J2Tg+nlBSgSg=; b=N4WogtbQ06wZtmHPt9UESxDinQUi0Mqqg67OVK9hPm2ut1AIeudtvPh9p5tsnDelTu /aUQ+sPalCe0syonez/bMitv32r3kKuRljnO/DcQEIDa7Ku7GOR9aTXPJz5i2j3KmHxa K4EF1Bc3r3rMMzsss6SxrSxJB0jMpnA98azQM= MIME-Version: 1.0 Received: by 10.52.24.70 with SMTP id s6mr7912210vdf.47.1328557463631; Mon, 06 Feb 2012 11:44:23 -0800 (PST) Received: by 10.52.31.167 with HTTP; Mon, 6 Feb 2012 11:44:23 -0800 (PST) Date: Tue, 7 Feb 2012 01:14:23 +0530 Message-ID: From: PRATIK MOHANTY To: freebsd-hackers@freebsd.org X-Mailman-Approved-At: Mon, 06 Feb 2012 20:20:49 +0000 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ioctl, copy structure from user X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 20:09:27 -0000 Hello sir, I need some example for ioctl to copy structure from user space to kernel space From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 6 21:26:09 2012 Return-Path: Delivered-To: hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BB501065672 for ; Mon, 6 Feb 2012 21:26:09 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from mx0.hoeg.nl (mx0.hoeg.nl [IPv6:2a01:4f8:101:5343::aa]) by mx1.freebsd.org (Postfix) with ESMTP id DFEF58FC15 for ; Mon, 6 Feb 2012 21:26:08 +0000 (UTC) Received: by mx0.hoeg.nl (Postfix, from userid 1000) id 174F32A28CD1; Mon, 6 Feb 2012 22:26:08 +0100 (CET) Date: Mon, 6 Feb 2012 22:26:08 +0100 From: Ed Schouten To: hackers@FreeBSD.org Message-ID: <20120206212608.GM1860@hoeg.nl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="iUV/lbBrmPtUT9dM" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Subject: Small tool: fixwhite(1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 21:26:09 -0000 --iUV/lbBrmPtUT9dM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi folks, I just added a small tool to the tools/ directory in src called fixwhite(1) that attempts to fix commonly made whitespace bugs in source files. I wrote it in such a way that it's quite conservative, to make sure it won't make too many annoying changes that you have to roll back manually. It should be useful especially when copy-pasting code between terminals. As mentioned in the commit message, you can just use :%!fixwhite if you're a vi(1) user. Other editors probably support a similar construct. Have fun! --=20 Ed Schouten WWW: http://80386.nl/ ----- Forwarded message from Ed Schouten ----- > Date: Mon, 6 Feb 2012 10:23:11 +0000 (UTC) > From: Ed Schouten > To: src-committers@freebsd.org, svn-src-all@freebsd.org, > svn-src-head@freebsd.org > Subject: svn commit: r231071 - head/tools/tools/fixwhite > > Author: ed > Date: Mon Feb 6 10:23:11 2012 > New Revision: 231071 > URL: http://svn.freebsd.org/changeset/base/231071 > > Log: > Add fixwhite(1). > > This small utility can be used to `sanitize' the whitespace in source > code. It does the following things: > > Global: > - Remove empty lines at the beginning and the end of a file. > - Merge successive empty lines into a single empty line. > > Per-line: > - Remove trailing whitespace. > - Merge spaces preceeding tabs into the tabs. > > It operated on stdin/stdout. This means that if you use vi(1), you can > just run :%!fixwhite to reorganize the file. ----- End forwarded message ----- --iUV/lbBrmPtUT9dM Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iQIcBAEBAgAGBQJPMEVvAAoJEG5e2P40kaK7C/UP/3obwMHb3ho+4WOKdmQZc++i YPMN39PhVqUNKyvvpxkyAongacdavJUIrKCNEeIb7/yqiqrmwEfOPQIWOhYZiCM4 O64EQVP+G4fi0VNhIhgSqI8FGd6x7/DTEtGob3g3H4TeODdydPAiTYLKujiGn0ib pvQpf3nv6Pppwxc/fqv1b1ra9677ljmciT5MDvAOdsqtOS7/QntgBM9xf982N/K0 1GgQaeGWLuFLkVyBMF9/vH3zV0gydg0UyABkKzYWOB6qCQueBADqdFXM9mdkb2Ch QS92gctWuFVc7Wc9kV27WWnO0Ni4Y1GuevJfUh8EALyNNezQ9QMDywgkiEXJkw0x pbb8VsX2FKvC2oOHCP7BGB7e3fmCGeLT4OJK0qA+FrHQX8Wxn6Qjmb/s4oYKEwz+ brxLNre9vKDW0KHnMlOlPrBUcs19XCCbCPU15JqkiyVTAa33zIqAiRTPQSrcUSXZ tblRLBQSl1SQAFuGPV5St4WQu8F4lXaVYsGi7RZmkpmWJiuwYoa62RgXtTTueQvT nlL3LFtw/VPi+LbrnOdBjSqdtHB1CovjWuz8JNyMB8fq6MCjbmVnacJhTbpAsum1 AGLpd6nHNLmTXt3olzIVGzdpINgSdFWkOgoax8haWFARZx6uBxfUPSpWPqpqQ75C WuS7NRtBVQ3p4JZ6Cxf7 =buhm -----END PGP SIGNATURE----- --iUV/lbBrmPtUT9dM-- From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 7 01:55:52 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 832761065674; Tue, 7 Feb 2012 01:55:52 +0000 (UTC) (envelope-from rflynn@acsalaska.net) Received: from denali.acsalaska.net (denali.acsalaska.net [209.112.173.242]) by mx1.freebsd.org (Postfix) with ESMTP id 4C4F18FC08; Tue, 7 Feb 2012 01:55:52 +0000 (UTC) Received: from mymail.acsalaska.net (sheep.acsalaska.net [216.67.61.194]) by denali.acsalaska.net (8.14.4/8.14.4) with ESMTP id q171I1CY037066; Mon, 6 Feb 2012 16:18:01 -0900 (AKST) (envelope-from rflynn@acsalaska.net) Received: from 46.129.107.107 (SquirrelMail authenticated user rflynn@acsalaska.net) by mymail.acsalaska.net with HTTP; Mon, 6 Feb 2012 16:18:02 -0900 (AKST) Message-ID: <3942.46.129.107.107.1328577482.squirrel@mymail.acsalaska.net> In-Reply-To: References: <4415.46.129.107.107.1328479605.squirrel@mymail.acsalaska.net> Date: Mon, 6 Feb 2012 16:18:02 -0900 (AKST) From: rflynn@acsalaska.net To: "Bernhard Froehlich" User-Agent: SquirrelMail/1.4.13 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.3 (denali.acsalaska.net [209.112.168.121]); Mon, 06 Feb 2012 16:18:02 -0900 (AKST) X-ACS-Spam-Status: no X-ACS-Scanned-By: MD 2.67; SA 3.3.0; spamdefang 1.122 X-Mailman-Approved-At: Tue, 07 Feb 2012 02:32:19 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org Subject: xlocale support (Was: Re: Zarafa port) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 01:55:52 -0000 Hi, > Thanks for starting to work on it! I've added a link from WantedPorts > to that > mail. Currently stuck on the absence of xlocale(3). Is there any chance these get MFC'd to RELENG_9 and RELENG_8? Or should I try implementing them in platform*.cpp within Zarafa? The latter looks possible, yet not trivial (still have to figure out what it needs from libc_private.h). Also open to workarounds or ports that implement these interfaces. -- Mel From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 7 06:10:03 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 813FE1065670; Tue, 7 Feb 2012 06:10:03 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 4C0508FC16; Tue, 7 Feb 2012 06:10:03 +0000 (UTC) Received: from julian-mac.elischer.org (c-67-180-24-15.hsd1.ca.comcast.net [67.180.24.15]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id q1769uOQ043349 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 6 Feb 2012 22:09:58 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4F30C085.3070305@freebsd.org> Date: Mon, 06 Feb 2012 22:11:17 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.26) Gecko/20120129 Thunderbird/3.1.18 MIME-Version: 1.0 To: Alexander Best References: <4F2F7B7F.40508@FreeBSD.org> <20120206160136.GA35918@freebsd.org> <4F2FFFDA.2080608@FreeBSD.org> <201202061837.48946.tijl@coosemans.org> <4F301406.7080906@FreeBSD.org> <20120206191031.GA76990@freebsd.org> In-Reply-To: <20120206191031.GA76990@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Alexander Motin , Tijl Coosemans Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 06:10:03 -0000 On 2/6/12 11:10 AM, Alexander Best wrote: > On Mon Feb 6 12, Alexander Motin wrote: >> On 02/06/12 19:37, Tijl Coosemans wrote: >>> On Monday 06 February 2012 17:29:14 Alexander Motin wrote: >>>> On 02/06/12 18:01, Alexander Best wrote: >>>>> On Mon Feb 6 12, Alexander Motin wrote: >>>>>> I've analyzed scheduler behavior and think found the problem with HTT. >>>>>> SCHED_ULE knows about HTT and when doing load balancing once a second, >>>>>> it does right things. Unluckily, if some other thread gets in the way, >>>>>> process can be easily pushed out to another CPU, where it will stay for >>>>>> another second because of CPU affinity, possibly sharing physical core >>>>>> with something else without need. >>>>>> >>>>>> I've made a patch, reworking SCHED_ULE affinity code, to fix that: >>>>>> http://people.freebsd.org/~mav/sched.htt.patch >>>>>> >>>>>> This patch does three things: >>>>>> - Disables strict affinity optimization when HTT detected to let more >>>>>> sophisticated code to take into account load of other logical core(s). >>>>>> - Adds affinity support to the sched_lowest() function to prefer >>>>>> specified (last used) CPU (and CPU groups it belongs to) in case of >>>>>> equal load. Previous code always selected first valid CPU of evens. It >>>>>> caused threads migration to lower CPUs without need. >>>>>> - If current CPU group has no CPU where the process with its priority >>>>>> can run now, sequentially check parent CPU groups before doing global >>>>>> search. That should improve affinity for the next cache levels. >>>>>> >>>>>> Who wants to do independent testing to verify my results or do some more >>>>>> interesting benchmarks? :) >>>>> i don't have any benchmarks to offer, but i'm seeing a massive increase >>>>> in >>>>> responsiveness with your patch. with an unpatched kernel, opening xterm >>>>> while >>>>> unrar'ing some huge archive could take up to 3 minutes!!! with your >>>>> patch the >>>>> time it takes for xterm to start is never> 10 seconds!!! >>>> Thank you for the report. I can suggest explanation for this. Original >>>> code does only one pass looking for CPU where the thread can run >>>> immediately. That pass limited to the first level of CPU topology (for >>>> HTT systems it is one physical core). If it sees no good candidate, it >>>> just looks for the CPU with minimal load, ignoring thread priority. I >>>> suppose that may lead to priority violation, scheduling thread to CPU >>>> where higher-priority thread is running, where it may wait for a very >>>> long time, while there is some other CPU with minimal priority thread. >>>> My patch does more searches, that allows to handle priorities better. >>> But why would unrar have a higher priority? >> I am not good with ULE priority calculations yet, but I think there >> could be many factors. Both GUI and unrar probably running with the same >> nice level of 0, so initially they are equal. After they started, their >> priorities depend on spent CPU time, calculated "interactivity" and who >> knows what else. So possibly at some moments unrar may get priority >> higher then GUI. Also there can participate several kernel threads, that >> have higher priority by definition. > one factor might also be that by doing i/o, a process can gather a higher > priority compared to a task that's doing cpu intensive stuff. statistics have > shown (at least historically) that i/o intensive tasks tend to trigger i/o > bursts and then either quit or stay idle. so schedulers usually prefer those > processes, since there's quite a big chance their request can be delivered in > one time slice. cpu intensive tasks on the other hand usually require more than > one time slice. this might cause a situation, where the 'unrar'-process is > being handled at a very high priority in order to finish its i/o burst. however > since unrar'ing a big file isn't really an i/o burst, the cpu intensive process > (or any other process in general) will suffer from starvation. > > this is just a wild guess though. it might be complete nonsense, since i > haven't got a clue what kind of scheduling (rr, fcfs, ...) ULE is doing. > > btw: does anybody know, if there are plans to commit the BFS scheduler to HEAD > at some point? personally i think what be even better than a new schuduling > algorithm would be a scheduling infrastructure similar to GEOM. that way it > would be much easier to implement new algorithms (maybe in XML). a scheduler 'framework' is rather hard to do because the scheduler interface is "complicated".. I and others did try abstract teh scheduler a bit in about 2000. As you can see we did manage to have 2 separate schedulers.. adding a third should be easier but the current interface is still slightly tied to the things that those two schedulers need. > cheers. > alex > >> -- >> Alexander Motin > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 7 13:50:27 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE95110656D1 for ; Tue, 7 Feb 2012 13:50:27 +0000 (UTC) (envelope-from decke@FreeBSD.org) Received: from groupware.itac.at (groupware.itac.at [91.205.172.99]) by mx1.freebsd.org (Postfix) with ESMTP id 803938FC14 for ; Tue, 7 Feb 2012 13:50:27 +0000 (UTC) Received: from home.bluelife.at (93.104.210.95) by groupware.itac.at (Axigen) with (AES256-SHA encrypted) ESMTPSA id 209018; Tue, 7 Feb 2012 14:36:07 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 07 Feb 2012 14:35:19 +0100 From: Bernhard Froehlich To: In-Reply-To: <3942.46.129.107.107.1328577482.squirrel@mymail.acsalaska.net> References: <4415.46.129.107.107.1328479605.squirrel@mymail.acsalaska.net> <3942.46.129.107.107.1328577482.squirrel@mymail.acsalaska.net> Message-ID: <86f5029a8da18393296855b90ac9e876@bluelife.at> X-Sender: decke@FreeBSD.org User-Agent: Roundcube Webmail/0.7.1 X-AxigenSpam-Level: 1 X-CTCH-RefID: str=0001.0A0B020D.4F312897.00EE,ss=1,fgs=0 X-CTCH-VOD: Unknown X-CTCH-Spam: Unknown Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org Subject: Re: xlocale support (Was: Re: Zarafa port) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 13:50:28 -0000 On 07.02.2012 02:18, rflynn@acsalaska.net wrote: > Hi, > >> Thanks for starting to work on it! I've added a link from >> WantedPorts >> to that >> mail. > > Currently stuck on the absence of xlocale(3). Is there any chance > these > get MFC'd to RELENG_9 and RELENG_8? Or should I try implementing them > in platform*.cpp within Zarafa? The latter looks possible, yet not > trivial (still have to figure out what it needs from libc_private.h). > Also open to workarounds or ports that implement these interfaces. It looks like xlocale support in current still has some problems so it needs to mature a bit more before it gets MFCd. Many projects have some fallback if xlocale.h does not exist and just need a small patch to don't include xlocale.h on FreeBSD. An example is the patch for multimedia/mlt: http://www.freebsd.org/cgi/cvsweb.cgi/ports/multimedia/mlt/files/patch-src__framework__mlt_property.h?rev=1.1 -- Bernhard Froehlich http://www.bluelife.at/ From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 7 15:58:03 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D629106566B; Tue, 7 Feb 2012 15:58:03 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id C9C918FC0A; Tue, 7 Feb 2012 15:58:02 +0000 (UTC) Received: by obcwo16 with SMTP id wo16so12028951obc.13 for ; Tue, 07 Feb 2012 07:58:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=CJU+X9uPB5SJ15ewJcIuv85RbHFijc1LvGwxC+qi0OI=; b=VGJNrW1BOtufVDHs6j2luLSJRAYJQufg/OClJyrDaOYAVy7EAxkMh0tqIcLhRzNBvs yhut9Ribab9K47J/AZW9Ad250BCiq7DYPVjK0HXV9ottV9oiVTmDmXp5gijfH8V6QoKN WjKUfsUrJfXKocvOnqjxiccIspU3UOy61Bl0w= MIME-Version: 1.0 Received: by 10.182.11.71 with SMTP id o7mr21456261obb.58.1328630282447; Tue, 07 Feb 2012 07:58:02 -0800 (PST) Received: by 10.182.46.163 with HTTP; Tue, 7 Feb 2012 07:58:01 -0800 (PST) In-Reply-To: References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> Date: Tue, 7 Feb 2012 07:58:01 -0800 Message-ID: From: Garrett Cooper To: Jan Mikkelsen Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org, Attilio Rao , Ivan Voras , Xin LI , Jan Mikkelsen , davidxu@freebsd.org Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 15:58:03 -0000 On Tue, Feb 7, 2012 at 7:50 AM, Jan Mikkelsen wrote: > > This has just happened again, this time with MAKE_JOBS_NUMBER=1, so that workaround didn't work. You need to also issue DISABLE_MAKE_JOBS=1 (you can't just issue one of the items). devel/talloc and databases/tdb abuse these variables via _MAKE_JOBS (I think in part because their intent is not well documented in bsd.ports.mk). I have a similar PR filed for this: http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/164488 . Sorry for missing that note.. -Garrett From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 7 14:52:33 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A89D106566B; Tue, 7 Feb 2012 14:52:33 +0000 (UTC) (envelope-from rflynn@acsalaska.net) Received: from huffman.acsalaska.net (huffman.acsalaska.net [209.112.173.250]) by mx1.freebsd.org (Postfix) with ESMTP id 242068FC13; Tue, 7 Feb 2012 14:52:32 +0000 (UTC) Received: from mymail.acsalaska.net (sheep.acsalaska.net [216.67.61.194]) by huffman.acsalaska.net (8.14.4/8.14.4) with ESMTP id q17EqVRQ023016; Tue, 7 Feb 2012 05:52:32 -0900 (AKST) (envelope-from rflynn@acsalaska.net) Received: from 46.129.107.107 (SquirrelMail authenticated user rflynn@acsalaska.net) by mymail.acsalaska.net with HTTP; Tue, 7 Feb 2012 05:52:32 -0900 (AKST) Message-ID: <4853.46.129.107.107.1328626352.squirrel@mymail.acsalaska.net> In-Reply-To: <86f5029a8da18393296855b90ac9e876@bluelife.at> References: <4415.46.129.107.107.1328479605.squirrel@mymail.acsalaska.net> <3942.46.129.107.107.1328577482.squirrel@mymail.acsalaska.net> <86f5029a8da18393296855b90ac9e876@bluelife.at> Date: Tue, 7 Feb 2012 05:52:32 -0900 (AKST) From: rflynn@acsalaska.net To: "Bernhard Froehlich" User-Agent: SquirrelMail/1.4.13 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.3 (huffman.acsalaska.net [209.112.168.121]); Tue, 07 Feb 2012 05:52:32 -0900 (AKST) X-ACS-Spam-Status: no X-ACS-Scanned-By: MD 2.67; SA 3.3.0; spamdefang 1.122 X-Mailman-Approved-At: Tue, 07 Feb 2012 16:12:27 +0000 Cc: freebsd-hackers@FreeBSD.org, freebsd-ports@FreeBSD.org Subject: Re: xlocale support (Was: Re: Zarafa port) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 14:52:33 -0000 > On 07.02.2012 02:18, rflynn@acsalaska.net wrote: >> Hi, >> >>> Thanks for starting to work on it! I've added a link from >>> WantedPorts >>> to that >>> mail. >> >> Currently stuck on the absence of xlocale(3). Is there any chance >> these >> get MFC'd to RELENG_9 and RELENG_8? Or should I try implementing them >> in platform*.cpp within Zarafa? The latter looks possible, yet not >> trivial (still have to figure out what it needs from libc_private.h). >> Also open to workarounds or ports that implement these interfaces. > > It looks like xlocale support in current still has some problems so it > needs to mature a bit more before it gets MFCd. > > Many projects have some fallback if xlocale.h does not exist and just > need a small patch to don't include xlocale.h on FreeBSD. An example > is the patch for multimedia/mlt: I looked at the OpenBSD port patches and stripped out the xlocale interfaces, just a bit different (using HAVE_XLOCALE i.s.o __FreeBSD__) so it'll work on HEAD. Hopefully, I can get an autoconf patch for this xlocale(3) test back to upstream, else I'll set it based on $OSVERSION in the port's Makefile, cause so far I can get away with not using USE_AUTOTOOLS. For future reference, USE_GCC=4.6 didn't help. -- Mel From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 7 15:50:12 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 293CE1065673 for ; Tue, 7 Feb 2012 15:50:12 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from midgard.transactionware.com (mail2.transactionware.com [203.14.245.36]) by mx1.freebsd.org (Postfix) with SMTP id 9113A8FC16 for ; Tue, 7 Feb 2012 15:50:07 +0000 (UTC) Received: (qmail 33712 invoked by uid 907); 7 Feb 2012 15:50:05 -0000 Received: from midgard.transactionware.com (HELO [127.0.0.1]) (192.168.1.55) by midgard.transactionware.com (qpsmtpd/0.82) with ESMTP; Wed, 08 Feb 2012 02:50:05 +1100 Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Jan Mikkelsen In-Reply-To: Date: Wed, 8 Feb 2012 02:50:04 +1100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <201201110806.30620.jhb@freebsd.org> <5D37298B-9D68-4F0F-8AAB-E8F2DBB9D9C3@transactionware.com> To: Attilio Rao X-Mailer: Apple Mail (2.1084) X-Mailman-Approved-At: Tue, 07 Feb 2012 16:31:40 +0000 Cc: Garrett Cooper , freebsd-hackers@freebsd.org, Ivan Voras , Xin LI , Jan Mikkelsen , davidxu@freebsd.org Subject: Re: sem(4) lockup in python? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 15:50:12 -0000 On 06/02/2012, at 3:49 AM, Attilio Rao wrote: > 2012/2/5 Ivan Voras : >> On 5 February 2012 11:44, Garrett Cooper wrote: >>=20 >>>=20 >>> 'make MAKE_JOBS_NUMBER=3D1' is the workground used right now.. >>=20 >> David Xu suggested that it is a bug in Python - it doesn't set >> process-shared attribute when it calls sem_init(), but i've tried >> patching it (replacing the port patchfile file the one I've attached) >> and I still get the hang. >=20 > Guys, > it would be valuable if you do the following: > 1) recompile your kernel with INVARIANTS, WITNESS and without = WITNESS_SKIPSPIN > 2a) If you have a serial console, please run the DDB stuff through it > (go to point 3) > 2b) If you don't have a serial console please run the DDB stuff in > textdump (go to point 3) > 3) Collect the following informations: > - show allpcpu > - show alllocks > - ps > - alltrace > 3a) If you had the serial console (thus not textdump) please collect > the coredump with: call doadump > 4) reset your machine >=20 > You will end up with the textdump or coredump + all the serial logs > necessary to debug this. > If you cannot reproduce your issue with WITNESS enabled, please remove > from your kernel config and avoid to call 'show alllocks' when in DDB. > But try to leave INVARIANTS on. >=20 > Hope this helps, > Attilio This has just happened again, this time with MAKE_JOBS_NUMBER=3D1, so = that workaround didn't work. I don't have INVARIANTS or WITNESS compiled in, but I did fire up kgdb = to poke around. The stack traces look identical. I don't know what to = expect in these structures. If there's anything useful I can dig out = here, please let me know. However: A parent and child process both blocked waiting on semaphores = smells like an user level bug to me. Jan. (kgdb) proc 24969 [Switching to thread 648 (Thread 101022)]#0 sched_switch = (td=3D0xfffffe003de43000, newtd=3D0xfffffe000b501000, flags=3DVariable = "flags" is not available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/sched_ule.c:18= 54 1854 cpuid =3D PCPU_GET(cpuid); (kgdb) where #0 sched_switch (td=3D0xfffffe003de43000, newtd=3D0xfffffe000b501000, = flags=3DVariable "flags" is not available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/sched_ule.c:18= 54 #1 0xffffffff8083af24 in mi_switch (flags=3D260, newtd=3D0x0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_synch.c:4= 48 #2 0xffffffff80872644 in sleepq_catch_signals = (wchan=3D0xfffffe0015fca800, pri=3D0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/subr_sleepqueu= e.c:425 #3 0xffffffff80872fb6 in sleepq_wait_sig (wchan=3DVariable "wchan" is = not available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/subr_sleepqueu= e.c:631 #4 0xffffffff8083b599 in _sleep (ident=3D0xfffffe0015fca800, = lock=3D0xffffffff81114860, priority=3DVariable "priority" is not = available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_synch.c:2= 32 #5 0xffffffff8084ac69 in do_sem_wait (td=3DVariable "td" is not = available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_umtx.c:51= 3 #6 0xffffffff8084ad61 in __umtx_op_sem_wait (td=3D0xfffffe003de43000, = uap=3D0xffffff8693d85bc0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_umtx.c:32= 05 #7 0xffffffff80b17de0 in amd64_syscall (td=3D0xfffffe003de43000, = traced=3D0) at subr_syscall.c:131 #8 0xffffffff80b03517 in Xfast_syscall () at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/amd64/amd64/excepti= on.S:387 #9 0x00000008010277fc in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) proc 24970 [Switching to thread 665 (Thread 100553)]#0 sched_switch = (td=3D0xfffffe02f7240460, newtd=3D0xfffffe000b501460, flags=3DVariable = "flags" is not available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/sched_ule.c:18= 54 1854 cpuid =3D PCPU_GET(cpuid); (kgdb) where #0 sched_switch (td=3D0xfffffe02f7240460, newtd=3D0xfffffe000b501460, = flags=3DVariable "flags" is not available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/sched_ule.c:18= 54 #1 0xffffffff8083af24 in mi_switch (flags=3D260, newtd=3D0x0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_synch.c:4= 48 #2 0xffffffff80872644 in sleepq_catch_signals = (wchan=3D0xfffffe0015fd7380, pri=3D0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/subr_sleepqueu= e.c:425 #3 0xffffffff80872fb6 in sleepq_wait_sig (wchan=3DVariable "wchan" is = not available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/subr_sleepqueu= e.c:631 #4 0xffffffff8083b599 in _sleep (ident=3D0xfffffe0015fd7380, = lock=3D0xffffffff811145e0, priority=3DVariable "priority" is not = available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_synch.c:2= 32 #5 0xffffffff8084ac69 in do_sem_wait (td=3DVariable "td" is not = available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_umtx.c:51= 3 #6 0xffffffff8084ad61 in __umtx_op_sem_wait (td=3D0xfffffe02f7240460, = uap=3D0xffffff8694b04bc0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_umtx.c:32= 05 #7 0xffffffff80b17de0 in amd64_syscall (td=3D0xfffffe02f7240460, = traced=3D0) at subr_syscall.c:131 #8 0xffffffff80b03517 in Xfast_syscall () at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/amd64/amd64/excepti= on.S:387 #9 0x00000008010277fc in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) up #1 0xffffffff8083af24 in mi_switch (flags=3D260, newtd=3D0x0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_synch.c:4= 48 448 sched_switch(td, newtd, flags); (kgdb) up #2 0xffffffff80872644 in sleepq_catch_signals = (wchan=3D0xfffffe0015fd7380, pri=3D0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/subr_sleepqueu= e.c:425 425 sleepq_switch(wchan, pri); (kgdb) up #3 0xffffffff80872fb6 in sleepq_wait_sig (wchan=3DVariable "wchan" is = not available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/subr_sleepqueu= e.c:631 631 rcatch =3D sleepq_catch_signals(wchan, pri); (kgdb) up #4 0xffffffff8083b599 in _sleep (ident=3D0xfffffe0015fd7380, = lock=3D0xffffffff811145e0, priority=3DVariable "priority" is not = available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_synch.c:2= 32 232 rval =3D sleepq_wait_sig(ident, pri); (kgdb) up #5 0xffffffff8084ac69 in do_sem_wait (td=3DVariable "td" is not = available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_umtx.c:51= 3 warning: Source file is more recent than executable. 513 error =3D msleep(uq, &uc->uc_lock, PCATCH, wmesg, timo); (kgdb) p *uq $1 =3D {uq_link =3D {tqe_next =3D 0x0, tqe_prev =3D 0xfffffe0015fd2080}, = uq_key =3D {hash =3D 186, type =3D 2, shared =3D 0, info =3D {shared =3D = {object =3D 0xfffffe00162b0310, offset =3D 34380812388}, private =3D { vs =3D 0xfffffe00162b0310, addr =3D 34380812388}, both =3D {a =3D = 0xfffffe00162b0310, b =3D 34380812388}}}, uq_flags =3D 1, uq_thread =3D = 0xfffffe02f7240460, uq_pi_blocked =3D 0x0, uq_lockq =3D { tqe_next =3D 0x0, tqe_prev =3D 0x0}, uq_pi_contested =3D {tqh_first = =3D 0x0, tqh_last =3D 0xfffffe0015fd73d8}, uq_inherited_pri =3D 255 '?', = uq_spare_queue =3D 0x0, uq_cur_queue =3D 0xfffffe0015fd2080} (kgdb) proc 24969 [Switching to thread 648 (Thread 101022)]#0 sched_switch = (td=3D0xfffffe003de43000, newtd=3D0xfffffe000b501000, flags=3DVariable = "flags" is not available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/sched_ule.c:18= 54 1854 cpuid =3D PCPU_GET(cpuid); (kgdb) up #1 0xffffffff8083af24 in mi_switch (flags=3D260, newtd=3D0x0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_synch.c:4= 48 448 sched_switch(td, newtd, flags); (kgdb) up #2 0xffffffff80872644 in sleepq_catch_signals = (wchan=3D0xfffffe0015fca800, pri=3D0) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/subr_sleepqueu= e.c:425 425 sleepq_switch(wchan, pri); (kgdb) up #3 0xffffffff80872fb6 in sleepq_wait_sig (wchan=3DVariable "wchan" is = not available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/subr_sleepqueu= e.c:631 631 rcatch =3D sleepq_catch_signals(wchan, pri); (kgdb) up #4 0xffffffff8083b599 in _sleep (ident=3D0xfffffe0015fca800, = lock=3D0xffffffff81114860, priority=3DVariable "priority" is not = available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_synch.c:2= 32 232 rval =3D sleepq_wait_sig(ident, pri); (kgdb) up #5 0xffffffff8084ac69 in do_sem_wait (td=3DVariable "td" is not = available. ) at = /home/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/sys/kern/kern_umtx.c:51= 3 513 error =3D msleep(uq, &uc->uc_lock, PCATCH, wmesg, timo); (kgdb) p *uq $2 =3D {uq_link =3D {tqe_next =3D 0x0, tqe_prev =3D 0xfffffe04fc73c280}, = uq_key =3D {hash =3D 194, type =3D 2, shared =3D 0, info =3D {shared =3D = {object =3D 0xfffffe001628d188, offset =3D 34380814884}, private =3D { vs =3D 0xfffffe001628d188, addr =3D 34380814884}, both =3D {a =3D = 0xfffffe001628d188, b =3D 34380814884}}}, uq_flags =3D 1, uq_thread =3D = 0xfffffe003de43000, uq_pi_blocked =3D 0x0, uq_lockq =3D { tqe_next =3D 0x0, tqe_prev =3D 0x0}, uq_pi_contested =3D {tqh_first = =3D 0x0, tqh_last =3D 0xfffffe0015fca858}, uq_inherited_pri =3D 255 '?', = uq_spare_queue =3D 0x0, uq_cur_queue =3D 0xfffffe04fc73c280} (kgdb)=20 From owner-freebsd-hackers@FreeBSD.ORG Tue Feb 7 19:03:47 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B162E106564A for ; Tue, 7 Feb 2012 19:03:47 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 889968FC13 for ; Tue, 7 Feb 2012 19:03:47 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id 4310A46B2A; Tue, 7 Feb 2012 14:03:47 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id A3CAFB962; Tue, 7 Feb 2012 14:03:46 -0500 (EST) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Tue, 7 Feb 2012 13:57:39 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p10; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201202071357.39645.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 07 Feb 2012 14:03:46 -0500 (EST) Cc: PRATIK MOHANTY Subject: Re: ioctl, copy structure from user X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 19:03:47 -0000 On Monday, February 06, 2012 2:44:23 pm PRATIK MOHANTY wrote: > Hello sir, > I need some example for ioctl to copy structure from user space to kernel > space In BSD the kernel copies the immediate ioctl argument into and out of userland for you. Thus, you can do something like this: struct foo { ... }; #define MY_IOCTL _IOWR('M', 1, struct foo) And in your kernel code: int foo_ioctl(..., u_long cmd, caddr_t data) { struct foo *f; switch (cmd) { case MY_IOCTL: f = (struct foo *)data; /* * 'f' is now a pointer to an in-kernel copy of * the structure. Any changes made to it will * be copied back out to userland after your * routine returns. */ break; } } -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 8 11:06:59 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AD2D106564A for ; Wed, 8 Feb 2012 11:06:59 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) by mx1.freebsd.org (Postfix) with ESMTP id E5EC58FC17 for ; Wed, 8 Feb 2012 11:06:58 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1Rv5MX-0003cX-Dy for freebsd-hackers@freebsd.org; Wed, 08 Feb 2012 12:06:57 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 08 Feb 2012 12:06:57 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 08 Feb 2012 12:06:57 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Wed, 08 Feb 2012 12:06:47 +0100 Lines: 18 Message-ID: References: <4F2F7B7F.40508@FreeBSD.org> <20120206160136.GA35918@freebsd.org> <4F2FFFDA.2080608@FreeBSD.org> <201202061837.48946.tijl@coosemans.org> <4F301406.7080906@FreeBSD.org> <20120206191031.GA76990@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20120110 Thunderbird/9.0 In-Reply-To: <20120206191031.GA76990@freebsd.org> Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 11:06:59 -0000 On 06/02/2012 20:10, Alexander Best wrote: > btw: does anybody know, if there are plans to commit the BFS scheduler to HEAD BFS is available but I think it needs more work on it before it can be useful; it didn't explore some optimizations it could have and currently spends much more time in lock contention with itself that necessary. I.e. it's usable only in very borderline cases. > algorithm would be a scheduling infrastructure similar to GEOM. that way it > would be much easier to implement new algorithms (maybe in XML). I don't think XML would be applicable beyond fine-tuning an already existing scheduler (unless we implement a whole Turing-complete sublanguage in it :) ). Someone mentioned on a list that he has a "pluggable scheduler" infrastructure ready - this would be a necessary first step in modularization. From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 8 13:36:01 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A421C106564A; Wed, 8 Feb 2012 13:36:01 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.64.117]) by mx1.freebsd.org (Postfix) with ESMTP id 2161C8FC0C; Wed, 8 Feb 2012 13:36:00 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.5/8.14.5) with ESMTP id q18Da0G3023112; Wed, 8 Feb 2012 17:36:00 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.5/8.14.5/Submit) id q18DZx4R023109; Wed, 8 Feb 2012 17:35:59 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Wed, 8 Feb 2012 17:35:59 +0400 From: Gleb Smirnoff To: Luigi Rizzo Message-ID: <20120208133559.GK13554@FreeBSD.org> References: <20120131110204.GA95472@onelab2.iet.unipi.it> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <20120131110204.GA95472@onelab2.iet.unipi.it> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Ermal Lu?i , freebsd-net , freebsd-hackers@FreeBSD.org Subject: Re: [PATCH] multiple instances of ipfw(4) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 13:36:01 -0000 On Tue, Jan 31, 2012 at 12:02:04PM +0100, Luigi Rizzo wrote: L> if i understand what the patch does, i think it makes sense to be L> able to hook ipfw instances to specific interfaces/sets of interfaces, L> as it permits the writing of more readable rulesets. Right now the L> workaround is start the ruleset with skipto rules matching on L> interface names, and then use some discipline in "reserving" a range L> of rule numbers to each interface. This is definitely a desired feature, but it should be implemented on level of pfil(9). However, that would still require multiple instances of ipfw(4). -- Totus tuus, Glebius. From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 8 14:04:10 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CE33106564A; Wed, 8 Feb 2012 14:04:10 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id C23928FC1D; Wed, 8 Feb 2012 14:04:09 +0000 (UTC) Received: by iaeo4 with SMTP id o4so1256087iae.13 for ; Wed, 08 Feb 2012 06:04:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=+J0xlP6I1mhPtm6JmLkzX/MN5IuuYy8GGNP+ZtNP1Kg=; b=IcivfqBr1JdiPJTEbuhNzxbMoy0aPAGOXDihScdkTJl+EJJe4gSjJAyxiQ4Lwiphbx 4PKQM9TNS56QNSlMr89Ra2KehIAr3kf+Mih1iJn2pmFz+a7nkgW66YsVf1y0kHuqWO3H OxAXXJz5HETWCSfWxVMZwtxrYIBGs7vRRP0b0= MIME-Version: 1.0 Received: by 10.42.144.69 with SMTP id a5mr27139143icv.45.1328709849420; Wed, 08 Feb 2012 06:04:09 -0800 (PST) Sender: ermal.luci@gmail.com Received: by 10.231.134.198 with HTTP; Wed, 8 Feb 2012 06:04:09 -0800 (PST) In-Reply-To: <20120208133559.GK13554@FreeBSD.org> References: <20120131110204.GA95472@onelab2.iet.unipi.it> <20120208133559.GK13554@FreeBSD.org> Date: Wed, 8 Feb 2012 15:04:09 +0100 X-Google-Sender-Auth: 0aFIRkQDzHRwTd5nBZ_gaj2_wVQ Message-ID: From: =?ISO-8859-1?Q?Ermal_Lu=E7i?= To: Gleb Smirnoff Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net , Luigi Rizzo , freebsd-hackers@freebsd.org Subject: Re: [PATCH] multiple instances of ipfw(4) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 14:04:10 -0000 2012/2/8 Gleb Smirnoff : > On Tue, Jan 31, 2012 at 12:02:04PM +0100, Luigi Rizzo wrote: > L> if i understand what the patch does, i think it makes sense to be > L> able to hook ipfw instances to specific interfaces/sets of interfaces, > L> as it permits the writing of more readable rulesets. Right now the > L> workaround is start the ruleset with skipto rules matching on > L> interface names, and then use some discipline in "reserving" a range > L> of rule numbers to each interface. > > This is definitely a desired feature, but it should be implemented > on level of pfil(9). However, that would still require multiple > instances of ipfw(4). > This opens a discussion of architecture design. I do not think presently pfil(9) is designed to handle such thing! Regards, Ermal From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 8 14:09:23 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6AD3D1065781; Wed, 8 Feb 2012 14:09:23 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.64.117]) by mx1.freebsd.org (Postfix) with ESMTP id E54448FC14; Wed, 8 Feb 2012 14:09:22 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.5/8.14.5) with ESMTP id q18E9Lil023577; Wed, 8 Feb 2012 18:09:21 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.5/8.14.5/Submit) id q18E9Lfq023576; Wed, 8 Feb 2012 18:09:21 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Wed, 8 Feb 2012 18:09:21 +0400 From: Gleb Smirnoff To: Ermal Lu?i Message-ID: <20120208140921.GM13554@glebius.int.ru> References: <20120131110204.GA95472@onelab2.iet.unipi.it> <20120208133559.GK13554@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net , Luigi Rizzo , freebsd-hackers@FreeBSD.org Subject: Re: [PATCH] multiple instances of ipfw(4) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 14:09:23 -0000 On Wed, Feb 08, 2012 at 03:04:09PM +0100, Ermal Lu?i wrote: E> 2012/2/8 Gleb Smirnoff : E> > On Tue, Jan 31, 2012 at 12:02:04PM +0100, Luigi Rizzo wrote: E> > L> if i understand what the patch does, i think it makes sense to be E> > L> able to hook ipfw instances to specific interfaces/sets of interfaces, E> > L> as it permits the writing of more readable rulesets. Right now the E> > L> workaround is start the ruleset with skipto rules matching on E> > L> interface names, and then use some discipline in "reserving" a range E> > L> of rule numbers to each interface. E> > E> > This is definitely a desired feature, but it should be implemented E> > on level of pfil(9). However, that would still require multiple E> > instances of ipfw(4). E> > E> This opens a discussion of architecture design. E> I do not think presently pfil(9) is designed to handle such thing! Several years ago, I guess around 2005, a discussion on a per-interface packet filtering was taken on the net@ mailing list. In that time, it lead to nothing, several people were against the idea. Recently on IRC I had raised the discussion again. Today more people liked the idea and found it a desired feature. Many kinds of high end networking equipment have per-interface ACLs. I know that networking sysadmins would be happy if FreeBSD packet filters would get this feature, since maintaing such ACLs is much easier on a router with dozens of interfaces. -- Totus tuus, Glebius. From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 8 16:32:48 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71C30106564A for ; Wed, 8 Feb 2012 16:32:48 +0000 (UTC) (envelope-from ansarm@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4F7188FC13 for ; Wed, 8 Feb 2012 16:32:42 +0000 (UTC) Received: by pbdv10 with SMTP id v10so861325pbd.13 for ; Wed, 08 Feb 2012 08:32:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=HUnm2ZBm1/JVazvnF29AK/+gcX8RfI1O9MZKQBX3GGI=; b=iXAstwCSlSGcrzq9O5Upx2otDiK1/nnNNqzBWOtzGl0+dG2OAtIStDxcJWWJxM7NNh E9Qh41zGAHW/hwFKVvkQg9k1PP1c7sNNgn1hcQMzhM/OupC74hLv1ABw2uiVqLWuOnmv nRIm6jxqQu83wntpzCDbwpZ4x5Qnw3hbZ3n1s= MIME-Version: 1.0 Received: by 10.68.222.194 with SMTP id qo2mr69271375pbc.20.1328711849990; Wed, 08 Feb 2012 06:37:29 -0800 (PST) Received: by 10.68.223.101 with HTTP; Wed, 8 Feb 2012 06:37:29 -0800 (PST) Date: Wed, 8 Feb 2012 06:37:29 -0800 Message-ID: From: Ansar Mohammed To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Kerberos and FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 16:32:48 -0000 Hello All, Is the port of Heimdal on FreeBSD being maintained? The version that ships with 9.0 seems a bit old. #> /usr/libexec/kdc-v kdc (Heimdal 1.1.0) Copyright 1995-2008 Kungliga Tekniska H=F6gskolan Send bug-reports to heimdal-bugs@h5l.org From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 8 16:57:49 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45E301065673 for ; Wed, 8 Feb 2012 16:57:49 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-8.mit.edu (DMZ-MAILSEC-SCANNER-8.MIT.EDU [18.7.68.37]) by mx1.freebsd.org (Postfix) with ESMTP id D531E8FC13 for ; Wed, 8 Feb 2012 16:57:48 +0000 (UTC) X-AuditID: 12074425-b7f4a6d0000008e0-b9-4f32a6068047 Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) by dmz-mailsec-scanner-8.mit.edu (Symantec Messaging Gateway) with SMTP id 13.BB.02272.606A23F4; Wed, 8 Feb 2012 11:42:46 -0500 (EST) Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id q18Ggkk5014420; Wed, 8 Feb 2012 11:42:46 -0500 Received: from multics.mit.edu (MULTICS.MIT.EDU [18.187.1.73]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q18GgiIQ006221 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 8 Feb 2012 11:42:45 -0500 (EST) Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id q18GgivB007639; Wed, 8 Feb 2012 11:42:44 -0500 (EST) Date: Wed, 8 Feb 2012 11:42:44 -0500 (EST) From: Benjamin Kaduk To: Ansar Mohammed In-Reply-To: Message-ID: References: User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-242928814-1328719364=:882" X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprBKsWRmVeSWpSXmKPExsUixCmqrMu2zMjf4Po/fouV59YzWWzf/I/R gcljxqf5LB47Z91lD2CK4rJJSc3JLEst0rdL4MpYs+swS8FOjoqJPVdYGxi/snUxcnBICJhI 7HnL2MXICWSKSVy4t54NxBYS2Mco0T2HHcJezyjx9El6FyMXkL2fSeLPvE2MEIl6iY7tXcwg NouAlsSXy8fBGtgEVCRmvtkINkhEQFVi+fnzLCA2s4C8xIXNh8B6hQUUJa7e+Q5WzykQKPFt 1jqwOK+AvcSUvU+YIeYHSNz5txusV1RAR2L1/iksEDWCEidnPoGa6Sex9vUGtgmMgrOQpGYh Sc0CepNZwFri2XkLiLC2xP2bbWwLGFlWMcqm5Fbp5iZm5hSnJusWJyfm5aUW6Vro5WaW6KWm lG5iBIU0u4vqDsYJh5QOMQpwMCrx8HIcMvQXYk0sK67MPcQoycGkJMp7domRvxBfUn5KZUZi cUZ8UWlOavEhRgkOZiUR3qVBQOW8KYmVValF+TApaQ4WJXFeTa13fkIC6YklqdmpqQWpRTBZ dQ4OgavzfzFLseTl56UqSfByLwWaL1iUmp5akZaZU4JQycTBCbKHB2gPI0gNb3FBYm5xZjpE /hSjLsedc5/PMwqBDZIS5zUCKRIAKcoozYObA0tQrxjFgT4U5hUFqeIBJje4Sa+AljABLUlh AnmmuCQRISXVwHjIsab++lyzSX5mVl/7ZGdxr1B69DhvH8vnRWL35GZzBWb0+y7JfvFpQpSN 76xTJxrXnoqu4kri3LLSb+0nL8mAUzFb61VSAsRW38pr833SdJA7KOCfdePUPT4Xjp5suZDl 4HP8c0lzVPx5zYicZZqBaRe/xewRfW0ZcLJ3ZdOGWR7Sb2c+bVFiKc5INNRiLipOBAAfwaPa KwMAAA== Cc: freebsd-hackers@freebsd.org Subject: Re: Kerberos and FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 16:57:49 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---559023410-242928814-1328719364=:882 Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 8 Feb 2012, Ansar Mohammed wrote: > Hello All, > Is the port of Heimdal on FreeBSD being maintained? The version that > ships with 9.0 seems a bit old. > > #> /usr/libexec/kdc-v > kdc (Heimdal 1.1.0) > Copyright 1995-2008 Kungliga Tekniska H=F6gskolan > Send bug-reports to heimdal-bugs@h5l.org My understanding is that every five years or so, someone becomes fed up=20 enough with the staleness of the "current" version and puts in the effort= =20 to merge in a newer version. It looks like 3 years ago, dfr brought in that Heimdal 1.1 you see, to=20 replace the Heimdal 0.6 that nectar brought in 8 years ago. I don't know of anyone with active plans to bring in a new version, at=20 present. -Ben Kaduk ---559023410-242928814-1328719364=:882-- From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 8 22:00:28 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC6A5106564A for ; Wed, 8 Feb 2012 22:00:28 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id B99868FC0A for ; Wed, 8 Feb 2012 22:00:28 +0000 (UTC) Received: by pbdv10 with SMTP id v10so1167395pbd.13 for ; Wed, 08 Feb 2012 14:00:28 -0800 (PST) Received: by 10.68.75.199 with SMTP id e7mr72148013pbw.128.1328732030987; Wed, 08 Feb 2012 12:13:50 -0800 (PST) Received: from [10.254.254.77] (ppp95-165-143-249.pppoe.spdop.ru. [95.165.143.249]) by mx.google.com with ESMTPS id r10sm770578pbs.12.2012.02.08.12.13.48 (version=SSLv3 cipher=OTHER); Wed, 08 Feb 2012 12:13:50 -0800 (PST) Message-ID: <4F32D779.60209@zonov.org> Date: Thu, 09 Feb 2012 00:13:45 +0400 From: Andrey Zonov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.24) Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: =?UTF-8?B?xYF1a2FzeiBLdXJlaw==?= References: <192396d2.4ca0430d.4f0baa06.40411@o2.pl> In-Reply-To: <192396d2.4ca0430d.4f0baa06.40411@o2.pl> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org Subject: Re: backup BIOS settings X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 22:00:28 -0000 On 10.01.2012 7:01, Åukasz Kurek wrote: > Hi, > Is it possible to backup BIOS settings (CMOS configuration) to file and restore this settings on the other machine (the same hardware configuration and the same BIOS)? > > I try do it for this way: > > kldload nvram > > dd if=/dev/nvram of=nvram.bin (backup) > > dd if=nvram.bin of=/dev/nvram (restore) > > > but this way always load default BIOS settings, not my (probably there is some kind of error). Try sysutils/nvramtool instead. -- Andrey Zonov From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 9 01:41:36 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D86B9106566B for ; Thu, 9 Feb 2012 01:41:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 962B78FC12 for ; Thu, 9 Feb 2012 01:41:36 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EAJIjM0+DaFvO/2dsb2JhbABDDoUBqwuBcgEBBSNWGxgCAg0ZAkgRBhOvIpFngS+KDgYCBhoMAwIFBAEGBQIEBwIdAQMCFASDHgEGAQ0GAoNAgRYEiEaMZ5ImVQ X-IronPort-AV: E=Sophos;i="4.73,387,1325480400"; d="scan'208";a="158799809" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 08 Feb 2012 20:41:35 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 80C1BB4041; Wed, 8 Feb 2012 20:41:35 -0500 (EST) Date: Wed, 8 Feb 2012 20:41:35 -0500 (EST) From: Rick Macklem To: Benjamin Kaduk Message-ID: <487167524.1045003.1328751695510.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-hackers@freebsd.org, Ansar Mohammed Subject: Re: Kerberos and FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 01:41:36 -0000 Benjamin Kaduk wrote: > On Wed, 8 Feb 2012, Ansar Mohammed wrote: >=20 > > Hello All, > > Is the port of Heimdal on FreeBSD being maintained? The version that > > ships with 9.0 seems a bit old. > > > > #> /usr/libexec/kdc-v > > kdc (Heimdal 1.1.0) > > Copyright 1995-2008 Kungliga Tekniska H=C3=B6gskolan > > Send bug-reports to heimdal-bugs@h5l.org >=20 > My understanding is that every five years or so, someone becomes fed > up > enough with the staleness of the "current" version and puts in the > effort > to merge in a newer version. > It looks like 3 years ago, dfr brought in that Heimdal 1.1 you see, to > replace the Heimdal 0.6 that nectar brought in 8 years ago. > I don't know of anyone with active plans to bring in a new version, at > present. >=20 > -Ben Kaduk >=20 I think it's a little trickier than it sounds. The Kerberos in FreeBSD isn't vanilla Heimdal 1.1, but a somewhat modified variant. Heimdal libraries have a separate source file for each function, plus a source file that defines all global storage used by functions in the library. One difference w.r.t. the FreeBSD variant that I am aware of is: - Some of the functions were moved from one library to another. (I don't know why, but maybe it was to avoid a POLA violation which would require apps to be linked with additional libraries?) - To do this, some global variables were added to the source file in the library these functions were moved to. As such, if you statically link an app. to both libraries, the global varia= ble can come up "multiply defined". (I ran into this when I was developing a "g= ssd" prior to the one introduced as part of the kernel rpc.) You can get around = this by dynamically linking, being careful about the order in which the librarie= s are specified. (The command "krb5-config --libs" helps w.r.t. this.) I don't know what else was changed, but I do know that it isn't as trivial = as replacing the sources with ones from a newer Heimdal release. I think it would be nice if a newer Heimdal release was brought it, with th= e minimal changes required to make it work. (If that meant that apps. needed = more libraries, the make files could use "krb5-config --libs" to handle it, I th= ink?) Oh, and I'm not volunteering to try and do it;-) rick From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 9 06:34:01 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 745541065677 for ; Thu, 9 Feb 2012 06:34:01 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (kozubik.com [216.218.240.130]) by mx1.freebsd.org (Postfix) with ESMTP id 5D58F8FC16 for ; Thu, 9 Feb 2012 06:34:01 +0000 (UTC) Received: from kozubik.com (localhost [127.0.0.1]) by kozubik.com (8.14.3/8.14.3) with ESMTP id q196XxEt074583 for ; Wed, 8 Feb 2012 22:33:59 -0800 (PST) (envelope-from john@kozubik.com) Received: from localhost (john@localhost) by kozubik.com (8.14.3/8.14.3/Submit) with ESMTP id q196XsX0074580 for ; Wed, 8 Feb 2012 22:33:54 -0800 (PST) (envelope-from john@kozubik.com) Date: Wed, 8 Feb 2012 22:33:54 -0800 (PST) From: John Kozubik To: freebsd-hackers@freebsd.org In-Reply-To: <4F213CEB.4020207@herveybayaustralia.com.au> Message-ID: References: <20120119005658.218280@gmx.com> <4F19188A.4090907@herveybayaustralia.com.au> <4F213CEB.4020207@herveybayaustralia.com.au> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: Moving on ... (was: Re: FreeBSD has serious problems with...) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 06:34:01 -0000 Hello, On Thu, 26 Jan 2012, Da Rock wrote: > I normally hate to dredge up old threads, but this is like getting halfway > through a story and not finding out the ending... :) > > What is the answer? Is there a solution to this? When I wrote the original post, I was expecting at most a benign response, and at worst some real blowback ... but I was really pleasantly surprised to find that my complaints were very well received, and that a lot of folks were experiencing the same issues that I was. It appears to be a classic case of "if one customer complains, 99 others are thinking the same thing". > I have a string of questions on this: > > 1. Incidentally, what exactly does constitute a major release? I was defining major releases to be 4.x, 5.x, 6.x ... and so on. Currently 9.x is the latest major release. > 2. Is there a reason to update the numbers so quickly? I didn't think so, which was one of the main points of my post. A lot of other folks have agreed. BUT there were some counter arguments - specifically that fully new features would be delayed for a much longer time AND that there would be large architectural gulfs between major releases. For instance, we might have waited an extra 3-5 years for any ZFS support at all in a -RELEASE, and when it appeared, it would introduce a big upgrade hurdle, as it would be accompanied by major underlying changes in system architecture. But my counter to this was that a lot of these features that we did get introduced to, earlier, were in fact not really usable anyway. For instance, my firm(s) have not even considered production use of ZFS until 8.3-RELEASE. So the question remains, where do we set that dial ? If it were up to me, I think I would stake out a very loud and clear mission statement for FreeBSD that explicitly sacrifices new features for longer lifecycles and deeper investment in particular releases. I've thought seriously about the points that were made in this thread supporting faster releases, and I do appreciate what we would be giving up, but I continue to advocate for a new major release every 5 years. I don't think that's going to happen, but based on this discussion and feedback, it appears that we're going to get more frequent minor releases - possibly 3-4 per year - which makes me very happy. > 3. Could a higher bar be set to reach a major release than simply temporal > objectives? One of the differentiating factors between linux and FreeBSD is > the simple fact that linux distros tend to run so fast through the numbers- > and while just a matter of perspective, it could provide some sense of > stability to enterprise users. Weighed against, of course, the ability to > upgrade easily. I think temporal objectives are decent, provided they are long :) Long timelines give people and organizations incentive to invest time and money into a thing. I know very well of several investments in FreeBSD that my firm(s) did NOT make because we didn't think it was worth it, given that the release, and underlying arch, were going away. > 4. If in the case of the former, could some backporting to the stable and > release branches facilitate an easy upgrade to the next major release? > > 5. Could binaries be the answer to easier upgrades (customised for enterprise > users)? I think others have had better comments and insight as far as those two points are concerned. I think you would be well served to read the entire comment thread, and just ignore all of the posts that are speaking about PRs - they were off-topic almost immediately and devolved from there. A good tl;dr is from a post I made early on: You could progress 8.x along its current trajectory, possibly building 8.4 a year or so from now and then be done with it, and then that would be the last short/unfocused release. Then you postpone 10.0-RELEASE until January 2017. Instead of having a legacy branch and two production branches, you would have legacy (8) production (9) and ... nothing. Or if you need to have it out there, 10 is the development branch. Minor releases come out 2-3 times per year, which gets you to 9.10 or 9.15 at the end of the cycle. Again, I don't think this is how it will pan out, but it was my favorite scenario. I'd be very happy and satisfied if we just got: a) more frequent (3-4) minor releases b) just one version declared as "production" at any given time[1] And I'd be pretty happy if we just got (a), which I think we will. --john [1] There were a lot of "me too" posts about more frequent minor releases, and the short lifecycle of major releases, but I was surprised that nobody else was bothered by the existence of two simultaneous "production" releases. To me it seems obviously distracting and confusing - and results in new revisions of drivers, etc., skipping the earlier "production" release. No matter where we decide to set the lifecycle dial for major releases, I really think we should rename the later one "development" ... From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 9 07:35:25 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 016B5106566B for ; Thu, 9 Feb 2012 07:35:25 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id C7AFD8FC08 for ; Thu, 9 Feb 2012 07:35:24 +0000 (UTC) Received: by pbcxa7 with SMTP id xa7so252432pbc.13 for ; Wed, 08 Feb 2012 23:35:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=Kw6SYitbuoORSD9K1u+Iicalq1EzT5nIt5e39Mwh+q0=; b=PD43x3TPcx03zhvVexI9ro+YFVWpkn4MomXyQ23hTtzt2xX6o4rt1xtB39pCe4ldDe PbGgikKAAurQTaMo+yPlbkWJHw/ibphMzAWQA+TxZjrxhHd2Nw52zYmnWz7Li7S75Map +BnGWYDq+YREZg1ooog2c4HeL4MIIwgvdpHjk= MIME-Version: 1.0 Received: by 10.68.74.134 with SMTP id t6mr3136188pbv.26.1328771360423; Wed, 08 Feb 2012 23:09:20 -0800 (PST) Sender: mdf356@gmail.com Received: by 10.68.131.9 with HTTP; Wed, 8 Feb 2012 23:09:20 -0800 (PST) In-Reply-To: References: <20120119005658.218280@gmx.com> <4F19188A.4090907@herveybayaustralia.com.au> <4F213CEB.4020207@herveybayaustralia.com.au> Date: Wed, 8 Feb 2012 23:09:20 -0800 X-Google-Sender-Auth: vCyfmAzH3tvIA7k05QxQZg0-5P4 Message-ID: From: mdf@FreeBSD.org To: John Kozubik Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Moving on ... (was: Re: FreeBSD has serious problems with...) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 07:35:25 -0000 On Wed, Feb 8, 2012 at 10:33 PM, John Kozubik wrote: > If it were up to me, I think I would stake out a very loud and clear miss= ion > statement for FreeBSD that explicitly sacrifices new features for longer > lifecycles and deeper investment in particular releases. =A0I've thought > seriously about the points that were made in this thread supporting faste= r > releases, and I do appreciate what we would be giving up, but I continue = to > advocate for a new major release every 5 years. To summarize 7 years of working on AIX, where we upgrade the major version number every 5 years or so, in the interim we do deliver hundreds of other features. However, if we made a poor choice for interface, we're stuck with it. This leads to two things: (1) flexible interfaces, like leaving an unused flags field in a syscall so it can become variadic later, and (2) flexible structures, like leaving a few bytes in a struct that's part of the KBI so they can be filled in, and when that space is nearly out, malloc(9) more storage to hang off the struct. Sometimes this means a lot of extra testing effort to ensure the legacy interface works as well as any extensions that arise later. It's not ideal, it's more work when designing and testing, and when a major release does come around where we could break binary compatibility, we've often forgotten about a lot of these little bits and just leave things the way they are. With a more frequent release cycle FreeBSD can be more free about breaking KBI and ABI, and even the [AK]PI, which keeps the current code clean but makes upgrading more difficult. It's a hassle. It's doable, but it adds work, and that's a tough call to make for a volunteer effort. Increasing the barrier to entry, or reducing the "fun" part of working on FreeBSD doesn't serve the project either. Just my 2 cents, a month late. Cheers, matthew From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 9 13:28:17 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C714B106566C for ; Thu, 9 Feb 2012 13:28:17 +0000 (UTC) (envelope-from freebsd-hackers@herveybayaustralia.com.au) Received: from mail.unitedinsong.com.au (mail.unitedinsong.com.au [150.101.178.33]) by mx1.freebsd.org (Postfix) with ESMTP id 1CDF38FC0A for ; Thu, 9 Feb 2012 13:28:16 +0000 (UTC) Received: from mail.unitedinsong.com.au (bell.herveybayaustralia.com.au [192.168.0.40]) by mail.unitedinsong.com.au (Postfix) with ESMTP id 0C27F5C2B for ; Thu, 9 Feb 2012 23:40:52 +1000 (EST) Received: from laptop1.herveybayaustralia.com.au (laptop1.herveybayaustralia.com.au [192.168.0.177]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.unitedinsong.com.au (Postfix) with ESMTPSA id DA0025C29 for ; Thu, 9 Feb 2012 23:40:51 +1000 (EST) Message-ID: <4F33C8DE.8020609@herveybayaustralia.com.au> Date: Thu, 09 Feb 2012 23:23:42 +1000 From: Da Rock <9Phackers@herveybayaustralia.com.au> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111109 Thunderbird/7.0.1 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <20120119005658.218280@gmx.com> <4F19188A.4090907@herveybayaustralia.com.au> <4F213CEB.4020207@herveybayaustralia.com.au> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 09 Feb 2012 13:55:42 +0000 Subject: Re: Moving on ... X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-hackers@freebsd.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 13:28:18 -0000 On 02/09/12 16:33, John Kozubik wrote: > > Hello, > > On Thu, 26 Jan 2012, Da Rock wrote: > >> I normally hate to dredge up old threads, but this is like getting >> halfway through a story and not finding out the ending... :) >> >> What is the answer? Is there a solution to this? > > > When I wrote the original post, I was expecting at most a benign > response, and at worst some real blowback ... but I was really > pleasantly surprised to find that my complaints were very well > received, and that a lot of folks were experiencing the same issues > that I was. > > It appears to be a classic case of "if one customer complains, 99 > others are thinking the same thing". I was interested from the perspective of what would make FreeBSD a bigger player on the enterprise level of the market. I know ISP's and large organisations are not as concerned with "off the shelf" as much as scalability, automation and customisation, and so don't mind running a minor dev team/person to achieve what they want; but smaller organisations would rather have it just work without much input or technical ability. I don't know what it is like where you are, but here even larger organisations are biased in this way- even up to a government level (outsource!). >> I have a string of questions on this: >> >> 1. Incidentally, what exactly does constitute a major release? > I was defining major releases to be 4.x, 5.x, 6.x ... and so on. > Currently 9.x is the latest major release. To clarify my question (I thought it was a little corny to put it this way, a little too shakespearean :) ), what is in a major release? I suppose it is related to my later question regarding the objectives of a release. >> 2. Is there a reason to update the numbers so quickly? > I didn't think so, which was one of the main points of my post. A lot > of other folks have agreed. BUT there were some counter arguments - > specifically that fully new features would be delayed for a much > longer time AND that there would be large architectural gulfs between > major releases. > > For instance, we might have waited an extra 3-5 years for any ZFS > support at all in a -RELEASE, and when it appeared, it would introduce > a big upgrade hurdle, as it would be accompanied by major underlying > changes in system architecture. > > But my counter to this was that a lot of these features that we did > get introduced to, earlier, were in fact not really usable anyway. > For instance, my firm(s) have not even considered production use of > ZFS until 8.3-RELEASE. > > So the question remains, where do we set that dial ? > > If it were up to me, I think I would stake out a very loud and clear > mission statement for FreeBSD that explicitly sacrifices new features > for longer lifecycles and deeper investment in particular releases. > I've thought seriously about the points that were made in this thread > supporting faster releases, and I do appreciate what we would be > giving up, but I continue to advocate for a new major release every 5 > years. > > I don't think that's going to happen, but based on this discussion and > feedback, it appears that we're going to get more frequent minor > releases - possibly 3-4 per year - which makes me very happy. That sounds fair to increase the lifecycle to 5 years, but I think hardware support (at least) needs to be available much sooner; perhaps included in the minor releases. Otherwise users will just go elsewhere (large corps or desktop users- mostly desktop though), it was trying even for me as a stubborn mule. Most haven't the patience that I do. I'm not 100% sure how to do this though... >> 3. Could a higher bar be set to reach a major release than simply >> temporal objectives? One of the differentiating factors between linux >> and FreeBSD is the simple fact that linux distros tend to run so fast >> through the numbers- and while just a matter of perspective, it could >> provide some sense of stability to enterprise users. Weighed against, >> of course, the ability to upgrade easily. > I think temporal objectives are decent, provided they are long :) > Long timelines give people and organizations incentive to invest time > and money into a thing. I know very well of several investments in > FreeBSD that my firm(s) did NOT make because we didn't think it was > worth it, given that the release, and underlying arch, were going away. Fair enough. I left linux for the same reason (but that was exceedingly extreme). For the same reason drifting [ak]bi between major releases is probably a bad idea - that would be an imitation of what most of us have left in linux. >> 4. If in the case of the former, could some backporting to the stable >> and release branches facilitate an easy upgrade to the next major >> release? >> >> 5. Could binaries be the answer to easier upgrades (customised for >> enterprise users)? > I think others have had better comments and insight as far as those > two points are concerned. > > I think you would be well served to read the entire comment thread, > and just ignore all of the posts that are speaking about PRs - they > were off-topic almost immediately and devolved from there. I did. I came across it about a week in, and it was one of the pr comments that was cross posted which I stumbled on and I searched for its root. I've been following from the very first post. Hence my interest :) > A good tl;dr is from a post I made early on: > > > > > You could progress 8.x along its current trajectory, possibly building > 8.4 a year or so from now and then be done with it, and then that > would be the last short/unfocused release. > > Then you postpone 10.0-RELEASE until January 2017. > > Instead of having a legacy branch and two production branches, you > would have legacy (8) production (9) and ... nothing. Or if you need > to have it out there, 10 is the development branch. > > Minor releases come out 2-3 times per year, which gets you to 9.10 or > 9.15 at the end of the cycle. > > > > > Again, I don't think this is how it will pan out, but it was my > favorite scenario. I'd be very happy and satisfied if we just got: > > a) more frequent (3-4) minor releases > > b) just one version declared as "production" at any given time[1] > > And I'd be pretty happy if we just got (a), which I think we will. > > > --john > > > > > [1] There were a lot of "me too" posts about more frequent minor > releases, and the short lifecycle of major releases, but I was > surprised that nobody else was bothered by the existence of two > simultaneous "production" releases. To me it seems obviously > distracting and confusing - and results in new revisions of drivers, > etc., skipping the earlier "production" release. > > No matter where we decide to set the lifecycle dial for major > releases, I really think we should rename the later one "development" ... You know, I looked at the 7.x branch (I was sure I installed 7.2 back in 2009) and it only needs another year to reach your goal of 5 years. I know something like this was mentioned before, but... it does seem strange to fall short by just "that much" (to quote Smart). The multiple "cultivated" branches seem a little strange to me too, but I think there may yet another "ghost" branch which has all the "what ifs". It seems to me that some efforts towards the start of a major release "calving" that some sub projects are already distracted by getting the new features in the "ghost" in the next current (that was probably already mentioned too, come to think of it :) meh.. ). I think using svn may relieve some of those effects, allowing cherry picking. Perhaps provide more complete features than half done in any release? IF svn was used primarily, would this allow the development team to focus more on the production branch, with each developer allowed a branch to themselves that they can break whatever, and with new api, abi, kbi, kpi, and so on and so forth put to the test (once coordinated and ready) in the development branch? This would mean 2 branches (dev and production), long lifecycle, hardware support (and maybe feature) in the production branch, and a (relatively) stable development branch. Once time was up (say 5 years), then the dev branch can calve safely without being too broken (or at least find a snapshot that was stable) and release a new major branch, instead of waiting for things to finish. Obviously then any things that finish after this can be introduced in the next minor release safely. (some of this could already have been proposed- this is probably an amalgamation) This would probably drop legacy though- goes the other way, sorry :) What this means is there would be 9.x and 10.x, with 9.x available for 5 years with active support, and 10.x being basted in low hot oven throughout that time. That does provide a good focus, but there is the marketing for volunteer support to consider I suppose. I would think that private branches might alleviate that a bit though. No matter what the lifecycle/branches/etc hardware is a big barrier if you're a user at any level. If the hardware doesn't work you're screwed, if software doesn't you can patch it easier. I think its harder to patch a device driver than a software app if you don't have any experience in driver development. You can also find a work around easier (my experience, others may vary) for software than hardware. So if only 2 branches then hardware needs to be fully supported in the production version. What a dream... I think in the long run a longer cycle and less branches may possibly even help the driver development to better support hardware in any branch, so maybe our ideas aren't so far apart? I think I have let my brain ramble enough through my musings here... From owner-freebsd-hackers@FreeBSD.ORG Thu Feb 9 22:45:39 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A7A8E106566C; Thu, 9 Feb 2012 22:45:39 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 5A29B8FC08; Thu, 9 Feb 2012 22:45:38 +0000 (UTC) Received: from julian-mac.elischer.org (64.1.209.194.ptr.us.xo.net [64.1.209.194]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id q19MjZmu063692 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 9 Feb 2012 14:45:36 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4F344CE4.301@freebsd.org> Date: Thu, 09 Feb 2012 14:47:00 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.26) Gecko/20120129 Thunderbird/3.1.18 MIME-Version: 1.0 To: Gleb Smirnoff References: <20120131110204.GA95472@onelab2.iet.unipi.it> <20120208133559.GK13554@FreeBSD.org> <20120208140921.GM13554@glebius.int.ru> In-Reply-To: <20120208140921.GM13554@glebius.int.ru> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: Ermal Lu?i , freebsd-net , Luigi Rizzo , freebsd-hackers@freebsd.org Subject: Re: [PATCH] multiple instances of ipfw(4) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 22:45:39 -0000 On 2/8/12 6:09 AM, Gleb Smirnoff wrote: > On Wed, Feb 08, 2012 at 03:04:09PM +0100, Ermal Lu?i wrote: > E> 2012/2/8 Gleb Smirnoff: > E> > On Tue, Jan 31, 2012 at 12:02:04PM +0100, Luigi Rizzo wrote: > E> > L> if i understand what the patch does, i think it makes sense to be > E> > L> able to hook ipfw instances to specific interfaces/sets of interfaces, > E> > L> as it permits the writing of more readable rulesets. Right now the > E> > L> workaround is start the ruleset with skipto rules matching on > E> > L> interface names, and then use some discipline in "reserving" a range > E> > L> of rule numbers to each interface. > E> > > E> > This is definitely a desired feature, but it should be implemented > E> > on level of pfil(9). However, that would still require multiple > E> > instances of ipfw(4). > E> > > E> This opens a discussion of architecture design. > E> I do not think presently pfil(9) is designed to handle such thing! > > Several years ago, I guess around 2005, a discussion on a per-interface > packet filtering was taken on the net@ mailing list. In that time, it lead > to nothing, several people were against the idea. > > Recently on IRC I had raised the discussion again. Today more people liked > the idea and found it a desired feature. > > Many kinds of high end networking equipment have per-interface ACLs. I know > that networking sysadmins would be happy if FreeBSD packet filters would > get this feature, since maintaing such ACLs is much easier on a router with > dozens of interfaces. I think it is a good idea. not only for interfaces but certain routing and bridging paths too. From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 10 09:00:17 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA10E106566B for ; Fri, 10 Feb 2012 09:00:15 +0000 (UTC) (envelope-from ansarm@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 99D1F8FC0C for ; Fri, 10 Feb 2012 09:00:15 +0000 (UTC) Received: by daec6 with SMTP id c6so2665169dae.13 for ; Fri, 10 Feb 2012 01:00:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=jtv/bzTt2zwSwOvDdZKwbv55Krj5lUdbX+q/tBLIyPs=; b=IhjkZk19f0fg6+Olb6rbQTbmcXL1bvwdy0vuBtajH7iCwMCyzRZ7CUkxQBA5pj8UK+ u/fmSESXhDDJrCvzfnJ2QDwq5fcYegu7hkB3SIkdNpIR0KSJHa7yhLmEvDqUfYi2XkLl h2uhupXgwMAXT8UpRyBGyqV/Jbpn73ToUSjS8= MIME-Version: 1.0 Received: by 10.68.230.6 with SMTP id su6mr14389922pbc.54.1328864085331; Fri, 10 Feb 2012 00:54:45 -0800 (PST) Received: by 10.68.223.101 with HTTP; Fri, 10 Feb 2012 00:54:45 -0800 (PST) In-Reply-To: <487167524.1045003.1328751695510.JavaMail.root@erie.cs.uoguelph.ca> References: <487167524.1045003.1328751695510.JavaMail.root@erie.cs.uoguelph.ca> Date: Fri, 10 Feb 2012 00:54:45 -0800 Message-ID: From: Ansar Mohammed To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Benjamin Kaduk Subject: Re: Kerberos and FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 09:00:17 -0000 Thanks for the feedback. I built world and disabled Kerberos in src.conf. I will just install Heimdal via ports now. There seems to be alot of other rather old bits of software in a default installation. I noticed some old digiboard utility in a base 9.0 build. On Wed, Feb 8, 2012 at 5:41 PM, Rick Macklem wrote: > Benjamin Kaduk wrote: >> On Wed, 8 Feb 2012, Ansar Mohammed wrote: >> >> > Hello All, >> > Is the port of Heimdal on FreeBSD being maintained? The version that >> > ships with 9.0 seems a bit old. >> > >> > #> /usr/libexec/kdc-v >> > kdc (Heimdal 1.1.0) >> > Copyright 1995-2008 Kungliga Tekniska H=F6gskolan >> > Send bug-reports to heimdal-bugs@h5l.org >> >> My understanding is that every five years or so, someone becomes fed >> up >> enough with the staleness of the "current" version and puts in the >> effort >> to merge in a newer version. >> It looks like 3 years ago, dfr brought in that Heimdal 1.1 you see, to >> replace the Heimdal 0.6 that nectar brought in 8 years ago. >> I don't know of anyone with active plans to bring in a new version, at >> present. >> >> -Ben Kaduk >> > I think it's a little trickier than it sounds. The Kerberos in FreeBSD > isn't vanilla Heimdal 1.1, but a somewhat modified variant. > > Heimdal libraries have a separate source file for each function, plus > a source file that defines all global storage used by functions in the > library. > One difference w.r.t. the FreeBSD variant that I am aware of is: > - Some of the functions were moved from one library to another. (I don't > =A0know why, but maybe it was to avoid a POLA violation which would requi= re > =A0apps to be linked with additional libraries?) > =A0- To do this, some global variables were added to the source file in t= he > =A0 =A0library these functions were moved to. > As such, if you statically link an app. to both libraries, the global var= iable > can come up "multiply defined". (I ran into this when I was developing a = "gssd" > prior to the one introduced as part of the kernel rpc.) You can get aroun= d this > by dynamically linking, being careful about the order in which the librar= ies are > specified. (The command "krb5-config --libs" helps w.r.t. this.) > > I don't know what else was changed, but I do know that it isn't as trivia= l as > replacing the sources with ones from a newer Heimdal release. > > I think it would be nice if a newer Heimdal release was brought it, with = the > minimal changes required to make it work. (If that meant that apps. neede= d more > libraries, the make files could use "krb5-config --libs" to handle it, I = think?) > > Oh, and I'm not volunteering to try and do it;-) rick > From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 10 09:01:48 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E2E81065672 for ; Fri, 10 Feb 2012 09:01:48 +0000 (UTC) (envelope-from rank1seeker@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 97C7B8FC13 for ; Fri, 10 Feb 2012 09:01:44 +0000 (UTC) Received: by wgbdq11 with SMTP id dq11so2658839wgb.31 for ; Fri, 10 Feb 2012 01:01:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:from:to:subject:date:content-type :content-transfer-encoding:x-mailer; bh=oRzc45CmnH7LX6EOaw8JXK1TsI4cJauTiI3Xz6GPlec=; b=PEVqySNAquzaJOfFSA4CdGIX9gB54iSTBjO3ll5rhG6R9R1UCIwBLslFJuPpeUxeMX Z0SAoXrpmtbnWhn24tMLCTPAzUwNFq0i7OckeXCKEGp6GyrXeXTTgds/3T1YFxGeuRw4 d9uYl4X4jXEpnEwNofr989gWELC7THEANQvZU= Received: by 10.216.133.205 with SMTP id q55mr468457wei.6.1328864504067; Fri, 10 Feb 2012 01:01:44 -0800 (PST) Received: from DOMYPC ([82.193.208.173]) by mx.google.com with ESMTPS id ec3sm1846403wib.1.2012.02.10.01.01.29 (version=SSLv3 cipher=OTHER); Fri, 10 Feb 2012 01:01:41 -0800 (PST) Message-ID: <20120210.090141.590.3@DOMY-PC> From: rank1seeker@gmail.com To: hackers@freebsd.org Date: Fri, 10 Feb 2012 10:01:41 +0100 Content-Type: text/plain; charset="Windows-1250" Content-Transfer-Encoding: quoted-printable X-Mailer: POP Peeper (3.8.1.0) Cc: Subject: Tiny 'tunefs' bug X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 09:01:48 -0000 Seems like there is a little bug in 'tunefs' binary.=0D=0AWhen I strip = '/dev/' to have a shorter CMD, it reports an error, even it does finish = it's task.=0D=0A=0D=0A# tunefs -j enable = ufsid/4edc992e27d147ce=0D=0Atunefs: Can't stat ufsid/4edc992e27d147ce: No = such file or directory=0D=0AUsing inode 13 in cg 0 for 8388608 byte = journal=0D=0Atunefs: soft updates journaling set=0D=0A=0D=0A# tunefs -p = ufsid/4edc992e27d147ce=0D=0Atunefs: Can't stat ufsid/4edc992e27d147ce: No = such file or directory=0D=0Atunefs: POSIX.1e ACLs: (-a) = disabled=0D=0Atunefs: NFSv4 ACLs: (-N) = disabled=0D=0Atunefs: MAC multilabel: (-l) = disabled=0D=0Atunefs: soft updates: (-n) = enabled=0D=0Atunefs: soft update journaling: (-j) = enabled=0D=0Atunefs: gjournal: (-J) = disabled=0D=0Atunefs: trim: (-t) = disabled=0D=0Atunefs: maximum blocks per file in a cylinder group: = (-e) 2048=0D=0Atunefs: average file size: (-f) = 16384=0D=0Atunefs: average number of files in a directory: (-s) = 64=0D=0Atunefs: minimum percentage of free space: (-m) = 8%=0D=0Atunefs: optimization preference: (-o) = time=0D=0Atunefs: volume label: (-L)=0D=0A=0D=0A=0D=0ADomagoj Smol=E8i=E6 From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 10 10:54:12 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E851F106564A for ; Fri, 10 Feb 2012 10:54:12 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-lpp01m020-f182.google.com (mail-lpp01m020-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 63CFB8FC18 for ; Fri, 10 Feb 2012 10:54:11 +0000 (UTC) Received: by lbbgj3 with SMTP id gj3so1931171lbb.13 for ; Fri, 10 Feb 2012 02:54:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=OHQCJEowmhniGHJQtxDSM1bIHPfIpgcRixEexjbkEmY=; b=JFNCPLss9UpEi/dQBuQLC1eRvPz2H2dD59hEe6fZ97+MNDq791j6lyjpr1u+CqoO1s Z26teHSxftlvhrZP9V6lTqyerkSOjzSHhJTz39UBgX9ybFv5RYew9GKAbUxHmQnAc+gf ntEIejNrzGtI8KHuM63jzroPepgG2qNRyI8lc= MIME-Version: 1.0 Received: by 10.152.122.74 with SMTP id lq10mr4169933lab.7.1328869672618; Fri, 10 Feb 2012 02:27:52 -0800 (PST) Received: by 10.152.18.4 with HTTP; Fri, 10 Feb 2012 02:27:52 -0800 (PST) In-Reply-To: <20120210.090141.590.3@DOMY-PC> References: <20120210.090141.590.3@DOMY-PC> Date: Fri, 10 Feb 2012 13:27:52 +0300 Message-ID: From: Sergey Kandaurov To: rank1seeker@gmail.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: hackers@freebsd.org Subject: Re: Tiny 'tunefs' bug X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 10:54:13 -0000 2012/2/10 : > Seems like there is a little bug in 'tunefs' binary. > When I strip '/dev/' to have a shorter CMD, it reports an error, even it = does finish it's task. > > # tunefs -j enable ufsid/4edc992e27d147ce > tunefs: Can't stat ufsid/4edc992e27d147ce: No such file or directory > Using inode 13 in cg 0 for 8388608 byte journal > tunefs: soft updates journaling set > > # tunefs -p ufsid/4edc992e27d147ce > tunefs: Can't stat ufsid/4edc992e27d147ce: No such file or directory > tunefs: POSIX.1e ACLs: (-a) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0disabled > tunefs: NFSv4 ACLs: (-N) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 disabled > tunefs: MAC multilabel: (-l) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 disabled > tunefs: soft updates: (-n) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 enabled > tunefs: soft update journaling: (-j) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 enabled > tunefs: gjournal: (-J) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 disabled > tunefs: trim: (-t) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 disabled > tunefs: maximum blocks per file in a cylinder group: (-e) =A02048 > tunefs: average file size: (-f) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A016384 > tunefs: average number of files in a directory: (-s) =A0 =A0 =A0 64 > tunefs: minimum percentage of free space: (-m) =A0 =A0 =A0 =A0 =A0 =A0 8% > tunefs: optimization preference: (-o) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0time > tunefs: volume label: (-L) It seems like this was changed with svn rev 207421 in 9.x. --=20 wbr, pluknet From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 10 22:55:02 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BE321065672 for ; Fri, 10 Feb 2012 22:55:02 +0000 (UTC) (envelope-from sushanth_rai@yahoo.com) Received: from nm4-vm0.access.bullet.mail.sp2.yahoo.com (nm4-vm0.access.bullet.mail.sp2.yahoo.com [98.139.44.110]) by mx1.freebsd.org (Postfix) with SMTP id 1CC768FC0A for ; Fri, 10 Feb 2012 22:55:01 +0000 (UTC) Received: from [98.139.44.104] by nm4.access.bullet.mail.sp2.yahoo.com with NNFMP; 10 Feb 2012 22:42:14 -0000 Received: from [98.139.44.90] by tm9.access.bullet.mail.sp2.yahoo.com with NNFMP; 10 Feb 2012 22:42:14 -0000 Received: from [127.0.0.1] by omp1027.access.mail.sp2.yahoo.com with NNFMP; 10 Feb 2012 22:42:14 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 97484.74941.bm@omp1027.access.mail.sp2.yahoo.com Received: (qmail 92341 invoked by uid 60001); 10 Feb 2012 22:42:13 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1328913733; bh=JbPH7NzJvl75hXRy9K15uuYmGWMtMbRERqI255VE9sU=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=nLgTDdkpTVKv0rs9VQaMbntMy+fjFuhotICUzY7+mdLXeXl6FEp2HNwpijYyyS16AGasn9IGyyNCULSMT3ZZVRZzmQF9VYTHOk0fpQJGySJD16siFn1sJB1hvhaqH+JrwSeRoGz1aL+kpuWYaCE6lG3NfwWztOo1dpqh5AZ8STA= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=IcKFRzBl8IL8ildSmGH7CtJ6fuFoDUHgNenFayzoM8tLVneyZ3mFFcdWy8wT6E97UYL99zrvBbr5L9F9bjBbpKU4ekVS/K1hj88p7kUZ97006X2RNaTh/xIiz+HHxmbB+p9u5ZU/mUnPT3jWu26oewOO/VUas9jD3F7blOHMxW4=; X-YMail-OSG: YXwj1JEVM1mWVUR_Eb75aTKsOQ7i7dWERylJZoYadMUT0N8 z5cluQgmmw2dzODhNiR5Ro8T92HbAMsfhw8_6As7H6NEuOPGHK6H_Jmxe8uA ZxRqoq4atuyxJj_2yJ9SVwVQ1pBuV4PGubiip70inLWRtQpzXDdNepc7iuhI KTaLielfNd17z3xJEh2IzvuPg8Tu6XIMWdKpBhskCsGdQQMa2TutMns_e4T5 1QApbYCCvf0dZ_0pBcdayKU_va2l9vZAQ85e7OXLI1lTIwCYCEi5YZAmTffK A5ZE7ZQZzwNGO2H8zoY_JslL5ZwYIdt3R_ut_TGqTnKzafQfCpOF_yuDl5AN Z44P1In853VtM_1BFEnk4d525Lywi2CdfMxVLLIaaK3QM0VIR6vxJgE88h6X 42Mlv_YKJbs.V4FtTF3w_Pq6V9xB8edP0ZCMB9hGt8FXtlLoZtzja7yhOEgW kzLjYdSk- Received: from [209.119.38.67] by web180008.mail.gq1.yahoo.com via HTTP; Fri, 10 Feb 2012 14:42:13 PST X-Mailer: YahooMailClassic/15.0.4 YahooMailWebService/0.8.116.338427 Message-ID: <1328913733.43669.YahooMailClassic@web180008.mail.gq1.yahoo.com> Date: Fri, 10 Feb 2012 14:42:13 -0800 (PST) From: Sushanth Rai To: freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Subject: Generating NMI due to WDT expiry X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 22:55:02 -0000 Basically I would like to force system panic (and take kernel dump) when watchdog time expires. Assuming that timer expired due to some OS bug, kernel memory dump would be very useful. I'm running freebsd 7.2 on Intel IbexPeak chipset. According to specs, the watchdog timer on IbexPeak first generates an SMI and then resets the CPU. Since SMI is handled within the BIOS, is there a way to generate NMI from within BIOS SMI handler ? I see that kernel has support to either enter the debugger or force panic upon receipt of a NMI. This is not necessarily a FreeBSD question, but would like to hear any thoughts/pointers. Thanks, Sushanth From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 11 11:06:55 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6C9A106566B for ; Sat, 11 Feb 2012 11:06:55 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 109208FC19 for ; Sat, 11 Feb 2012 11:06:54 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA28861; Sat, 11 Feb 2012 13:06:50 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1RwAn3-000DCc-So; Sat, 11 Feb 2012 13:06:49 +0200 Message-ID: <4F364BC7.3030906@FreeBSD.org> Date: Sat, 11 Feb 2012 13:06:47 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0) Gecko/20120202 Thunderbird/10.0 MIME-Version: 1.0 To: Sushanth Rai References: <1328913733.43669.YahooMailClassic@web180008.mail.gq1.yahoo.com> In-Reply-To: <1328913733.43669.YahooMailClassic@web180008.mail.gq1.yahoo.com> X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org Subject: Re: Generating NMI due to WDT expiry X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 11:06:55 -0000 on 11/02/2012 00:42 Sushanth Rai said the following: > Basically I would like to force system panic (and take kernel dump) when > watchdog time expires. Assuming that timer expired due to some OS bug, kernel > memory dump would be very useful. I'm running freebsd 7.2 on Intel IbexPeak > chipset. According to specs, the watchdog timer on IbexPeak first generates > an SMI and then resets the CPU. Since SMI is handled within the BIOS, is > there a way to generate NMI from within BIOS SMI handler ? I see that kernel > has support to either enter the debugger or force panic upon receipt of a > NMI. > > This is not necessarily a FreeBSD question, but would like to hear any > thoughts/pointers. See this: http://www.intel.com/content/dam/doc/datasheet/5-chipset-3400-chipset-datasheet.pdf Search for NMI2SMI_EN. Maybe it's what you want. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 11 13:35:16 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 482D9106566C; Sat, 11 Feb 2012 13:35:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id D8B288FC12; Sat, 11 Feb 2012 13:35:14 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA00209; Sat, 11 Feb 2012 15:35:12 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1RwD6e-000DGf-8D; Sat, 11 Feb 2012 15:35:12 +0200 Message-ID: <4F366E8F.9060207@FreeBSD.org> Date: Sat, 11 Feb 2012 15:35:11 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0) Gecko/20120202 Thunderbird/10.0 MIME-Version: 1.0 To: Alexander Motin References: <4F2F7B7F.40508@FreeBSD.org> In-Reply-To: <4F2F7B7F.40508@FreeBSD.org> X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 13:35:16 -0000 on 06/02/2012 09:04 Alexander Motin said the following: > Hi. > > I've analyzed scheduler behavior and think found the problem with HTT. SCHED_ULE > knows about HTT and when doing load balancing once a second, it does right > things. Unluckily, if some other thread gets in the way, process can be easily > pushed out to another CPU, where it will stay for another second because of CPU > affinity, possibly sharing physical core with something else without need. > > I've made a patch, reworking SCHED_ULE affinity code, to fix that: > http://people.freebsd.org/~mav/sched.htt.patch > > This patch does three things: > - Disables strict affinity optimization when HTT detected to let more > sophisticated code to take into account load of other logical core(s). > - Adds affinity support to the sched_lowest() function to prefer specified > (last used) CPU (and CPU groups it belongs to) in case of equal load. Previous > code always selected first valid CPU of evens. It caused threads migration to > lower CPUs without need. > - If current CPU group has no CPU where the process with its priority can run > now, sequentially check parent CPU groups before doing global search. That > should improve affinity for the next cache levels. Alexander, I know that you are working on improving this patch and we have already discussed some ideas via out-of-band channels. Here's some additional ideas. They are in part inspired by inspecting OpenSolaris code. Let's assume that one of the goals of a scheduler is to maximize system performance / computational throughput[*]. I think that modern SMP-aware schedulers try to employ the following two SMP-specific techniques to achieve that: - take advantage of thread-to-cache affinity to minimize "cold cache" time - distribute the threads over logical CPUs to optimize system resource usage by minimizing[**] sharing of / contention over the resources, which could be caches, instruction pipelines (for HTT threads), FPUs (for AMD Bulldozer "cores"), etc. 1. Affinity. It seems that on modern CPUs the caches are either inclusive or some smart "as if inclusive" caches. As a result, if two cores have a shared cache at any level, then it should be relatively cheap to move a thread from one core to the other. E.g. if logical CPUs P0 and P1 have private L1 and L2 caches and a shared L3 cache, then on modern processors it should be much cheaper to move a thread from P0 to P1 than to some processor P2 that doesn't share the L3 cache. If this assumption is really true, then we can track only an affinity of a thread with relation to a top level shared cache. E.g. if migration within an L3 cache is cheap, then we don't have any reason to constrain a migration scope to an L2 cache, let alone L1. 2. Balancing. I think that the current balancing code is pretty good, but can be augmented with the following: A. The SMP topology in longer term should include other important shared resources, not only caches. We already have this in some form via CG_FLAG_THREAD, which implies instruction pipeline sharing. B. Given the affinity assumptions, sched_pickcpu can pick the best CPU only among CPUs sharing a top level cache if a thread still has an affinity to it or among all CPUs otherwise. This should reduce temporary imbalances. C. I think that we should eliminate the bias in the sched_lowest() family of functions. I like how your patch started addressing this. For the cases where the hint (cg_prefer) can not be reasonably picked it should be a pseudo-random value. OpenSolaris does it the following way: http://fxr.watson.org/fxr/ident?v=OPENSOLARIS;im=10;i=CPU_PSEUDO_RANDOM Footnotes: [*] Goals of a scheduler could be controlled via policies. E.g. there could be a policy to reduce power usage. [**] Given a possibility of different policies a scheduler may want to concentrate threads. E.g. if a system has two packages with two cores each and there are two CPU-hungry threads, then the system may place them both on the same package to reduce power usage. Another interesting case is threads that share a VM space or otherwise share some non-trivial amount of memory. As you have suggested, it might make sense to concentrate those threads so that they share a cache. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 11 14:21:29 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4F45106564A; Sat, 11 Feb 2012 14:21:29 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 49CBE8FC0C; Sat, 11 Feb 2012 14:21:29 +0000 (UTC) Received: by eekb47 with SMTP id b47so1448895eek.13 for ; Sat, 11 Feb 2012 06:21:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=dC8fmjhdwxKtmwL4zI3tqHKeh4c3V8/5mbVM72e4d1s=; b=II2c7vAir86WWdDECzyZeMHHv3ww4F4JlL+wKOqMoI0ZsIRUu6bdtyIWw8wHQxECYQ 8aGkQzUj780w8kd7DFyObhibm41KH1fOOV2cNlqkBaA9LvJ/57j3y19viuA3G4nPzemf VU/R1S1b6sAc4lovlBO5z8Rb2YlcjpgkB49Gs= Received: by 10.213.27.148 with SMTP id i20mr933629ebc.78.1328970088281; Sat, 11 Feb 2012 06:21:28 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id n17sm35856865eei.3.2012.02.11.06.21.26 (version=SSLv3 cipher=OTHER); Sat, 11 Feb 2012 06:21:27 -0800 (PST) Sender: Alexander Motin Message-ID: <4F367965.6000602@FreeBSD.org> Date: Sat, 11 Feb 2012 16:21:25 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: Andriy Gapon References: <4F2F7B7F.40508@FreeBSD.org> <4F366E8F.9060207@FreeBSD.org> In-Reply-To: <4F366E8F.9060207@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 14:21:30 -0000 On 02/11/12 15:35, Andriy Gapon wrote: > on 06/02/2012 09:04 Alexander Motin said the following: >> I've analyzed scheduler behavior and think found the problem with HTT. SCHED_ULE >> knows about HTT and when doing load balancing once a second, it does right >> things. Unluckily, if some other thread gets in the way, process can be easily >> pushed out to another CPU, where it will stay for another second because of CPU >> affinity, possibly sharing physical core with something else without need. >> >> I've made a patch, reworking SCHED_ULE affinity code, to fix that: >> http://people.freebsd.org/~mav/sched.htt.patch >> >> This patch does three things: >> - Disables strict affinity optimization when HTT detected to let more >> sophisticated code to take into account load of other logical core(s). >> - Adds affinity support to the sched_lowest() function to prefer specified >> (last used) CPU (and CPU groups it belongs to) in case of equal load. Previous >> code always selected first valid CPU of evens. It caused threads migration to >> lower CPUs without need. >> - If current CPU group has no CPU where the process with its priority can run >> now, sequentially check parent CPU groups before doing global search. That >> should improve affinity for the next cache levels. > > Alexander, > > I know that you are working on improving this patch and we have already > discussed some ideas via out-of-band channels. I've heavily rewritten the patch already. So at least some of the ideas are already addressed. :) At this moment I am mostly satisfied with results and after final tests today I'll probably publish new version. > Here's some additional ideas. They are in part inspired by inspecting > OpenSolaris code. > > Let's assume that one of the goals of a scheduler is to maximize system > performance / computational throughput[*]. I think that modern SMP-aware > schedulers try to employ the following two SMP-specific techniques to achieve that: > - take advantage of thread-to-cache affinity to minimize "cold cache" time > - distribute the threads over logical CPUs to optimize system resource usage by > minimizing[**] sharing of / contention over the resources, which could be > caches, instruction pipelines (for HTT threads), FPUs (for AMD Bulldozer > "cores"), etc. > > 1. Affinity. > It seems that on modern CPUs the caches are either inclusive or some smart "as > if inclusive" caches. As a result, if two cores have a shared cache at any > level, then it should be relatively cheap to move a thread from one core to the > other. E.g. if logical CPUs P0 and P1 have private L1 and L2 caches and a > shared L3 cache, then on modern processors it should be much cheaper to move a > thread from P0 to P1 than to some processor P2 that doesn't share the L3 cache. Absolutely true! On smack-mysql indexed select benchmarks I've found that on Atom CPU with two cores without L3 it is cheaper to move two mysql threads to one physical core (L2 cache) suffering from SMT, then bounce data between cores. Same time on Core i7 with shared L3 and also SMT results are strictly opposite. > If this assumption is really true, then we can track only an affinity of a > thread with relation to a top level shared cache. E.g. if migration within an > L3 cache is cheap, then we don't have any reason to constrain a migration scope > to an L2 cache, let alone L1. In present patch version I've implemented two different thresholds for the last level cache and for the rest. That's why I am waiting from you patch to properly detect cache topologies. :) > 2. Balancing. > I think that the current balancing code is pretty good, but can be augmented > with the following: > A. The SMP topology in longer term should include other important shared > resources, not only caches. We already have this in some form via > CG_FLAG_THREAD, which implies instruction pipeline sharing. At this moment I am using different penalty coefficients for SMT and shared caches (for unrelated processes sharing is is not good). No problem to add more types there. Separate flag for shared FPU could be used to have different penalty coefficients for usual threads and FPU-less kernel threads. > B. Given the affinity assumptions, sched_pickcpu can pick the best CPU only > among CPUs sharing a top level cache if a thread still has an affinity to it or > among all CPUs otherwise. This should reduce temporary imbalances. I've done it in more complicated way. I am doing cache affinity with weight 2 to all paths with running _now_ threads of the same process and with weight 1 to the previous path where thread was running. I believe that constant cache trashing between two running threads is much worse then single jump from one CPU to another on context some switches. Though it could be made configurable. > C. I think that we should eliminate the bias in the sched_lowest() family of > functions. I like how your patch started addressing this. For the cases where > the hint (cg_prefer) can not be reasonably picked it should be a pseudo-random > value. OpenSolaris does it the following way: > http://fxr.watson.org/fxr/ident?v=OPENSOLARIS;im=10;i=CPU_PSEUDO_RANDOM With new math, cases of equal load estimation are more rate, but I've also added alike mechanism. I've just made it not exactly random, but depending on calling thread ID to avoid extra jumping. > Footnotes: > [*] Goals of a scheduler could be controlled via policies. E.g. there could be > a policy to reduce power usage. Good idea. > [**] Given a possibility of different policies a scheduler may want to > concentrate threads. E.g. if a system has two packages with two cores each and > there are two CPU-hungry threads, then the system may place them both on the > same package to reduce power usage. Good idea, but if one CPU is so much burning, difference between C1 and C6 of idle cores/packages will be minimal for system's total. Another question is concentrating periodically called interactive to let other cores/packages to not wake up at all and go into the deepest sleep state. > Another interesting case is threads that share a VM space or otherwise share > some non-trivial amount of memory. As you have suggested, it might make sense > to concentrate those threads so that they share a cache. I am doing it now for threaded processes. It works in described smack-mysql on Atom case if I hint scheduler to be more aggressive with affinity and less with SMT penalties. Unluckily I have no real multisocket system to test that properly. Also I have no idea what to do with processes using shared memory but not using threads (IIRC like Postgress). If there would be some correlation key... -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 11 15:35:41 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44ED9106564A; Sat, 11 Feb 2012 15:35:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id A6EA08FC13; Sat, 11 Feb 2012 15:35:40 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q1BFZaj0040857 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 11 Feb 2012 17:35:36 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q1BFZagM058073; Sat, 11 Feb 2012 17:35:36 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q1BFZZPH058072; Sat, 11 Feb 2012 17:35:35 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 11 Feb 2012 17:35:35 +0200 From: Konstantin Belousov To: Alexander Motin Message-ID: <20120211153535.GT3283@deviant.kiev.zoral.com.ua> References: <4F2F7B7F.40508@FreeBSD.org> <4F366E8F.9060207@FreeBSD.org> <4F367965.6000602@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="RIfOR5N0tZJV8I5p" Content-Disposition: inline In-Reply-To: <4F367965.6000602@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-hackers@freebsd.org, Andriy Gapon Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 15:35:41 -0000 --RIfOR5N0tZJV8I5p Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Feb 11, 2012 at 04:21:25PM +0200, Alexander Motin wrote: > At this moment I am using different penalty coefficients for SMT and=20 > shared caches (for unrelated processes sharing is is not good). No=20 > problem to add more types there. Separate flag for shared FPU could be=20 > used to have different penalty coefficients for usual threads and=20 > FPU-less kernel threads. It is very easy to record the fact of FPU access during the quantum on the context switch-out. So you can at least distinguish numeric code. This can be useful for bulldozer-like machines, if we ever want to optimize for FPU on them. --RIfOR5N0tZJV8I5p Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk82iscACgkQC3+MBN1Mb4jZ0gCdF7mjb+P1V6sWc8+wdfEnlkVO YnMAn3u+jH7at7Ewnkk+l0UCHUeafwL4 =MELH -----END PGP SIGNATURE----- --RIfOR5N0tZJV8I5p-- From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 11 17:05:23 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4230A106564A; Sat, 11 Feb 2012 17:05:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 2BDA98FC12; Sat, 11 Feb 2012 17:05:21 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA01674; Sat, 11 Feb 2012 19:05:20 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1RwGO0-000DMZ-5P; Sat, 11 Feb 2012 19:05:20 +0200 Message-ID: <4F369FCE.7080408@FreeBSD.org> Date: Sat, 11 Feb 2012 19:05:18 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0) Gecko/20120202 Thunderbird/10.0 MIME-Version: 1.0 To: Alexander Motin , freebsd-hackers@FreeBSD.org References: <4F2F7B7F.40508@FreeBSD.org> <4F366E8F.9060207@FreeBSD.org> In-Reply-To: <4F366E8F.9060207@FreeBSD.org> X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 17:05:23 -0000 on 11/02/2012 15:35 Andriy Gapon said the following: > It seems that on modern CPUs the caches are either inclusive or some smart "as > if inclusive" caches. As a result, if two cores have a shared cache at any > level, then it should be relatively cheap to move a thread from one core to the > other. E.g. if logical CPUs P0 and P1 have private L1 and L2 caches and a > shared L3 cache, then on modern processors it should be much cheaper to move a > thread from P0 to P1 than to some processor P2 that doesn't share the L3 cache Having read this paper http://www.cs.uwaterloo.ca/~brecht/courses/856/Possible-Readings/multicore/cache-performance-x86-2009.pdf I think that I have been too optimistic about the smartness of caches in some processors... -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 11 20:45:51 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3004E106566B for ; Sat, 11 Feb 2012 20:45:51 +0000 (UTC) (envelope-from sushanth_rai@yahoo.com) Received: from nm16.access.bullet.mail.sp2.yahoo.com (nm16.access.bullet.mail.sp2.yahoo.com [98.139.44.143]) by mx1.freebsd.org (Postfix) with SMTP id 011048FC0A for ; Sat, 11 Feb 2012 20:45:50 +0000 (UTC) Received: from [98.139.44.105] by nm16.access.bullet.mail.sp2.yahoo.com with NNFMP; 11 Feb 2012 20:45:50 -0000 Received: from [98.139.44.67] by tm10.access.bullet.mail.sp2.yahoo.com with NNFMP; 11 Feb 2012 20:45:50 -0000 Received: from [127.0.0.1] by omp1004.access.mail.sp2.yahoo.com with NNFMP; 11 Feb 2012 20:45:50 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 694286.19581.bm@omp1004.access.mail.sp2.yahoo.com Received: (qmail 26875 invoked by uid 60001); 11 Feb 2012 20:45:50 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1328993150; bh=IxK0FFC5SN8hrRcp/tbBHalefoMGLKdGIrvnFC1MMu0=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=yJJplZdjbU9M6FzpkCG6m/r0kKAi1AZ1AXCT02c0DOt7Chm8FOzo6rpJWSLbXtcjQZ37tvsSauQslbCB2jjARVQ0e68ObvWBFMA92/K+h/pi4Qy919hIlVDxVdzT2SodmSsMP+g4Uwdfi3Jbw+4XE+Ws/5Pp9TAC035TAoU2vPE= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=sNTSEGlUge/o2shGqOsIwIuUpGbfWnFRtCEXuOUlUqEgGTKrCUM8GAmpUmefE4W6tEyj762kob+WlIA1rS4DNs4QQ8fy5eoVX7/g90epdnsSg36J6kvvMOKKAKjBC+tr8nMlFbVXXSv8v/9osgRCt8aNa+JoOX9xcJRmt0hq9cE=; X-YMail-OSG: KDoI7nsVM1llJtpe36spYhpVa_lrYpJmfSpZgcSyjfA5KKL 0_Zs.UFTpqyoWmT33.5qIi75G7fmtVEVucTqrBRxfh6s0wPgna9VGGEUlio0 HjQ3IQ8LIHOPXcf1.qvc3bgA4YhBLj8.YLAAWXNMDh13mwErdz1rRWBpEU1t lPoCctoI0qGRf6ufqsFnxxCWtln4Q85cHEcxp4cPePzPP79JdTjnqVLNQzSD QamBNe1RznMlzFK0Ycs4atX9B7kK.LjOSm2JJUGu8VGL_9HB3wZtAWJznhyZ mPyopDAomCK0mIbxa3bjO9Lghe5n66gCf1WQfQQK_e4ZVjfA96WXsP7eT1ii HBwCev0e7lYf4nCz7E2KS9OYKiE1OqFUWtSz_pln0KRhG4Rkl_N1yvX.q_SJ fKBiB41zbXoPGGekI7v2toheGRxXnNaYSUqvZfNbrLsNPtmy3Mwn83F9.dY6 VQ_2Gv_h9OLyR8tHllKqJM3eXsZeVW6trto2BcnHV_d9LNcr4kfF6eSYKS4k T1uuY7Gy18nn6tvBOf6eQqfzISh96xNYSp.9bJWVmnc52bM3pOxF_ApSew9T 6PnROL2IPKCCjFuIiIFWa0g_uDOU- Received: from [76.211.224.115] by web180005.mail.gq1.yahoo.com via HTTP; Sat, 11 Feb 2012 12:45:50 PST X-Mailer: YahooMailClassic/15.0.4 YahooMailWebService/0.8.116.338427 Message-ID: <1328993150.2019.YahooMailClassic@web180005.mail.gq1.yahoo.com> Date: Sat, 11 Feb 2012 12:45:50 -0800 (PST) From: Sushanth Rai To: Andriy Gapon In-Reply-To: <4F364BC7.3030906@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@FreeBSD.org Subject: Re: Generating NMI due to WDT expiry X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 20:45:51 -0000 I had looked at this. It seems to be doing the opposite of what I want. Tha= t is, it routes a NMI as an SMI and I need SMI to trigger an NMI. Watchdog = timer on 3100 chipset had the ability to send either an NMI or SMI when the= timer fired for the first time. I used NMI to generate kernel panic. With = 3400 no longer generating NMI on WDT expiry, I'm trying to figure out how I= can force memory dump on watchdog expiry. Sushanth =20 --- On Sat, 2/11/12, Andriy Gapon wrote: > From: Andriy Gapon > Subject: Re: Generating NMI due to WDT expiry > To: "Sushanth Rai" > Cc: freebsd-hackers@FreeBSD.org > Date: Saturday, February 11, 2012, 3:06 AM > on 11/02/2012 00:42 Sushanth Rai said > the following: > > Basically I would like to force system panic (and take > kernel dump) when > > watchdog time expires. Assuming that timer expired due > to some OS bug, kernel > > memory dump would be very useful. I'm running freebsd > 7.2 on Intel IbexPeak > > chipset. According to specs, the watchdog timer on > IbexPeak first generates > > an SMI and then resets the CPU. Since SMI is handled > within the BIOS, is > > there a way to generate NMI from within BIOS SMI > handler ? I see that kernel > > has support to either enter the debugger or force panic > upon receipt of a > > NMI. > >=20 > > This is not necessarily a FreeBSD question, but would > like to hear any > > thoughts/pointers. >=20 > See this: > http://www.intel.com/content/dam/doc/datasheet/5-chipset-3400-chipset-dat= asheet.pdf > Search for NMI2SMI_EN.=A0 Maybe it's what you want. >=20 >=20 > --=20 > Andriy Gapon > _______________________________________________ > freebsd-hackers@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " >