From owner-freebsd-arch@FreeBSD.ORG Sun May 20 23:07:57 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B77B616A468 for ; Sun, 20 May 2007 23:07:57 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 69EE913C45B for ; Sun, 20 May 2007 23:07:57 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.101] (c-71-231-138-78.hsd1.or.comcast.net [71.231.138.78]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l4KN7tWD002745 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO) for ; Sun, 20 May 2007 19:07:56 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Sun, 20 May 2007 16:07:53 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: arch@freebsd.org Message-ID: <20070520155103.K632@10.0.0.1> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 May 2007 23:07:57 -0000 Attilio and I have been working on addressing the increasing problem of sched_lock contention on -CURRENT. Attilio has been addressing the parts of the kernel which do not need to fall under the scheduler lock and moving them into seperate locks. For example, the ldt/gdt lock and clock lock which were committed earlier. Also, using atomics for the vmcnt structure. I have been working on an approach to using thread locks rather than a global scheduler lock. The design is similar to Solaris's container locks, but the details are different. The basic idea is to have a pointer in the thread structure that points at a spinlock that protects the thread. This spinlock may be one of the scheduler lock, a turnstile lock, or a sleep queue lock. As the thread changes state from running to blocked on a lock or sleeping the lock changes with it. This has several advantages. The majority of the kernel simply calls thread_lock() which figures out the details. The kernel then knows nothing of the particulars of the scheduler locks, and the schedulers are free to implement them in any way that they like. Furthermore, in some cases the locking is reduced, because locking the thread has the side effect of locking the container. This patch does not implement per-cpu scheduler locks. It just changes the kernel to support this model. I have a fork of ULE in development that runs with per-cpu locks, but it is not ready yet. This means that there should be very little change in system performance until the scheduler catches up. In fact, on a 2cpu system the difference is immeasurable or almost so on every workload I have tested. On an 8way opteron system the results vary between +10% on some reasonable workloads and -15% on super-smack, which has some inherent problems that I believe are not exposing real performance problems with this patch. This has also been tested extensively by Kris and myself on a variety of machines and I believe it to be fairly solid. The only thing remaining to do is fix rusage so that it does not rely on a global scheduler lock. I am posting the patch here in case anyone with specific knowledge of turnstiles, sleepqueues, or signals would like to review it, and as a general heads up to people interested in where the kernel is headed. This will apply to current just prior to my kern_clock.c commits. I will re-merge and update again in the next few days, probably after we sort out rusage. http://people.freebsd.org/~jeff/threadlock.diff Thanks, Jeff From owner-freebsd-arch@FreeBSD.ORG Mon May 21 04:37:31 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7A03116A400 for ; Mon, 21 May 2007 04:37:31 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 2BC6B13C44C for ; Mon, 21 May 2007 04:37:31 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.101] (c-71-231-138-78.hsd1.or.comcast.net [71.231.138.78]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l4L4bSkH055454 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Mon, 21 May 2007 00:37:29 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Sun, 20 May 2007 21:37:25 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: Bruce Evans In-Reply-To: <20070521113648.F86217@besplex.bde.org> Message-ID: <20070520213132.K632@10.0.0.1> References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 04:37:31 -0000 On Mon, 21 May 2007, Bruce Evans wrote: > On Sun, 20 May 2007, Jeff Roberson wrote: > >> Attilio and I have been working on addressing the increasing problem of >> sched_lock contention on -CURRENT. Attilio has been addressing the parts >> of the kernel which do not need to fall under the scheduler lock and moving >> them into seperate locks. For example, the ldt/gdt lock and clock lock >> which were committed earlier. Also, using atomics for the vmcnt structure. > > Using atomics in the vmmeter struct is mostly just a pessimization and > and obfuscation, since locks are still needed for accesses to more > than one variable at a time. For these cases, locks are needed for You are right, there are some cases which this pessimized. I wanted to make sure the cnt members that previously were protected by the sched_lock were still correct. However, I overlooked some of these which were accessed many at a time. What should happen is we should find out if any locks do protect the remaining members and if so, don't use VMCNT*, but mark the header describing how they are protected. > correctness, and are also probably more efficient if there are > 2 > accesses. Then lock poisoning requires the same locks to be held for > updates of the variables (to prevent variables changing while you are > using them), and atomic updates of single variables are a pessimization > too. The VMCNT*() ineterface encorages this pessimization and has other > problems (messes with volatile, and not actually doing atomic accesses > for VMCNT_GET()). Some of the cases where locks aren't needed (mainly > for userland-only statistics) were already handled better by > PCPU_LAZY_INC(). Unfortunately, the method used in PCPU_LAZY_INC() > doesn't work for variables that the kernel wants to access as globals. Depending on the architecture PCPU may be more expensive than atomics so we decied not to use it all over. I'll revisit the places with multiple accesses with Attilio and clean this up. Thanks, Jeff > > Example of code with multiple accesses: > > % /* > % * Return the number of pages we need to free-up or cache > % * A positive number indicates that we do not have enough free pages. > % */ > % % static __inline % int > % vm_paging_target(void) > % { > % return ( > % (VMCNT_GET(free_target) + VMCNT_GET(cache_min)) - % > (VMCNT_GET(free_count) + VMCNT_GET(cache_count)) > % ); > % } > > Without holding a lock throughout this, this returns a garbage value. > Without holding a lock throughout this and the actions taken depending > on the return value, garbage actions may be taken. Values that are > only slightly wrong might work OK, but this is not clear, and if it > does work then volatile variables and [non-]atomic accesses to the > variables to get it. > > Bruce > From owner-freebsd-arch@FreeBSD.ORG Mon May 21 08:48:43 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5F6BF16A469 for ; Mon, 21 May 2007 08:48:43 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.188]) by mx1.freebsd.org (Postfix) with ESMTP id E26B113C4B0 for ; Mon, 21 May 2007 08:48:42 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by mu-out-0910.google.com with SMTP id w8so960096mue for ; Mon, 21 May 2007 01:48:39 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=Ic6sVlWtlqOUMb7LkdAFbtr3tdzuhdxXvYuloTPBEBzatzQ3e+aD8WQmxPcYn19yJUKAiKKWpdFymu1kBD9jlkIuYKmt8UfkB5aJE8fGKlgYqCQGqfr+iJ93QOO/Oqi40TFzTS9E6RkbI4y4485dAR1acDVVW9LlG/RhSATgDIw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=iQnJEKfpspFf+lz9HGKBJqpFRmJujipmmRxvGwpAYYTBwAuzWxB3A/fzk0nEhDgOXt6M9CQozfrkOZdVAYr3//H5MD5YTORH1DRIpzTNW06LWsoRgSa7r3xr+myVJOQwCWABU3QIUyD1L7AWd/c4ARkFzM5B6fGIrdEDIJkSGps= Received: by 10.82.176.3 with SMTP id y3mr8126383bue.1179737319234; Mon, 21 May 2007 01:48:39 -0700 (PDT) Received: from ?172.31.5.25? ( [89.97.252.178]) by mx.google.com with ESMTP id z34sm133543ikz.2007.05.21.01.48.37; Mon, 21 May 2007 01:48:38 -0700 (PDT) Message-ID: <4651CE2F.8080908@FreeBSD.org> Date: Mon, 21 May 2007 18:51:59 +0200 From: Attilio Rao User-Agent: Thunderbird 1.5 (X11/20060526) MIME-Version: 1.0 To: attilio@FreeBSD.org References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> In-Reply-To: <4651CAB8.8070007@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: Attilio Rao Cc: arch@freebsd.org, Bruce Evans Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 08:48:43 -0000 Attilio Rao wrote: > Jeff Roberson wrote: >> >> On Mon, 21 May 2007, Bruce Evans wrote: >> >>> On Sun, 20 May 2007, Jeff Roberson wrote: >>> >>>> Attilio and I have been working on addressing the increasing problem >>>> of sched_lock contention on -CURRENT. Attilio has been addressing >>>> the parts of the kernel which do not need to fall under the >>>> scheduler lock and moving them into seperate locks. For example, >>>> the ldt/gdt lock and clock lock which were committed earlier. Also, >>>> using atomics for the vmcnt structure. >>> >>> Using atomics in the vmmeter struct is mostly just a pessimization and >>> and obfuscation, since locks are still needed for accesses to more >>> than one variable at a time. For these cases, locks are needed for >> >> You are right, there are some cases which this pessimized. I wanted >> to make sure the cnt members that previously were protected by the >> sched_lock were still correct. However, I overlooked some of these >> which were accessed many at a time. What should happen is we should >> find out if any locks do protect the remaining members and if so, >> don't use VMCNT*, but mark the header describing how they are protected. > > Sorry, but I strongly disagree. Ah, and about consistency of functions your previously described, I assume nothing vital is linked to it. vmmeter is just a statistics collector and nothing else, so I don't expect nothing critical/vital happens from its fields (I'm sure a lot of variables are just bumped up and never decreased, for example). If that really happens we should fix that behaviour really, instead than making things a lot heavier. Thanks, Attilio From owner-freebsd-arch@FreeBSD.ORG Mon May 21 09:05:59 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8AFEE16A41F for ; Mon, 21 May 2007 09:05:59 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.170]) by mx1.freebsd.org (Postfix) with ESMTP id 1FFB913C44B for ; Mon, 21 May 2007 09:05:58 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so870943ugh for ; Mon, 21 May 2007 02:05:57 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=ldrC+2cMWAm3TEfPXP9N7q7EDyPZeb6PKt0+ATEilrwdxSa7Y3VxpwUXwmzMnIydGCS2VvuKgMYPNzJh+B1eV8V/QYT9r9PFCn0zNDqaNGURKcxXHz0a9zjBrD+2EVPbg2iuaLwtTzucqx9dhjKg0eCnqdRHNU4oQ8ETuUG4+0s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=dQHsPDHPtHiGeIUNMDOQR/NTgVkVa5VFGWHl+dvVYPCabc1EIMhIRVZP6phVhROjq7Tz7I73/+Jp2IkejDWBESY5T2affXyJ888HJwccJckHuxAiuy/NitipoJL+SaDNxWWK/JBst0WE45PKd6uUafFCijsiloW4LUgDFcLvxTg= Received: by 10.82.136.4 with SMTP id j4mr6994784bud.1179736432759; Mon, 21 May 2007 01:33:52 -0700 (PDT) Received: from ?172.31.5.25? ( [89.97.252.178]) by mx.google.com with ESMTP id z40sm9453491ikz.2007.05.21.01.33.51; Mon, 21 May 2007 01:33:51 -0700 (PDT) Message-ID: <4651CAB8.8070007@FreeBSD.org> Date: Mon, 21 May 2007 18:37:12 +0200 From: Attilio Rao User-Agent: Thunderbird 1.5 (X11/20060526) MIME-Version: 1.0 To: Jeff Roberson References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> In-Reply-To: <20070520213132.K632@10.0.0.1> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: Attilio Rao Cc: arch@freebsd.org, Bruce Evans Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 09:05:59 -0000 Jeff Roberson wrote: > > On Mon, 21 May 2007, Bruce Evans wrote: > >> On Sun, 20 May 2007, Jeff Roberson wrote: >> >>> Attilio and I have been working on addressing the increasing problem >>> of sched_lock contention on -CURRENT. Attilio has been addressing >>> the parts of the kernel which do not need to fall under the scheduler >>> lock and moving them into seperate locks. For example, the ldt/gdt >>> lock and clock lock which were committed earlier. Also, using >>> atomics for the vmcnt structure. >> >> Using atomics in the vmmeter struct is mostly just a pessimization and >> and obfuscation, since locks are still needed for accesses to more >> than one variable at a time. For these cases, locks are needed for > > You are right, there are some cases which this pessimized. I wanted to > make sure the cnt members that previously were protected by the > sched_lock were still correct. However, I overlooked some of these > which were accessed many at a time. What should happen is we should > find out if any locks do protect the remaining members and if so, don't > use VMCNT*, but mark the header describing how they are protected. Sorry, but I strongly disagree. You have to keep in mind some tips here: 1) reads don't pay penalties and at 99% of times multiple accesses you describe are reads or mostly 1 write + several reads 2) If you want to use a spinlock (or a sleepable lock) for doing a write or a read you have, in the better case, 2 atomics ops and 1 normal op (the read or the write in question). So, without caring about the problem you will increase latency for all reads too (that the current patch doesn't do), you have a lot more overhead even in the case of writes too. 3) If you use a lock, you need always to use it, you cannot do that only for some paths. The choosen approach is certainly the faster for mantaining a consistent approach of all vmmeter fields. Attilio From owner-freebsd-arch@FreeBSD.ORG Mon May 21 09:32:00 2007 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4176116A41F; Mon, 21 May 2007 09:32:00 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id E51FC13C489; Mon, 21 May 2007 09:31:59 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.101] (c-71-231-138-78.hsd1.or.comcast.net [71.231.138.78]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l4L9Vvl5074944 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Mon, 21 May 2007 05:31:58 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Mon, 21 May 2007 02:31:54 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: Attilio Rao In-Reply-To: <4651CE2F.8080908@FreeBSD.org> Message-ID: <20070521022847.D679@10.0.0.1> References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@FreeBSD.org, Bruce Evans Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 09:32:00 -0000 On Mon, 21 May 2007, Attilio Rao wrote: > Attilio Rao wrote: >> Jeff Roberson wrote: >>> >>> On Mon, 21 May 2007, Bruce Evans wrote: >>> >>>> On Sun, 20 May 2007, Jeff Roberson wrote: >>>> >>>>> Attilio and I have been working on addressing the increasing problem of >>>>> sched_lock contention on -CURRENT. Attilio has been addressing the >>>>> parts of the kernel which do not need to fall under the scheduler lock >>>>> and moving them into seperate locks. For example, the ldt/gdt lock and >>>>> clock lock which were committed earlier. Also, using atomics for the >>>>> vmcnt structure. >>>> >>>> Using atomics in the vmmeter struct is mostly just a pessimization and >>>> and obfuscation, since locks are still needed for accesses to more >>>> than one variable at a time. For these cases, locks are needed for >>> >>> You are right, there are some cases which this pessimized. I wanted to >>> make sure the cnt members that previously were protected by the sched_lock >>> were still correct. However, I overlooked some of these which were >>> accessed many at a time. What should happen is we should find out if any >>> locks do protect the remaining members and if so, don't use VMCNT*, but >>> mark the header describing how they are protected. >> >> Sorry, but I strongly disagree. > > Ah, and about consistency of functions your previously described, I assume > nothing vital is linked to it. > vmmeter is just a statistics collector and nothing else, so I don't expect > nothing critical/vital happens from its fields (I'm sure a lot of variables > are just bumped up and never decreased, for example). If that really happens > we should fix that behaviour really, instead than making things a lot > heavier. Well, Attilio is right that in most cases using a lock to save a few atomics is going to be more expensive. There is also the procedural cost of the lock and the cache miss etc. However, in some cases there is already a lock available that is protecting the counter. Furthermore, there are a few cases, most notably paging targets, where code depends on the value of the counters. For most fields, I believe we have a good approach, however, there are a few that could be minorly improved. The question is whether it's worth inconsistently accessing the counters to save a few atomics which likely have an immeasurable performance impact. Thanks, Jeff > > Thanks, > Attilio > From owner-freebsd-arch@FreeBSD.ORG Mon May 21 10:43:20 2007 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7F76816A400 for ; Mon, 21 May 2007 10:43:20 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from mail32.syd.optusnet.com.au (mail32.syd.optusnet.com.au [211.29.132.63]) by mx1.freebsd.org (Postfix) with ESMTP id 1BC4513C45A for ; Mon, 21 May 2007 10:43:19 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from c211-30-216-190.carlnfd3.nsw.optusnet.com.au (c211-30-216-190.carlnfd3.nsw.optusnet.com.au [211.30.216.190]) by mail32.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id l4LAh35Y031616 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 21 May 2007 20:43:07 +1000 Date: Mon, 21 May 2007 20:43:04 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Jeff Roberson In-Reply-To: <20070521022847.D679@10.0.0.1> Message-ID: <20070521195811.G56785@delplex.bde.org> References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Attilio Rao , arch@FreeBSD.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 10:43:20 -0000 On Mon, 21 May 2007, Jeff Roberson wrote: > On Mon, 21 May 2007, Attilio Rao wrote: >>> Sorry, but I strongly disagree. >> >> Ah, and about consistency of functions your previously described, I assume >> nothing vital is linked to it. >> vmmeter is just a statistics collector and nothing else, so I don't expect >> nothing critical/vital happens from its fields (I'm sure a lot of variables >> are just bumped up and never decreased, for example). If that really >> happens we should fix that behaviour really, instead than making things a >> lot heavier. The function that I described is only linked to vm paging. Statistics are vital for vm paging, though perfectly up to date statistics might not be. > Well, Attilio is right that in most cases using a lock to save a few atomics > is going to be more expensive. There is also the procedural cost of the lock > and the cache miss etc. However, in some cases there is already a lock > available that is protecting the counter. Also, atomic ops don't help much for VMCNT_GET() and VMCNT_SET(): VMCNT_GET(): atomic ops are not used. If they were used, then they would just reduce race windows, since without a lock the data may change immediately after you read it, whether or not you read it atomically. Atomic ops are still strictly needed to give atomic accesses to variables that cannot naturally be accessed atomically (perhaps because they are wider than the bus width), but everything in struct vmmeter is u_int so this is only a problem on exotic hardware. VMCNT_SET(): atomic ops are used. This doesn't help, since with multiple writers, without locks there would be races deciding what to write (starting from unlocked values) and then races writing. I think this isn't a problem because VMCNT_SET() is only used in initialization code when only one thread is running. > Furthermore, there are a few cases, most notably paging targets, where code > depends on the value of the counters. For most fields, I believe we have a Not counting sysctls, these are most cases by static count of the number of accesses. Probably by dynamic count too (since changes to the page counts are much more common than things like forks). Probably not by static count of the number of accesses (agin not counting sysctls) since there are a lot of fields and most aren't used for paging. > good approach, however, there are a few that could be minorly improved. The > question is whether it's worth inconsistently accessing the counters to save > a few atomics which likely have an immeasurable performance impact. We already have inconsistent accesses via PCPU_LAZY_INC(), and I think all cases where you don't really care about the values can be handled better by PCPU_LAZY_INC(). (Though I don't like PCPU_LAZY_INC(), since it gives namespace pollution, a larger race window while combining the values, complications, and missing support for combining the values in dead kernels.) Cases that can be handled by PCPU_LAZY_INC() are distinguished by the variables being monotonically increasing with time (until wrapping at UINT_MAX breaks them) and nothing except userland looking at them. The userland accesses are full of races, but userland doesn't really care. If accessing the pcpu is inefficient, then PCPU_LAZY_INC() can always use atomic ops if that is less inefficient. Bruce From owner-freebsd-arch@FreeBSD.ORG Mon May 21 12:07:10 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8E05E16A421 for ; Mon, 21 May 2007 12:07:10 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.171]) by mx1.freebsd.org (Postfix) with ESMTP id 1EE8013C484 for ; Mon, 21 May 2007 12:07:09 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so903896ugh for ; Mon, 21 May 2007 05:07:08 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=EMzzKCPlzpVTPb9zZWHa+w/ksIlo5oGu1SEa386G9j4jFnp72ipPsGElQwil/ZyVxCehGKkqt91W5deUsXjMXv6Y1/sLujvsM3NY0ejRMGbzHjic5nwoII25ji82LUch9czPmk7Ozj4Uh/91VoAwbHs4dT+UCZ9qb/fBWIthe8U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=Sgeuq6gwmZEGfIah6nBp0q6Rw7qop1z1e+cZBv+3Ur8HlF+X1AMSX964qx7DrRTazgvUuc7DpWmff6efOuyqUM1IpH6tR9HSYE+/pAgUP/Tj9yl6CbJZxjDvCV5cpbZNkO8+Vk7hwCRG12asoPELtKQGh6EVAxVCML/Mzco+j6U= Received: by 10.82.113.6 with SMTP id l6mr8501805buc.1179749228355; Mon, 21 May 2007 05:07:08 -0700 (PDT) Received: from ?172.31.5.25? ( [89.97.252.178]) by mx.google.com with ESMTP id c25sm9977624ika.2007.05.21.05.07.06; Mon, 21 May 2007 05:07:07 -0700 (PDT) Message-ID: <4651FCB5.7070604@FreeBSD.org> Date: Mon, 21 May 2007 22:10:29 +0200 From: Attilio Rao User-Agent: Thunderbird 1.5 (X11/20060526) MIME-Version: 1.0 To: Bruce Evans References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> In-Reply-To: <20070521195811.G56785@delplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: Attilio Rao Cc: arch@FreeBSD.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 12:07:10 -0000 Bruce Evans wrote: > On Mon, 21 May 2007, Jeff Roberson wrote: > > > The function that I described is only linked to vm paging. Statistics are > vital for vm paging, though perfectly up to date statistics might not be. Exactly, this is what I meant too. > Also, atomic ops don't help much for VMCNT_GET() and VMCNT_SET(): > VMCNT_GET(): atomic ops are not used. If they were used, then they would > just reduce race windows, since without a lock the data may change > immediately after you read it, whether or not you read it atomically. > Atomic ops are still strictly needed to give atomic accesses to > variables that cannot naturally be accessed atomically (perhaps because > they are wider than the bus width), but everything in struct vmmeter > is u_int so this is only a problem on exotic hardware. > VMCNT_SET(): atomic ops are used. This doesn't help, since with multiple > writers, without locks there would be races deciding what to write > (starting from unlocked values) and then races writing. I think this > isn't a problem because VMCNT_SET() is only used in initialization code > when only one thread is running. I think your objection about VMCNT_SET() is pretty right, it should not use atomic_* since it is used just for initializations. We can probabilly change it in short times. And as you stated, assuming reads to be atomic is not a problem here since we use volatile and all vmmeter fields are on machine natural boundaries. > We already have inconsistent accesses via PCPU_LAZY_INC(), and I think > all cases where you don't really care about the values can be handled > better by PCPU_LAZY_INC(). (Though I don't like PCPU_LAZY_INC(), since > it gives namespace pollution, a larger race window while combining the > values, complications, and missing support for combining the values > in dead kernels.) Cases that can be handled by PCPU_LAZY_INC() are > distinguished by the variables being monotonically increasing with > time (until wrapping at UINT_MAX breaks them) and nothing except > userland looking at them. The userland accesses are full of races, > but userland doesn't really care. If accessing the pcpu is inefficient, > then PCPU_LAZY_INC() can always use atomic ops if that is less inefficient. Yes, but how do you manage to mantain per-cpu datas and not using per-cpu fields? Do you want it fitting a table? In this case probabilly you will need an extra-lock for the table that will increases overhead. Thanks, Attilio From owner-freebsd-arch@FreeBSD.ORG Mon May 21 14:38:52 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5627916A400; Mon, 21 May 2007 14:38:52 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from mail16.syd.optusnet.com.au (mail16.syd.optusnet.com.au [211.29.132.197]) by mx1.freebsd.org (Postfix) with ESMTP id E838A13C483; Mon, 21 May 2007 14:38:51 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from c211-30-216-190.carlnfd3.nsw.optusnet.com.au (c211-30-216-190.carlnfd3.nsw.optusnet.com.au [211.30.216.190]) by mail16.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id l4LEcmQq013128 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 May 2007 00:38:49 +1000 Date: Tue, 22 May 2007 00:38:49 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Attilio Rao In-Reply-To: <4651FCB5.7070604@FreeBSD.org> Message-ID: <20070521225032.C57233@delplex.bde.org> References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 14:38:52 -0000 On Mon, 21 May 2007, Attilio Rao wrote: > Bruce Evans wrote: >> We already have inconsistent accesses via PCPU_LAZY_INC(), and I think >> all cases where you don't really care about the values can be handled >> better by PCPU_LAZY_INC(). (Though I don't like PCPU_LAZY_INC(), since >> it gives namespace pollution, a larger race window while combining the >> values, complications, and missing support for combining the values >> in dead kernels.) Cases that can be handled by PCPU_LAZY_INC() are >> distinguished by the variables being monotonically increasing with >> time (until wrapping at UINT_MAX breaks them) and nothing except >> userland looking at them. The userland accesses are full of races, >> but userland doesn't really care. If accessing the pcpu is inefficient, >> then PCPU_LAZY_INC() can always use atomic ops if that is less inefficient. > > Yes, but how do you manage to mantain per-cpu datas and not using per-cpu > fields? Do you want it fitting a table? > In this case probabilly you will need an extra-lock for the table that will > increases overhead. Just change the PCPU_LAZY_INC() macro to access the non-per-cpu global, using the same locking as you use for the non-pcpu access method. I do this in some (old) sys trees for the !SMP case. This gives a small unpessimization and fixes reading the values from dead kernels, but only in the !SMP case. I also remove pc_cnt from the pcpu data in the !SMP case. This is a smaller optimization and requires larger changes to remove adding up the pcpu data in vm_meter.c. The adding up can be left unchanged since it becomes a no-op if the data is left at all zeros. In the old sys trees, there is no locking for individual accesses, so PCPU_LAZY_INC(var) becomes (++(var)). In the current tree, PCPU_LAZY_INC(var) is (++()) written in asm for some arches to ensure that it is atomic with respect to interrupts on these arches. Atomicness with respect to interrupts is necessary and sufficient for the non-per-cpu global !SMP case too. I now seem to remember some more history of this. A few vmmeter variables, e.g., v_intr, have no natural locking and were originally locked by sched_lock or atomic accesses. People didn't like the slowness from this, and removed some of the sched_locking and/or switched to PCPU_LAZY_INC(). The remaining sched locking seems to be mostly accidental. Bugs found while grepping for sched_locking of VMCNT fields: - v_trap is supposed to be updated by PCPU_LAZY_INC(), but subr_trap.c updates it by VMCNT_ADD(). The direct access used to be perfectly unlocked and non-atomic, since it occurs immediately after releasing sched_lock and just used "++". Now it is atomic and a micro-pessimization. - v_trap's name is now hidden by the macro so grepping for it doesn't work. Non(?)-bugs found while grepping for VMCNT: - most updates of v_free_count are via pq->lcnt++ and pq->lcnt-- in vm_page.c and vm_pageq. pq->lcnt is VMCNT_PTR(&cnt.vm_free_count) for all pq. The locking for this seems to be correct except of course in sysctls, since it uses vm_page_queue_free_mtx and never needed VMCNT access methods. The unneeded VMCNT_GET()s for this alone comprise about 1/3 of the VMCNT_GET()s in vm. Bruce From owner-freebsd-arch@FreeBSD.ORG Mon May 21 15:13:08 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8C5AD16A468 for ; Mon, 21 May 2007 15:13:08 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.185]) by mx1.freebsd.org (Postfix) with ESMTP id 0C1F113C4B9 for ; Mon, 21 May 2007 15:13:05 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by mu-out-0910.google.com with SMTP id w8so1017189mue for ; Mon, 21 May 2007 08:13:04 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=t/Zv342hRToloAYnInszY1704Mg6PsxP96Q4lNTP4aS7Or6hhRy9Uw1yHrRr7MKpzonv7R1tJurkbGd5F9nXeiuoBp0FkA//dA/+JGpCA0X9jcXPlmb4/aFPqZJEuIX76K6RwUlbHmIzwYaPzTAEr2o7WdtKYsVYYSm6Q9zMFIQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=I7wQvD5mropa8tIZkempdbSvPA3kQKjBhq3oA2CpbfES7tTTEnpwLZAy3e8BtpcWYW1UrUoxiv5BNywwKDC9qdwGLmZXkDSaWzeuTHaxbJnw81mpMlqe4GTl/vfKvMr/mfjxrA7vpwQ16/QqreWcbBEof25/9X7kOzbOrCHa9fQ= Received: by 10.82.156.12 with SMTP id d12mr8854303bue.1179760384355; Mon, 21 May 2007 08:13:04 -0700 (PDT) Received: from ?172.31.5.25? ( [89.97.252.178]) by mx.google.com with ESMTP id y34sm273009iky.2007.05.21.08.13.00; Mon, 21 May 2007 08:13:01 -0700 (PDT) Message-ID: <46522846.5000707@FreeBSD.org> Date: Tue, 22 May 2007 01:16:22 +0200 From: Attilio Rao User-Agent: Thunderbird 1.5 (X11/20060526) MIME-Version: 1.0 To: Bruce Evans References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> In-Reply-To: <20070521225032.C57233@delplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: Attilio Rao Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 15:13:08 -0000 Bruce Evans wrote: > Just change the PCPU_LAZY_INC() macro to access the non-per-cpu global, > using the same locking as you use for the non-pcpu access method. I > do this in some (old) sys trees for the !SMP case. This gives a small > unpessimization and fixes reading the values from dead kernels, but > only in the !SMP case. I also remove pc_cnt from the pcpu data in the > !SMP case. This is a smaller optimization and requires larger changes > to remove adding up the pcpu data in vm_meter.c. The adding up can > be left unchanged since it becomes a no-op if the data is left at all > zeros. In the old sys trees, there is no locking for individual accesses, > so PCPU_LAZY_INC(var) becomes (++(var)). In the current tree, > PCPU_LAZY_INC(var) is (++()) written in asm for some arches to > ensure that it is atomic with respect to interrupts on these arches. > Atomicness with respect to interrupts is necessary and sufficient for > the non-per-cpu global !SMP case too. So you want these fields to be nomore per-cpu entirely, ok, now I got what you mean. > Bugs found while grepping for sched_locking of VMCNT fields: > - v_trap is supposed to be updated by PCPU_LAZY_INC(), but subr_trap.c > updates it by VMCNT_ADD(). The direct access used to be perfectly > unlocked > and non-atomic, since it occurs immediately after releasing sched_lock > and just used "++". Now it is atomic and a micro-pessimization. > - v_trap's name is now hidden by the macro so grepping for it doesn't work. Great, this is a bug introduced in old code that I didn't catch. I'm going to switch it to PCPU_LAZY_INC. Anyways you can grap those in this way (assuming you are in /usr/src/sys/): $ grep -r trap * | grep VMCNT_ > Non(?)-bugs found while grepping for VMCNT: > - most updates of v_free_count are via pq->lcnt++ and pq->lcnt-- in > vm_page.c and vm_pageq. pq->lcnt is VMCNT_PTR(&cnt.vm_free_count) > for all pq. The locking for this seems to be correct except of course > in sysctls, since it uses vm_page_queue_free_mtx and never needed > VMCNT access methods. The unneeded VMCNT_GET()s for this alone comprise > about 1/3 of the VMCNT_GET()s in vm. You have to consider that VMCNT_*() is used also for dealing with 'volatilization' of struct vmmeter cnt. it, so, entirely hides all implementation details for vmmeter counter. VMCNT_PTR is currently the right way to access to these datas if you want to hide the volatile to sysctls (and please note that VMCNT_GET() doesn't add any pessimization to the current code). Thanks, Attilio From owner-freebsd-arch@FreeBSD.ORG Tue May 22 04:43:00 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7F9F616A468 for ; Tue, 22 May 2007 04:43:00 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from fallbackmx03.syd.optusnet.com.au (fallbackmx03.syd.optusnet.com.au [211.29.133.136]) by mx1.freebsd.org (Postfix) with ESMTP id 9966613C447 for ; Tue, 22 May 2007 04:42:59 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from mail35.syd.optusnet.com.au (mail35.syd.optusnet.com.au [211.29.133.51]) by fallbackmx03.syd.optusnet.com.au (8.12.11.20060308/8.12.11) with ESMTP id l4L38oQ9026226 for ; Mon, 21 May 2007 13:08:50 +1000 Received: from besplex.bde.org (c211-30-216-190.carlnfd3.nsw.optusnet.com.au [211.30.216.190]) by mail35.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id l4L38aEJ030040 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 21 May 2007 13:08:39 +1000 Date: Mon, 21 May 2007 13:08:37 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Jeff Roberson In-Reply-To: <20070520155103.K632@10.0.0.1> Message-ID: <20070521113648.F86217@besplex.bde.org> References: <20070520155103.K632@10.0.0.1> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 May 2007 04:43:00 -0000 On Sun, 20 May 2007, Jeff Roberson wrote: > Attilio and I have been working on addressing the increasing problem of > sched_lock contention on -CURRENT. Attilio has been addressing the parts of > the kernel which do not need to fall under the scheduler lock and moving them > into seperate locks. For example, the ldt/gdt lock and clock lock which were > committed earlier. Also, using atomics for the vmcnt structure. Using atomics in the vmmeter struct is mostly just a pessimization and and obfuscation, since locks are still needed for accesses to more than one variable at a time. For these cases, locks are needed for correctness, and are also probably more efficient if there are > 2 accesses. Then lock poisoning requires the same locks to be held for updates of the variables (to prevent variables changing while you are using them), and atomic updates of single variables are a pessimization too. The VMCNT*() ineterface encorages this pessimization and has other problems (messes with volatile, and not actually doing atomic accesses for VMCNT_GET()). Some of the cases where locks aren't needed (mainly for userland-only statistics) were already handled better by PCPU_LAZY_INC(). Unfortunately, the method used in PCPU_LAZY_INC() doesn't work for variables that the kernel wants to access as globals. Example of code with multiple accesses: % /* % * Return the number of pages we need to free-up or cache % * A positive number indicates that we do not have enough free pages. % */ % % static __inline % int % vm_paging_target(void) % { % return ( % (VMCNT_GET(free_target) + VMCNT_GET(cache_min)) - % (VMCNT_GET(free_count) + VMCNT_GET(cache_count)) % ); % } Without holding a lock throughout this, this returns a garbage value. Without holding a lock throughout this and the actions taken depending on the return value, garbage actions may be taken. Values that are only slightly wrong might work OK, but this is not clear, and if it does work then volatile variables and [non-]atomic accesses to the variables to get it. Bruce From owner-freebsd-arch@FreeBSD.ORG Tue May 22 06:45:22 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9A7E816A421; Tue, 22 May 2007 06:45:22 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from mail08.syd.optusnet.com.au (mail08.syd.optusnet.com.au [211.29.132.189]) by mx1.freebsd.org (Postfix) with ESMTP id 3A74013C4AE; Tue, 22 May 2007 06:45:22 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from besplex.bde.org (c211-30-216-190.carlnfd3.nsw.optusnet.com.au [211.30.216.190]) by mail08.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id l4M6jINU004740 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 May 2007 16:45:20 +1000 Date: Tue, 22 May 2007 16:45:19 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Attilio Rao In-Reply-To: <20070521225032.C57233@delplex.bde.org> Message-ID: <20070522162819.N5249@besplex.bde.org> References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 May 2007 06:45:22 -0000 On Tue, 22 May 2007, Bruce Evans wrote: > Bugs found while grepping for sched_locking of VMCNT fields: > - v_trap is supposed to be updated by PCPU_LAZY_INC(), but subr_trap.c > updates it by VMCNT_ADD(). The direct access used to be perfectly unlocked > and non-atomic, since it occurs immediately after releasing sched_lock > and just used "++". Now it is atomic and a micro-pessimization. > - v_trap's name is now hidden by the macro so grepping for it doesn't work. I'm still searching for a 1% performance loss in makeworld under UP that occurred just after the VMCNT changes. I've only found this harmless (?) bug so far: % Index: vnode_pager.c % =================================================================== % RCS file: /home/ncvs/src/sys/vm/vnode_pager.c,v % retrieving revision 1.232 % retrieving revision 1.233 % diff -u -2 -r1.232 -r1.233 % --- vnode_pager.c 14 Oct 2006 23:21:48 -0000 1.232 % +++ vnode_pager.c 18 May 2007 07:10:50 -0000 1.233 % @@ -910,6 +910,6 @@ % atomic_add_int(&runningbufspace, bp->b_runningbufspace); % % - cnt.v_vnodein++; % - cnt.v_vnodepgsin += count; ^^^^^ % + VMCNT_ADD(vnodein, 1); % + VMCNT_ADD(vnodepgsin, 1); ^ % % /* do the input */ Related API problem: v_vnodein and v_vnodepgsin are not used in the kernel except to increment them and to export them using sysctl, so they should be incremented using PCPY_LAZY_INC(). However, PCPU_LAZY_INC() can only do increments of 1. The increments of +1 also complicate searching for the regression. After removing `volatile', the VMCNT change should have no effect on the binary, but it has an effect since increments of 1 are replaced by "movl $1,%reg; addl %reg,mem" in i386 code, since the i386 atomic asms don't use incl and the "ir" constraint isn't working as intended (this is with gcc-3.4.6). The "ir" constraint should result in the "i" variant being chosen and generate "addl $1,mem". Bruce From owner-freebsd-arch@FreeBSD.ORG Tue May 22 07:48:27 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D0F7616A400 for ; Tue, 22 May 2007 07:48:27 +0000 (UTC) (envelope-from dds@aueb.gr) Received: from mx-out.forthnet.gr (mx-out.forthnet.gr [193.92.150.103]) by mx1.freebsd.org (Postfix) with ESMTP id 4EB1813C4AD for ; Tue, 22 May 2007 07:48:26 +0000 (UTC) (envelope-from dds@aueb.gr) Received: from mx-av-04.forthnet.gr (mx-av.forthnet.gr [193.92.150.27]) by mx-out-02.forthnet.gr (8.14.0/8.14.0) with ESMTP id l4M7VDDn011019; Tue, 22 May 2007 10:31:13 +0300 Received: from MX-IN-04.forthnet.gr (mx-in-04.forthnet.gr [193.92.150.163]) by mx-av-04.forthnet.gr (8.14.1/8.14.1) with ESMTP id l4M7VD41021961; Tue, 22 May 2007 10:31:13 +0300 Received: from [192.168.136.22] (ppp125-70.adsl.forthnet.gr [193.92.232.70]) by MX-IN-04.forthnet.gr (8.14.1/8.14.1) with ESMTP id l4M7V5Dp032661; Tue, 22 May 2007 10:31:06 +0300 Authentication-Results: MX-IN-04.forthnet.gr from=dds@aueb.gr; sender-id=neutral; spf=neutral Message-ID: <46529C30.2090906@aueb.gr> Date: Tue, 22 May 2007 10:30:56 +0300 From: Diomidis Spinellis User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2) Gecko/20070222 SeaMonkey/1.1.1 MIME-Version: 1.0 To: freebsd-arch@freebsd.org, freebsd-hackers@freebsd.org, freebsd-current@freebsd.org References: <461958CC.4040804@aueb.gr> <20070414170218.M76326@fledge.watson.org> <4621E826.6050306@aueb.gr> <20070415105157.J84174@fledge.watson.org> <46231C64.9010707@aueb.gr> <20070419101815.Y2913@fledge.watson.org> <4627A6C3.2070409@aueb.gr> <20070419212253.L2913@fledge.watson.org> <4627E311.6080500@aueb.gr> <463B581E.6070804@aueb.gr> In-Reply-To: <463B581E.6070804@aueb.gr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Process accounting changes X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 May 2007 07:48:27 -0000 The accounting changes discussed in arch@ about a month ago are now part of the CURRENT tree. The changes increase the precision of the time values stored in acct(5) from 15ms (which for most commands was too large) to 1 microsecond. The userland programs sa(8) and lastcomm(1) provide backward compatibility with existing record and file formats (including accounting files including both old and new records); regression tests are included to verify this. The changes have been tested on i386, amd64, and (partly) sparc64; please drop me an email if you suspect a problem with your particular setup. Many thanks to Garance A Drosehn, Bruce Evans, Peter Jeremy, Poul-Henning Kamp, Sam Leffler, Rick C. Petty, Erik Trulsson, and Robert Watson for their earlier comments; to Eric Anderson, Carl Johan Gustavsson, Larry Rosenman, and Derek Tattersall for responding to my hackers@ request with test data, and especially to Larry Rosenman for arranging access to an additional test machine. Diomidis Spinellis - http://www.spinellis.gr From owner-freebsd-arch@FreeBSD.ORG Tue May 22 10:44:39 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8EF4216A400; Tue, 22 May 2007 10:44:39 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from mail23.syd.optusnet.com.au (mail23.syd.optusnet.com.au [211.29.133.164]) by mx1.freebsd.org (Postfix) with ESMTP id 2AD6813C45B; Tue, 22 May 2007 10:44:38 +0000 (UTC) (envelope-from bde@optusnet.com.au) Received: from besplex.bde.org (c211-30-216-190.carlnfd3.nsw.optusnet.com.au [211.30.216.190]) by mail23.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id l4MAiZkM012014 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 May 2007 20:44:37 +1000 Date: Tue, 22 May 2007 20:44:36 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Attilio Rao In-Reply-To: <20070522162819.N5249@besplex.bde.org> Message-ID: <20070522201336.C87981@besplex.bde.org> References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> <20070522162819.N5249@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 May 2007 10:44:39 -0000 On Tue, 22 May 2007, Bruce Evans wrote: > I'm still searching for a 1% performance loss in makeworld under UP > that occurred just after the VMCNT changes. I've only found this > harmless (?) bug so far: > > % Index: vnode_pager.c > % =================================================================== > % RCS file: /home/ncvs/src/sys/vm/vnode_pager.c,v > % retrieving revision 1.232 > % retrieving revision 1.233 > % diff -u -2 -r1.232 -r1.233 > % --- vnode_pager.c 14 Oct 2006 23:21:48 -0000 1.232 > % +++ vnode_pager.c 18 May 2007 07:10:50 -0000 1.233 > % @@ -910,6 +910,6 @@ > % atomic_add_int(&runningbufspace, bp->b_runningbufspace); > % % - cnt.v_vnodein++; > % - cnt.v_vnodepgsin += count; > ^^^^^ > % + VMCNT_ADD(vnodein, 1); > % + VMCNT_ADD(vnodepgsin, 1); > ^ > % % /* do the input */ 4 more translation errors breaking 8 counters altogether (v_vnodepgsin is broken twice): % Index: kern_fork.c % =================================================================== % RCS file: /home/ncvs/src/sys/kern/kern_fork.c,v % retrieving revision 1.270 % retrieving revision 1.271 % diff -u -2 -r1.270 -r1.271 % --- kern_fork.c 4 Apr 2007 09:11:32 -0000 1.270 % +++ kern_fork.c 18 May 2007 07:10:45 -0000 1.271 % @@ -666,18 +666,18 @@ % % if (flags == (RFFDG | RFPROC)) { % - atomic_add_int(&cnt.v_forks, 1); % - atomic_add_int(&cnt.v_forkpages, p2->p_vmspace->vm_dsize + % + VMCNT_ADD(forks, 1); % + VMCNT_ADD(forkpages, p2->p_vmspace->vm_dsize + % p2->p_vmspace->vm_ssize); % } else if (flags == (RFFDG | RFPROC | RFPPWAIT | RFMEM)) { % - atomic_add_int(&cnt.v_vforks, 1); ^^^^^^ % - atomic_add_int(&cnt.v_vforkpages, p2->p_vmspace->vm_dsize + ^^^^^^^^^^ % + VMCNT_ADD(forks, 1); ^^^^^ % + VMCNT_ADD(forkpages, p2->p_vmspace->vm_dsize + ^^^^^^^^^ % p2->p_vmspace->vm_ssize); % } else if (p1 == &proc0) { % - atomic_add_int(&cnt.v_kthreads, 1); % - atomic_add_int(&cnt.v_kthreadpages, p2->p_vmspace->vm_dsize + % + VMCNT_ADD(kthreads, 1); % + VMCNT_ADD(kthreadpages, p2->p_vmspace->vm_dsize + % p2->p_vmspace->vm_ssize); % } else { % - atomic_add_int(&cnt.v_rforks, 1); % - atomic_add_int(&cnt.v_rforkpages, p2->p_vmspace->vm_dsize + % + VMCNT_ADD(rforks, 1); % + VMCNT_ADD(rforkpages, p2->p_vmspace->vm_dsize + % p2->p_vmspace->vm_ssize); % } % Index: vnode_pager.c % =================================================================== % RCS file: /home/ncvs/src/sys/vm/vnode_pager.c,v % retrieving revision 1.232 % retrieving revision 1.233 % diff -u -2 -r1.232 -r1.233 % --- vnode_pager.c 14 Oct 2006 23:21:48 -0000 1.232 % +++ vnode_pager.c 18 May 2007 07:10:50 -0000 1.233 % @@ -1158,6 +1159,6 @@ % auio.uio_td = (struct thread *) 0; % error = VOP_WRITE(vp, &auio, ioflags, curthread->td_ucred); % - cnt.v_vnodeout++; ^^^ % - cnt.v_vnodepgsout += ncount; ^^^ % + VMCNT_ADD(vnodein, 1); ^^ % + VMCNT_ADD(vnodepgsin, ncount); ^^ % % if (error) { > Related API problem: v_vnodein and v_vnodepgsin are not used in the kernel > except to increment them and to export them using sysctl, so they should > be incremented using PCPY_LAZY_INC(). However, PCPU_LAZY_INC() can only > do increments of 1. Another problem with using PCU_LAZY_INC() is that all counters might need to be exported to userland by emulators, but only the vm_meter sysctls understand PCPU_LAZY_INC(). At least the following counters are currently exported by at least the linprocfs emulator: v_intr, v_swtch, v_vnodepgsin, v_vnodepgsout linprocfs accesses all these counters using VMCNT_GET(), but VMCNT_GET() doesn't work for counters in the pcpu. v_intr is updated by PCPU_LAZY_INC() so it is broken in linprocfs. The others aren't yet updated by PCPU_LAZY_INC() so they aren't broken in linprocfs (except the above bugs break v_vnodepgs*). Apart from these problems and bugs, PCPU_LAZY_INC() should be used for all of the counters visible in the above patches. Bruce From owner-freebsd-arch@FreeBSD.ORG Tue May 22 10:52:29 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2DAA316A421 for ; Tue, 22 May 2007 10:52:29 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.171]) by mx1.freebsd.org (Postfix) with ESMTP id B63A813C44B for ; Tue, 22 May 2007 10:52:28 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so181137ugh for ; Tue, 22 May 2007 03:52:27 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=IXXqjJIGbMO+3BXnPuvK4s9IWdkbPnJgwYelBEUBJ/RqkBy11J1h9OHyKWjZmN4pk5+jk6bHCNvniYzzZ4FEaNc4UEtwo8DvrLT9W3A4fZT8aksgiqPliDK1wQgjxi019y/QteUpTLKHEA2FAsy20yi2s/IsnecLAXqjU+D835w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=azaB60hxHTOJ57kuOxnzNU8kDggfUOpSJdY7CLNUCk/HoRkb4Sy9IK4Rl0kUOnpK4VDFxgNKovlb/0hihM9AcNT4Vn0+nNNiDPAfBacoN1ehPTjZWdigNoG3deAemkC7Qcfumss+UhF6fgiTJdH6cE2wvzJ3hzTRmidpkY9HKVY= Received: by 10.82.148.7 with SMTP id v7mr10561364bud.1179831147474; Tue, 22 May 2007 03:52:27 -0700 (PDT) Received: from ?172.31.5.25? ( [89.97.252.178]) by mx.google.com with ESMTP id c22sm156809ika.2007.05.22.03.52.21; Tue, 22 May 2007 03:52:25 -0700 (PDT) Message-ID: <46533CAD.8030104@FreeBSD.org> Date: Tue, 22 May 2007 20:55:41 +0200 From: Attilio Rao User-Agent: Thunderbird 1.5 (X11/20060526) MIME-Version: 1.0 To: Bruce Evans References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> <20070522162819.N5249@besplex.bde.org> <20070522201336.C87981@besplex.bde.org> In-Reply-To: <20070522201336.C87981@besplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: Attilio Rao Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 May 2007 10:52:29 -0000 Bruce Evans wrote: > 4 more translation errors breaking 8 counters altogether (v_vnodepgsin > is broken twice): Thanks a lot for the revision, there will be a pending patch in the next hour. Attilio From owner-freebsd-arch@FreeBSD.ORG Wed May 23 03:00:29 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 76AFE16A41F for ; Wed, 23 May 2007 03:00:29 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 158E713C465 for ; Wed, 23 May 2007 03:00:28 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.8/8.13.4) with ESMTP id l4N30Dvs065018; Tue, 22 May 2007 21:00:14 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Tue, 22 May 2007 23:00:26 -0400 (EDT) Message-Id: <20070522.230026.1611667537.imp@bsdimp.com> To: peterjeremy@optushome.com.au From: "M. Warner Losh" In-Reply-To: <20070517094445.GD1149@turion.vk2pj.dyndns.org> References: <86fy5wkim5.fsf@dwp.des.no> <20070517094445.GD1149@turion.vk2pj.dyndns.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Tue, 22 May 2007 21:00:15 -0600 (MDT) Cc: des@des.no, antinvidia@gmail.com, freebsd-arch@freebsd.org Subject: Re: A problem with the select(2) interface X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 03:00:29 -0000 In message: <20070517094445.GD1149@turion.vk2pj.dyndns.org> Peter Jeremy writes: : On 2007-May-16 15:23:14 +0200, Dag-Erling Smrgrav wrote: : >MQ writes: : >> No, that is not what I want. I think we'd better add a comments that : >> we *DO NOT WRITE TO THAT ADDRESS*, what the manual describes is that : >> there may be some platforms which write to that address. It's not the : >> same thing. : > : >Some platforms update the timeval and some don't, so portable : >applications must simply assume that its contents are undefined after : >the select(2) call. It can not be relied on to contain either the : >initial value nor the amount of time remaining. Thus your proposed : >change is pointless. : : I disagree. The FreeBSD man pages should document the behaviour of : FreeBSD. It's all very nice stating that SUSv2 allows a system to : modify the value passed as timeout but (IMNSHO), it is more important : to document what FreeBSD actually does. : : I agree that a _portable_ application must assume that timeout will : be undefined but that's no reason for refusing to document what : FreeBSD's behaviour actually is. : : There are two situations where the actual behaviour matters: : 1) Porting a random application that assumes specific behaviour for : select(). I need to know how FreeBSD behaves to know whether I : need to patch the code or not. : 2) I'm writing code that is specifically for FreeBSD. If I know : timeout will not change, I can optimise the code to avoid having to : re-initialise timeout before each select call. Index: select.2 =================================================================== RCS file: /cache/ncvs/src/lib/libc/sys/select.2,v retrieving revision 1.33 diff -u -r1.33 select.2 --- select.2 9 Jan 2007 00:28:15 -0000 1.33 +++ select.2 23 May 2007 03:00:14 -0000 @@ -222,3 +222,6 @@ by the .Fn select system call. +.Fx +does not modify the return value, which can cause problems for applications +ported from other systems. From owner-freebsd-arch@FreeBSD.ORG Wed May 23 17:39:15 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6A5FD16A400 for ; Wed, 23 May 2007 17:39:15 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd2mo3so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id 46CE313C489 for ; Wed, 23 May 2007 17:39:15 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd4mr2so.prod.shaw.ca (pd4mr2so-qfe3.prod.shaw.ca [10.0.141.213]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JII00EWT693DAJN@l-daemon> for freebsd-arch@freebsd.org; Wed, 23 May 2007 10:39:03 -0600 (MDT) Received: from pn2ml1so.prod.shaw.ca ([10.0.121.145]) by pd4mr2so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JII0020B68YHLH1@pd4mr2so.prod.shaw.ca> for freebsd-arch@freebsd.org; Wed, 23 May 2007 10:38:59 -0600 (MDT) Received: from hexahedron.daemonology.net ([24.82.18.31]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with SMTP id <0JII00F1O68XZTQ1@l-daemon> for freebsd-arch@freebsd.org; Wed, 23 May 2007 10:38:58 -0600 (MDT) Received: (qmail 1618 invoked from network); Wed, 23 May 2007 16:38:46 +0000 Received: from unknown (HELO hexahedron.daemonology.net) (127.0.0.1) by localhost with SMTP; Wed, 23 May 2007 16:38:46 +0000 Date: Wed, 23 May 2007 09:38:46 -0700 From: Colin Percival To: "freebsd-arch@freebsd.org" Message-id: <46546E16.9070707@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.95.0 User-Agent: Thunderbird 2.0.0.0 (X11/20070511) Cc: Subject: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 17:39:15 -0000 FreeBSD architects and file(1) maintainer, I'd like to remove file(1) and libmagic(3) from the FreeBSD base system for the following reasons: 1. I don't see it as being a necessary component of a UNIX-like operating system. 2. It's available in the ports tree. 3. Due to its nature as a program which parses multiple data formats, it poses an unusually high risk of having security problems in the future (cf. ethereal/wireshark). The one redeeming feature of file/libmagic as far as security is concerned is that it doesn't act as a daemon, i.e., other code or user intervention is required for an attacker to exploit security issues. This is why I'm asking here rather than wielding the "Security Officer can veto code which he doesn't like" stick. :-) Can anyone make a strong argument for keeping this code in the base system? Colin Percival FreeBSD Security Officer From owner-freebsd-arch@FreeBSD.ORG Wed May 23 18:00:48 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 16F0C16A41F for ; Wed, 23 May 2007 18:00:48 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id D017C13C45D for ; Wed, 23 May 2007 18:00:47 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.14.1/8.14.1/NETPLEX) with ESMTP id l4NHnDNV006648; Wed, 23 May 2007 13:49:13 -0400 (EDT) X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-3.0 (mail.ntplx.net [204.213.176.10]); Wed, 23 May 2007 13:49:13 -0400 (EDT) Date: Wed, 23 May 2007 13:49:13 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Colin Percival In-Reply-To: <46546E16.9070707@freebsd.org> Message-ID: References: <46546E16.9070707@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 18:00:48 -0000 On Wed, 23 May 2007, Colin Percival wrote: > FreeBSD architects and file(1) maintainer, > > I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > for the following reasons: > 1. I don't see it as being a necessary component of a UNIX-like operating > system. > 2. It's available in the ports tree. > 3. Due to its nature as a program which parses multiple data formats, it > poses an unusually high risk of having security problems in the future > (cf. ethereal/wireshark). > > The one redeeming feature of file/libmagic as far as security is concerned > is that it doesn't act as a daemon, i.e., other code or user intervention > is required for an attacker to exploit security issues. This is why I'm > asking here rather than wielding the "Security Officer can veto code which > he doesn't like" stick. :-) > > Can anyone make a strong argument for keeping this code in the base system? Yes, because other OS's have it (file) in their base, and because it is a POSIX-defined utility. Please consider this a strong no. -- DE From owner-freebsd-arch@FreeBSD.ORG Wed May 23 18:05:35 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B235816A41F for ; Wed, 23 May 2007 18:05:35 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd4mo2so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id 8E12C13C458 for ; Wed, 23 May 2007 18:05:35 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd4mr1so.prod.shaw.ca (pd4mr1so-qfe3.prod.shaw.ca [10.0.141.212]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JII00APAA8MA8C0@l-daemon> for freebsd-arch@freebsd.org; Wed, 23 May 2007 12:05:10 -0600 (MDT) Received: from pn2ml9so.prod.shaw.ca ([10.0.121.7]) by pd4mr1so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JII00ICNA8BIX00@pd4mr1so.prod.shaw.ca> for freebsd-arch@freebsd.org; Wed, 23 May 2007 12:05:11 -0600 (MDT) Received: from hexahedron.daemonology.net ([24.82.18.31]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with SMTP id <0JII00AFOA8AJC80@l-daemon> for freebsd-arch@freebsd.org; Wed, 23 May 2007 12:04:58 -0600 (MDT) Received: (qmail 2033 invoked from network); Wed, 23 May 2007 18:04:47 +0000 Received: from unknown (HELO hexahedron.daemonology.net) (127.0.0.1) by localhost with SMTP; Wed, 23 May 2007 18:04:47 +0000 Date: Wed, 23 May 2007 11:04:47 -0700 From: Colin Percival In-reply-to: To: Daniel Eischen Message-id: <4654823F.8000502@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.95.0 References: <46546E16.9070707@freebsd.org> User-Agent: Thunderbird 2.0.0.0 (X11/20070511) Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 18:05:35 -0000 Daniel Eischen wrote: > On Wed, 23 May 2007, Colin Percival wrote: >> Can anyone make a strong argument for keeping this code in the base >> system? > > Yes, because other OS's have it (file) in their base, and because > it is a POSIX-defined utility. To be fair, POSIX does define it as being optional. > Please consider this a strong no. Noted. Colin Percival From owner-freebsd-arch@FreeBSD.ORG Wed May 23 18:08:08 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6BF5216A473 for ; Wed, 23 May 2007 18:08:08 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd4mo1so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id 47A2F13C48C for ; Wed, 23 May 2007 18:08:08 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd3mr5so.prod.shaw.ca (pd3mr5so-qfe3.prod.shaw.ca [10.0.141.12]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JII00BEOABVP530@l-daemon> for freebsd-arch@freebsd.org; Wed, 23 May 2007 12:07:07 -0600 (MDT) Received: from pn2ml6so.prod.shaw.ca ([10.0.121.150]) by pd3mr5so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JII009G6ABQIYI0@pd3mr5so.prod.shaw.ca> for freebsd-arch@freebsd.org; Wed, 23 May 2007 12:07:08 -0600 (MDT) Received: from hexahedron.daemonology.net ([24.82.18.31]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with SMTP id <0JII00FXUABPRW60@l-daemon> for freebsd-arch@freebsd.org; Wed, 23 May 2007 12:07:02 -0600 (MDT) Received: (qmail 2054 invoked from network); Wed, 23 May 2007 18:06:50 +0000 Received: from unknown (HELO hexahedron.daemonology.net) (127.0.0.1) by localhost with SMTP; Wed, 23 May 2007 18:06:50 +0000 Date: Wed, 23 May 2007 11:06:50 -0700 From: Colin Percival In-reply-to: <200705231753.l4NHrTEm025055@hergotha.csail.mit.edu> To: Garrett Wollman Message-id: <465482BA.4050607@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.95.0 References: <200705231753.l4NHrTEm025055@hergotha.csail.mit.edu> User-Agent: Thunderbird 2.0.0.0 (X11/20070511) Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 18:08:08 -0000 Garrett Wollman wrote: > In article you write: >> FreeBSD architects and file(1) maintainer, >> 3. Due to its nature as a program which parses multiple data formats, it >> poses an unusually high risk of having security problems in the future >> (cf. ethereal/wireshark). > > And this doesn't apply to, say, awk(1)? Eh? Unless I'm seriously confused, awk doesn't parse any data formats... Colin Percival From owner-freebsd-arch@FreeBSD.ORG Wed May 23 18:29:29 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0E51F16A41F for ; Wed, 23 May 2007 18:29:29 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from noop.in-addr.com (noop.in-addr.com [208.58.23.51]) by mx1.freebsd.org (Postfix) with ESMTP id D905F13C4AE for ; Wed, 23 May 2007 18:29:28 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by noop.in-addr.com with local (Exim 4.54 (FreeBSD)) id 1HqvaS-000N6N-1Q for freebsd-arch@freebsd.org; Wed, 23 May 2007 14:29:28 -0400 Date: Wed, 23 May 2007 14:29:27 -0400 From: Gary Palmer To: "freebsd-arch@freebsd.org" Message-ID: <20070523182927.GF28958@in-addr.com> Mail-Followup-To: "freebsd-arch@freebsd.org" References: <46546E16.9070707@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46546E16.9070707@freebsd.org> Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 18:29:29 -0000 On Wed, May 23, 2007 at 09:38:46AM -0700, Colin Percival wrote: > FreeBSD architects and file(1) maintainer, > > I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > for the following reasons: > 1. I don't see it as being a necessary component of a UNIX-like operating > system. > 2. It's available in the ports tree. > 3. Due to its nature as a program which parses multiple data formats, it > poses an unusually high risk of having security problems in the future > (cf. ethereal/wireshark). Do either component do much parsing/reformatting of data? The major drawback to wireshark is that it has to accept different formats and display them in human readable form. To do that it doesn't use a common translation codebase with mapping files (which is what 'file' does), it has different binary parsers which each introduce their own potential problems. Unless I'm missing something, all file/libmagic have to do is look for binary signitures in the file that identify it as being of a specific type. The scope for file may have creeped over the years, but I do not see the functionality needed in file as being anywhere close to the song & dance that wireshark goes through, and as such I am not sure I agree with your comparison. The "file" program has been around as a standard part of UNIX and UNIX clones for a long time, and unless there is an endemic problem with the way the code works I personally would not be supportive of this move. > The one redeeming feature of file/libmagic as far as security is concerned > is that it doesn't act as a daemon, i.e., other code or user intervention > is required for an attacker to exploit security issues. This is why I'm > asking here rather than wielding the "Security Officer can veto code which > he doesn't like" stick. :-) > > Can anyone make a strong argument for keeping this code in the base system? > > Colin Percival > FreeBSD Security Officer > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > From owner-freebsd-arch@FreeBSD.ORG Wed May 23 18:34:21 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 721BD16A421 for ; Wed, 23 May 2007 18:34:21 +0000 (UTC) (envelope-from mureninc@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.250]) by mx1.freebsd.org (Postfix) with ESMTP id 24D8213C465 for ; Wed, 23 May 2007 18:34:20 +0000 (UTC) (envelope-from mureninc@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so95711anc for ; Wed, 23 May 2007 11:34:15 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=NdUx1OTGznnkMtcBG6bIUAR5raoaIlisQCYI2Ej+CWQzrXq0V0bJTz9JUshucTtNYwcXlhRtWHnkbnbdPCrp6LzLe6i073Ue8p36KvbYOt6SQ6ILlrJNsQMninszvsDgMgMry+WlgIq3JCkHn8EAUITstMbpKjYK2mdrDEh1UoI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=KuEbNUEMK3WXa/ydetqjnEA6P4GiXaueO8a885qHbSlV/7olxU4lHn0+UELIKgRFDUMRepMV1iMMneGfdv5YG1OTB/Z8lCBKjIwFjrlkoxXJxQ/O2dIaXCojxWDPhqWxy9Gwn2Cwv8NI4YAlI6rJG7ern5ZHTFQFugAT9o5sWT0= Received: by 10.100.46.19 with SMTP id t19mr796318ant.1179943580467; Wed, 23 May 2007 11:06:20 -0700 (PDT) Received: by 10.100.189.14 with HTTP; Wed, 23 May 2007 11:06:20 -0700 (PDT) Message-ID: Date: Wed, 23 May 2007 14:06:20 -0400 From: "Constantine A. Murenin" To: "Colin Percival" In-Reply-To: <46546E16.9070707@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <46546E16.9070707@freebsd.org> Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 18:34:21 -0000 On 23/05/07, Colin Percival wrote: > FreeBSD architects and file(1) maintainer, > > I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > for the following reasons: > 1. I don't see it as being a necessary component of a UNIX-like operating > system. > 2. It's available in the ports tree. > 3. Due to its nature as a program which parses multiple data formats, it > poses an unusually high risk of having security problems in the future > (cf. ethereal/wireshark). > > The one redeeming feature of file/libmagic as far as security is concerned > is that it doesn't act as a daemon, i.e., other code or user intervention > is required for an attacker to exploit security issues. This is why I'm > asking here rather than wielding the "Security Officer can veto code which > he doesn't like" stick. :-) > > Can anyone make a strong argument for keeping this code in the base system? What about the manual page, History section? << There has been a file command in every UNIX since at least Research Ver- sion 4 (man page dated November, 1973). The System V version introduced one significant major change: the external list of magic number types. >> Cheers, Constantine. From owner-freebsd-arch@FreeBSD.ORG Wed May 23 19:12:32 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BFA4016A400; Wed, 23 May 2007 19:12:32 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 6173A13C4BC; Wed, 23 May 2007 19:12:31 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.61.3]) by phk.freebsd.dk (Postfix) with ESMTP id 49A00173B5; Wed, 23 May 2007 19:12:30 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.1/8.14.1) with ESMTP id l4NJCqrh007159; Wed, 23 May 2007 19:12:52 GMT (envelope-from phk@critter.freebsd.dk) To: Colin Percival From: "Poul-Henning Kamp" In-Reply-To: Your message of "Wed, 23 May 2007 09:38:46 MST." <46546E16.9070707@freebsd.org> Date: Wed, 23 May 2007 19:12:52 +0000 Message-ID: <7158.1179947572@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 19:12:32 -0000 In message <46546E16.9070707@freebsd.org>, Colin Percival writes: >FreeBSD architects and file(1) maintainer, > >I'd like to remove file(1) and libmagic(3) from the FreeBSD base system >for the following reasons: >1. I don't see it as being a necessary component of a UNIX-like operating >system. On this I would tend to disagree strongly. The ability to identify random files have been a key component of UNIX for many years and I think people would be significantly surprised if we stopped providing it. One mitigating option would be to open the magic file and input and sequester the file process in a jail. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Wed May 23 19:21:04 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E3E3716A421; Wed, 23 May 2007 19:21:04 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id D11D113C448; Wed, 23 May 2007 19:21:04 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id B49771A4D80; Wed, 23 May 2007 12:22:05 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id F340D513FC; Wed, 23 May 2007 15:21:03 -0400 (EDT) Date: Wed, 23 May 2007 15:21:03 -0400 From: Kris Kennaway To: Colin Percival Message-ID: <20070523192103.GA61937@xor.obsecurity.org> References: <46546E16.9070707@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="y0ulUmNC+osPPQO6" Content-Disposition: inline In-Reply-To: <46546E16.9070707@freebsd.org> User-Agent: Mutt/1.4.2.2i Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 19:21:05 -0000 --y0ulUmNC+osPPQO6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, May 23, 2007 at 09:38:46AM -0700, Colin Percival wrote: > FreeBSD architects and file(1) maintainer, >=20 > I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > for the following reasons: > 1. I don't see it as being a necessary component of a UNIX-like operating > system. > 2. It's available in the ports tree. > 3. Due to its nature as a program which parses multiple data formats, it > poses an unusually high risk of having security problems in the future > (cf. ethereal/wireshark). >=20 > The one redeeming feature of file/libmagic as far as security is concerned > is that it doesn't act as a daemon, i.e., other code or user intervention > is required for an attacker to exploit security issues. This is why I'm > asking here rather than wielding the "Security Officer can veto code which > he doesn't like" stick. :-) >=20 > Can anyone make a strong argument for keeping this code in the base syste= m? What is the threat you are defending against here: "Admin runs file(1) on untrusted binary"? If so, how does it differ from e.g. running cat(1) on an untrusted binary, which can reprogram your terminal emulation and in some cases take over your terminal; or from various other unprivileged user binaries that also crash when operating on corrupted data, possibly in an exploitable way? Last time I checked lots of our /usr/bin tools coredumped when you passed them unexpected input. Also, did coverity find the buffer overflow, and if so, what other bugs does it see in this tool, and have you fixed them? :) Kris --y0ulUmNC+osPPQO6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGVJQfWry0BWjoQKURAiLbAKCbrwOYZnLrG5P32sXJRpbZ5MrqcQCffbUq Z02hlXrmJMM1CC9ecw29igA= =3vkD -----END PGP SIGNATURE----- --y0ulUmNC+osPPQO6-- From owner-freebsd-arch@FreeBSD.ORG Wed May 23 19:50:47 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BA36B16A421 for ; Wed, 23 May 2007 19:50:47 +0000 (UTC) (envelope-from maxim@macomnet.ru) Received: from mp2.macomnet.net (mp2.macomnet.net [195.128.64.6]) by mx1.freebsd.org (Postfix) with ESMTP id 2EBBF13C468 for ; Wed, 23 May 2007 19:50:46 +0000 (UTC) (envelope-from maxim@macomnet.ru) Received: from localhost (localhost.int.ru [127.0.0.1] (may be forged)) by mp2.macomnet.net (8.13.7/8.13.8) with ESMTP id l4NJH9rG005880; Wed, 23 May 2007 23:17:09 +0400 (MSD) (envelope-from maxim@macomnet.ru) Date: Wed, 23 May 2007 23:17:09 +0400 (MSD) From: Maxim Konovalov To: Colin Percival In-Reply-To: <46546E16.9070707@freebsd.org> Message-ID: <20070523231602.G47130@mp2.macomnet.net> References: <46546E16.9070707@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 19:50:47 -0000 On Wed, 23 May 2007, 09:38-0700, Colin Percival wrote: > FreeBSD architects and file(1) maintainer, > > I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > for the following reasons: [...] I'm not a FreeBSD architect or file(1) maintainer but please don't. It is very useful to have file(1) in the base system. -- Maxim Konovalov From owner-freebsd-arch@FreeBSD.ORG Wed May 23 19:58:32 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 32E6B16A468; Wed, 23 May 2007 19:58:32 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 1767513C465; Wed, 23 May 2007 19:58:32 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 1B1191A4D80; Wed, 23 May 2007 12:59:33 -0700 (PDT) Date: Wed, 23 May 2007 12:59:33 -0700 From: Alfred Perlstein To: Daniel Eischen Message-ID: <20070523195933.GM21795@elvis.mu.org> References: <46546E16.9070707@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i Cc: Colin Percival , "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 19:58:32 -0000 * Daniel Eischen [070523 11:05] wrote: > On Wed, 23 May 2007, Colin Percival wrote: > > >FreeBSD architects and file(1) maintainer, > > > >I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > >for the following reasons: > >1. I don't see it as being a necessary component of a UNIX-like operating > >system. > >2. It's available in the ports tree. > >3. Due to its nature as a program which parses multiple data formats, it > >poses an unusually high risk of having security problems in the future > >(cf. ethereal/wireshark). > > > >The one redeeming feature of file/libmagic as far as security is concerned > >is that it doesn't act as a daemon, i.e., other code or user intervention > >is required for an attacker to exploit security issues. This is why I'm > >asking here rather than wielding the "Security Officer can veto code which > >he doesn't like" stick. :-) > > > >Can anyone make a strong argument for keeping this code in the base system? > > Yes, because other OS's have it (file) in their base, and because > it is a POSIX-defined utility. Please consider this a strong no. I agree with Daniel. -- - Alfred Perlstein From owner-freebsd-arch@FreeBSD.ORG Wed May 23 21:30:06 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A498D16A4AB for ; Wed, 23 May 2007 21:30:06 +0000 (UTC) (envelope-from lyndon@orthanc.ca) Received: from orthanc.ca (204-174-83-38.static569.dsl.ucc-net.ca [204.174.83.38]) by mx1.freebsd.org (Postfix) with ESMTP id 68A2013C487 for ; Wed, 23 May 2007 21:30:06 +0000 (UTC) (envelope-from lyndon@orthanc.ca) Received: from localhost (localhost [127.0.0.1]) by orthanc.ca (8.14.1/8.14.1) with ESMTP id l4NL0RC8043378 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 23 May 2007 14:00:27 -0700 (PDT) (envelope-from lyndon@orthanc.ca) Date: Wed, 23 May 2007 14:00:27 -0700 (PDT) From: Lyndon Nerenberg To: Colin Percival In-Reply-To: <46546E16.9070707@freebsd.org> Message-ID: <20070523135730.O43231@orthanc.ca> References: <46546E16.9070707@freebsd.org> Organization: The Frobozz Magic Homing Pigeon Company MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 21:30:06 -0000 > I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > for the following reasons: > 1. I don't see it as being a necessary component of a UNIX-like operating > system. I do, and so does POSIX. I use file(1) on a daily basis. So does every sysadmin I know. > 2. It's available in the ports tree. Everything is available in the ports tree. Please leave file alone. It's happy where it is. --lyndon The two most common elements in the universe are Hydrogen and stupidity. -- Harlan Ellison From owner-freebsd-arch@FreeBSD.ORG Wed May 23 21:47:59 2007 Return-Path: X-Original-To: freebsd-arch@FreeBSD.ORG Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ACC7916A469 for ; Wed, 23 May 2007 21:47:59 +0000 (UTC) (envelope-from das@FreeBSD.ORG) Received: from VARK.MIT.EDU (VARK.MIT.EDU [18.95.3.179]) by mx1.freebsd.org (Postfix) with ESMTP id 786F613C48A for ; Wed, 23 May 2007 21:47:59 +0000 (UTC) (envelope-from das@FreeBSD.ORG) Received: from VARK.MIT.EDU (localhost [127.0.0.1]) by VARK.MIT.EDU (8.14.1/8.13.1) with ESMTP id l4NLNP1n003084; Wed, 23 May 2007 17:23:25 -0400 (EDT) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by VARK.MIT.EDU (8.14.1/8.13.1/Submit) id l4NLNP6x003083; Wed, 23 May 2007 17:23:25 -0400 (EDT) (envelope-from das@FreeBSD.ORG) Date: Wed, 23 May 2007 17:23:25 -0400 From: David Schultz To: Colin Percival Message-ID: <20070523212325.GA3022@VARK.MIT.EDU> Mail-Followup-To: Colin Percival , "freebsd-arch@freebsd.org" References: <46546E16.9070707@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46546E16.9070707@freebsd.org> Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 21:47:59 -0000 On Wed, May 23, 2007, Colin Percival wrote: > Can anyone make a strong argument for keeping this code in the base system? Removing it from the base system would merely amount to a marketing ploy, wherein we get to say that FreeBSD has fewer security holes because file(1) is a "third-party package". Doing so wouldn't make FreeBSD installations any more secure in practice. Virtually everyone would have to install file(1) anyway, and those who didn't wouldn't care about security holes in it anyway. In fact, removing it from the base system could make FreeBSD's file(1) less secure because developing and disseminating patches for holes in ports is a lower priority than patching holes in the base system. From owner-freebsd-arch@FreeBSD.ORG Wed May 23 21:49:02 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 64C5B16A46E for ; Wed, 23 May 2007 21:49:02 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from keltia.freenix.fr (keltia.freenix.org [82.230.37.243]) by mx1.freebsd.org (Postfix) with ESMTP id 1D85C13C4D0 for ; Wed, 23 May 2007 21:49:01 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from localhost (localhost [127.0.0.1]) by keltia.freenix.fr (Postfix/TLS) with ESMTP id 43B0139550 for ; Wed, 23 May 2007 23:32:52 +0200 (CEST) Received: from keltia.freenix.fr ([127.0.0.1]) by localhost (keltia.freenix.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 14545-09 for ; Wed, 23 May 2007 23:32:51 +0200 (CEST) Received: by keltia.freenix.fr (Postfix/TLS, from userid 101) id 8D9CE394E6; Wed, 23 May 2007 23:32:51 +0200 (CEST) Date: Wed, 23 May 2007 23:32:51 +0200 From: Ollivier Robert To: freebsd-arch@freebsd.org Message-ID: <20070523213251.GA14733@keltia.freenix.fr> References: <46546E16.9070707@freebsd.org> <7158.1179947572@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7158.1179947572@critter.freebsd.dk> X-Operating-System: MacOS X / Macbook Pro - FreeBSD 6.2 / Dell D820 SMP User-Agent: Mutt/1.5.15 (2007-04-06) X-Virus-Scanned: amavisd-new at keltia.freenix.fr Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 21:49:02 -0000 According to Poul-Henning Kamp: > On this I would tend to disagree strongly. The ability to identify > random files have been a key component of UNIX for many years and > I think people would be significantly surprised if we stopped > providing it. Agreed, take this message as a strong no from myself as well. -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- roberto@keltia.freenix.fr Darwin sidhe.keltia.net Kernel Version 8.9.1: Thu Feb 22 20:55:00 PST 2007 i386 From owner-freebsd-arch@FreeBSD.ORG Wed May 23 22:03:28 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9FDE116A41F; Wed, 23 May 2007 22:03:28 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 4325113C46C; Wed, 23 May 2007 22:03:28 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.8/8.13.4) with ESMTP id l4NM3PZb078740; Wed, 23 May 2007 16:03:25 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Wed, 23 May 2007 16:03:39 -0600 (MDT) Message-Id: <20070523.160339.-1264103850.imp@bsdimp.com> To: cperciva@freebsd.org From: "M. Warner Losh" In-Reply-To: <46546E16.9070707@freebsd.org> References: <46546E16.9070707@freebsd.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Wed, 23 May 2007 16:03:26 -0600 (MDT) Cc: freebsd-arch@freebsd.org Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 22:03:28 -0000 In message: <46546E16.9070707@freebsd.org> Colin Percival writes: : FreeBSD architects and file(1) maintainer, : : I'd like to remove file(1) and libmagic(3) from the FreeBSD base system : for the following reasons: : 1. I don't see it as being a necessary component of a UNIX-like operating : system. : 2. It's available in the ports tree. : 3. Due to its nature as a program which parses multiple data formats, it : poses an unusually high risk of having security problems in the future : (cf. ethereal/wireshark). : : The one redeeming feature of file/libmagic as far as security is concerned : is that it doesn't act as a daemon, i.e., other code or user intervention : is required for an attacker to exploit security issues. This is why I'm : asking here rather than wielding the "Security Officer can veto code which : he doesn't like" stick. :-) : : Can anyone make a strong argument for keeping this code in the base system? Because it is so darn useful, people use it in scripting all the time and it has been there for a long time. Warner From owner-freebsd-arch@FreeBSD.ORG Wed May 23 22:06:36 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2EF2516A41F; Wed, 23 May 2007 22:06:36 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id E32B113C45B; Wed, 23 May 2007 22:06:35 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.8/8.13.4) with ESMTP id l4NM47tN078749; Wed, 23 May 2007 16:04:07 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Wed, 23 May 2007 16:04:20 -0600 (MDT) Message-Id: <20070523.160420.163264050.imp@bsdimp.com> To: cperciva@freebsd.org From: "M. Warner Losh" In-Reply-To: <465482BA.4050607@freebsd.org> References: <200705231753.l4NHrTEm025055@hergotha.csail.mit.edu> <465482BA.4050607@freebsd.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Wed, 23 May 2007 16:04:12 -0600 (MDT) Cc: wollman@hergotha.csail.mit.edu, freebsd-arch@freebsd.org Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 22:06:36 -0000 In message: <465482BA.4050607@freebsd.org> Colin Percival writes: : Garrett Wollman wrote: : > In article you write: : >> FreeBSD architects and file(1) maintainer, : >> 3. Due to its nature as a program which parses multiple data formats, it : >> poses an unusually high risk of having security problems in the future : >> (cf. ethereal/wireshark). : > : > And this doesn't apply to, say, awk(1)? : : Eh? Unless I'm seriously confused, awk doesn't parse any data formats... It handles arbitrary data from potentially hostile sources as well. But only when the users asks it to do so... Warner From owner-freebsd-arch@FreeBSD.ORG Wed May 23 22:12:28 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B430C16A421 for ; Wed, 23 May 2007 22:12:28 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 71CA213C46A for ; Wed, 23 May 2007 22:12:28 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.8/8.13.4) with ESMTP id l4NMAPhh078786; Wed, 23 May 2007 16:10:25 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Wed, 23 May 2007 16:10:38 -0600 (MDT) Message-Id: <20070523.161038.-1989860747.imp@bsdimp.com> To: roberto@keltia.freenix.fr From: "M. Warner Losh" In-Reply-To: <20070523213251.GA14733@keltia.freenix.fr> References: <46546E16.9070707@freebsd.org> <7158.1179947572@critter.freebsd.dk> <20070523213251.GA14733@keltia.freenix.fr> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Wed, 23 May 2007 16:10:25 -0600 (MDT) Cc: freebsd-arch@freebsd.org Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 22:12:28 -0000 In message: <20070523213251.GA14733@keltia.freenix.fr> Ollivier Robert writes: : According to Poul-Henning Kamp: : > On this I would tend to disagree strongly. The ability to identify : > random files have been a key component of UNIX for many years and : > I think people would be significantly surprised if we stopped : > providing it. : : Agreed, take this message as a strong no from myself as well. I would argue that it would make the system LESS secure, because one loses the ability to identify files on the system. People are going to install it anyway, and it is a jump ball as to whether having it in the base system would cause vulnerabilities to be updated faster than having it in ports (both the actual update in the system, as well as the user causing the update to happen: ports are a touch easier to update, but lag a bit both in terms of people updating their ports tree and ports committers updating the port). And for there to be any exploitable vulnerability, the attacker would need to feed the victum a bogusly formatted file, and cause the victum to run file on that file. I doubt that the latest security hole will ever result in a system compromise... I guess I fail to see how this is any different than the .gz bugs that were found a while ago. Nobody suggested removing .gz from the tree because a few bugs were found. Everybody suggested updating right away to fix those bugs. File is no different, and really should remain in the tree. In short: this is a silly idea. Don't do it. Warner From owner-freebsd-arch@FreeBSD.ORG Wed May 23 22:56:45 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 267F516A421 for ; Wed, 23 May 2007 22:56:45 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id F3F8813C489 for ; Wed, 23 May 2007 22:56:44 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.101] (c-71-231-138-78.hsd1.or.comcast.net [71.231.138.78]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l4NMugG4020543 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO) for ; Wed, 23 May 2007 18:56:43 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Wed, 23 May 2007 15:56:35 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: arch@freebsd.org In-Reply-To: <20070520155103.K632@10.0.0.1> Message-ID: <20070523155236.U9443@10.0.0.1> References: <20070520155103.K632@10.0.0.1> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 22:56:45 -0000 Resuming the original intent of this thread; http://www.chesapeake.net/~jroberson/threadlock.diff I have updated this patch to the most recent current. I have included a scheduler called sched_smp.c that is a copy of ULE using per-cpu scheduler spinlocks. There are also changes to be slightly more agressive with updating the td_lock pointer when it has been blocked. This continues to be stable in testing by myself and Kris Kennaway on 1 to 8 cpu machines. Attilio is working on addressing concerns with the vmmeter diff. It's my fault for not sending this around to arch@ before committing. I apologize. We will have one diff before threadlock goes in to fix rusage such that it doesn't depend on a gobal scheduler lock. I will mail that here for review. After that I intend to commit threadlock. Please complain sooner rather than later! Thanks, Jeff On Sun, 20 May 2007, Jeff Roberson wrote: > Attilio and I have been working on addressing the increasing problem of > sched_lock contention on -CURRENT. Attilio has been addressing the parts of > the kernel which do not need to fall under the scheduler lock and moving them > into seperate locks. For example, the ldt/gdt lock and clock lock which were > committed earlier. Also, using atomics for the vmcnt structure. > > I have been working on an approach to using thread locks rather than a global > scheduler lock. The design is similar to Solaris's container locks, but the > details are different. The basic idea is to have a pointer in the thread > structure that points at a spinlock that protects the thread. This spinlock > may be one of the scheduler lock, a turnstile lock, or a sleep queue lock. > As the thread changes state from running to blocked on a lock or sleeping the > lock changes with it. > > This has several advantages. The majority of the kernel simply calls > thread_lock() which figures out the details. The kernel then knows nothing > of the particulars of the scheduler locks, and the schedulers are free to > implement them in any way that they like. Furthermore, in some cases the > locking is reduced, because locking the thread has the side effect of locking > the container. > > This patch does not implement per-cpu scheduler locks. It just changes the > kernel to support this model. I have a fork of ULE in development that runs > with per-cpu locks, but it is not ready yet. This means that there should be > very little change in system performance until the scheduler catches up. In > fact, on a 2cpu system the difference is immeasurable or almost so on every > workload I have tested. On an 8way opteron system the results vary between > +10% on some reasonable workloads and -15% on super-smack, which has some > inherent problems that I believe are not exposing real performance problems > with this patch. > > This has also been tested extensively by Kris and myself on a variety of > machines and I believe it to be fairly solid. The only thing remaining to do > is fix rusage so that it does not rely on a global scheduler lock. > > I am posting the patch here in case anyone with specific knowledge of > turnstiles, sleepqueues, or signals would like to review it, and as a general > heads up to people interested in where the kernel is headed. > > This will apply to current just prior to my kern_clock.c commits. I will > re-merge and update again in the next few days, probably after we sort out > rusage. > > http://people.freebsd.org/~jeff/threadlock.diff > > Thanks, > Jeff > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Wed May 23 23:27:05 2007 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9D4D216A41F for ; Wed, 23 May 2007 23:27:05 +0000 (UTC) (envelope-from gad@FreeBSD.org) Received: from smtp7.server.rpi.edu (smtp7.server.rpi.edu [128.113.2.227]) by mx1.freebsd.org (Postfix) with ESMTP id 680E413C4C1 for ; Wed, 23 May 2007 23:27:05 +0000 (UTC) (envelope-from gad@FreeBSD.org) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by smtp7.server.rpi.edu (8.13.1/8.13.1) with ESMTP id l4NMQvhM004666; Wed, 23 May 2007 18:26:58 -0400 Mime-Version: 1.0 Message-Id: In-Reply-To: <7158.1179947572@critter.freebsd.dk> References: <7158.1179947572@critter.freebsd.dk> Date: Wed, 23 May 2007 18:26:56 -0400 To: Colin Percival From: Garance A Drosehn Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-RPI-SA-Score: undef - spam scanning disabled X-CanItPRO-Stream: default X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 128.113.2.227 Cc: freebsd-arch@FreeBSD.org Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 23:27:05 -0000 At 7:12 PM +0000 5/23/07, Poul-Henning Kamp wrote: >In message <46546E16.9070707@freebsd.org>, Colin Percival writes: >> FreeBSD architects and file(1) maintainer, >> > > I'd like to remove file(1) and libmagic(3) from the FreeBSD base > > system for the following reasons: > > > > 1. I don't see it as being a necessary component of a UNIX-like > > operating system. > >On this I would tend to disagree strongly. The ability to identify >random files have been a key component of UNIX for many years and >I think people would be significantly surprised if we stopped >providing it. I concur with PHK. There has been a 'file' command on every unix system I have used in the past 15 (or more) years. If FreeBSD removes the file(1) command, almost every sysadmin will simply install it from ports. The file(1) command does not run as a daemon, it is not setuid or setgid, and has no special access to any information which must be kept secure (such as /etc/passwd). I don't see why we would single out that command based on one buffer overflow. I realize that every security advisory involves a lot of rush work on the part of the security team, but I don't think that file(1) has been guilty often enough for us to consider removing it. And I think removing it for *security* reasons is particularly pointless when we know that every unix sysadmin is just going to install it from ports if it was not in the base system. Mark me as a strong vote against removing it from the base system. If we really think that file(1) command is a serious security problem, then we should do things to limit the damage it can do. Moving it into an always-installed port will not improve security (IMO). -- Garance Alistair Drosehn = drosehn@rpi.edu Senior Systems Programmer or gad@FreeBSD.org Rensselaer Polytechnic Institute; Troy, NY; USA From owner-freebsd-arch@FreeBSD.ORG Wed May 23 23:28:20 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 03B9316A41F for ; Wed, 23 May 2007 23:28:20 +0000 (UTC) (envelope-from xcllnt@mac.com) Received: from smtpout.mac.com (smtpout.mac.com [17.250.248.174]) by mx1.freebsd.org (Postfix) with ESMTP id CE23313C455 for ; Wed, 23 May 2007 23:28:19 +0000 (UTC) (envelope-from xcllnt@mac.com) Received: from mac.com (smtpin08-en2 [10.13.10.153]) by smtpout.mac.com (Xserve/smtpout04/MantshX 4.0) with ESMTP id l4NNH1pd026237; Wed, 23 May 2007 16:17:01 -0700 (PDT) Received: from [172.24.104.227] (natint3.juniper.net [66.129.224.36]) (authenticated bits=0) by mac.com (Xserve/smtpin08/MantshX 4.0) with ESMTP id l4NNGunP004187 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 23 May 2007 16:16:57 -0700 (PDT) In-Reply-To: <20070523155236.U9443@10.0.0.1> References: <20070520155103.K632@10.0.0.1> <20070523155236.U9443@10.0.0.1> Mime-Version: 1.0 (Apple Message framework v752.3) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <6A9BD12D-D93C-4AE8-B4F4-D59A0327032D@mac.com> Content-Transfer-Encoding: 7bit From: Marcel Moolenaar Date: Wed, 23 May 2007 16:16:53 -0700 To: Jeff Roberson X-Mailer: Apple Mail (2.752.3) X-Brightmail-Tracker: AAAAAA== X-Brightmail-scanned: yes Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 23:28:20 -0000 On May 23, 2007, at 3:56 PM, Jeff Roberson wrote: > Resuming the original intent of this thread; > > http://www.chesapeake.net/~jroberson/threadlock.diff 404 Not Found. The old patch was missing PowerPC & ia64. Will the final version include those as well? -- Marcel Moolenaar xcllnt@mac.com From owner-freebsd-arch@FreeBSD.ORG Wed May 23 23:34:00 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B539616A46C for ; Wed, 23 May 2007 23:34:00 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 809E413C465 for ; Wed, 23 May 2007 23:34:00 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.101] (c-71-231-138-78.hsd1.or.comcast.net [71.231.138.78]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l4NNXvYB028738 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Wed, 23 May 2007 19:33:58 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Wed, 23 May 2007 16:33:50 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: Marcel Moolenaar In-Reply-To: <6A9BD12D-D93C-4AE8-B4F4-D59A0327032D@mac.com> Message-ID: <20070523163109.X9443@10.0.0.1> References: <20070520155103.K632@10.0.0.1> <20070523155236.U9443@10.0.0.1> <6A9BD12D-D93C-4AE8-B4F4-D59A0327032D@mac.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 23:34:00 -0000 On Wed, 23 May 2007, Marcel Moolenaar wrote: > > On May 23, 2007, at 3:56 PM, Jeff Roberson wrote: > >> Resuming the original intent of this thread; >> >> http://www.chesapeake.net/~jroberson/threadlock.diff Woops! people.freebsd.org/~jeff/threadlock.diff > > 404 Not Found. > > The old patch was missing PowerPC & ia64. Will the final version > include those as well? There are a couple of uses of the global scheduler lock in some architecture specific locations. They will continue to be safe with the 4BSD scheduler. I intended to work on these issues with the architecture maintainers after the threadlock patch goes in. Can you suggest some alternative to sched_lock for pmap_switch in ia64? There are a couple of these small issues that should be perfectly safe that I was hoping to address outside of this patch so that it didn't get too big. Jeff > > -- > Marcel Moolenaar > xcllnt@mac.com > > From owner-freebsd-arch@FreeBSD.ORG Wed May 23 23:48:42 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 013FC16A46B for ; Wed, 23 May 2007 23:48:42 +0000 (UTC) (envelope-from xcllnt@mac.com) Received: from smtpout.mac.com (smtpout.mac.com [17.250.248.186]) by mx1.freebsd.org (Postfix) with ESMTP id E363213C46A for ; Wed, 23 May 2007 23:48:41 +0000 (UTC) (envelope-from xcllnt@mac.com) Received: from mac.com (smtpin08-en2 [10.13.10.153]) by smtpout.mac.com (Xserve/smtpout16/MantshX 4.0) with ESMTP id l4NNmZPl023159; Wed, 23 May 2007 16:48:35 -0700 (PDT) Received: from [172.24.104.227] (natint3.juniper.net [66.129.224.36]) (authenticated bits=0) by mac.com (Xserve/smtpin08/MantshX 4.0) with ESMTP id l4NNmSMY015192 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 23 May 2007 16:48:30 -0700 (PDT) In-Reply-To: <20070523163109.X9443@10.0.0.1> References: <20070520155103.K632@10.0.0.1> <20070523155236.U9443@10.0.0.1> <6A9BD12D-D93C-4AE8-B4F4-D59A0327032D@mac.com> <20070523163109.X9443@10.0.0.1> Mime-Version: 1.0 (Apple Message framework v752.3) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <38601004-BB95-4B8B-87A6-26E2D52B89BA@mac.com> Content-Transfer-Encoding: 7bit From: Marcel Moolenaar Date: Wed, 23 May 2007 16:48:25 -0700 To: Jeff Roberson X-Mailer: Apple Mail (2.752.3) X-Brightmail-Tracker: AAAAAA== X-Brightmail-scanned: yes Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 23:48:42 -0000 >> The old patch was missing PowerPC & ia64. Will the final version >> include those as well? > > There are a couple of uses of the global scheduler lock in some > architecture specific locations. They will continue to be safe > with the 4BSD scheduler. I intended to work on these issues with > the architecture maintainers after the threadlock patch goes in. > Can you suggest some alternative to sched_lock for pmap_switch in > ia64? pmap_switch() is called from cpu_switch() and from pmap_install(). So, currently, pmap_install() grabs sched_lock to mimic the cpu_switch() path and we assert having sched_lock in pmap_switch(). Basically, any lock that serializes cpu_switch() would work, because we don't want to switch the thread while in the middle of setting up the region registers. > There are a couple of these small issues that should be perfectly > safe that I was hoping to address outside of this patch so that it > didn't get too big. I noticed you introduced sched_throw(). Would it harm if ia64 doesn't yet use sched_throw() and instead has the sequence it replaces? In other words: is the initial implementation of sched_throw() the same as the current code? -- Marcel Moolenaar xcllnt@mac.com From owner-freebsd-arch@FreeBSD.ORG Wed May 23 23:53:51 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2CB5416A421 for ; Wed, 23 May 2007 23:53:51 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 1CA6E13C44C for ; Wed, 23 May 2007 23:53:51 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 328E11A4D82; Wed, 23 May 2007 16:54:52 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 2A37F513B7; Wed, 23 May 2007 19:53:50 -0400 (EDT) Date: Wed, 23 May 2007 19:53:50 -0400 From: Kris Kennaway To: Jeff Roberson Message-ID: <20070523235349.GA66762@xor.obsecurity.org> References: <20070520155103.K632@10.0.0.1> <20070523155236.U9443@10.0.0.1> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="vtzGhvizbBRQ85DL" Content-Disposition: inline In-Reply-To: <20070523155236.U9443@10.0.0.1> User-Agent: Mutt/1.4.2.2i Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 May 2007 23:53:51 -0000 --vtzGhvizbBRQ85DL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, May 23, 2007 at 03:56:35PM -0700, Jeff Roberson wrote: > Resuming the original intent of this thread; >=20 > http://www.chesapeake.net/~jroberson/threadlock.diff >=20 > I have updated this patch to the most recent current. I have included a= =20 > scheduler called sched_smp.c that is a copy of ULE using per-cpu schedule= r=20 > spinlocks. There are also changes to be slightly more agressive with=20 > updating the td_lock pointer when it has been blocked. I have not yet found an application benchmark that really demonstrates this (e.g. the SQL benchmark is now entirely bottlenecked by the global select lock), but on a microbenchmark designed to specifically test scheduler performance (sysbench --test=3Dthreads) this gives dramatic results on an 8-core opteron. -- This test mode was written to benchmark scheduler performance, more specifically the cases when a scheduler has a large number of threads competing for some set of mutexes. SysBench creates a specified number of threads and a specified number of mutexes. Then each thread starts running the requests consisting of locking the mutex, yielding the CPU, so the thread is placed in the run queue by the scheduler, then unlocking the mutex when the thread is rescheduled back to execution. For each request, the above actions are run several times in a loop, so the more iterations is performed, the more concurrency is placed on each mutex. -- With the threadlock.diff changes and sched_smp there is a factor of 3.8 performance improvement compared to sched_ule. 4BSD actually performs 30% better than ULE on this microbenchmark (it has been much slower on all the application benchmarks I've done on this system), but is still a factor of 2.8 slower than sched_smp. Indeed, profiling confirms that with ULE and 4BSD the global sched_lock is the only relevant lock, and is heavily contended. This contention largely goes away with the per-cpu scheduler locks in sched_smp (but there is still some contention). Profiling indicates there might be further scope to as much as double performance of this benchmark by improving the load balancing and other architectural changes (system is about 50% idle still). I am hoping to see some real application benchmark improvements on sun4v when Kip gets it up and running again (should be soon), since last time we looked the global sched_lock was a dominant effect there. Kris --vtzGhvizbBRQ85DL Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGVNQNWry0BWjoQKURAlVVAKDzB/DTbAQh5X6fbaN7JwdRjtH+lwCgj6uq PM/rKM5sVT3cEZwABTUgF8w= =whQ2 -----END PGP SIGNATURE----- --vtzGhvizbBRQ85DL-- From owner-freebsd-arch@FreeBSD.ORG Thu May 24 00:11:24 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5EFD416A421 for ; Thu, 24 May 2007 00:11:24 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 29EFC13C48A for ; Thu, 24 May 2007 00:11:24 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.101] (c-71-231-138-78.hsd1.or.comcast.net [71.231.138.78]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l4O0BLF1035696 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Wed, 23 May 2007 20:11:22 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Wed, 23 May 2007 17:11:15 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: Marcel Moolenaar In-Reply-To: <38601004-BB95-4B8B-87A6-26E2D52B89BA@mac.com> Message-ID: <20070523170449.L9443@10.0.0.1> References: <20070520155103.K632@10.0.0.1> <20070523155236.U9443@10.0.0.1> <6A9BD12D-D93C-4AE8-B4F4-D59A0327032D@mac.com> <20070523163109.X9443@10.0.0.1> <38601004-BB95-4B8B-87A6-26E2D52B89BA@mac.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 00:11:24 -0000 On Wed, 23 May 2007, Marcel Moolenaar wrote: > >>> The old patch was missing PowerPC & ia64. Will the final version >>> include those as well? >> >> There are a couple of uses of the global scheduler lock in some >> architecture specific locations. They will continue to be safe with the >> 4BSD scheduler. I intended to work on these issues with the architecture >> maintainers after the threadlock patch goes in. Can you suggest some >> alternative to sched_lock for pmap_switch in ia64? > > pmap_switch() is called from cpu_switch() and from pmap_install(). > So, currently, pmap_install() grabs sched_lock to mimic the > cpu_switch() path and we assert having sched_lock in pmap_switch(). > Basically, any lock that serializes cpu_switch() would work, because > we don't want to switch the thread while in the middle of setting up > the region registers. We could simply use thread_lock() now if this serialization only applies to preventing multiple access to the same thread. > >> There are a couple of these small issues that should be perfectly safe that >> I was hoping to address outside of this patch so that it didn't get too >> big. > > I noticed you introduced sched_throw(). Would it harm if ia64 > doesn't yet use sched_throw() and instead has the sequence it > replaces? In other words: is the initial implementation of > sched_throw() the same as the current code? The problem is that sched_throw() must acquire the correct scheduler lock before entering cpu_throw(). That's why I moved it into the per-scheduler code. sched_smp, which is the updated ule, acquires the correct lock for the current cpu. Jeff > > -- > Marcel Moolenaar > xcllnt@mac.com > From owner-freebsd-arch@FreeBSD.ORG Thu May 24 00:24:21 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CD67C16A400 for ; Thu, 24 May 2007 00:24:21 +0000 (UTC) (envelope-from xcllnt@mac.com) Received: from smtpout.mac.com (smtpout.mac.com [17.250.248.175]) by mx1.freebsd.org (Postfix) with ESMTP id BAB4713C45B for ; Thu, 24 May 2007 00:24:21 +0000 (UTC) (envelope-from xcllnt@mac.com) Received: from mac.com (smtpin08-en2 [10.13.10.153]) by smtpout.mac.com (Xserve/smtpout05/MantshX 4.0) with ESMTP id l4O0OFk7011619; Wed, 23 May 2007 17:24:15 -0700 (PDT) Received: from [172.24.104.227] (natint3.juniper.net [66.129.224.36]) (authenticated bits=0) by mac.com (Xserve/smtpin08/MantshX 4.0) with ESMTP id l4O0OEIx027964 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 23 May 2007 17:24:14 -0700 (PDT) In-Reply-To: <20070523170449.L9443@10.0.0.1> References: <20070520155103.K632@10.0.0.1> <20070523155236.U9443@10.0.0.1> <6A9BD12D-D93C-4AE8-B4F4-D59A0327032D@mac.com> <20070523163109.X9443@10.0.0.1> <38601004-BB95-4B8B-87A6-26E2D52B89BA@mac.com> <20070523170449.L9443@10.0.0.1> Mime-Version: 1.0 (Apple Message framework v752.3) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <15155940-CE7B-4E9F-9FD1-4C83A3EA5F48@mac.com> Content-Transfer-Encoding: 7bit From: Marcel Moolenaar Date: Wed, 23 May 2007 17:24:10 -0700 To: Jeff Roberson X-Mailer: Apple Mail (2.752.3) X-Brightmail-Tracker: AAAAAA== X-Brightmail-scanned: yes Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 00:24:21 -0000 On May 23, 2007, at 5:11 PM, Jeff Roberson wrote: >> pmap_switch() is called from cpu_switch() and from pmap_install(). >> So, currently, pmap_install() grabs sched_lock to mimic the >> cpu_switch() path and we assert having sched_lock in pmap_switch(). >> Basically, any lock that serializes cpu_switch() would work, because >> we don't want to switch the thread while in the middle of setting up >> the region registers. > > We could simply use thread_lock() now if this serialization only > applies to preventing multiple access to the same thread. Yes, looks like it. >>> There are a couple of these small issues that should be perfectly >>> safe that I was hoping to address outside of this patch so that >>> it didn't get too big. >> >> I noticed you introduced sched_throw(). Would it harm if ia64 >> doesn't yet use sched_throw() and instead has the sequence it >> replaces? In other words: is the initial implementation of >> sched_throw() the same as the current code? > > The problem is that sched_throw() must acquire the correct > scheduler lock before entering cpu_throw(). That's why I moved it > into the per-scheduler code. sched_smp, which is the updated ule, > acquires the correct lock for the current cpu. Sounds like we want to keep ia64 in sync then. Please let me know before you commit if you found the time, motivation, whatever to include ia64 in the change or not. Either I want to test it or I want to fix it ;-) -- Marcel Moolenaar xcllnt@mac.com From owner-freebsd-arch@FreeBSD.ORG Thu May 24 00:47:17 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1D9FA16A400 for ; Thu, 24 May 2007 00:47:17 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd4mo1so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id F20C113C44B for ; Thu, 24 May 2007 00:47:16 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd2mr3so.prod.shaw.ca (pd2mr3so-qfe3.prod.shaw.ca [10.0.141.108]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JII00D2RSUPG5A0@l-daemon> for freebsd-arch@freebsd.org; Wed, 23 May 2007 18:47:13 -0600 (MDT) Received: from pn2ml10so.prod.shaw.ca ([10.0.121.80]) by pd2mr3so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JII00FCWSUNBF00@pd2mr3so.prod.shaw.ca> for freebsd-arch@freebsd.org; Wed, 23 May 2007 18:47:14 -0600 (MDT) Received: from hexahedron.daemonology.net ([24.82.18.31]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with SMTP id <0JII002U3SUMJC20@l-daemon> for freebsd-arch@freebsd.org; Wed, 23 May 2007 18:47:10 -0600 (MDT) Received: (qmail 3241 invoked from network); Thu, 24 May 2007 00:46:59 +0000 Received: from unknown (HELO hexahedron.daemonology.net) (127.0.0.1) by localhost with SMTP; Thu, 24 May 2007 00:46:59 +0000 Date: Wed, 23 May 2007 17:46:59 -0700 From: Colin Percival In-reply-to: <7158.1179947572@critter.freebsd.dk> To: Poul-Henning Kamp Message-id: <4654E083.10807@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.95.0 References: <7158.1179947572@critter.freebsd.dk> User-Agent: Thunderbird 2.0.0.0 (X11/20070511) Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 00:47:17 -0000 Poul-Henning Kamp wrote: > In message <46546E16.9070707@freebsd.org>, Colin Percival writes: >> I'd like to remove file(1) and libmagic(3) from the FreeBSD base system >> for the following reasons: > > One mitigating option would be to open the magic file and input > and sequester the file process in a jail. Last time I checked, unprivileged processes couldn't jail themselves. We could make file(1) setuid root and use a privilege separation approach, but I'm not convinced that would be a net win. Colin Percival From owner-freebsd-arch@FreeBSD.ORG Thu May 24 00:55:31 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2A4D216A400 for ; Thu, 24 May 2007 00:55:31 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outS.internet-mail-service.net (outS.internet-mail-service.net [216.240.47.242]) by mx1.freebsd.org (Postfix) with ESMTP id 1A6E313C455 for ; Thu, 24 May 2007 00:55:31 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Wed, 23 May 2007 17:55:30 -0700 Received: from julian-mac.elischer.org (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id 04294125A27; Wed, 23 May 2007 17:55:29 -0700 (PDT) Message-ID: <4654E287.3040206@elischer.org> Date: Wed, 23 May 2007 17:55:35 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.0 (Macintosh/20070326) MIME-Version: 1.0 To: Colin Percival References: <7158.1179947572@critter.freebsd.dk> <4654E083.10807@freebsd.org> In-Reply-To: <4654E083.10807@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Poul-Henning Kamp , "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 00:55:31 -0000 Colin Percival wrote: > Poul-Henning Kamp wrote: >> In message <46546E16.9070707@freebsd.org>, Colin Percival writes: >>> I'd like to remove file(1) and libmagic(3) from the FreeBSD base system >>> for the following reasons: >> One mitigating option would be to open the magic file and input >> and sequester the file process in a jail. > > Last time I checked, unprivileged processes couldn't jail themselves. We > could make file(1) setuid root and use a privilege separation approach, > but I'm not convinced that would be a net win. How about a bit in the headers of a program that are set by the Makefile. If the bit is not set then the elf program executor sets a bit that forbids exec from ever running.. how many programs actually need to be able to run exec.. the average exploit does an exec(/bin/sh) > > Colin Percival > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Thu May 24 01:17:49 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D401616A421; Thu, 24 May 2007 01:17:49 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id C5E2213C44B; Thu, 24 May 2007 01:17:49 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id EB9A91A3C19; Wed, 23 May 2007 18:18:50 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id E226D513FC; Wed, 23 May 2007 21:17:48 -0400 (EDT) Date: Wed, 23 May 2007 21:17:48 -0400 From: Kris Kennaway To: Julian Elischer Message-ID: <20070524011748.GA68201@xor.obsecurity.org> References: <7158.1179947572@critter.freebsd.dk> <4654E083.10807@freebsd.org> <4654E287.3040206@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4654E287.3040206@elischer.org> User-Agent: Mutt/1.4.2.2i Cc: Poul-Henning Kamp , Colin Percival , "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 01:17:49 -0000 On Wed, May 23, 2007 at 05:55:35PM -0700, Julian Elischer wrote: > Colin Percival wrote: > >Poul-Henning Kamp wrote: > >>In message <46546E16.9070707@freebsd.org>, Colin Percival writes: > >>>I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > >>>for the following reasons: > >>One mitigating option would be to open the magic file and input > >>and sequester the file process in a jail. > > > >Last time I checked, unprivileged processes couldn't jail themselves. We > >could make file(1) setuid root and use a privilege separation approach, > >but I'm not convinced that would be a net win. > > How about a bit in the headers of a program that are set by the Makefile. > If the bit is not set then the elf program executor sets a bit that > forbids exec from ever running.. > > how many programs actually need to be able to run exec.. > the average exploit does an exec(/bin/sh) Cart before horse. Colin needs to first tell us what attack he is trying to stop before we can figure out how to stop it. Kris P.S. Thesedays we have the MAC subsystem, no need for magic hacks of this nature. From owner-freebsd-arch@FreeBSD.ORG Thu May 24 01:36:44 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 94D2016A400 for ; Thu, 24 May 2007 01:36:44 +0000 (UTC) (envelope-from grog@lemis.com) Received: from ipmail02.adl2.internode.on.net (ipmail02.adl2.internode.on.net [203.16.214.141]) by mx1.freebsd.org (Postfix) with ESMTP id 1896E13C447 for ; Thu, 24 May 2007 01:36:43 +0000 (UTC) (envelope-from grog@lemis.com) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAKaEVEbAbcWH/2dsb2JhbAA X-IronPort-AV: E=Sophos;i="4.14,571,1170595800"; d="scan'208";a="128630270" Received: from wantadilla.lemis.com ([192.109.197.135]) by ipmail02.adl2.internode.on.net with ESMTP; 24 May 2007 10:50:07 +0930 Received: by wantadilla.lemis.com (Postfix, from userid 1004) id E0C1B1A989C; Thu, 24 May 2007 10:28:17 +0930 (CST) Date: Thu, 24 May 2007 10:28:17 +0930 From: Greg 'groggy' Lehey To: Alfred Perlstein Message-ID: <20070524005817.GD46113@wantadilla.lemis.com> References: <46546E16.9070707@freebsd.org> <20070523195933.GM21795@elvis.mu.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="YToU2i3Vx8H2dn7O" Content-Disposition: inline In-Reply-To: <20070523195933.GM21795@elvis.mu.org> User-Agent: Mutt/1.4.2.1i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 VoIP: sip:0871270137@sip.internode.on.net WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 Cc: Daniel Eischen , Colin Percival , "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 01:36:44 -0000 --YToU2i3Vx8H2dn7O Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wednesday, 23 May 2007 at 12:59:33 -0700, Alfred Perlstein wrote: > * Daniel Eischen [070523 11:05] wrote: >> On Wed, 23 May 2007, Colin Percival wrote: >> >>> FreeBSD architects and file(1) maintainer, >>> >>> I'd like to remove file(1) and libmagic(3) from the FreeBSD base system >>> >>> Can anyone make a strong argument for keeping this code in the base system? >> >> Yes, because other OS's have it (file) in their base, and because >> it is a POSIX-defined utility. Please consider this a strong no. > > I agree with Daniel. Me too! One of the most stupid things I know in the Microsoft space is to identify files by external features such as their name; IIRC this has opened the way for trojans such as executables posing as images, etc. The obvious alternative is the "UNIX way": identify the files by their content, not their name. And that's precisely the purpose of file(1). Removing it seems counterproductive. Greg -- See complete headers for address and phone numbers. --YToU2i3Vx8H2dn7O Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFGVOMpIubykFB6QiMRAsvdAKCZBGY7kFhGoZ4pmQgSwDvfeURq7QCfScJU Vy9LCtzi7QnT3EuhkSm7LlY= =HnW3 -----END PGP SIGNATURE----- --YToU2i3Vx8H2dn7O-- From owner-freebsd-arch@FreeBSD.ORG Thu May 24 03:23:58 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 149ED16A421 for ; Thu, 24 May 2007 03:23:58 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id D368013C465 for ; Thu, 24 May 2007 03:23:57 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.101] (c-71-231-138-78.hsd1.or.comcast.net [71.231.138.78]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l4O3Nost073031 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Wed, 23 May 2007 23:23:51 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Wed, 23 May 2007 20:23:43 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: Marcel Moolenaar In-Reply-To: <15155940-CE7B-4E9F-9FD1-4C83A3EA5F48@mac.com> Message-ID: <20070523202038.M9443@10.0.0.1> References: <20070520155103.K632@10.0.0.1> <20070523155236.U9443@10.0.0.1> <6A9BD12D-D93C-4AE8-B4F4-D59A0327032D@mac.com> <20070523163109.X9443@10.0.0.1> <38601004-BB95-4B8B-87A6-26E2D52B89BA@mac.com> <20070523170449.L9443@10.0.0.1> <15155940-CE7B-4E9F-9FD1-4C83A3EA5F48@mac.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 03:23:58 -0000 On Wed, 23 May 2007, Marcel Moolenaar wrote: > On May 23, 2007, at 5:11 PM, Jeff Roberson wrote: > >>> pmap_switch() is called from cpu_switch() and from pmap_install(). >>> So, currently, pmap_install() grabs sched_lock to mimic the >>> cpu_switch() path and we assert having sched_lock in pmap_switch(). >>> Basically, any lock that serializes cpu_switch() would work, because >>> we don't want to switch the thread while in the middle of setting up >>> the region registers. >> >> We could simply use thread_lock() now if this serialization only applies to >> preventing multiple access to the same thread. > > Yes, looks like it. > >>>> There are a couple of these small issues that should be perfectly safe >>>> that I was hoping to address outside of this patch so that it didn't get >>>> too big. >>> >>> I noticed you introduced sched_throw(). Would it harm if ia64 >>> doesn't yet use sched_throw() and instead has the sequence it >>> replaces? In other words: is the initial implementation of >>> sched_throw() the same as the current code? >> >> The problem is that sched_throw() must acquire the correct scheduler lock >> before entering cpu_throw(). That's why I moved it into the per-scheduler >> code. sched_smp, which is the updated ule, acquires the correct lock for >> the current cpu. > > Sounds like we want to keep ia64 in sync then. Please let me know > before you commit if you found the time, motivation, whatever to > include ia64 in the change or not. Either I want to test it or > I want to fix it ;-) I updated the patch at people.freebsd.org/~jeff/threadlock.diff Can you try this on ia64 marcel? You can try with 4BSD and ULE. ULE may not work if IPI_PREEMPT is not implemented however. I think it may be missing there. I changed the locks and asserts in pmap.c and moved over to the new sched_throw() interface. The other change in this diff is that I moved thread_lock_flags() into kern_mutex.c and fixed the thread locking loop so it is aware of td_lock transitions before it's successfully acquired a lock that may no longer be pointed at by the thread. This also removes the special case for blocked_lock and simply does the equivalent of a trylock() spin until it changes. Thanks, Jeff > > -- > Marcel Moolenaar > xcllnt@mac.com > From owner-freebsd-arch@FreeBSD.ORG Thu May 24 07:00:37 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E979F16A400 for ; Thu, 24 May 2007 07:00:37 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.NUXI.org (trang.nuxi.org [64.81.59.225]) by mx1.freebsd.org (Postfix) with ESMTP id C428F13C44C for ; Thu, 24 May 2007 07:00:37 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.NUXI.org (obrien@localhost [127.0.0.1]) by dragon.NUXI.org (8.13.8/8.13.8) with ESMTP id l4O6Lu53040619; Wed, 23 May 2007 23:21:56 -0700 (PDT) (envelope-from obrien@dragon.NUXI.org) Received: (from obrien@localhost) by dragon.NUXI.org (8.14.1/8.13.7/Submit) id l4O6LuRw040612; Wed, 23 May 2007 23:21:56 -0700 (PDT) (envelope-from obrien) Date: Wed, 23 May 2007 23:21:55 -0700 From: "David O'Brien" To: Colin Percival Message-ID: <20070524062155.GA39887@dragon.NUXI.org> Mail-Followup-To: obrien@freebsd.org, Colin Percival , "freebsd-arch@freebsd.org" References: <46546E16.9070707@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46546E16.9070707@freebsd.org> X-Operating-System: FreeBSD 7.0-CURRENT Organization: The NUXI BSD Group X-Pgp-Rsa-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Rsa-Keyid: 1024/34F9F9D5 User-Agent: Mutt/1.5.11 Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: obrien@freebsd.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 07:00:38 -0000 On Wed, May 23, 2007 at 09:38:46AM -0700, Colin Percival wrote: > FreeBSD architects and file(1) maintainer, > > I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > for the following reasons: > 1. I don't see it as being a necessary component of a UNIX-like operating > system. All I can say is "Wow - mind blowing". I do find it one of the essential components of a Unix system. > 2. It's available in the ports tree. So are many of things in /usr/bin > 3. Due to its nature as a program which parses multiple data formats, it > poses an unusually high risk of having security problems in the future > (cf. ethereal/wireshark). I think that is a stretch - so many utilities in the base system parse its input (just another word for processing input) > The one redeeming feature of file/libmagic as far as security is concerned > is that it doesn't act as a daemon, i.e., other code or user intervention > is required for an attacker to exploit security issues. And I think that is sufficient to make this idea a little over the top. thanks, -- -- David (obrien@FreeBSD.org) Q: Because it reverses the logical flow of conversation. A: Why is top-posting (putting a reply at the top of the message) frowned upon? Let's not play "Jeopardy-style quoting" From owner-freebsd-arch@FreeBSD.ORG Thu May 24 07:02:18 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6366B16A421 for ; Thu, 24 May 2007 07:02:18 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd4mo2so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id 3E99613C43E for ; Thu, 24 May 2007 07:02:18 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd3mr3so.prod.shaw.ca (pd3mr3so-qfe3.prod.shaw.ca [10.0.141.179]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JIJ00DQXA7U4I40@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:02:18 -0600 (MDT) Received: from pn2ml8so.prod.shaw.ca ([10.0.121.152]) by pd3mr3so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JIJ00I8MA7VWPV0@pd3mr3so.prod.shaw.ca> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:02:19 -0600 (MDT) Received: from hexahedron.daemonology.net ([24.82.18.31]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with SMTP id <0JIJ0080PA7THX30@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:02:18 -0600 (MDT) Received: (qmail 4082 invoked from network); Thu, 24 May 2007 07:02:07 +0000 Received: from unknown (HELO hexahedron.daemonology.net) (127.0.0.1) by localhost with SMTP; Thu, 24 May 2007 07:02:07 +0000 Date: Thu, 24 May 2007 00:02:06 -0700 From: Colin Percival In-reply-to: <20070523192103.GA61937@xor.obsecurity.org> To: Kris Kennaway Message-id: <4655386E.2000605@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.95.0 References: <46546E16.9070707@freebsd.org> <20070523192103.GA61937@xor.obsecurity.org> User-Agent: Thunderbird 2.0.0.0 (X11/20070511) Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 07:02:18 -0000 Kris Kennaway wrote: > What is the threat you are defending against here: "Admin runs file(1) > on untrusted binary"? Yes, or "user runs script(s) which run file(1) on untrusted binaries". > If so, how does it differ from e.g. running cat(1) on an untrusted > binary, which can reprogram your terminal emulation and in some cases > take over your terminal; or from various other unprivileged user > binaries that also crash when operating on corrupted data, possibly in > an exploitable way? Last time I checked lots of our /usr/bin tools > coredumped when you passed them unexpected input. What do you mean by "unexpected input"? Do you mean unexpected data on stdin to tools like b64decode, comm, cut, diff, and fold, which might reasonably be run on untrusted data, or do you mean wacky command lines to utilities like awk or c99 (where control over said command line would innately give an attacker the ability to run code of his choosing)? > Also, did coverity find the buffer overflow No. The overflow resulted from failing to correctly keep track of how much space was left in a buffer, so it wasn't something which Coverity (or any other similar tool) really had any chance to find. Colin Percival From owner-freebsd-arch@FreeBSD.ORG Thu May 24 07:10:47 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A3F0416A468 for ; Thu, 24 May 2007 07:10:47 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd2mo3so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id 7D5DB13C480 for ; Thu, 24 May 2007 07:10:47 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd4mr2so.prod.shaw.ca (pd4mr2so-qfe3.prod.shaw.ca [10.0.141.213]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JIJ001ZKAM0P490@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:10:48 -0600 (MDT) Received: from pn2ml4so.prod.shaw.ca ([10.0.121.148]) by pd4mr2so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JIJ002F6AM06BW0@pd4mr2so.prod.shaw.ca> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:10:48 -0600 (MDT) Received: from hexahedron.daemonology.net ([24.82.18.31]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with SMTP id <0JIJ00IYIALYXNJ2@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:10:47 -0600 (MDT) Received: (qmail 4104 invoked from network); Thu, 24 May 2007 07:10:35 +0000 Received: from unknown (HELO hexahedron.daemonology.net) (127.0.0.1) by localhost with SMTP; Thu, 24 May 2007 07:10:35 +0000 Date: Thu, 24 May 2007 00:10:35 -0700 From: Colin Percival In-reply-to: <20070523.161038.-1989860747.imp@bsdimp.com> To: "M. Warner Losh" Message-id: <46553A6B.7070904@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.95.0 References: <46546E16.9070707@freebsd.org> <7158.1179947572@critter.freebsd.dk> <20070523213251.GA14733@keltia.freenix.fr> <20070523.161038.-1989860747.imp@bsdimp.com> User-Agent: Thunderbird 2.0.0.0 (X11/20070511) Cc: freebsd-arch@freebsd.org Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 07:10:47 -0000 M. Warner Losh wrote: > I would argue that it would make the system LESS secure, because one > loses the ability to identify files on the system. People are going > to install it anyway, and it is a jump ball as to whether having it in > the base system would cause vulnerabilities to be updated faster than > having it in ports (both the actual update in the system, as well as > the user causing the update to happen: ports are a touch easier to > update, but lag a bit both in terms of people updating their ports > tree and ports committers updating the port). Interestingly, my experience from portsnap is that people tend to update ports more frequently than they apply security patches to the base system. > And for there to be any exploitable vulnerability, the attacker would > need to feed the victum a bogusly formatted file, and cause the victum > to run file on that file. I doubt that the latest security hole will > ever result in a system compromise... You're more optimistic than I am, then. This latest advisory was issued on the basis of "it's a heap overflow in rather messy code, so we really have no idea if it's exploitable". > I guess I fail to see how this is any different than the .gz bugs that > were found a while ago. Nobody suggested removing .gz from the tree > because a few bugs were found. Everybody suggested updating right > away to fix those bugs. File is no different, and really should > remain in the tree. Deflate is one file format which is used quite often. File parses several different formats, including several which are not tested often (i.e., have a much higher chance of including parse bugs). Colin Percival From owner-freebsd-arch@FreeBSD.ORG Thu May 24 07:14:45 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F114C16A46C for ; Thu, 24 May 2007 07:14:45 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd3mo3so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id C7B9913C4B0 for ; Thu, 24 May 2007 07:14:45 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd2mr6so.prod.shaw.ca (pd2mr6so-qfe3.prod.shaw.ca [10.0.141.9]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JIJ00GGXASMTV60@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:14:46 -0600 (MDT) Received: from pn2ml7so.prod.shaw.ca ([10.0.121.151]) by pd2mr6so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JIJ00JA9ASL8GN0@pd2mr6so.prod.shaw.ca> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:14:46 -0600 (MDT) Received: from hexahedron.daemonology.net ([24.82.18.31]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with SMTP id <0JIJ00K5AASKU2J1@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:14:45 -0600 (MDT) Received: (qmail 4122 invoked from network); Thu, 24 May 2007 07:14:33 +0000 Received: from unknown (HELO hexahedron.daemonology.net) (127.0.0.1) by localhost with SMTP; Thu, 24 May 2007 07:14:33 +0000 Date: Thu, 24 May 2007 00:14:33 -0700 From: Colin Percival In-reply-to: <20070524005817.GD46113@wantadilla.lemis.com> To: Greg 'groggy' Lehey Message-id: <46553B59.5030501@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.95.0 References: <46546E16.9070707@freebsd.org> <20070523195933.GM21795@elvis.mu.org> <20070524005817.GD46113@wantadilla.lemis.com> User-Agent: Thunderbird 2.0.0.0 (X11/20070511) Cc: Daniel Eischen , Alfred Perlstein , "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 07:14:46 -0000 Greg 'groggy' Lehey wrote: > One of the most stupid things I know in the Microsoft space is to > identify files by external features such as their name; IIRC this has > opened the way for trojans such as executables posing as images, etc. > The obvious alternative is the "UNIX way": identify the files by their > content, not their name. And that's precisely the purpose of > file(1). Removing it seems counterproductive. >From a security perspective, the only thing I can imagine which is worse than identifying the data type of a file based on the file name is to look at the file contents and try to guess. This lends itself to attacks against firewall systems by constructing a file which the firewall decides looks like a harmless file type, but the target host decides is something different. External metadata -- using MIME types, ideally -- is the only secure way to define file types. Colin Percival From owner-freebsd-arch@FreeBSD.ORG Thu May 24 07:17:49 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CE95816A46E; Thu, 24 May 2007 07:17:49 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 8604413C448; Thu, 24 May 2007 07:17:49 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 3B5391A4D80; Thu, 24 May 2007 00:18:44 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id A7435513FC; Thu, 24 May 2007 03:17:32 -0400 (EDT) Date: Thu, 24 May 2007 03:17:32 -0400 From: Kris Kennaway To: Colin Percival Message-ID: <20070524071732.GA80416@xor.obsecurity.org> References: <46546E16.9070707@freebsd.org> <20070523192103.GA61937@xor.obsecurity.org> <4655386E.2000605@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/04w6evG8XlLl3ft" Content-Disposition: inline In-Reply-To: <4655386E.2000605@freebsd.org> User-Agent: Mutt/1.4.2.2i Cc: "freebsd-arch@freebsd.org" , Kris Kennaway Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 07:17:49 -0000 --/04w6evG8XlLl3ft Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 24, 2007 at 12:02:06AM -0700, Colin Percival wrote: > Kris Kennaway wrote: > > What is the threat you are defending against here: "Admin runs file(1) > > on untrusted binary"? >=20 > Yes, or "user runs script(s) which run file(1) on untrusted binaries". OK. > > If so, how does it differ from e.g. running cat(1) on an untrusted > > binary, which can reprogram your terminal emulation and in some cases > > take over your terminal; or from various other unprivileged user > > binaries that also crash when operating on corrupted data, possibly in > > an exploitable way? Last time I checked lots of our /usr/bin tools > > coredumped when you passed them unexpected input. >=20 > What do you mean by "unexpected input"? Do you mean unexpected data on > stdin to tools like b64decode, comm, cut, diff, and fold, which might > reasonably be run on untrusted data, or do you mean wacky command lines > to utilities like awk or c99 (where control over said command line would > innately give an attacker the ability to run code of his choosing)? It was both (sed was another utility that liked to crash when fed "badly-formed" input via stdin). It's in the same problem class as your attack model, so any solution you come up with needs to also include other similar utilities. I think it's pretty clear that "remove them" is a non-starter, so you should try to focus on ways to contain or mitigate the effects of the class of threat you are concerned about. There is probably a lot of scope for using the trustedbsd security features for hardening systems that are at risk for this threat model. > > Also, did coverity find the buffer overflow >=20 > No. The overflow resulted from failing to correctly keep track of how > much space was left in a buffer, so it wasn't something which Coverity > (or any other similar tool) really had any chance to find. OK. Kris --/04w6evG8XlLl3ft Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGVTwMWry0BWjoQKURAkg/AKDKZj723zUXqKkjiyVOTWiZ5mI8JgCgr74c Bj+SVSev27qIJecp8TLgACU= =DhJR -----END PGP SIGNATURE----- --/04w6evG8XlLl3ft-- From owner-freebsd-arch@FreeBSD.ORG Thu May 24 07:19:07 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C4A8516A421; Thu, 24 May 2007 07:19:07 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id B164213C44C; Thu, 24 May 2007 07:19:07 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 3BBD51A4D80; Thu, 24 May 2007 00:20:09 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id E5735513FC; Thu, 24 May 2007 03:19:06 -0400 (EDT) Date: Thu, 24 May 2007 03:19:06 -0400 From: Kris Kennaway To: Colin Percival Message-ID: <20070524071906.GB80416@xor.obsecurity.org> References: <46546E16.9070707@freebsd.org> <7158.1179947572@critter.freebsd.dk> <20070523213251.GA14733@keltia.freenix.fr> <20070523.161038.-1989860747.imp@bsdimp.com> <46553A6B.7070904@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46553A6B.7070904@freebsd.org> User-Agent: Mutt/1.4.2.2i Cc: freebsd-arch@freebsd.org Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 07:19:07 -0000 On Thu, May 24, 2007 at 12:10:35AM -0700, Colin Percival wrote: > M. Warner Losh wrote: > > I would argue that it would make the system LESS secure, because one > > loses the ability to identify files on the system. People are going > > to install it anyway, and it is a jump ball as to whether having it in > > the base system would cause vulnerabilities to be updated faster than > > having it in ports (both the actual update in the system, as well as > > the user causing the update to happen: ports are a touch easier to > > update, but lag a bit both in terms of people updating their ports > > tree and ports committers updating the port). > > Interestingly, my experience from portsnap is that people tend to update > ports more frequently than they apply security patches to the base system. ...with freebsd update. Important qualification. > > And for there to be any exploitable vulnerability, the attacker would > > need to feed the victum a bogusly formatted file, and cause the victum > > to run file on that file. I doubt that the latest security hole will > > ever result in a system compromise... > > You're more optimistic than I am, then. This latest advisory was issued > on the basis of "it's a heap overflow in rather messy code, so we really > have no idea if it's exploitable". The only way I can think of is if there is a MIME email scanner out there that uses file(1) to identify attachment types. Kris From owner-freebsd-arch@FreeBSD.ORG Thu May 24 07:24:17 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B3E1216A421 for ; Thu, 24 May 2007 07:24:17 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd2mo1so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id 8B66313C48A for ; Thu, 24 May 2007 07:24:17 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd3mr5so.prod.shaw.ca (pd3mr5so-qfe3.prod.shaw.ca [10.0.141.12]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JIJ00CQ3B6V4L60@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:23:19 -0600 (MDT) Received: from pn2ml7so.prod.shaw.ca ([10.0.121.151]) by pd3mr5so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JIJ00KXOB6ULXO0@pd3mr5so.prod.shaw.ca> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:23:19 -0600 (MDT) Received: from hexahedron.daemonology.net ([24.82.18.31]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with SMTP id <0JIJ001T6B6T7W71@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:23:18 -0600 (MDT) Received: (qmail 4183 invoked from network); Thu, 24 May 2007 07:23:07 +0000 Received: from unknown (HELO hexahedron.daemonology.net) (127.0.0.1) by localhost with SMTP; Thu, 24 May 2007 07:23:07 +0000 Date: Thu, 24 May 2007 00:23:06 -0700 From: Colin Percival In-reply-to: <20070524062155.GA39887@dragon.NUXI.org> To: obrien@freebsd.org, Colin Percival , "freebsd-arch@freebsd.org" Message-id: <46553D5A.9090304@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.95.0 References: <46546E16.9070707@freebsd.org> <20070524062155.GA39887@dragon.NUXI.org> User-Agent: Thunderbird 2.0.0.0 (X11/20070511) Cc: Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 07:24:17 -0000 > On Wed, May 23, 2007 at 09:38:46AM -0700, Colin Percival wrote: >> FreeBSD architects and file(1) maintainer, >> >> I'd like to remove file(1) and libmagic(3) from the FreeBSD base system >> for the following reasons: >> 1. I don't see it as being a necessary component of a UNIX-like operating >> system. Ok, so far I've seen 3 votes in favour (including my own) and 15 votes against removing file; so I guess it's not time to swing the axe quite yet. David, could you please import file-4.21 to the vendor branch once it is released? Or if you don't mind importing something onto the vendor branch which isn't quite what the vendor distributed, I can give you a security patch generated against file-4.20. Colin Percival From owner-freebsd-arch@FreeBSD.ORG Thu May 24 07:27:26 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 62B5416A469 for ; Thu, 24 May 2007 07:27:26 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd2mo1so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id 3C4F313C46C for ; Thu, 24 May 2007 07:27:26 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from pd3mr4so.prod.shaw.ca (pd3mr4so-qfe3.prod.shaw.ca [10.0.141.180]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JIJ00CXKBDM4W60@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:27:22 -0600 (MDT) Received: from pn2ml7so.prod.shaw.ca ([10.0.121.151]) by pd3mr4so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JIJ003AYBDLN241@pd3mr4so.prod.shaw.ca> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:27:22 -0600 (MDT) Received: from hexahedron.daemonology.net ([24.82.18.31]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with SMTP id <0JIJ001ZBBDK7W71@l-daemon> for freebsd-arch@freebsd.org; Thu, 24 May 2007 01:27:21 -0600 (MDT) Received: (qmail 4193 invoked from network); Thu, 24 May 2007 07:27:06 +0000 Received: from unknown (HELO hexahedron.daemonology.net) (127.0.0.1) by localhost with SMTP; Thu, 24 May 2007 07:27:06 +0000 Date: Thu, 24 May 2007 00:27:06 -0700 From: Colin Percival In-reply-to: <20070524071906.GB80416@xor.obsecurity.org> To: Kris Kennaway Message-id: <46553E4A.1060008@freebsd.org> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.95.0 References: <46546E16.9070707@freebsd.org> <7158.1179947572@critter.freebsd.dk> <20070523213251.GA14733@keltia.freenix.fr> <20070523.161038.-1989860747.imp@bsdimp.com> <46553A6B.7070904@freebsd.org> <20070524071906.GB80416@xor.obsecurity.org> User-Agent: Thunderbird 2.0.0.0 (X11/20070511) Cc: freebsd-arch@freebsd.org Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 07:27:26 -0000 Kris Kennaway wrote: > On Thu, May 24, 2007 at 12:10:35AM -0700, Colin Percival wrote: >> Interestingly, my experience from portsnap is that people tend to update >> ports more frequently than they apply security patches to the base system. > > ...with freebsd update. Important qualification. No, I was looking at version numbers reported by portsnap: Over half of the systems running FreeBSD 6.0 or FreeBSD 6.1 are still running the RELEASE with no security patches (or no kernel patches, at least), while systems running old versions of portsnap were upgraded to newer versions of portsnap far more quickly. Admittedly, there is a bias here in that people running portsnap are likely to be more interested in updating their installed ports than most FreeBSD users; but I still think it's a significant difference. Colin Percival From owner-freebsd-arch@FreeBSD.ORG Thu May 24 07:42:37 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4556516A468; Thu, 24 May 2007 07:42:37 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 0304313C465; Thu, 24 May 2007 07:42:36 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.8/8.13.4) with ESMTP id l4O7fI9W081827; Thu, 24 May 2007 01:41:18 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Thu, 24 May 2007 01:41:31 -0600 (MDT) Message-Id: <20070524.014131.756908450.imp@bsdimp.com> To: cperciva@freebsd.org From: "M. Warner Losh" In-Reply-To: <46553A6B.7070904@freebsd.org> References: <20070523213251.GA14733@keltia.freenix.fr> <20070523.161038.-1989860747.imp@bsdimp.com> <46553A6B.7070904@freebsd.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Thu, 24 May 2007 01:41:19 -0600 (MDT) Cc: freebsd-arch@freebsd.org Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 07:42:37 -0000 In message: <46553A6B.7070904@freebsd.org> Colin Percival writes: : > I guess I fail to see how this is any different than the .gz bugs that : > were found a while ago. Nobody suggested removing .gz from the tree : > because a few bugs were found. Everybody suggested updating right : > away to fix those bugs. File is no different, and really should : > remain in the tree. : : Deflate is one file format which is used quite often. File parses several : different formats, including several which are not tested often (i.e., have : a much higher chance of including parse bugs). Now you are really reaching... File does it work by reading in meta data (in one format, well tested) that describes how to parse the files using a simple meta-language to describe offsets. I think you are really reaching here, unless you can show some conclusive evidence to support that this can cause problems (apart from core dumps from memory exhaustion, and the like). Can you point to one of the magic meta files that has a description so complex that it will likely cause these kinds of things to happen? Warner From owner-freebsd-arch@FreeBSD.ORG Thu May 24 08:05:11 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7142D16A468 for ; Thu, 24 May 2007 08:05:11 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id 26F6513C484 for ; Thu, 24 May 2007 08:05:10 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (unknown [84.165.196.247]) by redbull.bpaserver.net (Postfix) with ESMTP id 44E042E13F; Thu, 24 May 2007 09:44:37 +0200 (CEST) Received: from webmail.leidinger.net (webmail.Leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id DE6B05B545F; Thu, 24 May 2007 09:44:20 +0200 (CEST) Received: (from www@localhost) by webmail.leidinger.net (8.13.8/8.13.8/Submit) id l4O7iKAd057175; Thu, 24 May 2007 09:44:20 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde MIME library) with HTTP; Thu, 24 May 2007 09:44:20 +0200 Message-ID: <20070524094420.de73ozujr4sccc0o@webmail.leidinger.net> X-Priority: 3 (Normal) Date: Thu, 24 May 2007 09:44:20 +0200 From: Alexander Leidinger To: Colin Percival References: <46546E16.9070707@freebsd.org> <7158.1179947572@critter.freebsd.dk> <20070523213251.GA14733@keltia.freenix.fr> <20070523.161038.-1989860747.imp@bsdimp.com> <46553A6B.7070904@freebsd.org> In-Reply-To: <46553A6B.7070904@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.4) / FreeBSD-7.0 X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-14.364, required 8, BAYES_00 -15.00, DK_POLICY_SIGNSOME 0.00, FORGED_RCVD_HELO 0.14, NO_RDNS 0.50) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No Cc: freebsd-arch@freebsd.org Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 08:05:11 -0000 Quoting Colin Percival (from Thu, 24 May 2007 =20 00:10:35 -0700): > M. Warner Losh wrote: >> I would argue that it would make the system LESS secure, because one >> loses the ability to identify files on the system. People are going >> to install it anyway, and it is a jump ball as to whether having it in >> the base system would cause vulnerabilities to be updated faster than >> having it in ports (both the actual update in the system, as well as >> the user causing the update to happen: ports are a touch easier to >> update, but lag a bit both in terms of people updating their ports >> tree and ports committers updating the port). > > Interestingly, my experience from portsnap is that people tend to update > ports more frequently than they apply security patches to the base system. You can say people tend to update the ports collection, you don't know =20 about the ports (as in "installed ports"). I have several systems =20 (behind a proxy) which update the ports collection every day. But the =20 ports there are not updated that often. I also know about several =20 systems where the ports collection is updated every day, but the =20 installed ports are only touched if a client ask about a new software =20 or an update, which is maybe once a year. I also like to keep file in the base. It's too damn useful there. Bye, Alexander. --=20 He who hesitates is last. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID =3D B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID =3D 72077137 From owner-freebsd-arch@FreeBSD.ORG Thu May 24 09:04:58 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 535D016A46B for ; Thu, 24 May 2007 09:04:58 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.185]) by mx1.freebsd.org (Postfix) with ESMTP id CD17913C458 for ; Thu, 24 May 2007 09:04:57 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by mu-out-0910.google.com with SMTP id w9so59280mue for ; Thu, 24 May 2007 02:04:56 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=kQkxLWXMYh/OzM8SeTNRz4fBtl3ONmvDOBymhVNCQwPV+Ls5FYYh9YkIJm3AUwaASr1AzbKh1+zIaUegnGE95dIMBJjoFxQ09mncw6escK46WZs7bQq0axrT8nlBaFFQWVxDrNIgPyAWu/COs9+I+pryJfynNf6TfWrVPPqt1zE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=Jgbi1FVAFf2LBxnx2aF3YkPPACelnbGmdwf6VSnDHGImpyH1tsruv2EKZ7OaWs7SMAQxa02sm0lp34UJFNg7tYZBuBziEoUjiSC7sPegJGx412tKEnf/3kaJU1xUmt9HRjyGrO8QGGCWZDBNmOEPSQtkgH1Dkz4JPCzyIs2Z9vU= Received: by 10.82.191.3 with SMTP id o3mr2797451buf.1179997494148; Thu, 24 May 2007 02:04:54 -0700 (PDT) Received: from ?172.31.5.25? ( [89.97.252.178]) by mx.google.com with ESMTP id z40sm1687480ikz.2007.05.24.02.04.52; Thu, 24 May 2007 02:04:53 -0700 (PDT) Message-ID: <4655C67A.9060000@FreeBSD.org> Date: Thu, 24 May 2007 19:08:10 +0200 From: Attilio Rao User-Agent: Thunderbird 1.5 (X11/20060526) MIME-Version: 1.0 To: arch@freebsd.org References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> <20070522162819.N5249@besplex.bde.org> <20070522201336.C87981@besplex.bde.org> <46533CAD.8030104@FreeBSD.org> In-Reply-To: <46533CAD.8030104@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: Attilio Rao Cc: alc@freebsd.org, Jeff Roberson , Bruce Evans Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 09:04:58 -0000 Attilio Rao wrote: > Bruce Evans wrote: >> 4 more translation errors breaking 8 counters altogether (v_vnodepgsin >> is broken twice): > > Thanks a lot for the revision, there will be a pending patch in the next > hour. Hello, Let me know if this patch is right for you and if you have feedbacks, comments, etc: http://users.gufi.org/~rookie/works/patches/schedlock/vmmeter3.diff This should fix translation errors Bruce has found and switching the _SET() method in order to being a simple assignment (as Bruce has suggested). Thanks, Attilio From owner-freebsd-arch@FreeBSD.ORG Thu May 24 12:31:49 2007 Return-Path: X-Original-To: freebsd-arch@FreeBSD.ORG Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C1FFF16A468; Thu, 24 May 2007 12:31:49 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id 2CF8C13C457; Thu, 24 May 2007 12:31:49 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l4OCVm22002306; Thu, 24 May 2007 22:31:48 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l4OCVmqn002305; Thu, 24 May 2007 22:31:48 +1000 (EST) (envelope-from peter) Date: Thu, 24 May 2007 22:31:48 +1000 From: Peter Jeremy To: Colin Percival , "freebsd-arch@freebsd.org" Message-ID: <20070524123148.GC1160@turion.vk2pj.dyndns.org> References: <46546E16.9070707@freebsd.org> <20070523212325.GA3022@VARK.MIT.EDU> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zCKi3GIZzVBPywwA" Content-Disposition: inline In-Reply-To: <20070523212325.GA3022@VARK.MIT.EDU> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.15 (2007-04-06) Cc: Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 12:31:49 -0000 --zCKi3GIZzVBPywwA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-May-23 17:23:25 -0400, David Schultz wrote: >On Wed, May 23, 2007, Colin Percival wrote: >> Can anyone make a strong argument for keeping this code in the base syst= em? > >Removing it from the base system would merely amount to a >marketing ploy, wherein we get to say that FreeBSD has fewer >security holes because file(1) is a "third-party package". Doing >so wouldn't make FreeBSD installations any more secure in >practice. My thoughts as well. The way I see it, file(1) is an interpreter for the language defined in magic(5). For most purposes (particularly when processing untrusted input), the "program" that file(1) will execute is /usr/share/misc/magic Viewed this way, I do not see it as any different to awk or sed. =46rom a security aspect, file(1) can extract C-style strings and offsets from the untrusted input - and these obviously need careful sanity checks in addition to the normal error checking. Rather than treating ports as a ghetto for potentially unsafe utilities, I believe the Project would be better off making those utilities more robust. Has the OpenBSD project got an 'audited' file(1)? If so, can we import it or the fixes? --=20 Peter Jeremy --zCKi3GIZzVBPywwA Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGVYW0/opHv/APuIcRAqJPAJ9MkMaaA0FG8PTJ6W9nK3m00KCrwACdFRc4 yEfxXMi0wmBaij6wusS3eqA= =ZRq1 -----END PGP SIGNATURE----- --zCKi3GIZzVBPywwA-- From owner-freebsd-arch@FreeBSD.ORG Thu May 24 13:26:06 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1C38B16A469 for ; Thu, 24 May 2007 13:26:06 +0000 (UTC) (envelope-from alc@cs.rice.edu) Received: from mail.cs.rice.edu (mail.cs.rice.edu [128.42.1.31]) by mx1.freebsd.org (Postfix) with ESMTP id E3DC713C457 for ; Thu, 24 May 2007 13:26:05 +0000 (UTC) (envelope-from alc@cs.rice.edu) Received: from mail.cs.rice.edu (localhost.localdomain [127.0.0.1]) by mail.cs.rice.edu (Postfix) with ESMTP id 31C9A2C2A94; Thu, 24 May 2007 08:08:28 -0500 (CDT) X-Virus-Scanned: by amavis-2.4.0 at mail.cs.rice.edu Received: from mail.cs.rice.edu ([127.0.0.1]) by mail.cs.rice.edu (mail.cs.rice.edu [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 29BjG3PL4-mK; Thu, 24 May 2007 08:08:20 -0500 (CDT) Received: from [216.63.78.18] (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.cs.rice.edu (Postfix) with ESMTP id 5D1762C2A92; Thu, 24 May 2007 08:08:20 -0500 (CDT) Message-ID: <46558E43.8040608@cs.rice.edu> Date: Thu, 24 May 2007 08:08:19 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.13) Gecko/20070328 X-Accept-Language: en-us, en MIME-Version: 1.0 To: attilio@FreeBSD.org References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> <20070522162819.N5249@besplex.bde.org> <20070522201336.C87981@besplex.bde.org> <46533CAD.8030104@FreeBSD.org> <4655C67A.9060000@FreeBSD.org> In-Reply-To: <4655C67A.9060000@FreeBSD.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: alc@freebsd.org, arch@freebsd.org, Jeff Roberson , Bruce Evans Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 13:26:06 -0000 Attilio Rao wrote: > Attilio Rao wrote: > >> Bruce Evans wrote: >> >>> 4 more translation errors breaking 8 counters altogether (v_vnodepgsin >>> is broken twice): >> >> >> Thanks a lot for the revision, there will be a pending patch in the >> next hour. > > > Hello, > Let me know if this patch is right for you and if you have feedbacks, > comments, etc: > http://users.gufi.org/~rookie/works/patches/schedlock/vmmeter3.diff > > This should fix translation errors Bruce has found and switching the > _SET() method in order to being a simple assignment (as Bruce has > suggested). Let me offer a simple rule of thumb for VMCNT_ADD() vs. PCPU_LAZY_INC(): If the field is NOT under the section labeled "Distribution of page usages." in vmmeter, then PCPU_LAZY_INC() is preferable to VMCNT_ADD() implemented with an atomic op. Alan From owner-freebsd-arch@FreeBSD.ORG Thu May 24 14:30:31 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4CEF216A41F for ; Thu, 24 May 2007 14:30:31 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.169]) by mx1.freebsd.org (Postfix) with ESMTP id 9BDF313C458 for ; Thu, 24 May 2007 14:30:30 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by ug-out-1314.google.com with SMTP id u2so53131uge for ; Thu, 24 May 2007 07:30:29 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=JII7lSrENS/GB+BPTS93jO3jehHoMO3b9BQli98LnTIW9NTpZFEScYGSOZ0Vo33DLbKej9Y8yS4qU07yshOhdTtjZq77fppztztGRdQlRYJXI7ObrelDMXUelyEB/3s/DdbL+ibCGZYkQNx3f7JXXJnylRWKWGT5DBcLlg04F+E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=s/yjfdTKAy3pooprUJXNmFna/+XDMh+ObHx/0FU/vRDP+R+viynHRhFN9k30R/jjGsvv9izKRZEPGkZJ66E6N43+26Maua271n0Alg1q4CcRw71CTK8Koc+slCg8Qvaceo1rdE5uXYzH3axnRhOA6yZl3I1Q4Y/cntLmKGicd2I= Received: by 10.82.191.3 with SMTP id o3mr3390477buf.1180017027232; Thu, 24 May 2007 07:30:27 -0700 (PDT) Received: from ?172.31.5.25? ( [89.97.252.178]) by mx.google.com with ESMTP id b33sm2584703ika.2007.05.24.07.30.24; Thu, 24 May 2007 07:30:26 -0700 (PDT) Message-ID: <465612C4.3040400@FreeBSD.org> Date: Fri, 25 May 2007 00:33:40 +0200 From: Attilio Rao User-Agent: Thunderbird 1.5 (X11/20060526) MIME-Version: 1.0 To: Alan Cox References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> <20070522162819.N5249@besplex.bde.org> <20070522201336.C87981@besplex.bde.org> <46533CAD.8030104@FreeBSD.org> <4655C67A.9060000@FreeBSD.org> <46558E43.8040608@cs.rice.edu> In-Reply-To: <46558E43.8040608@cs.rice.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: Attilio Rao Cc: alc@freebsd.org, arch@freebsd.org, Jeff Roberson , Bruce Evans Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 14:30:31 -0000 Alan Cox wrote: > Attilio Rao wrote: > >> Attilio Rao wrote: >> >>> Bruce Evans wrote: >>> >>>> 4 more translation errors breaking 8 counters altogether (v_vnodepgsin >>>> is broken twice): >>> >>> >>> Thanks a lot for the revision, there will be a pending patch in the >>> next hour. >> >> >> Hello, >> Let me know if this patch is right for you and if you have feedbacks, >> comments, etc: >> http://users.gufi.org/~rookie/works/patches/schedlock/vmmeter3.diff >> >> This should fix translation errors Bruce has found and switching the >> _SET() method in order to being a simple assignment (as Bruce has >> suggested). > > > Let me offer a simple rule of thumb for VMCNT_ADD() vs. PCPU_LAZY_INC(): > If the field is NOT under the section labeled "Distribution of page > usages." in vmmeter, then PCPU_LAZY_INC() is preferable to VMCNT_ADD() > implemented with an atomic op. Ok, I've updated the patch following your suggestion. I just left out that vmmeter fields which needs to be incremented not by one but by another value (since PCPU_LAZY_INC() just increments by 1). Do you think it is more appropriate to expand the PCPU_LAZY_*() interface and let it cover increments not by 1 too? It would let grow the patch notably since we need to touch all architectures for that however... Thanks, Attilio From owner-freebsd-arch@FreeBSD.ORG Thu May 24 14:59:23 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D417816A400; Thu, 24 May 2007 14:59:23 +0000 (UTC) (envelope-from ache@nagual.pp.ru) Received: from nagual.pp.ru (nagual.pp.ru [194.87.13.69]) by mx1.freebsd.org (Postfix) with ESMTP id 4C79A13C489; Thu, 24 May 2007 14:59:22 +0000 (UTC) (envelope-from ache@nagual.pp.ru) Received: from nagual.pp.ru (ache@localhost [127.0.0.1]) by nagual.pp.ru (8.14.1/8.14.1) with ESMTP id l4OExLwr031747; Thu, 24 May 2007 18:59:21 +0400 (MSD) (envelope-from ache@nagual.pp.ru) Received: (from ache@localhost) by nagual.pp.ru (8.14.1/8.14.1/Submit) id l4OExLDD031746; Thu, 24 May 2007 18:59:21 +0400 (MSD) (envelope-from ache) Date: Thu, 24 May 2007 18:59:20 +0400 From: Andrey Chernov To: Daniel Eischen Message-ID: <20070524145920.GB31367@nagual.pp.ru> Mail-Followup-To: Andrey Chernov , Daniel Eischen , Colin Percival , "freebsd-arch@freebsd.org" References: <46546E16.9070707@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.15 (2007-04-06) Cc: Colin Percival , "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 14:59:23 -0000 On Wed, May 23, 2007 at 01:49:13PM -0400, Daniel Eischen wrote: > On Wed, 23 May 2007, Colin Percival wrote: > > > FreeBSD architects and file(1) maintainer, > > > > I'd like to remove file(1) and libmagic(3) from the FreeBSD base system > > for the following reasons: > > 1. I don't see it as being a necessary component of a UNIX-like operating > > system. > > 2. It's available in the ports tree. > > 3. Due to its nature as a program which parses multiple data formats, it > > poses an unusually high risk of having security problems in the future > > (cf. ethereal/wireshark). > > > > The one redeeming feature of file/libmagic as far as security is concerned > > is that it doesn't act as a daemon, i.e., other code or user intervention > > is required for an attacker to exploit security issues. This is why I'm > > asking here rather than wielding the "Security Officer can veto code which > > he doesn't like" stick. :-) > > > > Can anyone make a strong argument for keeping this code in the base system? > > Yes, because other OS's have it (file) in their base, and because > it is a POSIX-defined utility. Please consider this a strong no. I agree with Daniel. Better way is to add a big warning to file(1) manpage about running it against untrusted sources. -- http://ache.pp.ru/ From owner-freebsd-arch@FreeBSD.ORG Thu May 24 15:59:38 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 30C4E16A41F; Thu, 24 May 2007 15:59:38 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.NUXI.org (trang.nuxi.org [64.81.59.225]) by mx1.freebsd.org (Postfix) with ESMTP id E74F313C483; Thu, 24 May 2007 15:59:37 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.NUXI.org (obrien@localhost [127.0.0.1]) by dragon.NUXI.org (8.13.8/8.13.8) with ESMTP id l4OFxaGc073488; Thu, 24 May 2007 08:59:37 -0700 (PDT) (envelope-from obrien@dragon.NUXI.org) Received: (from obrien@localhost) by dragon.NUXI.org (8.14.1/8.13.7/Submit) id l4OFxaI2073487; Thu, 24 May 2007 08:59:36 -0700 (PDT) (envelope-from obrien) Date: Thu, 24 May 2007 08:59:36 -0700 From: "David O'Brien" To: Colin Percival Message-ID: <20070524155936.GA41522@dragon.NUXI.org> Mail-Followup-To: obrien@nuxi.org, Colin Percival , "freebsd-arch@freebsd.org" References: <46546E16.9070707@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46546E16.9070707@freebsd.org> X-Operating-System: FreeBSD 7.0-CURRENT Organization: The NUXI BSD Group X-Pgp-Rsa-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Rsa-Keyid: 1024/34F9F9D5 User-Agent: Mutt/1.5.11 Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: obrien@NUXI.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 15:59:38 -0000 On Wed, May 23, 2007 at 09:38:46AM -0700, Colin Percival wrote: > FreeBSD architects and file(1) maintainer, Actually I was in progress of updating file(1)/libmagic(3) in -CURRENT yesterday morning (before seeing any of this thread). I delayed due to needing to get my system updated to GCC 4.2 in the base + library symbolizing. Based on the responses to this thread I'm going to go ahead with the import. If file(1)/libmagic(3) gets later removed so be it. The Attic will be more up to date. :-) I'm not working on it due to trying to start a fight - but rather there seems to be sufficient support to go ahead with the upgrade. -- -- David (obrien@NUXI.org) NOT Sent via my Crackberry Handheld From owner-freebsd-arch@FreeBSD.ORG Thu May 24 17:11:08 2007 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C244716A400 for ; Thu, 24 May 2007 17:11:08 +0000 (UTC) (envelope-from me@johnea.net) Received: from mail.johnea.net (johnea.net [70.167.123.7]) by mx1.freebsd.org (Postfix) with ESMTP id AE8E313C46E for ; Thu, 24 May 2007 17:11:08 +0000 (UTC) (envelope-from me@johnea.net) Received: from [192.168.100.211] (unknown [192.168.100.211]) by mail.johnea.net (Postfix) with ESMTP id 9B91D7B1 for ; Thu, 24 May 2007 09:51:31 -0700 (PDT) Message-ID: <4655C238.7000809@johnea.net> Date: Thu, 24 May 2007 09:50:00 -0700 From: johnea User-Agent: Mozilla Thunderbird 1.0.7 (X11/20051002) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-arch@FreeBSD.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: i386? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 17:11:08 -0000 Hi, Don't know if this will make it through your spam filter, or if you'll have the bandwidth to respond, but here goes. Why doesn't the FreeBSD project support a 32bit intel architecture newer than 386? Pretty much every PC in operation today is at least 686/athalon. Wouldn't the architechtural improvements to the x86 architecture over the last 10 years provide performance improvements to FreeBDSD? Just wondering, couldn't really find a forum to post such a question to. Thank you for reading my email! And thank you for your hard work on the FreeBSD project! And if you have just a minute to craft an answer, Thank you for your reply! ciao... johnea From owner-freebsd-arch@FreeBSD.ORG Thu May 24 17:19:32 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F3B5E16A46D for ; Thu, 24 May 2007 17:19:31 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (grnl-static-02-0046.dsl.iowatelecom.net [69.66.56.110]) by mx1.freebsd.org (Postfix) with ESMTP id AFBFF13C455 for ; Thu, 24 May 2007 17:19:31 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.13.8/8.13.8) with ESMTP id l4OHJUdd000533; Thu, 24 May 2007 12:19:30 -0500 (CDT) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.13.8/8.13.8/Submit) id l4OHJUni000532; Thu, 24 May 2007 12:19:30 -0500 (CDT) (envelope-from brooks) Date: Thu, 24 May 2007 12:19:30 -0500 From: Brooks Davis To: johnea Message-ID: <20070524171930.GA348@lor.one-eyed-alien.net> References: <4655C238.7000809@johnea.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="sdtB3X0nJg68CQEu" Content-Disposition: inline In-Reply-To: <4655C238.7000809@johnea.net> User-Agent: Mutt/1.5.15 (2007-04-06) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (lor.one-eyed-alien.net [127.0.0.1]); Thu, 24 May 2007 12:19:30 -0500 (CDT) Cc: freebsd-arch@freebsd.org Subject: Re: i386? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 17:19:32 -0000 --sdtB3X0nJg68CQEu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 24, 2007 at 09:50:00AM -0700, johnea wrote: >=20 > Hi, >=20 > Don't know if this will make it through your spam filter, or if you'll h= ave=20 > the bandwidth to respond, but here goes. >=20 > Why doesn't the FreeBSD project support a 32bit intel architecture newer= =20 > than 386? FreeBSD does support all x86 platforms newer than the 80386. We do not support the 80386 at all. For historical reasons we call the platform i386. Changing the name would be difficult and largely pointless i386. > Pretty much every PC in operation today is at least 686/athalon. In the desktop and server market, mostly. In the embedded and thin-client market, no. There are plenty of 486 based platforms today that are critical support including the lower end of the Soekris line. > Wouldn't the architechtural improvements to the x86 architecture over th= e=20 > last 10 years provide performance improvements to FreeBDSD? They do and we take advantage of them. We just don't gratuitously exclude the possibility of running on older, but still viable platforms. > Just wondering, couldn't really find a forum to post such a question to. You didn't try hard enough. This thread comes up several times year. -- Brooks --sdtB3X0nJg68CQEu Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFGVckhXY6L6fI4GtQRAnC4AKCtpupMcxgXdj06fuH2z7s5nHZ9hACghWVw IgzTw3/Z+WJyLFtB+bllvVw= =qMkn -----END PGP SIGNATURE----- --sdtB3X0nJg68CQEu-- From owner-freebsd-arch@FreeBSD.ORG Thu May 24 17:48:06 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E014616A469 for ; Thu, 24 May 2007 17:48:06 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.236]) by mx1.freebsd.org (Postfix) with ESMTP id 9CBFB13C457 for ; Thu, 24 May 2007 17:48:06 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: by wr-out-0506.google.com with SMTP id 70so167725wra for ; Thu, 24 May 2007 10:48:06 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=IyG/8deXDjbRzz5cY46lSOny38d4+hwP3lDFc09NBdwoMfXlXNarJ2altuu+bJ5TMaZheneBtp2UeHtT5WRP3NDNj2xDqrZJWARDhYcYTLSDUZCYPyFqQxGwOT2NhsJNg0EeOjA2sJ5PDUOzuM1R3fZrQKXhYgQVPbn3/BGqbuI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=kvMejXUYb16IKnF76L/wLRET6cBjsfEdUfRwIcv2yD2C9NQp1f516LadvYEUGm/6TxVNDQ6TvrQ2tHonYlBILqWeu2wDiniGTqMl/KZtHcW0bhikLjOCMEZo+VHKBRcGrMEO4GaBA3aLITCLXlAFqNIGIXQ7qt3ehTp0HpSuCHw= Received: by 10.78.131.8 with SMTP id e8mr616997hud.1180027154209; Thu, 24 May 2007 10:19:14 -0700 (PDT) Received: by 10.78.107.13 with HTTP; Thu, 24 May 2007 10:19:14 -0700 (PDT) Message-ID: Date: Thu, 24 May 2007 10:19:14 -0700 From: "Kip Macy" To: johnea , freebsd-arch@freebsd.org In-Reply-To: <4655C238.7000809@johnea.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4655C238.7000809@johnea.net> Cc: Subject: Re: i386? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 17:48:07 -0000 A) this is not an appropriate forum, see -hackers B) this misconception has been brought up before please see the archives The GENERIC kernel needs to be a one size fits all ... On 5/24/07, johnea wrote: > > Hi, > > Don't know if this will make it through your spam filter, or if you'll have > the bandwidth to respond, but here goes. > > Why doesn't the FreeBSD project support a 32bit intel architecture newer > than 386? > > Pretty much every PC in operation today is at least 686/athalon. > > Wouldn't the architechtural improvements to the x86 architecture over the > last 10 years provide performance improvements to FreeBDSD? > > Just wondering, couldn't really find a forum to post such a question to. > > Thank you for reading my email! And thank you for your hard work on the > FreeBSD project! > > And if you have just a minute to craft an answer, Thank you for your reply! > > ciao... > > johnea > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Thu May 24 18:06:07 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E813F16A400; Thu, 24 May 2007 18:06:07 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.NUXI.org (trang.nuxi.org [64.81.59.225]) by mx1.freebsd.org (Postfix) with ESMTP id B5B4013C487; Thu, 24 May 2007 18:06:07 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.NUXI.org (obrien@localhost [127.0.0.1]) by dragon.NUXI.org (8.13.8/8.13.8) with ESMTP id l4OI67N8076858; Thu, 24 May 2007 11:06:07 -0700 (PDT) (envelope-from obrien@dragon.NUXI.org) Received: (from obrien@localhost) by dragon.NUXI.org (8.14.1/8.13.7/Submit) id l4OI67jB076857; Thu, 24 May 2007 11:06:07 -0700 (PDT) (envelope-from obrien) Date: Thu, 24 May 2007 11:06:07 -0700 From: "David O'Brien" To: Colin Percival Message-ID: <20070524180607.GA76638@dragon.NUXI.org> Mail-Followup-To: obrien@freebsd.org, Colin Percival , "freebsd-arch@freebsd.org" References: <46546E16.9070707@freebsd.org> <20070524062155.GA39887@dragon.NUXI.org> <46553D5A.9090304@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46553D5A.9090304@freebsd.org> X-Operating-System: FreeBSD 7.0-CURRENT Organization: The NUXI BSD Group X-Pgp-Rsa-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Rsa-Keyid: 1024/34F9F9D5 User-Agent: Mutt/1.5.11 Cc: "freebsd-arch@freebsd.org" Subject: Re: RFC: Removing file(1)+libmagic(3) from the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: obrien@freebsd.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 18:06:08 -0000 On Thu, May 24, 2007 at 12:23:06AM -0700, Colin Percival wrote: > David, could you please import file-4.21 to the vendor branch once it is > released? Will do - I just got the release announcement from Christos. -- -- David (obrien@FreeBSD.org) Q: Because it reverses the logical flow of conversation. A: Why is top-posting (putting a reply at the top of the message) frowned upon? Let's not play "Jeopardy-style quoting" From owner-freebsd-arch@FreeBSD.ORG Thu May 24 18:08:16 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1C90A16A4CC; Thu, 24 May 2007 18:08:16 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.freebsd.org (Postfix) with ESMTP id 76AE313C45E; Thu, 24 May 2007 18:08:15 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.8/8.13.8) with ESMTP id l4OI82Ln029981; Thu, 24 May 2007 14:08:02 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-arch@freebsd.org Date: Thu, 24 May 2007 11:13:56 -0400 User-Agent: KMail/1.9.6 References: <20070520155103.K632@10.0.0.1> <38601004-BB95-4B8B-87A6-26E2D52B89BA@mac.com> <20070523170449.L9443@10.0.0.1> In-Reply-To: <20070523170449.L9443@10.0.0.1> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200705241113.57372.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Thu, 24 May 2007 14:08:02 -0400 (EDT) X-Virus-Scanned: ClamAV 0.88.3/3290/Thu May 24 07:22:02 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Marcel Moolenaar , arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 18:08:16 -0000 On Wednesday 23 May 2007 08:11:15 pm Jeff Roberson wrote: > On Wed, 23 May 2007, Marcel Moolenaar wrote: > > > > >>> The old patch was missing PowerPC & ia64. Will the final version > >>> include those as well? > >> > >> There are a couple of uses of the global scheduler lock in some > >> architecture specific locations. They will continue to be safe with the > >> 4BSD scheduler. I intended to work on these issues with the architecture > >> maintainers after the threadlock patch goes in. Can you suggest some > >> alternative to sched_lock for pmap_switch in ia64? > > > > pmap_switch() is called from cpu_switch() and from pmap_install(). > > So, currently, pmap_install() grabs sched_lock to mimic the > > cpu_switch() path and we assert having sched_lock in pmap_switch(). > > Basically, any lock that serializes cpu_switch() would work, because > > we don't want to switch the thread while in the middle of setting up > > the region registers. > > We could simply use thread_lock() now if this serialization only applies > to preventing multiple access to the same thread. You have to have a spin lock with membar's across cpu_switch() so that curthread-private data is consistent after a thread migrates CPUs. You only really need it for migrations if all your migrations are explicit, in which case a per-CPU lock for a migration queue where both CPUs (old first, then new second) lock the lock during the handoff. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu May 24 18:08:16 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1C90A16A4CC; Thu, 24 May 2007 18:08:16 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.freebsd.org (Postfix) with ESMTP id 76AE313C45E; Thu, 24 May 2007 18:08:15 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.8/8.13.8) with ESMTP id l4OI82Ln029981; Thu, 24 May 2007 14:08:02 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-arch@freebsd.org Date: Thu, 24 May 2007 11:13:56 -0400 User-Agent: KMail/1.9.6 References: <20070520155103.K632@10.0.0.1> <38601004-BB95-4B8B-87A6-26E2D52B89BA@mac.com> <20070523170449.L9443@10.0.0.1> In-Reply-To: <20070523170449.L9443@10.0.0.1> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200705241113.57372.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Thu, 24 May 2007 14:08:02 -0400 (EDT) X-Virus-Scanned: ClamAV 0.88.3/3290/Thu May 24 07:22:02 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Marcel Moolenaar , arch@freebsd.org Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2007 18:08:16 -0000 On Wednesday 23 May 2007 08:11:15 pm Jeff Roberson wrote: > On Wed, 23 May 2007, Marcel Moolenaar wrote: > > > > >>> The old patch was missing PowerPC & ia64. Will the final version > >>> include those as well? > >> > >> There are a couple of uses of the global scheduler lock in some > >> architecture specific locations. They will continue to be safe with the > >> 4BSD scheduler. I intended to work on these issues with the architecture > >> maintainers after the threadlock patch goes in. Can you suggest some > >> alternative to sched_lock for pmap_switch in ia64? > > > > pmap_switch() is called from cpu_switch() and from pmap_install(). > > So, currently, pmap_install() grabs sched_lock to mimic the > > cpu_switch() path and we assert having sched_lock in pmap_switch(). > > Basically, any lock that serializes cpu_switch() would work, because > > we don't want to switch the thread while in the middle of setting up > > the region registers. > > We could simply use thread_lock() now if this serialization only applies > to preventing multiple access to the same thread. You have to have a spin lock with membar's across cpu_switch() so that curthread-private data is consistent after a thread migrates CPUs. You only really need it for migrations if all your migrations are explicit, in which case a per-CPU lock for a migration queue where both CPUs (old first, then new second) lock the lock during the handoff. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Fri May 25 00:27:03 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 299B816A468; Fri, 25 May 2007 00:27:03 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx02.syd.optusnet.com.au (fallbackmx02.syd.optusnet.com.au [211.29.133.72]) by mx1.freebsd.org (Postfix) with ESMTP id B304313C45D; Fri, 25 May 2007 00:27:02 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail04.syd.optusnet.com.au (mail04.syd.optusnet.com.au [211.29.132.185]) by fallbackmx02.syd.optusnet.com.au (8.12.11.20060308/8.12.11) with ESMTP id l4OBR3nr010586; Thu, 24 May 2007 21:27:03 +1000 Received: from besplex.bde.org (c211-30-216-190.carlnfd3.nsw.optusnet.com.au [211.30.216.190]) by mail04.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id l4OBQurY010787 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 24 May 2007 21:27:01 +1000 Date: Thu, 24 May 2007 21:26:57 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Attilio Rao In-Reply-To: <4655C67A.9060000@FreeBSD.org> Message-ID: <20070524210030.J36089@besplex.bde.org> References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> <20070522162819.N5249@besplex.bde.org> <20070522201336.C87981@besplex.bde.org> <46533CAD.8030104@FreeBSD.org> <4655C67A.9060000@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: alc@freebsd.org, arch@freebsd.org, Jeff Roberson Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 May 2007 00:27:03 -0000 On Thu, 24 May 2007, Attilio Rao wrote: > Hello, > Let me know if this patch is right for you and if you have feedbacks, > comments, etc: > http://users.gufi.org/~rookie/works/patches/schedlock/vmmeter3.diff > > This should fix translation errors Bruce has found and switching the _SET() > method in order to being a simple assignment (as Bruce has suggested). OK. Note that I won't be really happy until the macros and volatiles go away. But keep at least the macros for development. The volatiles are easier to remove: put them in the access macros instead of putting them in the declarations and cancelling them in the access macros where they are most needed. This would leave them broken in the access macros where they are most needed, the same as now (since the sysctl macros just don't support volatile, they must be cancelled now and cannot be added). Note that we use this method for almost all "volatile" variables in the kernel -- we don't declare quite enough things volatile, but we add volatiles in the atomic function calls and depend on a combination of atomic accesses and locking to avoid races. It is easy to get this wrong, but if we get it wrong then declaring things volatile only reduces the races. I forgot to mention another easy-to-fix bug: missing parentheses around the `val' arg in macro definitions. Bruce From owner-freebsd-arch@FreeBSD.ORG Fri May 25 06:57:11 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3913B16A421; Fri, 25 May 2007 06:57:11 +0000 (UTC) (envelope-from alc@cs.rice.edu) Received: from mail.cs.rice.edu (mail.cs.rice.edu [128.42.1.31]) by mx1.freebsd.org (Postfix) with ESMTP id 0EEF213C4C7; Fri, 25 May 2007 06:57:11 +0000 (UTC) (envelope-from alc@cs.rice.edu) Received: from mail.cs.rice.edu (localhost.localdomain [127.0.0.1]) by mail.cs.rice.edu (Postfix) with ESMTP id 818FC2C2A66; Fri, 25 May 2007 01:57:10 -0500 (CDT) X-Virus-Scanned: by amavis-2.4.0 at mail.cs.rice.edu Received: from mail.cs.rice.edu ([127.0.0.1]) by mail.cs.rice.edu (mail.cs.rice.edu [127.0.0.1]) (amavisd-new, port 10024) with LMTP id AHvi+OXkzc14; Fri, 25 May 2007 01:57:02 -0500 (CDT) Received: from [216.63.78.18] (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.cs.rice.edu (Postfix) with ESMTP id AAD6A2C2A65; Fri, 25 May 2007 01:57:02 -0500 (CDT) Message-ID: <465688BC.90308@cs.rice.edu> Date: Fri, 25 May 2007 01:57:00 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.13) Gecko/20070328 X-Accept-Language: en-us, en MIME-Version: 1.0 To: attilio@FreeBSD.org References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> <20070522162819.N5249@besplex.bde.org> <20070522201336.C87981@besplex.bde.org> <46533CAD.8030104@FreeBSD.org> <4655C67A.9060000@FreeBSD.org> <46558E43.8040608@cs.rice.edu> <465612C4.3040400@FreeBSD.org> In-Reply-To: <465612C4.3040400@FreeBSD.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: alc@freebsd.org, arch@freebsd.org, Jeff Roberson , Bruce Evans Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 May 2007 06:57:11 -0000 Attilio Rao wrote: > > Ok, I've updated the patch following your suggestion. > I just left out that vmmeter fields which needs to be incremented not > by one but by another value (since PCPU_LAZY_INC() just increments by 1). > > Do you think it is more appropriate to expand the PCPU_LAZY_*() > interface and let it cover increments not by 1 too? > Yes, but let's do that in a later patch, not this one. Alan From owner-freebsd-arch@FreeBSD.ORG Fri May 25 07:04:27 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A05ED16A468; Fri, 25 May 2007 07:04:27 +0000 (UTC) (envelope-from alc@cs.rice.edu) Received: from mail.cs.rice.edu (mail.cs.rice.edu [128.42.1.31]) by mx1.freebsd.org (Postfix) with ESMTP id 768F913C448; Fri, 25 May 2007 07:04:27 +0000 (UTC) (envelope-from alc@cs.rice.edu) Received: from mail.cs.rice.edu (localhost.localdomain [127.0.0.1]) by mail.cs.rice.edu (Postfix) with ESMTP id 2ECDF2C2A66; Fri, 25 May 2007 02:04:27 -0500 (CDT) X-Virus-Scanned: by amavis-2.4.0 at mail.cs.rice.edu Received: from mail.cs.rice.edu ([127.0.0.1]) by mail.cs.rice.edu (mail.cs.rice.edu [127.0.0.1]) (amavisd-new, port 10024) with LMTP id HCtgJYufctvJ; Fri, 25 May 2007 02:04:19 -0500 (CDT) Received: from [216.63.78.18] (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.cs.rice.edu (Postfix) with ESMTP id 531612C2A62; Fri, 25 May 2007 02:04:19 -0500 (CDT) Message-ID: <46568A72.3020406@cs.rice.edu> Date: Fri, 25 May 2007 02:04:18 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.13) Gecko/20070328 X-Accept-Language: en-us, en MIME-Version: 1.0 To: attilio@FreeBSD.org References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> <20070522162819.N5249@besplex.bde.org> <20070522201336.C87981@besplex.bde.org> <46533CAD.8030104@FreeBSD.org> <4655C67A.9060000@FreeBSD.org> In-Reply-To: <4655C67A.9060000@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: alc@freebsd.org, arch@freebsd.org, Jeff Roberson , Bruce Evans Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 May 2007 07:04:27 -0000 Attilio Rao wrote: > Attilio Rao wrote: > >> Bruce Evans wrote: >> >>> 4 more translation errors breaking 8 counters altogether (v_vnodepgsin >>> is broken twice): >> >> >> Thanks a lot for the revision, there will be a pending patch in the >> next hour. > > > Hello, > Let me know if this patch is right for you and if you have feedbacks, > comments, etc: > http://users.gufi.org/~rookie/works/patches/schedlock/vmmeter3.diff > > This should fix translation errors Bruce has found and switching the > _SET() method in order to being a simple assignment (as Bruce has > suggested). I think there are still translation errors in vnode_pager.c: This increment by 1 of cnt.v_vnodepgsin is incorrect: @@ -909,8 +909,8 @@ bp->b_runningbufspace = bp->b_bufsize; atomic_add_int(&runningbufspace, bp->b_runningbufspace); - VMCNT_ADD(vnodein, 1); - VMCNT_ADD(vnodepgsin, 1); + PCPU_LAZY_INC(cnt.v_vnodein); + PCPU_LAZY_INC(cnt.v_vnodepgsin); /* do the input */ bp->b_iooffset = dbtob(bp->b_blkno); Please double-check the others. Alan From owner-freebsd-arch@FreeBSD.ORG Sat May 26 00:03:15 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4106516A477 for ; Sat, 26 May 2007 00:03:15 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.171]) by mx1.freebsd.org (Postfix) with ESMTP id 599EB13C43E for ; Sat, 26 May 2007 00:03:11 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by ug-out-1314.google.com with SMTP id u2so435280uge for ; Fri, 25 May 2007 17:03:10 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=cGzmcfFVpTSw1Ej8wx4szYJvAl10YFFSoEa+NtZupUsbLdTqy+UT5jj+VPpEbT9o9ZR+y13khataFNiRlJ7ipNOV/3OmNOAFrpAjhAnYizkF7eHzvvFp19eeFcJfRkllSu74/Liq9gtaZSAEslVnqaAVLVAzgh/9yVkM3XT8Wz8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding:sender; b=WCNbHPvh462le5ERuZkLPjdKJ6QwqUB90VHmqen2LbpUNxgYWH20iTWqX3oKiHbyK7wDb0q3TWAikNjw03tbKm3o+Ynyq07K5Y68Xhi7Uf8lSDdouCczkR/MLI034IfgBxMu4Wmjcd5tD+tetBl7EH3Dug+LJeafCTuOcQiobE0= Received: by 10.67.97.18 with SMTP id z18mr3269247ugl.1180137790314; Fri, 25 May 2007 17:03:10 -0700 (PDT) Received: from ?151.75.248.62? ( [151.75.248.62]) by mx.google.com with ESMTP id y1sm1465987uge.2007.05.25.17.03.08; Fri, 25 May 2007 17:03:09 -0700 (PDT) Message-ID: <46577936.9060804@FreeBSD.org> Date: Sat, 26 May 2007 02:03:02 +0200 From: Attilio Rao User-Agent: Thunderbird 1.5 (X11/20060526) MIME-Version: 1.0 To: Alan Cox References: <20070520155103.K632@10.0.0.1> <20070521113648.F86217@besplex.bde.org> <20070520213132.K632@10.0.0.1> <4651CAB8.8070007@FreeBSD.org> <4651CE2F.8080908@FreeBSD.org> <20070521022847.D679@10.0.0.1> <20070521195811.G56785@delplex.bde.org> <4651FCB5.7070604@FreeBSD.org> <20070521225032.C57233@delplex.bde.org> <20070522162819.N5249@besplex.bde.org> <20070522201336.C87981@besplex.bde.org> <46533CAD.8030104@FreeBSD.org> <4655C67A.9060000@FreeBSD.org> <46568A72.3020406@cs.rice.edu> In-Reply-To: <46568A72.3020406@cs.rice.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: Attilio Rao Cc: alc@freebsd.org, arch@freebsd.org, Jeff Roberson , Bruce Evans Subject: Re: sched_lock && thread_lock() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 May 2007 00:03:15 -0000 Alan Cox wrote: > Attilio Rao wrote: > >> Attilio Rao wrote: >> >>> Bruce Evans wrote: >>> >>>> 4 more translation errors breaking 8 counters altogether (v_vnodepgsin >>>> is broken twice): >>> >>> >>> Thanks a lot for the revision, there will be a pending patch in the >>> next hour. >> >> >> Hello, >> Let me know if this patch is right for you and if you have feedbacks, >> comments, etc: >> http://users.gufi.org/~rookie/works/patches/schedlock/vmmeter3.diff >> >> This should fix translation errors Bruce has found and switching the >> _SET() method in order to being a simple assignment (as Bruce has >> suggested). > > > I think there are still translation errors in vnode_pager.c: > > This increment by 1 of cnt.v_vnodepgsin is incorrect: ok, following the last suggestions I updated the patch. The URL is above. Thanks, Attilio