From owner-freebsd-arch@FreeBSD.ORG Mon Jan 29 20:52:01 2007 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 67A0716A402 for ; Mon, 29 Jan 2007 20:52:01 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 9C3C013C4B2 for ; Mon, 29 Jan 2007 20:52:00 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 9CDD9487FE; Mon, 29 Jan 2007 21:20:30 +0100 (CET) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id E637045CD9; Mon, 29 Jan 2007 21:20:23 +0100 (CET) Date: Mon, 29 Jan 2007 21:19:38 +0100 From: Pawel Jakub Dawidek To: arch@FreeBSD.org Message-ID: <20070129201938.GF87767@garage.freebsd.pl> References: <20070128202917.5B67916A5A6@hub.freebsd.org> <45BD82D2.20301@root.org> <20070129175222.GA87767@garage.freebsd.pl> <45BE37DC.6080509@root.org> <20070129184522.GD87767@garage.freebsd.pl> <45BE46B7.8000406@samsco.org> <20070129193205.GE87767@garage.freebsd.pl> <20070129194158.N32458@fledge.watson.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="dWYAkE0V1FpFQHQ3" Content-Disposition: inline In-Reply-To: <20070129194158.N32458@fledge.watson.org> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: cvs-src@FreeBSD.org, Scott Long , src-committers@FreeBSD.org, cvs-all@FreeBSD.org, Nate Lawson Subject: Re: cvs commit: src/sys/geom/eli g_eli.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jan 2007 20:52:01 -0000 --dWYAkE0V1FpFQHQ3 Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 29, 2007 at 07:52:20PM +0000, Robert Watson wrote: >=20 > On Mon, 29 Jan 2007, Pawel Jakub Dawidek wrote: >=20 > >>Why? You're proposing yet another intrusive change to the kernel to ha= ndle yet another one-off requirement of your code. Why not do what I sugge= sted before with hooking=20 > >>the appropriate SYSINIT in your module? Or why not follow Robert's sugg= estion and implement a simple event mechanism so that any module can know w= hen a CPU has come=20 > >>online or offline. Heck, you probably don't even need to implement a n= ew mechanism, just hook the existing EVENTHANLER mechanism. That's what it= 's designed for!! > > > >I'm afraid Scott that your proposals are hacks. As a GEOM class I should= not use SYSINIT, EVENTHANDLER, etc. I shouldn't bother if CPUs are online = or not. All events I=20 > >need to implement a GEOM class I should receive from the infrastructure.= Also I shouldn't be called by the infrastructure when the system is not ye= t ready for my activity,=20 > >that's why I proposed to implement this functionality in the infrastruct= ure (ie. delay GEOM tasting machanism), that hack SYSINITs in every single = GEOM class that need to=20 > >bind to a CPU. >=20 > I guess I'm not sure I entirely agree. I think that we lack some importa= nt infrastructure, which we've been talking on and off for a dev summit or = two now, for handling=20 > the arrival and departure of CPU resources ("dynamic reconfiguration"). = While once this wasn't really an issue on PC hardware, it now is, with the = advent of hypervisors,=20 > virtualization, not to mention more multiprocessing, etc. We have quite = a few algorithms and data structures that assume that the set of CPUs is st= atic, and fail quite=20 > badly (i.e., memory leaks, work lost, etc) if a CPU were to stop scheduli= ng threads. Geli is not alone in wanting to know what and when CPUs are av= ailable for concurrent=20 > work, and like other pieces of code (UMA is the piece I have particular f= amiliarity with), finds our infrastructure lacking. I'm also not entirely = convinced I agree with=20 > you as by the same token that you might claim sysinits and event handlers= shouldn't be used by GEOM modules, perhaps kthreads should also not be use= d :-). Sysinits,=20 > eventhandlers, and kthreads are all ways for scheduling and dispatching w= ork. The infrastructure is also there to help, simplify the code and allow to avoid code duplications. I see no reason to start GEOM classes when the system is simply not ready. So instead of using yet another KPI in every GEOM class that would like to bind to CPU, I suggested to remove the code from geli and instruct GEOM to do it for all classes in one go. > So perhaps we need to start having the conversation about CPU events more= seriously now. What do you think of the idea of the following: two event = handlers, a CPU start=20 > event and a CPU stop event, which are guaranteed to run on each CPU as as= the CPU comes online, and just before the CPU goes offline. Kernel subsyst= ems could use these=20 > events to determine when CPU resources were arriving and departing in som= e serious sense (not just "busy") in order to initialize and tear down per-= CPU data structures,=20 > rebalance workloads, start or stop per-CPU works, etc. The example I hav= e in mind here is the network stack, which might reasonably wish to have pe= r-CPU netisr (worker)=20 > threads. When the set of CPUs changes, it would like to increase or decre= ase the number of workers -- having the same number of workers compressed d= own to a smaller number=20 > of CPUs by migration would be a disaster for performance. I fully agree that there should be a clean KPI for this. What you proposed if fine. Because of lack of such KPI geli has to handle HTT CPUs which are turned off by default in releases also by abusing scheduler internals. KPI you proposed would allow me to remove those hacks. And I'm really all for it. What you and Scott are missing is that when I implement a GEOM class, I'm using what is available to do my work. I'm not going to educate myself how schedulers work, implement nice and clean KPI to use it in my class. I'm not saying it wouldn't be great to be able to do so, but I don't have time for everything, unfortunately, and you guys should understand that very well. I had conversation with John (jhb@) on IRC when I asked him how can I skip CPUs that are turned off. He then mentioned that it should be handled by KPI you're proposing, but also mentioned that I should go with the solution I've now, because at this point there is nothing better than that. Anyway, I'd love to remove current hacks and use what you proposed, Robert. > How to handle the boot processor is an interesting question -- are we int= erested in configuring away the boot processor at run-time? If not, we pro= bably want to handle it=20 > as a special case via sysinit. If all CPUs are equal and any may go away= , then we might need to rework our notion of shutdown, and provide these sa= me events for the boot=20 > CPU (which does sound desirable so as not to end up with lots of special = casing in subsystems). Regardless, we are hardly the first OS to try to add= ress these issues via a=20 > clean architectural solution, and my thinking is we should do a bit of re= search. A first place to look would definitely be OpenSolaris. I'd prefer boot CPU to not be treated in any special way. If it has to be, it can be hidden from the subsystems, ie. by sending CPU-online event for the boot CPU, but never sending CPU-offline event or something like this. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --dWYAkE0V1FpFQHQ3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFFvlbaForvXbEpPzQRAp59AJwKUHqvYLo3o1+vdi85Ebv56B5F9QCgoGxX DwdHSiKwxUWBXOz9JZumOuw= =yN+t -----END PGP SIGNATURE----- --dWYAkE0V1FpFQHQ3-- From owner-freebsd-arch@FreeBSD.ORG Mon Jan 29 21:45:54 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3996A16A40A for ; Mon, 29 Jan 2007 21:45:54 +0000 (UTC) (envelope-from corsmith@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.232]) by mx1.freebsd.org (Postfix) with ESMTP id DAFA213C4C4 for ; Mon, 29 Jan 2007 21:45:53 +0000 (UTC) (envelope-from corsmith@gmail.com) Received: by wx-out-0506.google.com with SMTP id s18so1751650wxc for ; Mon, 29 Jan 2007 13:45:53 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=AVlzSFqh3OJ+CXERWyXnTeH34SodssBBu1thbB42HQDSSjHZ7Tz13gnCI7/WoypJ96nvI1OJozsovfK1psypPkHETCrKoRAc9mWTqrqPrBeKO23asF3EbLe6DlWZT5nUc4wm/JPWshXgBplgPc0kMsk8Kqh069KV2HDEg108dfs= Received: by 10.70.73.12 with SMTP id v12mr13741047wxa.1170105684936; Mon, 29 Jan 2007 13:21:24 -0800 (PST) Received: by 10.70.8.3 with HTTP; Mon, 29 Jan 2007 13:21:24 -0800 (PST) Message-ID: <8061fbee0701291321v5d127f47nb411093140d4c816@mail.gmail.com> Date: Mon, 29 Jan 2007 16:21:24 -0500 From: "Corey Smith" To: freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: carp support into GENERIC X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jan 2007 21:45:54 -0000 Please CC: me as I'm not on list. Please consider adding "device carp" to the GENERIC kernel. We have many servers using carp and it increases the complexity of installs and (freebsd-update) upgrades when you have to build a custom kernel only for this functionality. The kernel size increase is small and the benefits are (IMHO) worthwhile. device carp is not in the GENERIC kernel and (AFAIK) it is very difficult to make a module to add this support via kld. carp is also enabled by default in OpenBSD. -Corey Smith From owner-freebsd-arch@FreeBSD.ORG Mon Jan 29 22:14:39 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C949816A401 for ; Mon, 29 Jan 2007 22:14:39 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.183]) by mx1.freebsd.org (Postfix) with ESMTP id 3F38A13C49D for ; Mon, 29 Jan 2007 22:14:39 +0000 (UTC) (envelope-from max@love2party.net) Received: from [88.66.50.249] (helo=amd64.laiers.local) by mrelayeu.kundenserver.de (node=mrelayeu3) with ESMTP (Nemesis), id 0MKxQS-1HBeZZ1UIb-0004DB; Mon, 29 Jan 2007 23:01:58 +0100 From: Max Laier Organization: FreeBSD To: freebsd-arch@freebsd.org Date: Mon, 29 Jan 2007 23:01:48 +0100 User-Agent: KMail/1.9.5 References: <8061fbee0701291321v5d127f47nb411093140d4c816@mail.gmail.com> In-Reply-To: <8061fbee0701291321v5d127f47nb411093140d4c816@mail.gmail.com> X-Face: ,,8R(x[kmU]tKN@>gtH1yQE4aslGdu+2]; R]*pL,U>^H?)gW@49@wdJ`H<=?utf-8?q?=25=7D*=5FBD=0A=09U=5For=3D=5CmOZf764=26nYj=3DJYbR1PW0ud?=>|!~,,CPC.1-D$FG@0h3#'5"k{V]a~.<=?utf-8?q?mZ=7D44=23Se=7Em=0A=09Fe=7E=5C=5DX5B=5D=5Fxj?=(ykz9QKMw_l0C2AQ]}Ym8)fU MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1919175.1LV22hGIAh"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200701292301.54678.max@love2party.net> X-Provags-ID: kundenserver.de abuse@kundenserver.de login:61c499deaeeba3ba5be80f48ecc83056 X-Provags-ID2: V01U2FsdGVkX1/X59ZNfDTKSgqSLAqpgiwPF85pxzq5FYZAmbEDHOo2u7vYtjzuDaN/tLYn/6CSUwYn4bi7UEoTr3WOiS16jpC7QWMMb17fLfbesQiQa4caeg== Cc: Corey Smith Subject: Re: carp support into GENERIC X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jan 2007 22:14:39 -0000 --nextPart1919175.1LV22hGIAh Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Monday 29 January 2007 22:21, Corey Smith wrote: > Please CC: me as I'm not on list. > > Please consider adding "device carp" to the GENERIC kernel. We have > many servers using carp and it increases the complexity of installs > and (freebsd-update) upgrades when you have to build a custom kernel > only for this functionality. The kernel size increase is small and the > benefits are (IMHO) worthwhile. > > device carp is not in the GENERIC kernel and (AFAIK) it is very > difficult to make a module to add this support via kld. carp is also > enabled by default in OpenBSD. The problem with carp lies in the protocol number collision with vrrp. I=20 won't go into the details of why this is the case, but right now it seems=20 there is no simple sollution to this. If we are to build carp into=20 GENRIC, we must ensure that it plays nicely with vrrp. One possible way=20 would be to wrap the packet hooks - #ifdef'ed right now - under a sysctl. = =20 This is not too much work and seems the easiest way. The main problem why carp can't easily be made a module is the fact that=20 it has to register itself as an ip protocol in inetsw - which is not=20 really ready for dynamic changes, yet. =2D-=20 /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News --nextPart1919175.1LV22hGIAh Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQBFvm7SXyyEoT62BG0RAn8AAJ4k66Tm+KRggAdKMzcGWkcffxy4hACfSfDH ayzRGbAjaZYqJx26JdPE+xk= =j+qW -----END PGP SIGNATURE----- --nextPart1919175.1LV22hGIAh-- From owner-freebsd-arch@FreeBSD.ORG Tue Jan 30 22:25:01 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 06C7B16AB9B for ; Tue, 30 Jan 2007 22:25:00 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outL.internet-mail-service.net (outL.internet-mail-service.net [216.240.47.235]) by mx1.freebsd.org (Postfix) with ESMTP id CADE013C4B5 for ; Tue, 30 Jan 2007 22:25:00 +0000 (UTC) (envelope-from julian@elischer.org) Received: from shell.idiom.com (HELO idiom.com) (216.240.47.20) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Tue, 30 Jan 2007 13:48:50 -0800 Received: from [192.168.2.5] (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 8E83B125B3F; Tue, 30 Jan 2007 14:10:14 -0800 (PST) Message-ID: <45BFC246.2000005@elischer.org> Date: Tue, 30 Jan 2007 14:10:14 -0800 From: Julian Elischer User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: Marko Zec , Robert Watson , arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: vimage and modules X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Jan 2007 22:25:01 -0000 I've looked at the current vimage code and I'm impressed. The question remains to me that we need to have vimage and modules interact well. One question is, "should existing vimages suddenly get new capabilities when new kernel modules are loaded?" One alternative is to only allow them to have access to modules that were loaded before the creation of the vimage. An analogy in the TLS (thread-local storage) world is that when a new shared library is loaded, TLS variables are immediately available to it. However this may not be needed in a virtual machine. I heard someone mention the following idea in passing and the more I think about it, the more I like it.. Virtual machines are 'booted without loaded modules.' They have however, available to them all the modules loaded into the base system at the time that they are looking, and can 'load them' just as the base system can load kernel modules. The difference is that they are not able to load new modules, but rather, only to do the 'vm_linkage' stage required on already loaded modules. The vm linkage would be a separate subset of what needs to be done when a module is loaded. It would be a separate entry-point that would only be present on modules that supported vimages. An example would be some module 'x'. It would have a function that would set up any per-vimage linkages needed (mallocing and linking its own vimage-specific variables structure into the list of modules for that vimage. Currently we have a load and unload method. This would suggest adding a vm_load method as well. creating a new vimage could run through all the existing modules, and call that functions, or we could do it as part of the booting of the vimage from (say) /etc/rc or similar. If a module had no vimage-specific behaviour it would not have the extra entry-point. It does mean that we need to look at the inter-module dependencies as if you wanted to have one module call into another, you'd need to have dependencies well documented. In the current code you get a linkage failure, if there are global variables accessed between them but with a 'per vmimage' structure of variables this becomes a runtime problem, unless we explicitly have the dependencies registered. From owner-freebsd-arch@FreeBSD.ORG Tue Jan 30 22:53:17 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1438E16A400 for ; Tue, 30 Jan 2007 22:53:17 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.freebsd.org (Postfix) with ESMTP id B23D413C461 for ; Tue, 30 Jan 2007 22:53:16 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.6/8.13.6) with ESMTP id l0UMa9QD072006; Tue, 30 Jan 2007 17:36:15 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: Pawel Jakub Dawidek Date: Tue, 30 Jan 2007 16:36:36 -0500 User-Agent: KMail/1.9.1 References: <20070128202917.5B67916A5A6@hub.freebsd.org> <20070129194158.N32458@fledge.watson.org> <20070129201938.GF87767@garage.freebsd.pl> In-Reply-To: <20070129201938.GF87767@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200701301636.38175.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Tue, 30 Jan 2007 17:36:16 -0500 (EST) X-Virus-Scanned: ClamAV 0.88.3/2506/Tue Jan 30 14:50:40 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Scott Long , src-committers@freebsd.org, cvs-src@freebsd.org, cvs-all@freebsd.org, arch@freebsd.org, Nate Lawson Subject: Re: cvs commit: src/sys/geom/eli g_eli.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Jan 2007 22:53:17 -0000 On Monday 29 January 2007 15:19, Pawel Jakub Dawidek wrote: > I fully agree that there should be a clean KPI for this. What you > proposed if fine. Because of lack of such KPI geli has to handle HTT > CPUs which are turned off by default in releases also by abusing > scheduler internals. KPI you proposed would allow me to remove those > hacks. And I'm really all for it. > > What you and Scott are missing is that when I implement a GEOM class, > I'm using what is available to do my work. I'm not going to educate > myself how schedulers work, implement nice and clean KPI to use it in > my class. I'm not saying it wouldn't be great to be able to do so, but I > don't have time for everything, unfortunately, and you guys should > understand that very well. Something you seem to be missing though is that in general it's nice to not just endulge in an endless series of hacks in your specific area of the system. IWBN to at some point devote some time to help fix more general problems that will benefit the system as a whole rather than adding hacks that just benefit yourself and require someone else to clean up when they eventually add the more-architected solution. We are all very busy, but if everyone just did hacks and never put any effort into improving the general infrastructure, we'd have a royal mess. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Tue Jan 30 23:33:36 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E6C8816A402; Tue, 30 Jan 2007 23:33:36 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 359BE13C494; Tue, 30 Jan 2007 23:33:35 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id ABFC946DA5; Wed, 31 Jan 2007 00:33:33 +0100 (CET) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id E34D945681; Wed, 31 Jan 2007 00:33:26 +0100 (CET) Date: Wed, 31 Jan 2007 00:32:38 +0100 From: Pawel Jakub Dawidek To: John Baldwin Message-ID: <20070130233238.GB94650@garage.freebsd.pl> References: <20070128202917.5B67916A5A6@hub.freebsd.org> <20070129194158.N32458@fledge.watson.org> <20070129201938.GF87767@garage.freebsd.pl> <200701301636.38175.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Bn2rw/3z4jIqBvZU" Content-Disposition: inline In-Reply-To: <200701301636.38175.jhb@freebsd.org> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: Scott Long , src-committers@freebsd.org, cvs-src@freebsd.org, cvs-all@freebsd.org, arch@freebsd.org, Nate Lawson Subject: Re: cvs commit: src/sys/geom/eli g_eli.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Jan 2007 23:33:37 -0000 --Bn2rw/3z4jIqBvZU Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 30, 2007 at 04:36:36PM -0500, John Baldwin wrote: > On Monday 29 January 2007 15:19, Pawel Jakub Dawidek wrote: > > I fully agree that there should be a clean KPI for this. What you > > proposed if fine. Because of lack of such KPI geli has to handle HTT > > CPUs which are turned off by default in releases also by abusing > > scheduler internals. KPI you proposed would allow me to remove those > > hacks. And I'm really all for it. > >=20 > > What you and Scott are missing is that when I implement a GEOM class, > > I'm using what is available to do my work. I'm not going to educate > > myself how schedulers work, implement nice and clean KPI to use it in > > my class. I'm not saying it wouldn't be great to be able to do so, but I > > don't have time for everything, unfortunately, and you guys should > > understand that very well. >=20 > Something you seem to be missing though is that in general it's nice to n= ot=20 > just endulge in an endless series of hacks in your specific area of the= =20 > system. IWBN to at some point devote some time to help fix more general= =20 > problems that will benefit the system as a whole rather than adding hacks= =20 > that just benefit yourself and require someone else to clean up when they= =20 > eventually add the more-architected solution. We are all very busy, but = if=20 > everyone just did hacks and never put any effort into improving the gener= al=20 > infrastructure, we'd have a royal mess. To improve the infrastructure, one should feel strong about areas he want to improve. I do infrastructual work in areas, where I think I've a clue. Asking me to work on schedulers or SMP architecture where I don't feel competent enough is just unfair, John. The time I need to spend on educating myself in those areas will be much longer than I can afford. So my choice is leave broken code and ignore the users, or fix it using tools available now and fix it better when the infrastructure is ready or I've time to work on the infrastructure in this area. If someone competent will provide infrastructural improvements I can use, I'd love to use it. I try hard not to leave my code behind. Can we move on now? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --Bn2rw/3z4jIqBvZU Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFFv9WWForvXbEpPzQRAiaWAKCxGs6PnzgK4pRZsow3wLpXrIowcACfUPis OQ+OTCOmO+Ac/nqsQ2neu6E= =nR8H -----END PGP SIGNATURE----- --Bn2rw/3z4jIqBvZU-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 07:05:54 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2BB0016A403 for ; Wed, 31 Jan 2007 07:05:54 +0000 (UTC) (envelope-from vtheile@gmx.de) Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by mx1.freebsd.org (Postfix) with SMTP id C035413C478 for ; Wed, 31 Jan 2007 07:05:53 +0000 (UTC) (envelope-from vtheile@gmx.de) Received: (qmail 28271 invoked by uid 0); 31 Jan 2007 06:39:11 -0000 Received: from 217.111.66.66 by www092.gmx.net with HTTP; Wed, 31 Jan 2007 07:39:11 +0100 (CET) Content-Type: text/plain; charset="us-ascii" Date: Wed, 31 Jan 2007 07:39:11 +0100 From: "Volker Theile" Message-ID: <20070131063911.50650@gmx.net> MIME-Version: 1.0 To: freebsd-arch@freebsd.org X-Authenticated: #1412882 X-Flags: 0001 X-Mailer: WWW-Mail 6100 (Global Message Exchange) X-Priority: 3 Content-Transfer-Encoding: 7bit Subject: Any plans to implement OpenBSD sensor framework into FreeBSD? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 07:05:54 -0000 Hello all, i just want to ask if there are any plans to implement such a feature as the sensor framework in OpenBSD. I'm asking because it seems to me that there is currently no centralized architecture to handle hardware sensors like cpu temp or fan speed. Thanks && Greetings Volker -- "Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ... Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 08:36:23 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6F45316A4C5; Wed, 31 Jan 2007 08:36:23 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 2F30313C481; Wed, 31 Jan 2007 08:36:23 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (critter.freebsd.dk [192.168.48.2]) by phk.freebsd.dk (Postfix) with ESMTP id 6106C1747C; Wed, 31 Jan 2007 08:36:21 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.8/8.13.8) with ESMTP id l0V8aItf041207; Wed, 31 Jan 2007 08:36:20 GMT (envelope-from phk@critter.freebsd.dk) To: John Baldwin From: "Poul-Henning Kamp" In-Reply-To: Your message of "Tue, 30 Jan 2007 16:36:36 EST." <200701301636.38175.jhb@freebsd.org> Date: Wed, 31 Jan 2007 08:36:18 +0000 Message-ID: <41206.1170232578@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: arch@freebsd.org Subject: Re: cvs commit: src/sys/geom/eli g_eli.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 08:36:23 -0000 In message <200701301636.38175.jhb@freebsd.org>, John Baldwin writes: >Something you seem to be missing though is that in general it's nice to not >just endulge in an endless series of hacks in your specific area of the >system. IWBN to at some point devote some time to help fix more general >problems that will benefit the system as a whole rather than adding hacks >that just benefit yourself and require someone else to clean up when they >eventually add the more-architected solution. We are all very busy, but if >everyone just did hacks and never put any effort into improving the general >infrastructure, we'd have a royal mess. Seconded. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 12:59:12 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 397DE16A401 for ; Wed, 31 Jan 2007 12:59:12 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id E503113C467 for ; Wed, 31 Jan 2007 12:59:11 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p54A5CE3E.dip.t-dialin.net [84.165.206.62]) by redbull.bpaserver.net (Postfix) with ESMTP id F1E642E1A2 for ; Wed, 31 Jan 2007 13:49:58 +0100 (CET) Received: from webmail.leidinger.net (webmail.Leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 8B6485B4CBF for ; Wed, 31 Jan 2007 13:38:38 +0100 (CET) Received: (from www@localhost) by webmail.leidinger.net (8.13.8/8.13.8/Submit) id l0VCccLb061953 for freebsd-arch@freebsd.org; Wed, 31 Jan 2007 13:38:38 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde MIME library) with HTTP; Wed, 31 Jan 2007 13:38:38 +0100 Message-ID: <20070131133838.zkeoxoekkk80w0kw@webmail.leidinger.net> X-Priority: 3 (Normal) Date: Wed, 31 Jan 2007 13:38:38 +0100 From: Alexander Leidinger To: freebsd-arch@freebsd.org References: <20070131063911.50650@gmx.net> In-Reply-To: <20070131063911.50650@gmx.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.1.3) / FreeBSD-7.0 X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-14.864, required 6, autolearn=not spam, BAYES_00 -15.00, DK_POLICY_SIGNSOME 0.00, FORGED_RCVD_HELO 0.14) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No Subject: Re: Any plans to implement OpenBSD sensor framework into FreeBSD? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 12:59:12 -0000 Quoting Volker Theile (from Wed, 31 Jan 2007 07:39:11 +0100): > Hello all, > > i just want to ask if there are any plans to implement such a > feature as the sensor framework in OpenBSD. I'm asking because it > seems to me that there is currently no centralized architecture to > handle hardware sensors like cpu temp or fan speed. Sounds interesting. Links for people which want to give it a try or want an overview: - http://www.openbsd.org/papers/bsdcan06-biosensors.pdf - http://www.openbsd.org/cgi-bin/cvsweb/src/usr.sbin/sensorsd/ - http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/bioctl/ - http://www.openbsd.org/cgi-bin/cvsweb/src/sys/scsi/safte.c Bye, Alexander. -- A small town that cannot support one lawyer can always support two. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 15:03:35 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4009716A406 for ; Wed, 31 Jan 2007 15:03:35 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [210.51.165.229]) by mx1.freebsd.org (Postfix) with ESMTP id E249213C47E for ; Wed, 31 Jan 2007 15:03:34 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from localhost (tarsier.geekcn.org [210.51.165.229]) by tarsier.geekcn.org (Postfix) with ESMTP id 52CE6EB79D6; Wed, 31 Jan 2007 22:35:34 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([210.51.165.229]) by localhost (mail.geekcn.org [210.51.165.229]) (amavisd-new, port 10024) with ESMTP id pG-crk-tN6zk; Wed, 31 Jan 2007 22:35:19 +0800 (CST) Received: from [10.217.99.27] (unknown [61.135.152.195]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTP id 96DD9EB7963; Wed, 31 Jan 2007 22:35:19 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:organization:user-agent:mime-version:to:cc: subject:references:in-reply-to:x-enigmail-version:content-type; b=cMnvKdrl4R3tAIleTT/EAFmZhVmRb20RMFdl8ViVVxcx6bJBySpOjqMgSNMUL+JCc GF6aRK2zfGxbgA+aYSPdg== Message-ID: <45C0A924.6070204@delphij.net> Date: Wed, 31 Jan 2007 22:35:16 +0800 From: LI Xin Organization: The FreeBSD Project User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: Alexander Leidinger References: <20070131063911.50650@gmx.net> <20070131133838.zkeoxoekkk80w0kw@webmail.leidinger.net> In-Reply-To: <20070131133838.zkeoxoekkk80w0kw@webmail.leidinger.net> X-Enigmail-Version: 0.94.1.0 Content-Type: multipart/signed; micalg=pgp-ripemd160; protocol="application/pgp-signature"; boundary="------------enig01012FBEB80FAFCA8108AE67" Cc: Joel Dahl , freebsd-arch@freebsd.org Subject: Re: Any plans to implement OpenBSD sensor framework into FreeBSD? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 15:03:35 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig01012FBEB80FAFCA8108AE67 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Alexander Leidinger wrote: > Quoting Volker Theile (from Wed, 31 Jan 2007 07:39:11 > +0100): >=20 >> Hello all, >> >> i just want to ask if there are any plans to implement such a feature= >> as the sensor framework in OpenBSD. I'm asking because it seems to me= >> that there is currently no centralized architecture to handle >> hardware sensors like cpu temp or fan speed. >=20 > Sounds interesting. >=20 > Links for people which want to give it a try or want an overview: > - http://www.openbsd.org/papers/bsdcan06-biosensors.pdf > - http://www.openbsd.org/cgi-bin/cvsweb/src/usr.sbin/sensorsd/ > - http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/bioctl/ > - http://www.openbsd.org/cgi-bin/cvsweb/src/sys/scsi/safte.c Perhaps we want to list it on the ideas page? Cheers, --=20 Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! --------------enig01012FBEB80FAFCA8108AE67 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFwKkkOfuToMruuMARAx/fAJ4iCrHu+wNbQwaboE5X/Bo+XrFFoACfeCw4 gWTcD2cnruhw4ejfL6Mq1UU= =zI3L -----END PGP SIGNATURE----- --------------enig01012FBEB80FAFCA8108AE67-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 15:44:43 2007 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 181A816A402 for ; Wed, 31 Jan 2007 15:44:43 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id C4C3513C48D for ; Wed, 31 Jan 2007 15:44:42 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id l0VFBvrt085006; Wed, 31 Jan 2007 08:12:02 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <45C0B1B1.8000103@samsco.org> Date: Wed, 31 Jan 2007 08:11:45 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2pre) Gecko/20070111 SeaMonkey/1.1 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20070128202917.5B67916A5A6@hub.freebsd.org> <20070129194158.N32458@fledge.watson.org> <20070129201938.GF87767@garage.freebsd.pl> <200701301636.38175.jhb@freebsd.org> <20070130233238.GB94650@garage.freebsd.pl> In-Reply-To: <20070130233238.GB94650@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (pooker.samsco.org [168.103.85.57]); Wed, 31 Jan 2007 08:12:02 -0700 (MST) X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on pooker.samsco.org Cc: src-committers@FreeBSD.org, John Baldwin , cvs-src@FreeBSD.org, cvs-all@FreeBSD.org, arch@FreeBSD.org, Nate Lawson Subject: Re: cvs commit: src/sys/geom/eli g_eli.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 15:44:43 -0000 Pawel Jakub Dawidek wrote: > On Tue, Jan 30, 2007 at 04:36:36PM -0500, John Baldwin wrote: >> On Monday 29 January 2007 15:19, Pawel Jakub Dawidek wrote: >>> I fully agree that there should be a clean KPI for this. What you >>> proposed if fine. Because of lack of such KPI geli has to handle HTT >>> CPUs which are turned off by default in releases also by abusing >>> scheduler internals. KPI you proposed would allow me to remove those >>> hacks. And I'm really all for it. >>> >>> What you and Scott are missing is that when I implement a GEOM class, >>> I'm using what is available to do my work. I'm not going to educate >>> myself how schedulers work, implement nice and clean KPI to use it in >>> my class. I'm not saying it wouldn't be great to be able to do so, but I >>> don't have time for everything, unfortunately, and you guys should >>> understand that very well. >> Something you seem to be missing though is that in general it's nice to not >> just endulge in an endless series of hacks in your specific area of the >> system. IWBN to at some point devote some time to help fix more general >> problems that will benefit the system as a whole rather than adding hacks >> that just benefit yourself and require someone else to clean up when they >> eventually add the more-architected solution. We are all very busy, but if >> everyone just did hacks and never put any effort into improving the general >> infrastructure, we'd have a royal mess. > > To improve the infrastructure, one should feel strong about areas he > want to improve. I do infrastructual work in areas, where I think I've a > clue. Asking me to work on schedulers or SMP architecture where I don't > feel competent enough is just unfair, John. The time I need to spend on > educating myself in those areas will be much longer than I can afford. > So my choice is leave broken code and ignore the users, or fix it using > tools available now and fix it better when the infrastructure is ready > or I've time to work on the infrastructure in this area. > If someone competent will provide infrastructural improvements I can > use, I'd love to use it. I try hard not to leave my code behind. > > Can we move on now? > You're quite naive if you think that people like John and myself work on infrastructure for any other reason than that it needs to be done. You may not feel that it's your job to do good architecture, but whatever you propose to put into FreeBSD is still subject to peer review. Keep this in mind. Scott From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 17:08:02 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9F6F716A401 for ; Wed, 31 Jan 2007 17:08:02 +0000 (UTC) (envelope-from ducrot@poupinou.org) Received: from poup.poupinou.org (poup.poupinou.org [195.101.94.96]) by mx1.freebsd.org (Postfix) with ESMTP id 6978B13C442 for ; Wed, 31 Jan 2007 17:08:01 +0000 (UTC) (envelope-from ducrot@poupinou.org) Received: from ducrot by poup.poupinou.org with local (Exim) id 1HCIh9-0004Rc-00; Wed, 31 Jan 2007 17:52:27 +0100 Date: Wed, 31 Jan 2007 17:52:27 +0100 To: Alexander Leidinger Message-ID: <20070131165227.GG12197@poupinou.org> References: <20070131063911.50650@gmx.net> <20070131133838.zkeoxoekkk80w0kw@webmail.leidinger.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070131133838.zkeoxoekkk80w0kw@webmail.leidinger.net> User-Agent: Mutt/1.5.9i From: Bruno Ducrot Cc: freebsd-arch@freebsd.org Subject: Re: Any plans to implement OpenBSD sensor framework into FreeBSD? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 17:08:02 -0000 On Wed, Jan 31, 2007 at 01:38:38PM +0100, Alexander Leidinger wrote: > Quoting Volker Theile (from Wed, 31 Jan 2007 07:39:11 > +0100): > > >Hello all, > > > >i just want to ask if there are any plans to implement such a > >feature as the sensor framework in OpenBSD. I'm asking because it > >seems to me that there is currently no centralized architecture to > >handle hardware sensors like cpu temp or fan speed. > > Sounds interesting. Indeed. I'll take a look. Thanks for the links. > Links for people which want to give it a try or want an overview: > - http://www.openbsd.org/papers/bsdcan06-biosensors.pdf > - http://www.openbsd.org/cgi-bin/cvsweb/src/usr.sbin/sensorsd/ > - http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/bioctl/ > - http://www.openbsd.org/cgi-bin/cvsweb/src/sys/scsi/safte.c > -- Bruno Ducrot -- Which is worse: ignorance or apathy? -- Don't know. Don't care. From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 19:51:50 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 017B716A402 for ; Wed, 31 Jan 2007 19:51:50 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id E6D8013C491 for ; Wed, 31 Jan 2007 19:51:49 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l0VJpmvs061245; Wed, 31 Jan 2007 11:51:48 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l0VJpmca061244; Wed, 31 Jan 2007 11:51:48 -0800 (PST) (envelope-from rizzo) Date: Wed, 31 Jan 2007 11:51:48 -0800 From: Luigi Rizzo To: arch@freebsd.org Message-ID: <20070131115148.A60420@xorpc.icir.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i Cc: Subject: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 19:51:50 -0000 I think there was some discussion on the topic some time ago but cannot remember if there was any outcome so let me ask. We have several tables in our kernel and modules, containing at least device-id tables (pci, usb), quirks for certain devices, and maybe more (i am excluding firmware because it is not something where you may want to change a few lines manually). Right now we compile them statically in the code. However sometimes it is useful to update them 'on the fly' without having to rebuild a kernel/module - e.g. because an entry (e.g. a quirk for a specific device) cannot be safely included in the distribution, etc.etc. What are the options to implement a mechanism that lets userland update such tables, and maybe in a way that is accessible to the boot loader ? The (obvious) requirements are: + support for multiple tables with variable (maybe defined at compile time) structure; + initial values should be compiled in; + the access mechanism should provide mechanism to read/modify/write/delete entries; + human-readable (i.e. textual) representation of entries in userland, with hooks to translate them in a machine-friendly format on writes, and back on reads; sysctl is the first method that comes to my mind, but perhaps there is some better way ? cheers luigi From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 20:16:16 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 253DD16A52A; Wed, 31 Jan 2007 20:16:08 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.freebsd.org (Postfix) with ESMTP id 6FA7913C442; Wed, 31 Jan 2007 20:16:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.6/8.13.6) with ESMTP id l0VKFpZJ084277; Wed, 31 Jan 2007 15:15:59 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: Pawel Jakub Dawidek Date: Wed, 31 Jan 2007 13:08:14 -0500 User-Agent: KMail/1.9.1 References: <20070128202917.5B67916A5A6@hub.freebsd.org> <200701301636.38175.jhb@freebsd.org> <20070130233238.GB94650@garage.freebsd.pl> In-Reply-To: <20070130233238.GB94650@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200701311308.16810.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Wed, 31 Jan 2007 15:16:01 -0500 (EST) X-Virus-Scanned: ClamAV 0.88.3/2508/Wed Jan 31 11:42:50 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Scott Long , src-committers@freebsd.org, cvs-src@freebsd.org, cvs-all@freebsd.org, arch@freebsd.org, Nate Lawson Subject: Re: cvs commit: src/sys/geom/eli g_eli.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 20:16:16 -0000 On Tuesday 30 January 2007 18:32, Pawel Jakub Dawidek wrote: > On Tue, Jan 30, 2007 at 04:36:36PM -0500, John Baldwin wrote: > > On Monday 29 January 2007 15:19, Pawel Jakub Dawidek wrote: > > > I fully agree that there should be a clean KPI for this. What you > > > proposed if fine. Because of lack of such KPI geli has to handle HTT > > > CPUs which are turned off by default in releases also by abusing > > > scheduler internals. KPI you proposed would allow me to remove those > > > hacks. And I'm really all for it. > > > > > > What you and Scott are missing is that when I implement a GEOM class, > > > I'm using what is available to do my work. I'm not going to educate > > > myself how schedulers work, implement nice and clean KPI to use it in > > > my class. I'm not saying it wouldn't be great to be able to do so, but I > > > don't have time for everything, unfortunately, and you guys should > > > understand that very well. > > > > Something you seem to be missing though is that in general it's nice to not > > just endulge in an endless series of hacks in your specific area of the > > system. IWBN to at some point devote some time to help fix more general > > problems that will benefit the system as a whole rather than adding hacks > > that just benefit yourself and require someone else to clean up when they > > eventually add the more-architected solution. We are all very busy, but if > > everyone just did hacks and never put any effort into improving the general > > infrastructure, we'd have a royal mess. > > To improve the infrastructure, one should feel strong about areas he > want to improve. I do infrastructual work in areas, where I think I've a > clue. Asking me to work on schedulers or SMP architecture where I don't > feel competent enough is just unfair, John. The time I need to spend on > educating myself in those areas will be much longer than I can afford. > So my choice is leave broken code and ignore the users, or fix it using > tools available now and fix it better when the infrastructure is ready > or I've time to work on the infrastructure in this area. > If someone competent will provide infrastructural improvements I can > use, I'd love to use it. I try hard not to leave my code behind. > > Can we move on now? I wouldn't sell yourself so short actually. :) You're probably more competent than you think. I'm not really an expert on SMP or schedulers either (I've not had any formal training, and if you were to review some of my earlier SMP stuff it shows), but ended up working on it because it needed to be done. Most of what I've learned is by just working on stuff. Anyways, maybe this issue will bubble up to someone's todo list at BSDCan to settle how to handle CPU's arriving and departing. One big thing that worries me is how to handle pinned and bound threads when a CPU goes away. Also, it may be more useful to think of CPUs not as just present/not present, but more in terms of: - not present - present but offline - present and online And most code would really only care about offline/online events. We could maybe allow pinned and bound threads to still run on an offline CPU (and the idlethread for that CPU of course) but require that there be no pinned bound threads to completely detach a CPU (in the case of systems with removable CPUs). It would be useful to at least handle taking CPUs offline and then back online though. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 22:20:41 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BE9A416A405; Wed, 31 Jan 2007 22:20:41 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 21A5A13C49D; Wed, 31 Jan 2007 22:20:41 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 23EE045CDA; Wed, 31 Jan 2007 23:20:40 +0100 (CET) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 9B4F045681; Wed, 31 Jan 2007 23:20:32 +0100 (CET) Date: Wed, 31 Jan 2007 23:19:44 +0100 From: Pawel Jakub Dawidek To: John Baldwin Message-ID: <20070131221944.GD487@garage.freebsd.pl> References: <20070128202917.5B67916A5A6@hub.freebsd.org> <200701301636.38175.jhb@freebsd.org> <20070130233238.GB94650@garage.freebsd.pl> <200701311308.16810.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="a2FkP9tdjPU2nyhF" Content-Disposition: inline In-Reply-To: <200701311308.16810.jhb@freebsd.org> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: Scott Long , src-committers@freebsd.org, cvs-src@freebsd.org, cvs-all@freebsd.org, arch@freebsd.org, Nate Lawson Subject: Re: cvs commit: src/sys/geom/eli g_eli.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 22:20:41 -0000 --a2FkP9tdjPU2nyhF Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jan 31, 2007 at 01:08:14PM -0500, John Baldwin wrote: > [...] Anyways, maybe this=20 > issue will bubble up to someone's todo list at BSDCan to settle how to ha= ndle=20 > CPU's arriving and departing. One big thing that worries me is how to ha= ndle=20 > pinned and bound threads when a CPU goes away. Also, it may be more usef= ul=20 > to think of CPUs not as just present/not present, but more in terms of: >=20 > - not present > - present but offline > - present and online >=20 > And most code would really only care about offline/online events. We cou= ld=20 > maybe allow pinned and bound threads to still run on an offline CPU (and = the=20 > idlethread for that CPU of course) but require that there be no pinned bo= und=20 > threads to completely detach a CPU (in the case of systems with removable= =20 > CPUs). It would be useful to at least handle taking CPUs offline and the= n=20 > back online though. As a consumer of such functionality I'd like something like this: - I'd like to register myself as interested in receiving "CPU-online" and "CPU-offline" events. The EVENTHANDLER(9) KPI seems to be ok. When my registration is from a kernel module (all CPUs are already online), I'd like to still receive a fake CPU-online event for all of them. - When my CPU-online handler is called, I'd like to start my thread from there and be sure that CPU won't go offline before I return from the handler. It will be nice to be able to specify CPU ID I want to bind to in kthread_create(). This would save me from the kthread_create-sleep-wakeup dance. - When someone asks for a CPU to go offline, all CPU-offline handlers are called one by one and CPU will work (and stay online) until the last handler returns. - It'll be nice to get an error in return from sched_bind() return if CPU is going offline or is already offline. - Would be nice if there will be no need for the consumers to handle the boot CPU somehow specially. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --a2FkP9tdjPU2nyhF Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFFwRYAForvXbEpPzQRApUxAJ9VgE0alhpkwP8dhX+0lNF+sZzX4gCeLrZ6 OFTwuTgWVa+ZA2DX1B6HDhE= =x1C6 -----END PGP SIGNATURE----- --a2FkP9tdjPU2nyhF-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 31 23:43:57 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 26EF716A405 for ; Wed, 31 Jan 2007 23:43:57 +0000 (UTC) (envelope-from nate@root.org) Received: from root.org (root.org [67.118.192.226]) by mx1.freebsd.org (Postfix) with ESMTP id 0AEED13C49D for ; Wed, 31 Jan 2007 23:43:56 +0000 (UTC) (envelope-from nate@root.org) Received: (qmail 43496 invoked from network); 31 Jan 2007 23:32:42 -0000 Received: from adsl-67-119-74-222.dsl.sntc01.pacbell.net (HELO ?10.0.0.44?) (nate-mail@67.119.74.222) by root.org with ESMTPA; 31 Jan 2007 23:32:42 -0000 Message-ID: <45C1270C.1050308@root.org> Date: Wed, 31 Jan 2007 15:32:28 -0800 From: Nate Lawson User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) MIME-Version: 1.0 To: John Baldwin References: <20070128202917.5B67916A5A6@hub.freebsd.org> <200701301636.38175.jhb@freebsd.org> <20070130233238.GB94650@garage.freebsd.pl> <200701311308.16810.jhb@freebsd.org> In-Reply-To: <200701311308.16810.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: Scott Long , src-committers@freebsd.org, Pawel Jakub Dawidek , cvs-src@freebsd.org, cvs-all@freebsd.org, arch@freebsd.org Subject: Re: cvs commit: src/sys/geom/eli g_eli.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 23:43:57 -0000 John Baldwin wrote: > I wouldn't sell yourself so short actually. :) You're probably more competent > than you think. I'm not really an expert on SMP or schedulers either (I've > not had any formal training, and if you were to review some of my earlier SMP > stuff it shows), but ended up working on it because it needed to be done. > Most of what I've learned is by just working on stuff. Anyways, maybe this > issue will bubble up to someone's todo list at BSDCan to settle how to handle > CPU's arriving and departing. One big thing that worries me is how to handle > pinned and bound threads when a CPU goes away. Also, it may be more useful > to think of CPUs not as just present/not present, but more in terms of: > > - not present > - present but offline > - present and online > > And most code would really only care about offline/online events. We could > maybe allow pinned and bound threads to still run on an offline CPU (and the > idlethread for that CPU of course) but require that there be no pinned bound > threads to completely detach a CPU (in the case of systems with removable > CPUs). It would be useful to at least handle taking CPUs offline and then > back online though. I think this is a good design, assuming that all code that creates a thread and pins it are forced to register to receive events. That way you don't end up with a thread pinned and blocking the CPU going away. This capability will also be useful for suspend/resume on SMP since we need to stop CPUs and then restart them after resume. -- Nate From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 06:42:24 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 28F5116A403; Thu, 1 Feb 2007 06:42:24 +0000 (UTC) (envelope-from ssouhlal@FreeBSD.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 179F413C467; Thu, 1 Feb 2007 06:42:24 +0000 (UTC) (envelope-from ssouhlal@FreeBSD.org) Received: from ssl.mu.org (localhost [127.0.0.1]) by elvis.mu.org (Postfix) with ESMTP id 30B5B1A4D81; Wed, 31 Jan 2007 22:14:54 -0800 (PST) Received: from 67.188.127.3 (SquirrelMail authenticated user refugee) by ssl.mu.org with HTTP; Wed, 31 Jan 2007 22:14:54 -0800 (PST) Message-ID: <25784.67.188.127.3.1170310494.squirrel@ssl.mu.org> Date: Wed, 31 Jan 2007 22:14:54 -0800 (PST) From: "Suleiman Souhlal" To: alc@freebsd.org User-Agent: SquirrelMail/1.5.1 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: arch@freebsd.org Subject: Reducing vm page queue mutex contention X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 06:42:24 -0000 Hello Alan, Profiling shows that the vm page queue mutex is the most contended lock in the kernel, maybe apart from sched_lock. It seems that this is in part because this lock protects a lot of things: page queues, pv entries, page flags, page hold count, page wired count.. I came up with a possible plan to reduce contention on this lock, concentrating on the amd64 pmap (although these should be applicable to the other architectures as well): - Make vm_page_flag_set/clear() just use atomic operations to get rid of the page queues lock dependency. I'm still not entirely convinced this is entirely safe. - vm_page_hold and vm_page_unhold can be made not acquire the queues lock in the common case. I already have a patch for this, although it increases the size of vm_page_t (I have some other ideas to reduce the size of vm_page_t, but that's for another time): http://people.freebsd.org/~ssouhlal/testing/vm_page_hold-20070131.diff - Add a mutex pool for vm pages to protect the pv entries lists. I'm currently working on this. My current approach makes struct pv_entry larger because it needs to store a pointer to the pte in each pv_entry. Another way that might be better is to move to per-object pv entries, which is what Linux does. It would greatly reduce memory usage when mapping large objects in a lot of processes, although it might be slower for sparsely faulted objects mapped in a large number of processes. This approach would be a lot of work, which is why I'm leaning towards keeping per-page pv entries. - It should be possible to make vm_page->wired_count use atomic operations instead of needing a lock, similarly to what I did for the hold_count. This might be a bit tricky, but hopefully possible. Alternatively, we could use the mutex pool described above to protect it. - We can change pmap_unuse_pt and free_pv_entry to just mark the pages they want to free in an array allocated by the caller. The caller will then free those pages after it drops the pmap lock. For example: struct pages_to_free { vm_page_t *page[MAX_PAGES]; int num_pages; }; void pmap_remove(...) { struct pages_to_free pages; PMAP_LOCK(pmap); ... pmap_unuse_pt(..., &pages); ... PMAP_UNLOCK(pmap); vm_page_lock_queues(); for (i = 0; i < pages.num_pages; i++) vm_page_free(pages.page[i]); vm_page_unlock_queues(); } This way, pmap_remove can be mostly without queues lock. - Once the above are done, it should be possible to make pmap_enter() run mostly queue lock free by: - Pre-allocating a pv chunk early in pmap_enter, if there are no free ones, so that we never have to allocate new chunks in pmap_insert_entry. - Dropping the page queues lock immediately after the pmap_allocpte in pmap_enter. Any thoughts/comments? -- Suleiman From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 06:47:41 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 33CDF16A492; Thu, 1 Feb 2007 06:47:41 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id DDAD113C461; Thu, 1 Feb 2007 06:47:40 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p54A5F3B4.dip.t-dialin.net [84.165.243.180]) by redbull.bpaserver.net (Postfix) with ESMTP id 6B3DB2E1D7; Thu, 1 Feb 2007 07:59:05 +0100 (CET) Received: from webmail.leidinger.net (webmail.Leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 57BE45B4DC7; Thu, 1 Feb 2007 07:47:33 +0100 (CET) Received: (from www@localhost) by webmail.leidinger.net (8.13.8/8.13.8/Submit) id l116lXWp064037; Thu, 1 Feb 2007 07:47:33 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde MIME library) with HTTP; Thu, 01 Feb 2007 07:47:33 +0100 Message-ID: <20070201074733.ra9g8bl9a8cs84w4@webmail.leidinger.net> X-Priority: 3 (Normal) Date: Thu, 01 Feb 2007 07:47:33 +0100 From: Alexander Leidinger To: LI Xin References: <20070131063911.50650@gmx.net> <20070131133838.zkeoxoekkk80w0kw@webmail.leidinger.net> <45C0A924.6070204@delphij.net> In-Reply-To: <45C0A924.6070204@delphij.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.1.3) / FreeBSD-7.0 X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-14.864, required 6, autolearn=not spam, BAYES_00 -15.00, DK_POLICY_SIGNSOME 0.00, FORGED_RCVD_HELO 0.14) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No Cc: Joel Dahl , freebsd-arch@freebsd.org Subject: Re: Any plans to implement OpenBSD sensor framework into FreeBSD? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 06:47:41 -0000 Quoting LI Xin (from Wed, 31 Jan 2007 22:35:16 +0800): > Perhaps we want to list it on the ideas page? If nobody says he is willing to give it a try, it will show up there. I already talked with Joel about it via ICQ shortly after I did send the mail with the links. Bye, Alexander. -- So far as I can remember, there is not one word in the Gospels in praise of intelligence. -- Bertrand Russell http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 10:09:43 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AED1A16A403 for ; Thu, 1 Feb 2007 10:09:43 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id 279C313C442 for ; Thu, 1 Feb 2007 10:09:42 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.13.8/8.13.8) with ESMTP id l119al2S029863; Thu, 1 Feb 2007 20:36:47 +1100 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.13.8/8.13.8/Submit) id l119al9C029862; Thu, 1 Feb 2007 20:36:47 +1100 (EST) (envelope-from peter) Date: Thu, 1 Feb 2007 20:36:47 +1100 From: Peter Jeremy To: Luigi Rizzo Message-ID: <20070201093647.GA909@turion.vk2pj.dyndns.org> References: <20070131115148.A60420@xorpc.icir.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="vtzGhvizbBRQ85DL" Content-Disposition: inline In-Reply-To: <20070131115148.A60420@xorpc.icir.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.13 (2006-08-11) Cc: arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 10:09:43 -0000 --vtzGhvizbBRQ85DL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, 2007-Jan-31 11:51:48 -0800, Luigi Rizzo wrote: >What are the options to implement a mechanism that lets >userland update such tables, and maybe in a way that >is accessible to the boot loader ? It doesn't help the bootloader but /dev/FOO.ctl may be an option for dumping configuration data into a device. >The (obvious) requirements are: > + support for multiple tables with variable (maybe defined > at compile time) structure; Passing binary blobs back and forth through sysctl requires code in sysctl(8) to {en,de}code whereas the all-text approach makes the OID tree look messy. It would be nice if sysctl grew a configuration file (or maybe magic OIDs that let it retrieve the description of opaque OIDs). --=20 Peter Jeremy --vtzGhvizbBRQ85DL Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFFwbSv/opHv/APuIcRApDgAJ9isxPxReoVDG32LozlCQk3gPLgAwCgpGiD n3I86K80wy5wj0yNoVqnnI0= =0+SL -----END PGP SIGNATURE----- --vtzGhvizbBRQ85DL-- From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 16:21:13 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D3BFB16A40B for ; Thu, 1 Feb 2007 16:21:13 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.freebsd.org (Postfix) with ESMTP id 65F3713C4C9 for ; Thu, 1 Feb 2007 16:21:13 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.6/8.13.6) with ESMTP id l11GL73B091825; Thu, 1 Feb 2007 11:21:09 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-arch@freebsd.org Date: Thu, 1 Feb 2007 11:09:12 -0500 User-Agent: KMail/1.9.1 References: <20070131115148.A60420@xorpc.icir.org> In-Reply-To: <20070131115148.A60420@xorpc.icir.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200702011109.12821.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Thu, 01 Feb 2007 11:21:10 -0500 (EST) X-Virus-Scanned: ClamAV 0.88.3/2511/Thu Feb 1 09:55:18 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 16:21:14 -0000 On Wednesday 31 January 2007 14:51, Luigi Rizzo wrote: > I think there was some discussion on the topic some time ago > but cannot remember if there was any outcome so let me ask. > > We have several tables in our kernel and modules, > containing at least device-id tables (pci, usb), > quirks for certain devices, and maybe more (i am excluding > firmware because it is not something where you may want > to change a few lines manually). > > Right now we compile them statically in the code. > However sometimes it is useful to update them 'on the fly' > without having to rebuild a kernel/module - e.g. because > an entry (e.g. a quirk for a specific device) cannot be > safely included in the distribution, etc.etc. > > What are the options to implement a mechanism that lets > userland update such tables, and maybe in a way that > is accessible to the boot loader ? > > The (obvious) requirements are: > + support for multiple tables with variable (maybe defined > at compile time) structure; > + initial values should be compiled in; > + the access mechanism should provide mechanism to read/modify/write/delete > entries; > + human-readable (i.e. textual) representation of entries in userland, > with hooks to translate them in a machine-friendly format on writes, > and back on reads; > > sysctl is the first method that comes to my mind, but > perhaps there is some better way ? Windows and MAC OS X both use plain text files to hold things like PCI device IDs to match drivers to devices. OS X uses an XML file format that lets you list the PCI ID's a driver supports and the kernel auto-loads driver modules by matching on PCI IDs. Many drivers can't be helped by this though, as they use the device ID for for device-specific behavior (such as em(4) or brgpy(4)). -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 17:16:06 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3A21316A40A; Thu, 1 Feb 2007 17:16:06 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id 29CD013C491; Thu, 1 Feb 2007 17:16:06 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l11HG51E082464; Thu, 1 Feb 2007 09:16:05 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l11HG5fG082463; Thu, 1 Feb 2007 09:16:05 -0800 (PST) (envelope-from rizzo) Date: Thu, 1 Feb 2007 09:16:05 -0800 From: Luigi Rizzo To: John Baldwin Message-ID: <20070201091605.A82313@xorpc.icir.org> References: <20070131115148.A60420@xorpc.icir.org> <200702011109.12821.jhb@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <200702011109.12821.jhb@freebsd.org>; from jhb@freebsd.org on Thu, Feb 01, 2007 at 11:09:12AM -0500 Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 17:16:06 -0000 On Thu, Feb 01, 2007 at 11:09:12AM -0500, John Baldwin wrote: > On Wednesday 31 January 2007 14:51, Luigi Rizzo wrote: ... > > We have several tables in our kernel and modules, > > containing at least device-id tables (pci, usb), > > quirks for certain devices, and maybe more (i am excluding ... > Windows and MAC OS X both use plain text files to hold things like PCI device > IDs to match drivers to devices. OS X uses an XML file format that lets you plain text files! too obvious to think of it :) but, where can i find an example of a piece of kernel code that can read from a file "safely" (i.e. say in the modevent handler or maybe at device probe time) ? Something like char *load_file_into_kernel_memody(filename, max_size, &error); I have looked at the kernel side of execve and kldload, they are not exactly straightforward (at least there are seveal indirections). Maybe there are other simpler ones ? > list the PCI ID's a driver supports and the kernel auto-loads driver modules > by matching on PCI IDs. Many drivers can't be helped by this though, as they > use the device ID for for device-specific behavior (such as em(4) or > brgpy(4)). well, that's a two-phase process, if you have the above table-based mechanism, the kernel (actually the bus driver i would say) can load the [device_id -> driver] table (or tables, assuming each driver has its own file) to select and load the correct driver, and then the driver itself could use the same mechanism to load a [device_id -> quirks] table for internal use ? cheers luigi From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 18:03:52 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1399B16A400; Thu, 1 Feb 2007 18:03:52 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (vc4-2-0-87.dsl.netrack.net [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id B347813C4A6; Thu, 1 Feb 2007 18:03:51 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.4/8.13.4) with ESMTP id l11I1ZAU062378; Thu, 1 Feb 2007 11:01:35 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Thu, 01 Feb 2007 11:02:06 -0700 (MST) Message-Id: <20070201.110206.1102529050.imp@bsdimp.com> To: rizzo@icir.org From: "M. Warner Losh" In-Reply-To: <20070201091605.A82313@xorpc.icir.org> References: <20070131115148.A60420@xorpc.icir.org> <200702011109.12821.jhb@freebsd.org> <20070201091605.A82313@xorpc.icir.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Thu, 01 Feb 2007 11:01:36 -0700 (MST) Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 18:03:52 -0000 In message: <20070201091605.A82313@xorpc.icir.org> Luigi Rizzo writes: : On Thu, Feb 01, 2007 at 11:09:12AM -0500, John Baldwin wrote: : > On Wednesday 31 January 2007 14:51, Luigi Rizzo wrote: : ... : > > We have several tables in our kernel and modules, : > > containing at least device-id tables (pci, usb), : > > quirks for certain devices, and maybe more (i am excluding : ... : > Windows and MAC OS X both use plain text files to hold things like PCI device : > IDs to match drivers to devices. OS X uses an XML file format that lets you : : plain text files! : : too obvious to think of it :) : : but, where can i find an example of a piece of kernel code that can : read from a file "safely" (i.e. say in the modevent handler or maybe : at device probe time) ? : Something like : : char *load_file_into_kernel_memody(filename, max_size, &error); : : I have looked at the kernel side of execve and kldload, they are not : exactly straightforward (at least there are seveal indirections). : Maybe there are other simpler ones ? Look at the firmware routines. However, they won't work until / is mounted, which is after all the device probing happens. : > list the PCI ID's a driver supports and the kernel auto-loads driver modules : > by matching on PCI IDs. Many drivers can't be helped by this though, as they : > use the device ID for for device-specific behavior (such as em(4) or : > brgpy(4)). : : well, that's a two-phase process, if you have the above table-based : mechanism, the kernel (actually the bus driver i would say) can : load the [device_id -> driver] table (or tables, assuming : each driver has its own file) to select and load the : correct driver, and then the driver itself could use the same : mechanism to load a [device_id -> quirks] table for internal use ? On the one hand, I like the flexibility of having the ability to look at a device and know what driver(s) may attach to it w/o having the drivers in memory, since that allows us to move to a demand load model for those people that want it. Right now, there's no way to smartly extract the plug and play info used by the device probe routines to bid on a device from the drivers to try to smartly load, say, the atheros driver when an atheros cardbus card is inserted. However, only pccard and miibus drivers seem to religiously use tables to do their configuration. No other busses do that currently, although many have support for doing so. usb and pci are the big offenders whereby the probe is handled in many ad-hoc hackish ways. usb is bad in another way: it doesn't grok reprobing and has a 'vacuum' device that attaches to everything (ugen). The other issue is that there's many different parts of the PnP information that are used to bind devices to drivers. In pccard, for example, sio doesn't have a list of all modems, but instead attaches to all serial cards based on function id rather than the typcial vendor/model identifiers that other drivers use. This multiplicity of selection methods complicates any solution. On the other hand, the reason that we have the device driver tables compiled into the kernel is because when the kernel is doing its initial probe/attach of devices, it can't touch the root file system. This is the classic chicken and egg problem. The loader would have to assist the kernel in loading these tables, but there's complications for situations where one is booting over the network, or one is booting from flash. In these situations, it can be impossible to load another module until more of the infrastructure is in place (either disk drivers + busses + cam/ata or nfs + network interface). I've long wanted to have the ability to tell a bus to 'pretend' that the device ID for a given device is XXXX rather than the YYYY read from the device's enumeration mechanism. This would allow us to 'quirk' drivers for new hardware, as well as allow for users to get their devices up and running more quickly than having to rebuild the kernel. While this is the safest solution, it doesn't solve the 'which module to load' issue. It also doesn't force users to report these device IDs back to us, which may make fewer devices work out of the box in the long run. There's also only so much one can do with this mechanism if the driver is smart and has workarounds for different models and such that aren't needed/are harmful to newer silicon. Warner From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 18:25:05 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1F47516A405 for ; Thu, 1 Feb 2007 18:25:05 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outV.internet-mail-service.net (outV.internet-mail-service.net [216.240.47.245]) by mx1.freebsd.org (Postfix) with ESMTP id D6AA413C467 for ; Thu, 1 Feb 2007 18:25:04 +0000 (UTC) (envelope-from julian@elischer.org) Received: from shell.idiom.com (HELO idiom.com) (216.240.47.20) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Thu, 01 Feb 2007 10:03:24 -0800 Received: from [192.168.2.5] (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id D7840125B31; Thu, 1 Feb 2007 10:24:58 -0800 (PST) Message-ID: <45C2307A.8000206@elischer.org> Date: Thu, 01 Feb 2007 10:24:58 -0800 From: Julian Elischer User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: Luigi Rizzo References: <20070131115148.A60420@xorpc.icir.org> <200702011109.12821.jhb@freebsd.org> <20070201091605.A82313@xorpc.icir.org> In-Reply-To: <20070201091605.A82313@xorpc.icir.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 18:25:05 -0000 Luigi Rizzo wrote: > On Thu, Feb 01, 2007 at 11:09:12AM -0500, John Baldwin wrote: >> On Wednesday 31 January 2007 14:51, Luigi Rizzo wrote: > ... >>> We have several tables in our kernel and modules, >>> containing at least device-id tables (pci, usb), >>> quirks for certain devices, and maybe more (i am excluding > ... >> Windows and MAC OS X both use plain text files to hold things like PCI device >> IDs to match drivers to devices. OS X uses an XML file format that lets you > > plain text files! > > too obvious to think of it :) > > but, where can i find an example of a piece of kernel code that can > read from a file "safely" (i.e. say in the modevent handler or maybe > at device probe time) ? > Something like > > char *load_file_into_kernel_memody(filename, max_size, &error); > > I have looked at the kernel side of execve and kldload, they are not > exactly straightforward (at least there are seveal indirections). > Maybe there are other simpler ones ? > >> list the PCI ID's a driver supports and the kernel auto-loads driver modules >> by matching on PCI IDs. Many drivers can't be helped by this though, as they >> use the device ID for for device-specific behavior (such as em(4) or >> brgpy(4)). I like the idea of specifying aliases.. "if you see XXXX then pretend you saw YYYY and treat it the same." > > well, that's a two-phase process, if you have the above table-based > mechanism, the kernel (actually the bus driver i would say) can > load the [device_id -> driver] table (or tables, assuming > each driver has its own file) to select and load the > correct driver, and then the driver itself could use the same > mechanism to load a [device_id -> quirks] table for internal use ? > > cheers > luigi > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 18:45:12 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2B54916A403; Thu, 1 Feb 2007 18:45:12 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id CEFEA13C471; Thu, 1 Feb 2007 18:45:10 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l11Ij6LJ083328; Thu, 1 Feb 2007 10:45:06 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l11Ij6pS083327; Thu, 1 Feb 2007 10:45:06 -0800 (PST) (envelope-from rizzo) Date: Thu, 1 Feb 2007 10:45:06 -0800 From: Luigi Rizzo To: "M. Warner Losh" Message-ID: <20070201104506.B82313@xorpc.icir.org> References: <20070131115148.A60420@xorpc.icir.org> <200702011109.12821.jhb@freebsd.org> <20070201091605.A82313@xorpc.icir.org> <20070201.110206.1102529050.imp@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20070201.110206.1102529050.imp@bsdimp.com>; from imp@bsdimp.com on Thu, Feb 01, 2007 at 11:02:06AM -0700 Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 18:45:12 -0000 On Thu, Feb 01, 2007 at 11:02:06AM -0700, M. Warner Losh wrote: ... > : plain text files! > : > : too obvious to think of it :) > : > : but, where can i find an example of a piece of kernel code that can > : read from a file "safely" (i.e. say in the modevent handler or maybe > : at device probe time) ? > : Something like > : > : char *load_file_into_kernel_memody(filename, max_size, &error); > : > : I have looked at the kernel side of execve and kldload, they are not > : exactly straightforward (at least there are seveal indirections). > : Maybe there are other simpler ones ? > > Look at the firmware routines. However, they won't work until / is > mounted, which is after all the device probing happens. unfortunately firmare images are embedded in .ko files, so the loading is done elsewhere - but ok, i can spend some time figuring out what LINKER_LOAD_FILE() does and whether it is just plain loading of the file in memory or more than that, whether it can be made to work even with an unstructured file, and so on. Re. the availability of / - one of the requirements i had written was the ability to preload the table at compile time - that's the easy part, in the end it is just some macro/scripting magic to embed the initial table in the object. Short of putting into the table some hooks to give control to the console and ask the user to manually type in the 'alias ID' you were referring to (in the good old times maybe someone would have even conceived a 'please type the full driver image in hex') Surely it requires some form of rebuild of the kernel but there doesn't seem to be any other way to possibly solve the problem... cheers luigi From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 19:47:48 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7B5F116A405; Thu, 1 Feb 2007 19:47:48 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (vc4-2-0-87.dsl.netrack.net [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 1774F13C4BB; Thu, 1 Feb 2007 19:47:47 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.4/8.13.4) with ESMTP id l11Ji6m1063383; Thu, 1 Feb 2007 12:44:07 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Thu, 01 Feb 2007 12:44:37 -0700 (MST) Message-Id: <20070201.124437.1474621167.imp@bsdimp.com> To: rizzo@icir.org From: "M. Warner Losh" In-Reply-To: <20070201104506.B82313@xorpc.icir.org> References: <20070201091605.A82313@xorpc.icir.org> <20070201.110206.1102529050.imp@bsdimp.com> <20070201104506.B82313@xorpc.icir.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Thu, 01 Feb 2007 12:44:07 -0700 (MST) Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 19:47:48 -0000 In message: <20070201104506.B82313@xorpc.icir.org> Luigi Rizzo writes: : On Thu, Feb 01, 2007 at 11:02:06AM -0700, M. Warner Losh wrote: : ... : > : plain text files! : > : : > : too obvious to think of it :) : > : : > : but, where can i find an example of a piece of kernel code that can : > : read from a file "safely" (i.e. say in the modevent handler or maybe : > : at device probe time) ? : > : Something like : > : : > : char *load_file_into_kernel_memody(filename, max_size, &error); : > : : > : I have looked at the kernel side of execve and kldload, they are not : > : exactly straightforward (at least there are seveal indirections). : > : Maybe there are other simpler ones ? : > : > Look at the firmware routines. However, they won't work until / is : > mounted, which is after all the device probing happens. : : unfortunately firmare images are embedded in .ko files, so the loading : is done elsewhere - but ok, i can spend some time figuring out : what LINKER_LOAD_FILE() does and whether it is just plain loading : of the file in memory or more than that, whether it can be made to : work even with an unstructured file, and so on. I'd forgotten about that. : Re. the availability of / - one of the requirements i had written : was the ability to preload the table at compile time - that's the : easy part, in the end it is just some macro/scripting magic to embed : the initial table in the object. : Short of putting into the table some hooks to give control : to the console and ask the user to manually type in : the 'alias ID' you were referring to (in the good old times : maybe someone would have even conceived a 'please type : the full driver image in hex') I'm not sure that I follow you here. : Surely it requires some form of rebuild of the kernel : but there doesn't seem to be any other way to possibly solve : the problem... The more serious problem is the need for someone who has a moderate clue and a lot of time to do grunt work to go through the tree and regularize the pci and usb busses, plus provide some infrasturcture to allow this information to be exported (currently the pnpinfo provided by busses gives one the device info, the other side of the coin is what's needed). Until we have that, talking about anything other than simple aliasing isn't going to get very far. Warner From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 22:54:28 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 290CB16A40D for ; Thu, 1 Feb 2007 22:54:28 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.freebsd.org (Postfix) with ESMTP id AA3FA13C4B4 for ; Thu, 1 Feb 2007 22:54:27 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.6/8.13.6) with ESMTP id l11MsNRJ094270; Thu, 1 Feb 2007 17:54:25 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: Luigi Rizzo Date: Thu, 1 Feb 2007 16:13:02 -0500 User-Agent: KMail/1.9.1 References: <20070131115148.A60420@xorpc.icir.org> <200702011109.12821.jhb@freebsd.org> <20070201091605.A82313@xorpc.icir.org> In-Reply-To: <20070201091605.A82313@xorpc.icir.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200702011613.02809.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Thu, 01 Feb 2007 17:54:25 -0500 (EST) X-Virus-Scanned: ClamAV 0.88.3/2513/Thu Feb 1 11:48:53 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 22:54:28 -0000 On Thursday 01 February 2007 12:16, Luigi Rizzo wrote: > On Thu, Feb 01, 2007 at 11:09:12AM -0500, John Baldwin wrote: > > On Wednesday 31 January 2007 14:51, Luigi Rizzo wrote: > ... > > > We have several tables in our kernel and modules, > > > containing at least device-id tables (pci, usb), > > > quirks for certain devices, and maybe more (i am excluding > ... > > Windows and MAC OS X both use plain text files to hold things like PCI device > > IDs to match drivers to devices. OS X uses an XML file format that lets you > > plain text files! > > too obvious to think of it :) > > but, where can i find an example of a piece of kernel code that can > read from a file "safely" (i.e. say in the modevent handler or maybe > at device probe time) ? > Something like > > char *load_file_into_kernel_memody(filename, max_size, &error); > > I have looked at the kernel side of execve and kldload, they are not > exactly straightforward (at least there are seveal indirections). > Maybe there are other simpler ones ? I think this would be a bad idea (kernel loading files). I think instead you should depend on userland to hand stuff to the kernel. For example, if you want to autoload drivers, you should have some sort of database devd consults to find a driver to kldload to see if it attaches to the device. > > list the PCI ID's a driver supports and the kernel auto-loads driver modules > > by matching on PCI IDs. Many drivers can't be helped by this though, as they > > use the device ID for for device-specific behavior (such as em(4) or > > brgpy(4)). > > well, that's a two-phase process, if you have the above table-based > mechanism, the kernel (actually the bus driver i would say) can > load the [device_id -> driver] table (or tables, assuming > each driver has its own file) to select and load the > correct driver, and then the driver itself could use the same > mechanism to load a [device_id -> quirks] table for internal use ? You missed the much more glaring problem. Suppose a new device comes out, and you want to add support to it to an existing driver, so you add the PCI ID of the card and load the driver. Only, since there's no _code_ to handle the quirk (think about the case of a quirk that didn't exist before), then it doesn't work, or worse, it sort of works until it hoses a bunch of data. I really don't think this is a good idea. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 22:54:36 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6F07416A400 for ; Thu, 1 Feb 2007 22:54:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.freebsd.org (Postfix) with ESMTP id 00A6313C47E for ; Thu, 1 Feb 2007 22:54:35 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.6/8.13.6) with ESMTP id l11MsNRK094270; Thu, 1 Feb 2007 17:54:27 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: Luigi Rizzo Date: Thu, 1 Feb 2007 16:14:48 -0500 User-Agent: KMail/1.9.1 References: <20070131115148.A60420@xorpc.icir.org> <20070201.110206.1102529050.imp@bsdimp.com> <20070201104506.B82313@xorpc.icir.org> In-Reply-To: <20070201104506.B82313@xorpc.icir.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200702011614.48712.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Thu, 01 Feb 2007 17:54:28 -0500 (EST) X-Virus-Scanned: ClamAV 0.88.3/2513/Thu Feb 1 11:48:53 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 22:54:36 -0000 On Thursday 01 February 2007 13:45, Luigi Rizzo wrote: > On Thu, Feb 01, 2007 at 11:02:06AM -0700, M. Warner Losh wrote: > ... > > : plain text files! > > : > > : too obvious to think of it :) > > : > > : but, where can i find an example of a piece of kernel code that can > > : read from a file "safely" (i.e. say in the modevent handler or maybe > > : at device probe time) ? > > : Something like > > : > > : char *load_file_into_kernel_memody(filename, max_size, &error); > > : > > : I have looked at the kernel side of execve and kldload, they are not > > : exactly straightforward (at least there are seveal indirections). > > : Maybe there are other simpler ones ? > > > > Look at the firmware routines. However, they won't work until / is > > mounted, which is after all the device probing happens. > > unfortunately firmare images are embedded in .ko files, so the loading > is done elsewhere - but ok, i can spend some time figuring out > what LINKER_LOAD_FILE() does and whether it is just plain loading > of the file in memory or more than that, whether it can be made to > work even with an unstructured file, and so on. > > Re. the availability of / - one of the requirements i had written > was the ability to preload the table at compile time - that's the > easy part, in the end it is just some macro/scripting magic to embed > the initial table in the object. > Short of putting into the table some hooks to give control > to the console and ask the user to manually type in > the 'alias ID' you were referring to (in the good old times > maybe someone would have even conceived a 'please type > the full driver image in hex') Why not do what firmware(9) does? You can take a table and turn it into a module and then either load that via the loader or kldload it at runtime (or use linker_reference_module(9) at runtime). Granted, I still think the concept is not really a good idea, but I think firmware(9) is actually a good model for this sort of thing. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 23:01:05 2007 Return-Path: X-Original-To: freebsd-arch@FreeBSD.ORG Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4785716A405; Thu, 1 Feb 2007 23:01:05 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (vc4-2-0-87.dsl.netrack.net [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id DF86713C491; Thu, 1 Feb 2007 23:01:04 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.4/8.13.4) with ESMTP id l11MwbQ0065961; Thu, 1 Feb 2007 15:58:37 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Thu, 01 Feb 2007 15:59:08 -0700 (MST) Message-Id: <20070201.155908.-957832847.imp@bsdimp.com> To: jhb@FreeBSD.ORG From: "M. Warner Losh" In-Reply-To: <200702011614.48712.jhb@freebsd.org> References: <20070201.110206.1102529050.imp@bsdimp.com> <20070201104506.B82313@xorpc.icir.org> <200702011614.48712.jhb@freebsd.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Thu, 01 Feb 2007 15:58:37 -0700 (MST) Cc: freebsd-arch@FreeBSD.ORG Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 23:01:05 -0000 In message: <200702011614.48712.jhb@freebsd.org> John Baldwin writes: : On Thursday 01 February 2007 13:45, Luigi Rizzo wrote: : > On Thu, Feb 01, 2007 at 11:02:06AM -0700, M. Warner Losh wrote: : > ... : > > : plain text files! : > > : : > > : too obvious to think of it :) : > > : : > > : but, where can i find an example of a piece of kernel code that can : > > : read from a file "safely" (i.e. say in the modevent handler or maybe : > > : at device probe time) ? : > > : Something like : > > : : > > : char *load_file_into_kernel_memody(filename, max_size, &error); : > > : : > > : I have looked at the kernel side of execve and kldload, they are not : > > : exactly straightforward (at least there are seveal indirections). : > > : Maybe there are other simpler ones ? : > > : > > Look at the firmware routines. However, they won't work until / is : > > mounted, which is after all the device probing happens. : > : > unfortunately firmare images are embedded in .ko files, so the loading : > is done elsewhere - but ok, i can spend some time figuring out : > what LINKER_LOAD_FILE() does and whether it is just plain loading : > of the file in memory or more than that, whether it can be made to : > work even with an unstructured file, and so on. : > : > Re. the availability of / - one of the requirements i had written : > was the ability to preload the table at compile time - that's the : > easy part, in the end it is just some macro/scripting magic to embed : > the initial table in the object. : > Short of putting into the table some hooks to give control : > to the console and ask the user to manually type in : > the 'alias ID' you were referring to (in the good old times : > maybe someone would have even conceived a 'please type : > the full driver image in hex') : : Why not do what firmware(9) does? You can take a table and turn it into a : module and then either load that via the loader or kldload it at runtime (or : use linker_reference_module(9) at runtime). Granted, I still think the : concept is not really a good idea, but I think firmware(9) is actually a good : model for this sort of thing. Assuming that you have the infrastructure to produce the tables, then yes. See my other posts for the issues with that assumption Warner From owner-freebsd-arch@FreeBSD.ORG Thu Feb 1 23:07:36 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C3C0516A40D; Thu, 1 Feb 2007 23:07:36 +0000 (UTC) (envelope-from babkin@verizon.net) Received: from vms042pub.verizon.net (vms042pub.verizon.net [206.46.252.42]) by mx1.freebsd.org (Postfix) with ESMTP id AB67013C4AC; Thu, 1 Feb 2007 23:07:36 +0000 (UTC) (envelope-from babkin@verizon.net) Received: from vms227.mailsrvcs.net ([192.168.1.2]) by vms042.mailsrvcs.net (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPA id <0JCT00GNI48GPJ0B@vms042.mailsrvcs.net>; Thu, 01 Feb 2007 17:07:28 -0600 (CST) Received: from 208.253.138.194 ([208.253.138.194]) by vms227.mailsrvcs.net (Verizon Webmail) with HTTP; Thu, 01 Feb 2007 17:07:28 -0600 (CST) Date: Thu, 01 Feb 2007 17:07:28 -0600 (CST) From: Sergey Babkin X-Originating-IP: [208.253.138.194] To: Luigi Rizzo , John Baldwin Message-id: <25887312.2602951170371248250.JavaMail.root@vms227.mailsrvcs.net> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit Cc: freebsd-arch@freebsd.org Subject: Re: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: babkin@users.sf.net List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2007 23:07:36 -0000 >From: Luigi Rizzo >Date: 2007/02/01 Thu AM 11:16:05 CST >To: John Baldwin >Cc: freebsd-arch@freebsd.org >Subject: Re: configurable device (and other) tables in the kernel ? >On Thu, Feb 01, 2007 at 11:09:12AM -0500, John Baldwin wrote: >> On Wednesday 31 January 2007 14:51, Luigi Rizzo wrote: >... >> > We have several tables in our kernel and modules, >> > containing at least device-id tables (pci, usb), >> > quirks for certain devices, and maybe more (i am excluding >... >> Windows and MAC OS X both use plain text files to hold things like PCI device >> IDs to match drivers to devices. OS X uses an XML file format that lets you > >plain text files! > >too obvious to think of it :) They way UnixWare works is that it has plain text files with PCI ID info and such but this information doesn't go directly into the kernel. Instead an user-space process reads the PCI config info from the kernel and these tables, matches them up, and loads the drivers. Well, in reality things are a bit more complicated there: there is an in-kernel resource manager database that matches the devices (PCI and such are auto-detected, ISA are entered by scripts when building the kernel) with the appropriate drivers. This database is dumped to a file before reboot and then loaded back by the boot loader, so it's persistent between the boots. This user-space program goes through the entries in the database that don't have a driver assigned, tries to find the stuff in the plain text files, and if found then inserts the driver names into the appropriate database records. Than the kernel can take care of loading the modules. Now, sysctl looks like a very similar database to me, and we already have the boot loader configuration done through a similar way. Can it be reused for this purpose too? -SB From owner-freebsd-arch@FreeBSD.ORG Fri Feb 2 08:57:52 2007 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8E1D516A400 for ; Fri, 2 Feb 2007 08:57:52 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 1BBF213C467 for ; Fri, 2 Feb 2007 08:57:52 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id D353847661; Fri, 2 Feb 2007 03:25:20 -0500 (EST) Date: Fri, 2 Feb 2007 08:25:20 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: John Baldwin In-Reply-To: <200701311308.16810.jhb@freebsd.org> Message-ID: <20070202075421.A91177@fledge.watson.org> References: <20070128202917.5B67916A5A6@hub.freebsd.org> <200701301636.38175.jhb@freebsd.org> <20070130233238.GB94650@garage.freebsd.pl> <200701311308.16810.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org, Scott Long , Pawel Jakub Dawidek , Nate Lawson Subject: Re: cvs commit: src/sys/geom/eli g_eli.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Feb 2007 08:57:52 -0000 (commit mailing lists dropped, just on arch now) On Wed, 31 Jan 2007, John Baldwin wrote: > I wouldn't sell yourself so short actually. :) You're probably more > competent than you think. I'm not really an expert on SMP or schedulers > either (I've not had any formal training, and if you were to review some of > my earlier SMP stuff it shows), but ended up working on it because it needed > to be done. Most of what I've learned is by just working on stuff. > Anyways, maybe this issue will bubble up to someone's todo list at BSDCan to > settle how to handle CPU's arriving and departing. One big thing that > worries me is how to handle pinned and bound threads when a CPU goes away. > Also, it may be more useful to think of CPUs not as just present/not > present, but more in terms of: > > - not present > - present but offline > - present and online > > And most code would really only care about offline/online events. We could > maybe allow pinned and bound threads to still run on an offline CPU (and the > idlethread for that CPU of course) but require that there be no pinned bound > threads to completely detach a CPU (in the case of systems with removable > CPUs). It would be useful to at least handle taking CPUs offline and then > back online though. Well, I think bound threads are the easy case, as they involve an explicit CPU-use strategy for a kernel subsystem. Any subsystem using CPU binding must register with the event handler to provide a facility for unbinding (perhaps binding to another CPU, perhaps exiting). Depending on what the thread is doing, the event handler might have to sleep waiting for the thread to migrate by itself (part of a CPU rotation that will start skipping that CPU once it's removed from the list of available CPUs), or perhaps flag, signal, or otherwise take action before waiting. We should expect shutdown of bound services to potentially take "quite a while" -- i.e., wait on a disk I/O or sx lock to become available. Not so sure about the pinned case -- we may need to evaluate what sort of things are currently covered by pinning. Are they all "short sleep" activities, such as might be safe under a mutex, in which case perhaps we simply wait for the number of threads pinned on the CPU to hit zero, having decided no longer to schedule new threads there? Or do we need a more explicit strategy? Regardless, pinning is a strategy used by subsystems that do not own the thread they are running in, so can't use sched_bind(), etc. Pinning may, for example, be used by a subsystem invoked in a thread that is already bound, since pinning is similar to a critical section. Either way, it sounds like there might well be a multiphase shutdown on the order of the following: - The CPU is marked as "going down", so that new services starting no longer get assigned to the CPU, and any further events in the shutdown can know that the CPU is going away. - Eventhandler runs on the CPU to cancel bound CPU services -- things like per-CPU crypto or worker functions, which may take some time to shut down. Each subsystem prevents the eventhandler from returning until it is shut down, so this is a synchronous shutdown process. This would include making sure that device driver interrupts stop going to the CPU. - "Critical" subsystems, such as UMA, shut down. Since memory allocation is depended on by most other subsystems, shutting down UMA on the CPU needs to happen very close to the CPU going offline, perhaps as the last stage before the scheduler goes offline. - Scheduler shutdown, which arranges to make sure no further clock interrupts are disabled, that it is the last thing running, and a panic if there is anything bound to the CPU still. And a similar, but reversed, startup, in which UMA is started, the scheduler becomes happy, interrupts are brought up, it is marked as available for use, and event handlers are run so that subsystems can start pushing load to it by creating per-CPU workers (netisr, crypto, etc). Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-arch@FreeBSD.ORG Sat Feb 3 11:55:03 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F157F16A400 for ; Sat, 3 Feb 2007 11:55:03 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id B976013C442 for ; Sat, 3 Feb 2007 11:55:03 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 61FF846C4A; Sat, 3 Feb 2007 06:55:03 -0500 (EST) Date: Sat, 3 Feb 2007 11:55:03 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Luigi Rizzo In-Reply-To: <20070201104506.B82313@xorpc.icir.org> Message-ID: <20070203115235.W91177@fledge.watson.org> References: <20070131115148.A60420@xorpc.icir.org> <200702011109.12821.jhb@freebsd.org> <20070201091605.A82313@xorpc.icir.org> <20070201.110206.1102529050.imp@bsdimp.com> <20070201104506.B82313@xorpc.icir.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Feb 2007 11:55:04 -0000 On Thu, 1 Feb 2007, Luigi Rizzo wrote: >> Look at the firmware routines. However, they won't work until / is >> mounted, which is after all the device probing happens. > > unfortunately firmare images are embedded in .ko files, so the loading is > done elsewhere - but ok, i can spend some time figuring out what > LINKER_LOAD_FILE() does and whether it is just plain loading of the file in > memory or more than that, whether it can be made to work even with an > unstructured file, and so on. > > Re. the availability of / - one of the requirements i had written was the > ability to preload the table at compile time - that's the easy part, in the > end it is just some macro/scripting magic to embed the initial table in the > object. Short of putting into the table some hooks to give control to the > console and ask the user to manually type in the 'alias ID' you were > referring to (in the good old times maybe someone would have even conceived > a 'please type the full driver image in hex') > > Surely it requires some form of rebuild of the kernel but there doesn't seem > to be any other way to possibly solve the problem... The preferred way to make configuration frobs available during early boot is via the hints mechanism, which supports both loading data via the loader, compiling it into the kernel, and updating it using kenv(8). I'd really like us avoid adding yet more file access dependencies in the kernel. These tend to be fragile, can only run in certain contexts, run into issues with changing roots, etc. Could we add /boot/deviceids.hints to match /boot/device.hints? Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-arch@FreeBSD.ORG Sat Feb 3 12:32:52 2007 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B9C0116A407; Sat, 3 Feb 2007 12:32:52 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id 8BBAE13C428; Sat, 3 Feb 2007 12:32:52 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l13CWowg008697; Sat, 3 Feb 2007 04:32:50 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l13CWoEI008696; Sat, 3 Feb 2007 04:32:50 -0800 (PST) (envelope-from rizzo) Date: Sat, 3 Feb 2007 04:32:50 -0800 From: Luigi Rizzo To: Robert Watson Message-ID: <20070203043250.A8294@xorpc.icir.org> References: <20070131115148.A60420@xorpc.icir.org> <200702011109.12821.jhb@freebsd.org> <20070201.110206.1102529050.imp@bsdimp.com> <20070201104506.B82313@xorpc.icir.org> <20070203115235.W91177@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20070203115235.W91177@fledge.watson.org>; from rwatson@FreeBSD.org on Sat, Feb 03, 2007 at 11:55:03AM +0000 Cc: freebsd-arch@FreeBSD.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Feb 2007 12:32:52 -0000 On Sat, Feb 03, 2007 at 11:55:03AM +0000, Robert Watson wrote: ... > The preferred way to make configuration frobs available during early boot is > via the hints mechanism, which supports both loading data via the loader, > compiling it into the kernel, and updating it using kenv(8). I'd really like > us avoid adding yet more file access dependencies in the kernel. These tend > to be fragile, can only run in certain contexts, run into issues with changing > roots, etc. Could we add /boot/deviceids.hints to match /boot/device.hints? ok i was just asking for what the available options are. But in another private email i was mentioning that the bootloader "load" mechanism also already support loading opaque files and seems to pass the info to the kernel (preloaded_files ?), and this would overcome what i think is a limitation of the hints/kenv mechanism (more below). Following your principle (which i agree with) there would be no reason to have a separate the firmware(9) mechanism except that: + the hints/kenv/loader variables/resource mechanism has too many different names so people get confused on what it really is or can do; + documentation is also lacking a lot. E.g. "man -k hints" does not mention kenv(2), which in turn does not mention any of the resource_*(9) calls ("man -k resource" lists a few but not all of them, but you have to know in the first place that 'resource' and 'hint' are related, see previous point). + it seems to me that hints are only good at storing C strings i.e. single lines of plain text. There is no support for opaque binary data files. + the internal organization of hints is just a single list. Even if one forgets about binary data and tries to store some large tables (device ids, quirks etc) as multiple one-line name=value entries, the mechanism doesn't scale. All of the above can be fixed - especially the documentation part, but that doesn't mean that the hints mechanism can already do what i was asking; there is still a bit of work to do... cheers luigi From owner-freebsd-arch@FreeBSD.ORG Sat Feb 3 15:31:41 2007 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C05DF16A403; Sat, 3 Feb 2007 15:31:41 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (vc4-2-0-87.dsl.netrack.net [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 6210313C442; Sat, 3 Feb 2007 15:31:41 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.4/8.13.4) with ESMTP id l13FT9q0097115; Sat, 3 Feb 2007 08:29:14 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Sat, 03 Feb 2007 08:29:41 -0700 (MST) Message-Id: <20070203.082941.1649771040.imp@bsdimp.com> To: rizzo@icir.org From: "M. Warner Losh" In-Reply-To: <20070203043250.A8294@xorpc.icir.org> References: <20070201104506.B82313@xorpc.icir.org> <20070203115235.W91177@fledge.watson.org> <20070203043250.A8294@xorpc.icir.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Sat, 03 Feb 2007 08:29:14 -0700 (MST) Cc: rwatson@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Feb 2007 15:31:41 -0000 In message: <20070203043250.A8294@xorpc.icir.org> Luigi Rizzo writes: : On Sat, Feb 03, 2007 at 11:55:03AM +0000, Robert Watson wrote: : ... : > The preferred way to make configuration frobs available during early boot is : > via the hints mechanism, which supports both loading data via the loader, : > compiling it into the kernel, and updating it using kenv(8). I'd really like : > us avoid adding yet more file access dependencies in the kernel. These tend : > to be fragile, can only run in certain contexts, run into issues with changing : > roots, etc. Could we add /boot/deviceids.hints to match /boot/device.hints? : : ok i was just asking for what the available options are. : : But in another private email i was mentioning that the bootloader : "load" mechanism also already support loading opaque files : and seems to pass the info to the kernel (preloaded_files ?), : and this would overcome what i think is a limitation of the hints/kenv : mechanism (more below). : : Following your principle (which i agree with) there would be no reason to : have a separate the firmware(9) mechanism except that: : : + the hints/kenv/loader variables/resource mechanism has too many : different names so people get confused on what it really is or can do; : : + documentation is also lacking a lot. E.g. "man -k hints" does not : mention kenv(2), which in turn does not mention any of the : resource_*(9) calls ("man -k resource" lists a few but not all : of them, but you have to know in the first place that 'resource' : and 'hint' are related, see previous point). : : + it seems to me that hints are only good at storing C strings i.e. : single lines of plain text. There is no support for opaque binary : data files. : : + the internal organization of hints is just a single list. Even if : one forgets about binary data and tries to store some large tables : (device ids, quirks etc) as multiple one-line name=value entries, : the mechanism doesn't scale. : : All of the above can be fixed - especially the documentation part, : but that doesn't mean that the hints mechanism can already do : what i was asking; there is still a bit of work to do... True. Things could be better documented. However, the more fundamental problems remain. It does no good to have a fancy binary loader with every kind of plug and play info if few of the drivers in the system use tables in a consistant way. That's a big problem to solve... Warner From owner-freebsd-arch@FreeBSD.ORG Sat Feb 3 15:37:10 2007 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9FC2616A403; Sat, 3 Feb 2007 15:37:10 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id 87D7E13C491; Sat, 3 Feb 2007 15:37:10 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l13Fb9o5010493; Sat, 3 Feb 2007 07:37:09 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l13Fb9ln010492; Sat, 3 Feb 2007 07:37:09 -0800 (PST) (envelope-from rizzo) Date: Sat, 3 Feb 2007 07:37:09 -0800 From: Luigi Rizzo To: "M. Warner Losh" Message-ID: <20070203073709.B10325@xorpc.icir.org> References: <20070201104506.B82313@xorpc.icir.org> <20070203115235.W91177@fledge.watson.org> <20070203043250.A8294@xorpc.icir.org> <20070203.082941.1649771040.imp@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20070203.082941.1649771040.imp@bsdimp.com>; from imp@bsdimp.com on Sat, Feb 03, 2007 at 08:29:41AM -0700 Cc: rwatson@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Feb 2007 15:37:10 -0000 On Sat, Feb 03, 2007 at 08:29:41AM -0700, M. Warner Losh wrote: ... > : All of the above can be fixed - especially the documentation part, > : but that doesn't mean that the hints mechanism can already do > : what i was asking; there is still a bit of work to do... > > True. Things could be better documented. > > However, the more fundamental problems remain. It does no good to > have a fancy binary loader with every kind of plug and play info if > few of the drivers in the system use tables in a consistant way. > That's a big problem to solve... well but that also can be fixed. e.g. many of the usb and quirks tables, i think, are already reasonably structured so if a mechanism is in place it won't take much work to adapt them, maybe one at a time. cheers luigi From owner-freebsd-arch@FreeBSD.ORG Sat Feb 3 16:15:54 2007 Return-Path: X-Original-To: freebsd-arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 745DA16A400; Sat, 3 Feb 2007 16:15:54 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (vc4-2-0-87.dsl.netrack.net [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 310C213C467; Sat, 3 Feb 2007 16:15:54 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.4/8.13.4) with ESMTP id l13GFV42097491; Sat, 3 Feb 2007 09:15:31 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Sat, 03 Feb 2007 09:16:03 -0700 (MST) Message-Id: <20070203.091603.1136083763.imp@bsdimp.com> To: rizzo@icir.org From: "M. Warner Losh" In-Reply-To: <20070203073709.B10325@xorpc.icir.org> References: <20070203043250.A8294@xorpc.icir.org> <20070203.082941.1649771040.imp@bsdimp.com> <20070203073709.B10325@xorpc.icir.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Sat, 03 Feb 2007 09:15:31 -0700 (MST) Cc: rwatson@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Feb 2007 16:15:54 -0000 In message: <20070203073709.B10325@xorpc.icir.org> Luigi Rizzo writes: : On Sat, Feb 03, 2007 at 08:29:41AM -0700, M. Warner Losh wrote: : ... : > : All of the above can be fixed - especially the documentation part, : > : but that doesn't mean that the hints mechanism can already do : > : what i was asking; there is still a bit of work to do... : > : > True. Things could be better documented. : > : > However, the more fundamental problems remain. It does no good to : > have a fancy binary loader with every kind of plug and play info if : > few of the drivers in the system use tables in a consistant way. : > That's a big problem to solve... : : well but that also can be fixed. e.g. many of the usb and quirks : tables, i think, are already reasonably structured so if : a mechanism is in place it won't take much work to adapt : them, maybe one at a time. Quirks, yes. Device tables, no. I guess it depends on how generic you want it, and how universal. Warner From owner-freebsd-arch@FreeBSD.ORG Sat Feb 3 17:41:01 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BBFB916A403 for ; Sat, 3 Feb 2007 17:41:01 +0000 (UTC) (envelope-from sam@errno.com) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id 5592A13C4C5 for ; Sat, 3 Feb 2007 17:41:01 +0000 (UTC) (envelope-from sam@errno.com) Received: from [10.212.129.70] ([10.0.0.221]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id l13HEfFG014709 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 3 Feb 2007 09:14:41 -0800 (PST) (envelope-from sam@errno.com) Message-ID: <45C4C301.3040603@errno.com> Date: Sat, 03 Feb 2007 09:14:41 -0800 From: Sam Leffler Organization: Errno Consulting User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: Luigi Rizzo References: <20070131115148.A60420@xorpc.icir.org> <200702011109.12821.jhb@freebsd.org> <20070201.110206.1102529050.imp@bsdimp.com> <20070201104506.B82313@xorpc.icir.org> <20070203115235.W91177@fledge.watson.org> <20070203043250.A8294@xorpc.icir.org> In-Reply-To: <20070203043250.A8294@xorpc.icir.org> X-Enigmail-Version: 0.94.1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Robert Watson , freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Feb 2007 17:41:01 -0000 Luigi Rizzo wrote: > On Sat, Feb 03, 2007 at 11:55:03AM +0000, Robert Watson wrote: > ... >> The preferred way to make configuration frobs available during early boot is >> via the hints mechanism, which supports both loading data via the loader, >> compiling it into the kernel, and updating it using kenv(8). I'd really like >> us avoid adding yet more file access dependencies in the kernel. These tend >> to be fragile, can only run in certain contexts, run into issues with changing >> roots, etc. Could we add /boot/deviceids.hints to match /boot/device.hints? > > ok i was just asking for what the available options are. > > But in another private email i was mentioning that the bootloader > "load" mechanism also already support loading opaque files > and seems to pass the info to the kernel (preloaded_files ?), > and this would overcome what i think is a limitation of the hints/kenv > mechanism (more below). > > Following your principle (which i agree with) there would be no reason to > have a separate the firmware(9) mechanism except that: > > + the hints/kenv/loader variables/resource mechanism has too many > different names so people get confused on what it really is or can do; > > + documentation is also lacking a lot. E.g. "man -k hints" does not > mention kenv(2), which in turn does not mention any of the > resource_*(9) calls ("man -k resource" lists a few but not all > of them, but you have to know in the first place that 'resource' > and 'hint' are related, see previous point). > > + it seems to me that hints are only good at storing C strings i.e. > single lines of plain text. There is no support for opaque binary > data files. > > + the internal organization of hints is just a single list. Even if > one forgets about binary data and tries to store some large tables > (device ids, quirks etc) as multiple one-line name=value entries, > the mechanism doesn't scale. > > All of the above can be fixed - especially the documentation part, > but that doesn't mean that the hints mechanism can already do > what i was asking; there is still a bit of work to do... As I said to you privately, another motivation for firmware was to allow code in the firmware module to do interesting things like decode/decompress/etc. firmware data before handing it to the caller. Otherwise firmware data tends to have slightly different needs because it can be large. Sam From owner-freebsd-arch@FreeBSD.ORG Sat Feb 3 20:13:18 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C137516A401 for ; Sat, 3 Feb 2007 20:13:18 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.freebsd.org (Postfix) with ESMTP id 98E0613C428 for ; Sat, 3 Feb 2007 20:13:18 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (jj1llm1e7h2dmap2@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id l13Jw0h4097674; Sat, 3 Feb 2007 11:58:00 -0800 (PST) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id l13JvxdS097667; Sat, 3 Feb 2007 11:57:59 -0800 (PST) (envelope-from jmg) Date: Sat, 3 Feb 2007 11:57:59 -0800 From: John-Mark Gurney To: "M. Warner Losh" Message-ID: <20070203195759.GG779@funkthat.com> Mail-Followup-To: "M. Warner Losh" , rizzo@icir.org, freebsd-arch@freebsd.org References: <20070131115148.A60420@xorpc.icir.org> <200702011109.12821.jhb@freebsd.org> <20070201091605.A82313@xorpc.icir.org> <20070201.110206.1102529050.imp@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070201.110206.1102529050.imp@bsdimp.com> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Feb 2007 20:13:18 -0000 Warner Losh wrote this message on Thu, Feb 01, 2007 at 11:02 -0700: > On the one hand, I like the flexibility of having the ability to look > at a device and know what driver(s) may attach to it w/o having the > drivers in memory, since that allows us to move to a demand load > model for those people that want it. Right now, there's no way to > smartly extract the plug and play info used by the device probe > routines to bid on a device from the drivers to try to smartly load, > say, the atheros driver when an atheros cardbus card is inserted. Hmmm.. couldn't we do something like dedicate a section to a simple table that contains the PCI id's and related data to match.. Then we could have a tool that embeds the PCI id data from a plain text file and recreates the plain text file from the section... This doesn't solve the cases where the driver needs to handle a device specially, but w/ the text file, and the driver using the table, it'd be easy to do an on demand load pass... (and for new devices, users would be able to test drivers w/o having to recompile the module).. And as part of installation, it lists the modules that are necessary to boot in loader.conf... Just a thought... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Sat Feb 3 23:16:02 2007 Return-Path: X-Original-To: freebsd-arch@freebsd.org Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D509616A406 for ; Sat, 3 Feb 2007 23:16:02 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (vc4-2-0-87.dsl.netrack.net [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 9308B13C441 for ; Sat, 3 Feb 2007 23:16:02 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.4/8.13.4) with ESMTP id l13NDxaw001243; Sat, 3 Feb 2007 16:13:59 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Sat, 03 Feb 2007 16:14:31 -0700 (MST) Message-Id: <20070203.161431.-2001112364.imp@bsdimp.com> To: gurney_j@resnet.uoregon.edu From: "M. Warner Losh" In-Reply-To: <20070203195759.GG779@funkthat.com> References: <20070201091605.A82313@xorpc.icir.org> <20070201.110206.1102529050.imp@bsdimp.com> <20070203195759.GG779@funkthat.com> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Sat, 03 Feb 2007 16:14:00 -0700 (MST) Cc: freebsd-arch@freebsd.org Subject: Re: configurable device (and other) tables in the kernel ? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Feb 2007 23:16:02 -0000 In message: <20070203195759.GG779@funkthat.com> John-Mark Gurney writes: : Warner Losh wrote this message on Thu, Feb 01, 2007 at 11:02 -0700: : > On the one hand, I like the flexibility of having the ability to look : > at a device and know what driver(s) may attach to it w/o having the : > drivers in memory, since that allows us to move to a demand load : > model for those people that want it. Right now, there's no way to : > smartly extract the plug and play info used by the device probe : > routines to bid on a device from the drivers to try to smartly load, : > say, the atheros driver when an atheros cardbus card is inserted. : : Hmmm.. couldn't we do something like dedicate a section to a simple : table that contains the PCI id's and related data to match.. Then : we could have a tool that embeds the PCI id data from a plain text : file and recreates the plain text file from the section... : : This doesn't solve the cases where the driver needs to handle a device : specially, but w/ the text file, and the driver using the table, it'd : be easy to do an on demand load pass... (and for new devices, users : would be able to test drivers w/o having to recompile the module).. : : And as part of installation, it lists the modules that are necessary : to boot in loader.conf... You could do all these things... However, somebody has to take it upon themselves to spearhead fixing all the drivers to use the new infrastructure. The higher level 'how to pass the table' in is the interesting problem. Until we have more drivers using tables, however, the uninteresting problem of driver conversion remains a show stopper. Warner