From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 00:41:05 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 495A9F51 for ; Sun, 22 Mar 2015 00:41:05 +0000 (UTC) Received: from mail-pd0-x233.google.com (mail-pd0-x233.google.com [IPv6:2607:f8b0:400e:c02::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1869112D for ; Sun, 22 Mar 2015 00:41:05 +0000 (UTC) Received: by pdbop1 with SMTP id op1so146893358pdb.2 for ; Sat, 21 Mar 2015 17:41:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=fZ+Hn326gOXaqv0b+XjKNtClvSe3RJ1w9Ur9UCKnEvY=; b=JUOLfmuIaVxQZT5WCAfrqm3dfmAGQhjt4ewT+cXmbttylmn+qUeMMYzFbBREr3Vn0Q Ui79ZWCTZjAeV58zMx3ncozPDrX53yfAc/ukXJY5Ep1evp2KT419qgKuNlBy2icw9Iaq ccS8MUqQeT8HwB97FzUo0fmXvF90Wf07MQprjOR2wAUDlaZnsGG5lUx0ygG38DF+S193 6CI60X3/tzJRbbfYE5l9PBWR/VuQPKYacFp1O5gG8r73l5JPeemRC6iSZfCs6dtxJQ/x p6prVwYM4RXaqAehn31SoC5Y91uhbxO6ODcfw83zpBs2kU62eUjE47b1T5LlPLQuIAJs ejBg== X-Received: by 10.70.125.129 with SMTP id mq1mr199617674pdb.70.1426984864569; Sat, 21 Mar 2015 17:41:04 -0700 (PDT) MIME-Version: 1.0 Received: by 10.66.179.203 with HTTP; Sat, 21 Mar 2015 17:40:34 -0700 (PDT) From: Yue Chen Date: Sat, 21 Mar 2015 20:40:34 -0400 Message-ID: Subject: How to traverse all resident pages in kernel? To: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 00:41:05 -0000 Hi all, For research purposes, I am trying to get all the virtual memory starting address of every page in kernel. First I tried to traverse vm_map_entry and its sub-entries to get the address range. However, not all the pages are present between the ``start'' and the ``end''. (struct vm_map_entry *node, node->start, node->end). Then I tried to get the virtual address of each page by vm_object and vm_page, but it seems nothing describing the page virtual address in these structures. Is there any way to get all the resident pages' virtual addresses in kernel? Best regards, Yue From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 09:20:11 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5CD9117E for ; Sun, 22 Mar 2015 09:20:11 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id A8D5D6E6 for ; Sun, 22 Mar 2015 09:20:10 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygCniBU9iQ5V_MnCAw--.41901S2; Sun, 22 Mar 2015 17:20:04 +0800 (CST) Date: Sun, 22 Mar 2015 17:19:40 +0800 From: Tiwei Bie To: Mateusz Guzik Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150322091853.GA89976@freebsd> References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150321200500.GC14650@dft-labs.eu> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygCniBU9iQ5V_MnCAw--.41901S2 X-Coremail-Antispam: 1UD129KBjvJXoWxWryUCF47JF4xJr47Jr1UZFb_yoWrXFW8pF 98uFW5JrZ8Wr43Zrsav3y5ZFyFyr4kKr4Uu39xAr4ayr10qryrXr1rGryF9Fs7Wrs2gFn5 Za15Xry7K3yUZFJanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUyEb7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Xr0_Ar1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4 A2jsIEc7CjxVAFwI0_GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IE w4CE5I8CrVC2j2WlYx0E2Ix0cI8IcVAFwI0_JrI_JrylYx0Ex4A2jsIE14v26r4j6F4UMc vjeVCFs4IE7xkEbVWUJVW8JwACjcxG0xvEwIxGrwCY02Avz4vE14v_GFWl42xK82IYc2Ij 64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x 8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1Y6r17MIIYrxkI7VAKI48JMIIF0xvE 2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r1j6r4UMIIF0xvE42 xK8VAvwI8IcIk0rVWrZr1j6s0DMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIE c7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU5t8n7UUUUU== X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUQAVQhl-peFAAAsH Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 09:20:11 -0000 Sorry, I introduced a bug... allproc_lock could not be used to protect the access to corefilename[]. Because, sysctl_kern_corefile() could be called very early: static void sysctl_register_all(void *arg) { struct sysctl_oid **oidp; sx_init(&sysctlmemlock, "sysctl mem"); SYSCTL_INIT(); SYSCTL_XLOCK(); SET_FOREACH(oidp, sysctl_set) sysctl_register_oid(*oidp); SYSCTL_XUNLOCK(); } SYSINIT(sysctl, SI_SUB_KMEM, SI_ORDER_FIRST, sysctl_register_all, 0); That is to say, when the tunable `kern.corefile' is set in loader.conf, sysctl_kern_corefile() will be called as the priority of (SI_SUB_KMEM, SI_ORDER_FIRST). At this time, allproc_lock is not initialized. allproc_lock is initialized as the priority of (SI_SUB_INTRINSIC, SI_ORDER_FIRST): void procinit() { ...... sx_init(&allproc_lock, "allproc"); ...... } static void proc0_init(void *dummy __unused) { ...... procinit(); ...... } SYSINIT(p0init, SI_SUB_INTRINSIC, SI_ORDER_FIRST, proc0_init, NULL); Sorry... I couldn't find a proper existing lock for this task. Maybe a dedicated lock needs to be created. And initialize it together with sysctlmemlock: static void sysctl_register_all(void *arg) { struct sysctl_oid **oidp; sx_init(&sysctlmemlock, "sysctl mem"); SYSCTL_INIT(); SYSCTL_XLOCK(); SET_FOREACH(oidp, sysctl_set) sysctl_register_oid(*oidp); SYSCTL_XUNLOCK(); } Or maybe sysctlmemlock could be used, which is only acuqired when req.oldlen > PAGE_SIZE. On Sat, Mar 21, 2015 at 09:05:00PM +0100, Mateusz Guzik wrote: > On Sat, Mar 21, 2015 at 09:59:05PM +0800, Tiwei Bie wrote: > > Hi, Mateusz! > > > > I have finished the task: Validate coredump format string [1]. > > > > Following is my patch: > > > > --- > > sys/kern/kern_sig.c | 31 ++++++++++++++++++++++++++++--- > > 1 file changed, 28 insertions(+), 3 deletions(-) > > > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > > index 8410d9d..52f05be 100644 > > --- a/sys/kern/kern_sig.c > > +++ b/sys/kern/kern_sig.c > > @@ -3099,13 +3099,38 @@ static char corefilename[MAXPATHLEN] = {"%N.core"}; > > static int > > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > > { > > - int error; > > + char *format; > > + int i, error; > > + > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > > + > > + sx_slock(&corefilename_lock); > > + strncpy(format, corefilename, MAXPATHLEN); > > + sx_sunlock(&corefilename_lock); > > + > > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > > + if (error != 0 || strcmp(format, corefilename) == 0) > > + goto out; > > + > > + for (i = 0; format[i] != '\0'; i++) { > > + if (format[i] == '%') { > > + char ch = format[++i]; > > + if (ch != '%' && ch != 'H' && ch != 'I' && > > + ch != 'N' && ch != 'P' && ch != 'U') { > > + error = EINVAL; > > + printf("Unknown format character %c in " > > + "corename `%s'\n", ch, format); > > + goto out; > > + } > > + } > > + } > > Code traversing the string uses 'switch'. Any reason to deviate from that? > > It also uses log(LOG_ERR,) so why is printf is used here? > > corefilename can be also set with a bootloader tunable, so we have to > validate what is being passed there and possibly reject it. > > When we know that the string we have set in corefilename is valid, there > is no reason to have aforementioned log() in corefile_open(). > > As a side note 'I' more than once in the format is not really supported, > so I would check for that too. > > > > > sx_xlock(&corefilename_lock); > > - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), > > - req); > > + strncpy(corefilename, format, sizeof(corefilename)); > > sx_xunlock(&corefilename_lock); > > > > +out: > > + free(format, M_TEMP); > > return (error); > > } > > SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | > > -- > > 2.1.2 > > > > [1] https://wiki.freebsd.org/JuniorJobs#Validate_coredump_format_string > > > > Best regards, > > Tiwei Bie > > > > -- > Mateusz Guzik From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 10:14:07 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5D8B4FD5 for ; Sun, 22 Mar 2015 10:14:07 +0000 (UTC) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DDD9AC3E for ; Sun, 22 Mar 2015 10:14:06 +0000 (UTC) Received: by wixw10 with SMTP id w10so31946825wix.0 for ; Sun, 22 Mar 2015 03:14:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=enmNX/pC0tvXxC/D7TESbcanoyus2UR7Do5U0E8QlaA=; b=CGzH8Gn25JdC6PaVZ6YnBTSqbNAXnfAHZrSLM0PqLV4xsWztFclQnkxBk1pSwH0Yov nZqoo+orXGMksTDz+YPmOVvvtjP/+4XeT1iv+gwCqq+9awOMowUGq4KK/5ABvAeZBdm6 bMsmxInl/OrpMiv4L/k2crEXiUmyv08PEba1DIrbUE4uGF+sGqkdhyr5GRu9Q9JI0/c+ bTNvMs2huso9VyONLaluPUiiJ7wvXV2RPwil0GiwmmshakZNd9DOJaX17Dn2jFggD5Mf 0MQ3VrCOJESuH/ssfc1Bm9DJJ+LZWkS5C6IXICwSLTRg1rr7/ebyim/6Bznym8SiKCxe xPqA== X-Received: by 10.180.24.162 with SMTP id v2mr10211097wif.80.1427019245330; Sun, 22 Mar 2015 03:14:05 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id ub1sm14233711wjc.43.2015.03.22.03.14.03 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Sun, 22 Mar 2015 03:14:03 -0700 (PDT) Date: Sun, 22 Mar 2015 11:14:01 +0100 From: Mateusz Guzik To: Tiwei Bie Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150322101401.GH14650@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Tiwei Bie , freebsd-hackers@freebsd.org, Konstantin Belousov References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150322091853.GA89976@freebsd> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 10:14:07 -0000 On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > Sorry, I introduced a bug... allproc_lock could not be used to protect > the access to corefilename[]. > First off I committed the code, so the fault is on me. > Because, sysctl_kern_corefile() could be called very early: > [..] > That is to say, when the tunable `kern.corefile' is set in loader.conf, > sysctl_kern_corefile() will be called as the priority of (SI_SUB_KMEM, > SI_ORDER_FIRST). > > At this time, allproc_lock is not initialized. > > I couldn't find a proper existing lock for this task. Maybe a dedicated > lock needs to be created. And initialize it together with sysctlmemlock: > [..] > Or maybe sysctlmemlock could be used, which is only acuqired when > req.oldlen > PAGE_SIZE. > > I was somehow convinced that tunables are dealt with other code. If such sysctl handler is also called for tunables, the kernel should pass a flag or some other indicator so that the function knows it is dealing with a tunable and that would avoid locking and thus solve the problem. I'm wondering if we should go a little bit further and get rid of static char corefilename[MAXPATHLEN] and have a static char *corefilename instead. A dedicated sysinit func could fetch and validate the tunable, if any. If no tunable was provided it would alloc memory for the default. -- Mateusz Guzik From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 10:24:34 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6129936D for ; Sun, 22 Mar 2015 10:24:34 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C4ADBD42 for ; Sun, 22 Mar 2015 10:24:33 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t2MAOSkP029197 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 22 Mar 2015 12:24:28 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t2MAOSkP029197 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t2MAOSEc029196; Sun, 22 Mar 2015 12:24:28 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 22 Mar 2015 12:24:28 +0200 From: Konstantin Belousov To: Mateusz Guzik , Tiwei Bie , freebsd-hackers@freebsd.org Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150322102428.GZ2379@kib.kiev.ua> References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150322101401.GH14650@dft-labs.eu> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 10:24:34 -0000 On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > > Sorry, I introduced a bug... allproc_lock could not be used to protect > > the access to corefilename[]. > > > > First off I committed the code, so the fault is on me. > > > Because, sysctl_kern_corefile() could be called very early: > > > [..] > > That is to say, when the tunable `kern.corefile' is set in loader.conf, > > sysctl_kern_corefile() will be called as the priority of (SI_SUB_KMEM, > > SI_ORDER_FIRST). > > > > At this time, allproc_lock is not initialized. > > > > I couldn't find a proper existing lock for this task. Maybe a dedicated > > lock needs to be created. And initialize it together with sysctlmemlock: > > > [..] > > Or maybe sysctlmemlock could be used, which is only acuqired when > > req.oldlen > PAGE_SIZE. > > > > > > I was somehow convinced that tunables are dealt with other code. > > If such sysctl handler is also called for tunables, the kernel should > pass a flag or some other indicator so that the function knows it is > dealing with a tunable and that would avoid locking and thus solve the > problem. > > I'm wondering if we should go a little bit further and get rid of > static char corefilename[MAXPATHLEN] > > and have a static char *corefilename instead. Accessing the array through the pointer dereference is micro-pessimization, as well as having to maintain metadata for the malloced memory, isn't it ? > > A dedicated sysinit func could fetch and validate the tunable, if any. > If no tunable was provided it would alloc memory for the default. Or you could move initialization of the sx in question earlier. From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 10:27:24 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EE256533 for ; Sun, 22 Mar 2015 10:27:23 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id 061FAD5D for ; Sun, 22 Mar 2015 10:27:22 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygBXCD38mA5VbvfCAw--.23144S2; Sun, 22 Mar 2015 18:27:13 +0800 (CST) Date: Sun, 22 Mar 2015 18:26:50 +0800 From: Tiwei Bie To: Mateusz Guzik Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150322102650.GA14629@freebsd> References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150322101401.GH14650@dft-labs.eu> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygBXCD38mA5VbvfCAw--.23144S2 X-Coremail-Antispam: 1UD129KBjvJXoW7tF17XryDtr45Cr18tr1DZFb_yoW8tr15p3 y8Wry0yr4Dtw1Uua4xAw1UGa4rtw4ktFWjyrn8G398Zw17WryrXr4vqF1FqFZFgr4Sg3yS va18Was093yYva7anT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUyEb7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Xr0_Ar1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4 A2jsIEc7CjxVAFwI0_GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IE w4CE5I8CrVC2j2WlYx0E2Ix0cI8IcVAFwI0_JrI_JrylYx0Ex4A2jsIE14v26r4j6F4UMc vjeVCFs4IE7xkEbVWUJVW8JwACjcxG0xvEwIxGrwCY02Avz4vE14v_Gw1l42xK82IYc2Ij 64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x 8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1Y6r17MIIYrxkI7VAKI48JMIIF0xvE 2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r1j6r4UMIIF0xvE42 xK8VAvwI8IcIk0rVWrZr1j6s0DMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIE c7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU5QyCtUUUUU== X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUQAVQhl-peFAAFsC Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 10:27:24 -0000 On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > [..] > > I was somehow convinced that tunables are dealt with other code. > > If such sysctl handler is also called for tunables, the kernel should > pass a flag or some other indicator so that the function knows it is > dealing with a tunable and that would avoid locking and thus solve the > problem. > If a sysctl is registered as a tunable (CTLFLAG_TUN), when calling sysctl_register_oid(), sysctl_load_tunable_by_oid_locked() will be called to fetch the kenv setting (i.e. tunable setting defined in loader.conf) to update the sysctl setting: void sysctl_register_oid(struct sysctl_oid *oidp) { ...... if ((oidp->oid_kind & CTLTYPE) != CTLTYPE_NODE && #ifdef VIMAGE (oidp->oid_kind & CTLFLAG_VNET) == 0 && #endif (oidp->oid_kind & CTLFLAG_TUN) != 0 && (oidp->oid_kind & CTLFLAG_NOFETCH) == 0) { sysctl_load_tunable_by_oid_locked(oidp); } } sysctl_load_tunable_by_oid_locked() will fetch the kenv setting, and call the sysctl handler, e.g. sysctl_kern_corefile() to update the sysctl setting: static void sysctl_load_tunable_by_oid_locked(struct sysctl_oid *oidp) { ...... case CTLTYPE_STRING: penv = kern_getenv(path + rem); if (penv == NULL) return; req.newlen = strlen(penv); req.newptr = penv; break; default: return; } error = sysctl_root_handler_locked(oidp, oidp->oid_arg1, oidp->oid_arg2, &req); if (error != 0) printf("Setting sysctl %s failed: %d\n", path, error); ...... } The tunable setting is just a "name"=value format in kenv. And the kenv codes have no idea of the meaning of the "name". It can also be modified later when system is up by `kenv' command. And the corresponding sysctl setting won't be affected any more. > I'm wondering if we should go a little bit further and get rid of > static char corefilename[MAXPATHLEN] > > and have a static char *corefilename instead. > > A dedicated sysinit func could fetch and validate the tunable, if any. > If no tunable was provided it would alloc memory for the default. > > -- > Mateusz Guzik Best regards, Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 11:26:25 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 10F3E378 for ; Sun, 22 Mar 2015 11:26:25 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id 085712DD for ; Sun, 22 Mar 2015 11:26:23 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygAX+hbVpg5VeiHDAw--.49826S2; Sun, 22 Mar 2015 19:26:20 +0800 (CST) Date: Sun, 22 Mar 2015 19:25:55 +0800 From: Tiwei Bie To: Mateusz Guzik Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150322112555.GA44277@freebsd> References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150322101401.GH14650@dft-labs.eu> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygAX+hbVpg5VeiHDAw--.49826S2 X-Coremail-Antispam: 1UD129KBjvJXoWxJw4kArW5urW3GF48ZF13urg_yoW5Jw4UpF yrCr95ArsYkr43CFn3Z3y8A3WYywn5A3yUW347XrnxCryFgryUXr1Fgr1FvF1kGr1vqas8 Ja15WFy7KryUZ3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUk0b7Iv0xC_tr1lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Ar0_tr1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Cr0_Gr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I 8E87Iv6xkF7I0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI 64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r106r15McIj6I8E87Iv67AKxVW8JVWxJw Am72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IY64vIr41lc2xSY4AK67AK6r4kMxAIw28IcxkI 7VAKI48JMxC20s026xCaFVCjc4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxV Cjr7xvwVAFwI0_JrI_JrWlx4CE17CEb7AF67AKxVWUXVWUAwCIc40Y0x0EwIxGrwCI42IY 6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVWUJVW8JwCI42IY6x AIw20EY4v20xvaj40_Wr1j6rW3Jr1lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280 aVCY1x0267AKxVWUJVW8JbIYCTnIWIevJa73UjIFyTuYvjxU4NtxDUUUU X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUQAVQhl-peFAAIsP Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 11:26:25 -0000 On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: [..] > > A dedicated sysinit func could fetch and validate the tunable, if any. > If no tunable was provided it would alloc memory for the default. > My new patch, that uses a dedicated sysinit func to fetch and validate the tunable: --- sys/kern/kern_sig.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c index 8410d9d..e2a00ba 100644 --- a/sys/kern/kern_sig.c +++ b/sys/kern/kern_sig.c @@ -3096,19 +3096,77 @@ static int compress_user_cores = 0; static char corefilename[MAXPATHLEN] = {"%N.core"}; +static bool +corefilename_check(const char *format) +{ + int i, counti; + + counti = 0; + for (i = 0; format[i] != '\0'; i++) { + if (format[i] == '%') { + i++; + switch (format[i]) { + case 'I': + counti++; + if (counti > 1) { + printf("Too many index flags specified " + "in corename `%s'\n", format); + return (false); + } + case '%': + case 'H': + case 'N': + case 'P': + case 'U': + break; + default: + printf("Unknown format character %c in " + "corename `%s'\n", format[i], format); + return (false); + } + } + } + + return (true); +} + +static void +corefilename_init(void *arg) +{ + char *format = kern_getenv("kern.corefile"); + + if (format != NULL && corefilename_check(format)) + strncpy(corefilename, format, sizeof(corefilename)); +} +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); + static int sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) { + char *format; int error; + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); + + sx_slock(&corefilename_lock); + strncpy(format, corefilename, MAXPATHLEN); + sx_sunlock(&corefilename_lock); + + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); + if (error != 0 || strcmp(format, corefilename) == 0 || + !corefilename_check(format)) + goto out; + sx_xlock(&corefilename_lock); error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), req); sx_xunlock(&corefilename_lock); +out: + free(format, M_TEMP); return (error); } -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", "Process corefile name format string"); -- 2.1.2 Best regards, Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 11:38:30 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 442994B2 for ; Sun, 22 Mar 2015 11:38:30 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C08073C5 for ; Sun, 22 Mar 2015 11:38:29 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t2MBcMv7046313 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 22 Mar 2015 13:38:22 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t2MBcMv7046313 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t2MBcMth046312; Sun, 22 Mar 2015 13:38:22 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 22 Mar 2015 13:38:22 +0200 From: Konstantin Belousov To: Tiwei Bie Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150322113822.GB2379@kib.kiev.ua> References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150322112555.GA44277@freebsd> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-hackers@freebsd.org, Mateusz Guzik X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 11:38:30 -0000 On Sun, Mar 22, 2015 at 07:25:55PM +0800, Tiwei Bie wrote: > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > [..] > > > > A dedicated sysinit func could fetch and validate the tunable, if any. > > If no tunable was provided it would alloc memory for the default. > > > > My new patch, that uses a dedicated sysinit func to fetch and validate > the tunable: > > --- > sys/kern/kern_sig.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 59 insertions(+), 1 deletion(-) > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > index 8410d9d..e2a00ba 100644 > --- a/sys/kern/kern_sig.c > +++ b/sys/kern/kern_sig.c > @@ -3096,19 +3096,77 @@ static int compress_user_cores = 0; > > static char corefilename[MAXPATHLEN] = {"%N.core"}; Are {} around string literal needed ? > > +static bool > +corefilename_check(const char *format) > +{ > + int i, counti; > + > + counti = 0; > + for (i = 0; format[i] != '\0'; i++) { > + if (format[i] == '%') { > + i++; > + switch (format[i]) { > + case 'I': > + counti++; > + if (counti > 1) { > + printf("Too many index flags specified " > + "in corename `%s'\n", format); I very much dislike the kernel lecturing the user about things. Kernel must silently return an error code, with man pages providing explanation of errors. > + return (false); > + } > + case '%': > + case 'H': > + case 'N': > + case 'P': > + case 'U': > + break; > + default: > + printf("Unknown format character %c in " > + "corename `%s'\n", format[i], format); Same there. > + return (false); > + } > + } > + } > + > + return (true); > +} > + > +static void > +corefilename_init(void *arg) > +{ > + char *format = kern_getenv("kern.corefile"); Do not use initialization at the local variable declaration. It is a requirement of style(9). > + > + if (format != NULL && corefilename_check(format)) > + strncpy(corefilename, format, sizeof(corefilename)); Use of strncpy() is almost always an indication of the bug. Is it acceptable for corefilename be non-null terminated ? > +} > +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); > + > static int > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > { > + char *format; > int error; > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > + > + sx_slock(&corefilename_lock); > + strncpy(format, corefilename, MAXPATHLEN); > + sx_sunlock(&corefilename_lock); > + > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > + if (error != 0 || strcmp(format, corefilename) == 0 || > + !corefilename_check(format)) > + goto out; This code somehow misses the check for req->newptr. Did you verified that simply fetching current value for kern.corefile works after the patch ? > + > sx_xlock(&corefilename_lock); > error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), > req); > sx_xunlock(&corefilename_lock); > > +out: > + free(format, M_TEMP); > return (error); > } > -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | > +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | > CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", > "Process corefile name format string"); > > -- > 2.1.2 > > Best regards, > Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 11:48:13 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B2BF7603 for ; Sun, 22 Mar 2015 11:48:13 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id C2C7867D for ; Sun, 22 Mar 2015 11:48:11 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygA3FxT0qw5VcTHDAw--.32466S2; Sun, 22 Mar 2015 19:48:08 +0800 (CST) Date: Sun, 22 Mar 2015 19:47:46 +0800 From: Tiwei Bie To: Mateusz Guzik Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150322114746.GA53624@freebsd> References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150322112555.GA44277@freebsd> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygA3FxT0qw5VcTHDAw--.32466S2 X-Coremail-Antispam: 1UD129KBjvJXoWxJw4kJr43ArW8Kr15tF4rGrg_yoW5ury8pF yrCr98ArZYkr43CFn3Z3yrAa4Yy3s5A3yUW343Xw1akryFgry8Xr1rKr1FvF1kGr4vqasx Ja15WFy7Kryjv3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUv2b7Iv0xC_tr1lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Xr0_Ar1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Cr0_Gr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I 8E87Iv6xkF7I0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI 64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVW8JVWxJw Am72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IY64vIr41lc2xSY4AK67AK6r4kMxAIw28IcxkI 7VAKI48JMxC20s026xCaFVCjc4AY6r1j6r4UMxCIbckI1I0E14v26r1Y6r17MI8I3I0E5I 8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4CE17CEb7AF67AKxVWUXVWU AwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x 0267AKxVWUJVW8JwCI42IY6xAIw20EY4v20xvaj40_Wr1j6rW3Jr1lIxAIcVC2z280aVAF wI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa 7IU0co7tUUUUU== X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUQAVQhl-peFAAMsL Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 11:48:13 -0000 On Sun, Mar 22, 2015 at 07:25:55PM +0800, Tiwei Bie wrote: > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > [..] > > > > A dedicated sysinit func could fetch and validate the tunable, if any. > > If no tunable was provided it would alloc memory for the default. > > > > My new patch, that uses a dedicated sysinit func to fetch and validate > the tunable: > [..] > static int > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > { > + char *format; > int error; > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > + > + sx_slock(&corefilename_lock); > + strncpy(format, corefilename, MAXPATHLEN); > + sx_sunlock(&corefilename_lock); > + > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > + if (error != 0 || strcmp(format, corefilename) == 0 || > + !corefilename_check(format)) > + goto out; > + > sx_xlock(&corefilename_lock); > error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), > req); Sorry, I was too hurried. ;-( Fix typo: --- sys/kern/kern_sig.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 60 insertions(+), 3 deletions(-) diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c index 8410d9d..1ade17f 100644 --- a/sys/kern/kern_sig.c +++ b/sys/kern/kern_sig.c @@ -3096,19 +3096,76 @@ static int compress_user_cores = 0; static char corefilename[MAXPATHLEN] = {"%N.core"}; +static bool +corefilename_check(const char *format) +{ + int i, counti; + + counti = 0; + for (i = 0; format[i] != '\0'; i++) { + if (format[i] == '%') { + i++; + switch (format[i]) { + case 'I': + counti++; + if (counti > 1) { + printf("Too many index flags specified " + "in corename `%s'\n", format); + return (false); + } + case '%': + case 'H': + case 'N': + case 'P': + case 'U': + break; + default: + printf("Unknown format character %c in " + "corename `%s'\n", format[i], format); + return (false); + } + } + } + + return (true); +} + +static void +corefilename_init(void *arg) +{ + char *format = kern_getenv("kern.corefile"); + + if (format != NULL && corefilename_check(format)) + strncpy(corefilename, format, sizeof(corefilename)); +} +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); + static int sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) { + char *format; int error; + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); + + sx_slock(&corefilename_lock); + strncpy(format, corefilename, MAXPATHLEN); + sx_sunlock(&corefilename_lock); + + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); + if (error != 0 || strcmp(format, corefilename) == 0 || + !corefilename_check(format)) + goto out; + sx_xlock(&corefilename_lock); - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), - req); + strncpy(corefilename, format, sizeof(corefilename)); sx_xunlock(&corefilename_lock); +out: + free(format, M_TEMP); return (error); } -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", "Process corefile name format string"); -- 2.1.2 Best regards, Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 12:07:24 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BA9D8C6D for ; Sun, 22 Mar 2015 12:07:24 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id DAA9D88C for ; Sun, 22 Mar 2015 12:07:23 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygCHaRZxsA5V_T7DAw--.37634S2; Sun, 22 Mar 2015 20:07:18 +0800 (CST) Date: Sun, 22 Mar 2015 20:06:55 +0800 From: Tiwei Bie To: Konstantin Belousov Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150322120655.GA59757@freebsd> References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> <20150322113822.GB2379@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150322113822.GB2379@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygCHaRZxsA5V_T7DAw--.37634S2 X-Coremail-Antispam: 1UD129KBjvJXoWxWrW8WF4rGF43tF4UGF4xCrg_yoW5ZrWxpF WFkrZYyF4YyrW3Cr1av3y8Aa4Fy3s5Jr45W347JFn8KF9Y9r1kJr1Fgw1F9F1kWr1vgwnF vay5WFy7K345Z3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUvFb7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Xr0_Ar1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4 A2jsIEc7CjxVAFwI0_GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IE w4CE5I8CrVC2j2WlYx0E2Ix0cI8IcVAFwI0_JrI_JrylYx0Ex4A2jsIE14v26r4j6F4UMc vjeVCFs4IE7xkEbVWUJVW8JwACjcxG0xvEwIxGrwCY02Avz4vE14v_Xr1l42xK82IYc2Ij 64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1l4IxYO2xFxVAFwI0_Jrv_JF1lx2IqxVAqx4 xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1Y6r17 MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I 0E14v26r1j6r4UMIIF0xvE42xK8VAvwI8IcIk0rVWrJr0_WFyUJwCI42IY6I8E87Iv67AK xVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvj xUy3l8UUUUU X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUQAVQhl-peFAAOsJ Cc: freebsd-hackers@freebsd.org, Mateusz Guzik X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 12:07:24 -0000 On Sun, Mar 22, 2015 at 01:38:22PM +0200, Konstantin Belousov wrote: > On Sun, Mar 22, 2015 at 07:25:55PM +0800, Tiwei Bie wrote: > > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > > [..] > > > > > > A dedicated sysinit func could fetch and validate the tunable, if any. > > > If no tunable was provided it would alloc memory for the default. > > > > > > > My new patch, that uses a dedicated sysinit func to fetch and validate > > the tunable: > > > > --- > > sys/kern/kern_sig.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++- > > 1 file changed, 59 insertions(+), 1 deletion(-) > > > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > > index 8410d9d..e2a00ba 100644 > > --- a/sys/kern/kern_sig.c > > +++ b/sys/kern/kern_sig.c > > @@ -3096,19 +3096,77 @@ static int compress_user_cores = 0; > > > > static char corefilename[MAXPATHLEN] = {"%N.core"}; > Are {} around string literal needed ? > It is not needed. But I didn't modify this line. > > > > +static bool > > +corefilename_check(const char *format) > > +{ > > + int i, counti; > > + > > + counti = 0; > > + for (i = 0; format[i] != '\0'; i++) { > > + if (format[i] == '%') { > > + i++; > > + switch (format[i]) { > > + case 'I': > > + counti++; > > + if (counti > 1) { > > + printf("Too many index flags specified " > > + "in corename `%s'\n", format); > I very much dislike the kernel lecturing the user about things. > Kernel must silently return an error code, with man pages providing > explanation of errors. > I will remove it. I was not sure about it. Thank you very much for telling me about this! > > + return (false); > > + } > > + case '%': > > + case 'H': > > + case 'N': > > + case 'P': > > + case 'U': > > + break; > > + default: > > + printf("Unknown format character %c in " > > + "corename `%s'\n", format[i], format); > Same there. > > > + return (false); > > + } > > + } > > + } > > + > > + return (true); > > +} > > + > > +static void > > +corefilename_init(void *arg) > > +{ > > + char *format = kern_getenv("kern.corefile"); > Do not use initialization at the local variable declaration. It is > a requirement of style(9). > Sorry... > > + > > + if (format != NULL && corefilename_check(format)) > > + strncpy(corefilename, format, sizeof(corefilename)); > Use of strncpy() is almost always an indication of the bug. Is it acceptable > for corefilename be non-null terminated ? > No, it is NOT. Sorry... > > +} > > +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); > > + > > static int > > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > > { > > + char *format; > > int error; > > > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > > + > > + sx_slock(&corefilename_lock); > > + strncpy(format, corefilename, MAXPATHLEN); > > + sx_sunlock(&corefilename_lock); > > + > > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > > + if (error != 0 || strcmp(format, corefilename) == 0 || > > + !corefilename_check(format)) > > + goto out; > This code somehow misses the check for req->newptr. Did you verified > that simply fetching current value for kern.corefile works after the > patch ? > I'm so sorry, I was too hurried. I will submit my new patch later. From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 13:15:53 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 68218A2E for ; Sun, 22 Mar 2015 13:15:53 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id 7C528EBA for ; Sun, 22 Mar 2015 13:15:51 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygB3Jzx_wA5Vxm7DAw--.49545S2; Sun, 22 Mar 2015 21:15:48 +0800 (CST) Date: Sun, 22 Mar 2015 21:15:24 +0800 From: Tiwei Bie To: Konstantin Belousov Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150322131524.GA95795@freebsd> References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> <20150322113822.GB2379@kib.kiev.ua> <20150322120655.GA59757@freebsd> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150322120655.GA59757@freebsd> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygB3Jzx_wA5Vxm7DAw--.49545S2 X-Coremail-Antispam: 1UD129KBjvJXoWxGFW5ZFWkCF4xJrWDuFW8Xrb_yoW5Ww4xpr WrCr95ArsakF43CFsav3yrZa4Yy3s5Jw4UW3y7XrsIkr1FgrykXr1rKr1FvF1kWrWvqr98 Jan8WFy7KrW7Z3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUkYb7Iv0xC_tr1lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Ar0_tr1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwV C2z280aVCY1x0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC 0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Gr0_Cr 1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0Y48IcVAKI48JMxkIecxEwVAFwVW5GwCF04k20xvY 0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I 0E7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jrv_JF1lIxkGc2Ij64vIr41lIxAI cVC0I7IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Jr0_Gr1lIxAIcV CF04k26cxKx2IYs7xG6rW3Jr0E3s1lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280 aVCY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7IU0DUUUUUUUU== X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUQAVQhl-peFAAQsX Cc: freebsd-hackers@freebsd.org, Mateusz Guzik X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 13:15:53 -0000 On Sun, Mar 22, 2015 at 08:06:55PM +0800, Tiwei Bie wrote: > On Sun, Mar 22, 2015 at 01:38:22PM +0200, Konstantin Belousov wrote: > > On Sun, Mar 22, 2015 at 07:25:55PM +0800, Tiwei Bie wrote: > > > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > [..] > I will submit my new patch later. > Here is my new patch: --- sys/kern/kern_sig.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 64 insertions(+), 9 deletions(-) diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c index 8410d9d..5162ab6 100644 --- a/sys/kern/kern_sig.c +++ b/sys/kern/kern_sig.c @@ -3094,21 +3094,81 @@ static int compress_user_cores = 0; */ #define corefilename_lock allproc_lock -static char corefilename[MAXPATHLEN] = {"%N.core"}; +static char corefilename[MAXPATHLEN] = "%N.core"; + +static bool +corefilename_check(const char *format) +{ + int i, counti; + + counti = 0; + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { + if (format[i] == '%') { + i++; + switch (format[i]) { + case 'I': + counti++; + if (counti > 1) + return (false); + case '%': + case 'H': + case 'N': + case 'P': + case 'U': + break; + default: + return (false); + } + } + } + + if (i == MAXPATHLEN) + return (false); + + return (true); +} + +static void +corefilename_init(void *arg) +{ + char *format; + + format = kern_getenv("kern.corefile"); + if (format != NULL && corefilename_check(format)) + strcpy(corefilename, format); +} +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); static int sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) { + char *format; int error; + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); + + sx_slock(&corefilename_lock); + strcpy(format, corefilename); + sx_sunlock(&corefilename_lock); + + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); + if (error || !req->newptr || strcmp(format, corefilename) == 0) + goto out; + + if (!corefilename_check(format)) { + error = EINVAL; + goto out; + } + sx_xlock(&corefilename_lock); - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), - req); + strcpy(corefilename, format); sx_xunlock(&corefilename_lock); +out: + free(format, M_TEMP); return (error); } -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", "Process corefile name format string"); @@ -3170,11 +3230,6 @@ corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, case 'U': /* user id */ sbuf_printf(&sb, "%u", uid); break; - default: - log(LOG_ERR, - "Unknown format character %c in " - "corename `%s'\n", format[i], format); - break; } break; default: -- 2.1.2 Best regards, Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 13:52:18 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 71004176; Sun, 22 Mar 2015 13:52:18 +0000 (UTC) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id 2704B23C; Sun, 22 Mar 2015 13:52:18 +0000 (UTC) Received: from [10.0.1.17] (host81-156-92-60.range81-156.btcentralplus.com [81.156.92.60]) by cyrus.watson.org (Postfix) with ESMTPSA id 93F1D46B2A; Sun, 22 Mar 2015 09:52:10 -0400 (EDT) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: Self introduciton of Hao Sun and thoughts on LibNetstat From: "Robert N. M. Watson" In-Reply-To: Date: Sun, 22 Mar 2015 13:52:07 +0000 Message-Id: <826E80C6-55BE-4690-B35B-14670EE482AD@FreeBSD.org> References: To: Hao Sun X-Mailer: Apple Mail (2.2070.6) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Gabor Pali , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 13:52:18 -0000 Tuning in slightly late: I=E2=80=99m not sure if the bsnmp=E2=80=99s = plugins might also want to learn about libnetstat and friends? Robert > On 22 Mar 2015, at 12:09, Hao Sun wrote: >=20 > Hi G=C3=A1bor, >=20 > Thanks a lot for the relpy. Your comments provide me a lot of valuable = information, and now I'm getting clearer about what the target is and = what need I do in the future.=20 >=20 > Following your guidance, I've cloned the FreeBSD mirror from GitHub = and will get down to have an initial scratch with the latest version. = Yeah, 6 years passed by, a lot have been changed and updated during that = period. Also I had a glance over the existing user-space code and get = cleared about the basic code strcture and abstratactions of the previous = version. Now I think it's time to propose my ideas about the project and = write the proposal. >=20 > However there is one more quesiton. On the project's wiki page = (https://wiki.freebsd.org/LibNetstat = ) , I think the target for GSoC = 2015 is to finish the tasks haven't been done in the following table. = But accroding to your comments in the emails, it seems like I need to = start the job from scratch. Thus the question is should I keep the = existing code and add new features to the previous version or just start = the project from the very beginning? >=20 > =20 > >=20 > Thanks, > Hao >=20 >=20 > 2015-03-21 16:57 GMT+08:00 Gabor Pali >: > [Please CC me in your replies, I am not on freebsd-hackers.] >=20 > Hi Hao, >=20 > 2015-03-19 1:53 GMT+01:00 Hao Sun >: > > I saw the project introduction of LibNetstat on > > FreeBSD=E2=80=99s GSoC 15=E2=80=99 homepage and was attracted by the = project. >=20 > Thank you for contacting us, it is always good to see fresh people > around who would like to contribute to the Project, especially as part > of the Google Summer of Code program. >=20 > > I think the LibNetstat would be fitful for me because I have related = rich > > project experiences on C, namely the FontDesigner project in the = lab, the > > face recognition plugin in Muticoreware and other course projects. >=20 > I believe this project is mostly about refactoring the netstat(1) > utility into a library and make the utility its client. This could > come with many advantages, such as other programs could easily access > the services it would offer and this would also help with accessing > all the networking-related statistics in an platform-independent way, > even through the network. The goal of the project is to come up with > an API and ABI that is convenient to use and captures all the concepts > that are currently used in netstat(1). >=20 > > I read the project description carefully and have done the following = jobs > > since the monitoring organisations were published. >=20 > I think those are indeed good first steps in order to get involved. >=20 > > 1. Check out the code from //depot/projects/soc2009/pgj_libstat/. As = the p4 > > introduction article shows, maybe I need a FreeBSD.org account to = get > > access into our depot. Thus would you please offer helps to create = an > > account? >=20 > Please note that project you are talking about was done almost 6 years > ago. Things can change a lot even in a year, so you may find yourself > starting again from scratch (which may be equally either good or bad > news for you). One of those changes is that Perforce has shifted out > of the focus in the recent years, students have started to use the > Project's Subversion repositories for their works, or I believe, now > they can even choose to work with git, through GitHub. >=20 > So, I guess you would only need a GitHub account and you are ready to > fork the Git mirror of the FreeBSD src repository there: >=20 > https://github.com/freebsd/freebsd = >=20 > > 2. I plan to run some demo codes to have an insight into the current = version > > of LibNetstat. Do we have demo codes or test cases which could help = me > > get familir with the code? >=20 > It is also keep in mind that the original libnetstat code was written > and kept updated until 2011, which assumes an older base system (and > kernel) version of FreeBSD. Again, many changes might have changed > (and I am sure they have indeed changed) in the recent years, like the > kernel now has nice atomic counters for networking statistics (thanks > to Gleb Smirnoff) which was one of the blocker issues when I stopped > working on the project. >=20 > Of course, if you would like to study the code that we wrote and you > have questions about it, I am happy answer them -- note that you can > access all the sources through the P4DB web, you do not have to check > out anything. However, please also note I am not officially a src > committer so my comments may not be as precise as for example, > Robert's. I have gained some experienced in working with the > networking parts of the FreeBSD kernel and I have a few years of > experience in hacking on various projects ranging from computer games > to compilers, but that is not my area of expertise therefore I may not > be up-to-date enough on the subject. >=20 > > 3. After Step #2, I want to read some existing modules, for = instance, > > routing abstractions. I believe this step would help me get clear = how to > > make the original interface less ABI-sensitive. So do you have any > > suggestions where to start this step? >=20 > Most of the userspace code can be found here: >=20 > = https://p4web.freebsd.org/@md=3Dd&cd=3D//depot/projects/soc2009/pgj_libsta= t/src/lib/&c=3DhL4@//depot/projects/soc2009/pgj_libstat/src/lib/libnetstat= /?ac=3D83 = >=20 > They may have related kernel-side changes, but for the first stab, I > think that is what you may want to see. "netstat.h" and > "netstat_internal.h" may tell you more about the abstractions I > created. Feel free to dump them, and start from scratch, perhaps I > would do them differently myself if I started to work on this project > today. >=20 > For further reference, you can also study the sister libraries of > libnetstat, libprocstat and libmemstat. They are probably much more > up-to-date with the current state of the development: >=20 > https://github.com/freebsd/freebsd/tree/master/lib/libmemstat = > https://github.com/freebsd/freebsd/tree/master/lib/libprocstat = >=20 > Cheers, > G=C3=A1bor >=20 From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 12:09:12 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0D7D4D54; Sun, 22 Mar 2015 12:09:12 +0000 (UTC) Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com [IPv6:2a00:1450:400c:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA6D6899; Sun, 22 Mar 2015 12:09:10 +0000 (UTC) Received: by wixw10 with SMTP id w10so18759909wix.0; Sun, 22 Mar 2015 05:09:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=BrYyz0zEtgYA1KjjuKP9xrOacZKOgf967HzWI/USoXQ=; b=te0ppDeyjzycP6dvs8al/KESjtEZOxGs+rSU8KMVunVP1FCovt7n3wAOwisU5y0TmC 0sRbYGQXsxzOq44ywyML/LRbvpiQaSRCtfVwmGJNHQA6lRQmpO0Nk3O/toIzLKRdCdIj VTML/X1+7qNcVJYOre4XJxEHeYriN3SaS/0nnKKAnlqwfVo+JmGIGkeRW9HCdx8AJRZ4 20MsvD8tmN+tcgVnmxtXUnV6cUI3KkIfH9EU1bbGiG924nJbe4UwwNOtYCv16YmJQHuI +hcWysKZVFhIaOqv8s4+YEzCLOp4wvRUMuH3HB0tvM3Yb2JS2Nj2HuWjVExsWCkX2clg Wk6Q== MIME-Version: 1.0 X-Received: by 10.194.57.170 with SMTP id j10mr104783943wjq.102.1427026149249; Sun, 22 Mar 2015 05:09:09 -0700 (PDT) Received: by 10.28.98.70 with HTTP; Sun, 22 Mar 2015 05:09:08 -0700 (PDT) In-Reply-To: References: Date: Sun, 22 Mar 2015 20:09:08 +0800 Message-ID: Subject: Re: Self introduciton of Hao Sun and thoughts on LibNetstat From: Hao Sun To: Gabor Pali X-Mailman-Approved-At: Sun, 22 Mar 2015 14:49:11 +0000 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-hackers@freebsd.org, Robert Watson X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 12:09:12 -0000 Hi G=C3=A1bor, Thanks a lot for the relpy. Your comments provide me a lot of valuable information, and now I'm getting clearer about what the target is and what need I do in the future. Following your guidance, I've cloned the FreeBSD mirror from GitHub and will get down to have an initial scratch with the latest version. Yeah, 6 years passed by, a lot have been changed and updated during that period. Also I had a glance over the existing user-space code and get cleared about the basic code strcture and abstratactions of the previous version. Now I think it's time to propose my ideas about the project and write the proposal. However there is one more quesiton. On the project's wiki page ( https://wiki.freebsd.org/LibNetstat) , I think the target for GSoC 2015 is to finish the tasks haven't been done in the following table. But accroding to your comments in the emails, it seems like I need to start the job from scratch. Thus the question is should I keep the existing code and add new features to the previous version or just start the project from the very beginning? [image: =E5=86=85=E5=B5=8C=E5=9B=BE=E7=89=87 2] Thanks, Hao 2015-03-21 16:57 GMT+08:00 Gabor Pali : > [Please CC me in your replies, I am not on freebsd-hackers.] > > Hi Hao, > > 2015-03-19 1:53 GMT+01:00 Hao Sun : > > I saw the project introduction of LibNetstat on > > FreeBSD=E2=80=99s GSoC 15=E2=80=99 homepage and was attracted by the pr= oject. > > Thank you for contacting us, it is always good to see fresh people > around who would like to contribute to the Project, especially as part > of the Google Summer of Code program. > > > I think the LibNetstat would be fitful for me because I have related ri= ch > > project experiences on C, namely the FontDesigner project in the lab, t= he > > face recognition plugin in Muticoreware and other course projects. > > I believe this project is mostly about refactoring the netstat(1) > utility into a library and make the utility its client. This could > come with many advantages, such as other programs could easily access > the services it would offer and this would also help with accessing > all the networking-related statistics in an platform-independent way, > even through the network. The goal of the project is to come up with > an API and ABI that is convenient to use and captures all the concepts > that are currently used in netstat(1). > > > I read the project description carefully and have done the following jo= bs > > since the monitoring organisations were published. > > I think those are indeed good first steps in order to get involved. > > > 1. Check out the code from //depot/projects/soc2009/pgj_libstat/. As th= e > p4 > > introduction article shows, maybe I need a FreeBSD.org account to get > > access into our depot. Thus would you please offer helps to create an > > account? > > Please note that project you are talking about was done almost 6 years > ago. Things can change a lot even in a year, so you may find yourself > starting again from scratch (which may be equally either good or bad > news for you). One of those changes is that Perforce has shifted out > of the focus in the recent years, students have started to use the > Project's Subversion repositories for their works, or I believe, now > they can even choose to work with git, through GitHub. > > So, I guess you would only need a GitHub account and you are ready to > fork the Git mirror of the FreeBSD src repository there: > > https://github.com/freebsd/freebsd > > > 2. I plan to run some demo codes to have an insight into the current > version > > of LibNetstat. Do we have demo codes or test cases which could help me > > get familir with the code? > > It is also keep in mind that the original libnetstat code was written > and kept updated until 2011, which assumes an older base system (and > kernel) version of FreeBSD. Again, many changes might have changed > (and I am sure they have indeed changed) in the recent years, like the > kernel now has nice atomic counters for networking statistics (thanks > to Gleb Smirnoff) which was one of the blocker issues when I stopped > working on the project. > > Of course, if you would like to study the code that we wrote and you > have questions about it, I am happy answer them -- note that you can > access all the sources through the P4DB web, you do not have to check > out anything. However, please also note I am not officially a src > committer so my comments may not be as precise as for example, > Robert's. I have gained some experienced in working with the > networking parts of the FreeBSD kernel and I have a few years of > experience in hacking on various projects ranging from computer games > to compilers, but that is not my area of expertise therefore I may not > be up-to-date enough on the subject. > > > 3. After Step #2, I want to read some existing modules, for instance, > > routing abstractions. I believe this step would help me get clear how t= o > > make the original interface less ABI-sensitive. So do you have any > > suggestions where to start this step? > > Most of the userspace code can be found here: > > > https://p4web.freebsd.org/@md=3Dd&cd=3D//depot/projects/soc2009/pgj_libst= at/src/lib/&c=3DhL4@//depot/projects/soc2009/pgj_libstat/src/lib/libnetstat= /?ac=3D83 > > They may have related kernel-side changes, but for the first stab, I > think that is what you may want to see. "netstat.h" and > "netstat_internal.h" may tell you more about the abstractions I > created. Feel free to dump them, and start from scratch, perhaps I > would do them differently myself if I started to work on this project > today. > > For further reference, you can also study the sister libraries of > libnetstat, libprocstat and libmemstat. They are probably much more > up-to-date with the current state of the development: > > https://github.com/freebsd/freebsd/tree/master/lib/libmemstat > https://github.com/freebsd/freebsd/tree/master/lib/libprocstat > > Cheers, > G=C3=A1bor > From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 20:51:03 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F317974F for ; Sun, 22 Mar 2015 20:51:03 +0000 (UTC) Received: from dmz-mailsec-scanner-6.mit.edu (dmz-mailsec-scanner-6.mit.edu [18.7.68.35]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9640F37C for ; Sun, 22 Mar 2015 20:51:02 +0000 (UTC) X-AuditID: 12074423-f79536d000000e74-aa-550f2a014a0e Received: from mailhub-auth-2.mit.edu ( [18.7.62.36]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-6.mit.edu (Symantec Messaging Gateway) with SMTP id E8.79.03700.10A2F055; Sun, 22 Mar 2015 16:45:53 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id t2MKjqYG006902; Sun, 22 Mar 2015 16:45:53 -0400 Received: from multics.mit.edu (system-low-sipb.mit.edu [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id t2MKjoji007668 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sun, 22 Mar 2015 16:45:52 -0400 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id t2MKjoC5028417; Sun, 22 Mar 2015 16:45:50 -0400 (EDT) Date: Sun, 22 Mar 2015 16:45:50 -0400 (EDT) From: Benjamin Kaduk X-X-Sender: kaduk@multics.mit.edu To: Mateusz Guzik Subject: Re: How to traverse kernel threads? In-Reply-To: <20150321232622.GF14650@dft-labs.eu> Message-ID: References: <20150321220246.GE14650@dft-labs.eu> <20150321232622.GF14650@dft-labs.eu> User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupmleLIzCtJLcpLzFFi42IRYrdT0WXU4g81eNCpaLF98z9Gi8aDi1ks Dh8+werA7DHj03wWj52z7rIHMEVx2aSk5mSWpRbp2yVwZaz728pScISpYtUDwQbG34xdjJwc EgImEjdev2aGsMUkLtxbz9bFyMUhJLCYSWLh/oeMEM5GRokJm5axQjiHmCQOTPgJ5TQwSvS+ mwvWzyKgLfG1dQ3YXDYBNYnHe5tZIeYqSmw+NQmsRkRAVeL50fVgcWaBNIknr26C1QsL6Ei0 X5kJZnMKGEp0z+kCs3kFHCU6pz+Euuk+o8SrA/1MIAlRoIbV+6ewQBQJSpyc+YQFYqiWxPLp 21gmMArNQpKahSS1gJFpFaNsSm6Vbm5iZk5xarJucXJiXl5qka6ZXm5miV5qSukmRnA4uyjv YPxzUOkQowAHoxIPr0IuX6gQa2JZcWXuIUZJDiYlUd7tD4FCfEn5KZUZicUZ8UWlOanFhxgl OJiVRHjfSPCHCvGmJFZWpRblw6SkOViUxHk3/eALERJITyxJzU5NLUgtgsnKcHAoSfCu0wBq FCxKTU+tSMvMKUFIM3FwggznARqurwkyvLggMbc4Mx0if4pRl+POlP+LmIRY8vLzUqXEeReD DBIAKcoozYObA0tDrxjFgd4S5pUCGcUDTGFwk14BLWECWnIunw9kSUkiQkqqgVHUfIZmoMZW hU6dtMXqN5iCnhWf0Zn+fFHyTAbzxuJLhdeP3zVsTzvbadY5Mykr+o/UZKt0445/q72Vr35s NnpypavyQo+u5JN4W92/M570h4h+yPqZEh+dq/PILfiKmnaSpfsGlSTH4hc7Lqx4+ixkjY50 2uUDlnOmJ+w503m9MTfD7kPXaSWW4oxEQy3mouJEAJiDhOIeAwAA Cc: "freebsd-hackers@freebsd.org" , Yue Chen X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 20:51:04 -0000 On Sat, 21 Mar 2015, Mateusz Guzik wrote: > But once more the real question is what are you trying to do. I don't > see any use for stack info of random threads. One thing that comes to mind is for live binary-patching the kernel, to confirm that no thread is currently in a routine which would be patched. -Ben From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 22 20:58:51 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 463CFB7C for ; Sun, 22 Mar 2015 20:58:51 +0000 (UTC) Received: from mail-yk0-f174.google.com (mail-yk0-f174.google.com [209.85.160.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 057A03E8 for ; Sun, 22 Mar 2015 20:58:50 +0000 (UTC) Received: by ykcn8 with SMTP id n8so63504045ykc.3 for ; Sun, 22 Mar 2015 13:58:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=v8kGrE9is0tYmip61Nr/RoMSz/DmQKc4qS/ZgYgkMnA=; b=JHhlczbWjZDUjtRdTX3Vm8oC1nIbpdunJWlNFg9YAqpYz6u+YdaSrPpe4Cs0QA4uNd KSe9HUJcCY7GkoVGxSHW5omABjEDVdNO0k67C6O0wGWNU20wl+j0/iXWMf13BNdGO56l Xr/x3g0wrcUArCU6M4kcmY4Bc997mztNXBTUad9/g1NyfPBS3h/O5YRhEVEoOryL7I3b YcgdR5zxJ+VKb0ynN8FuKU3RJs3N2nvSG7lGvJuA8VG6739g1spCVPnshBgtRVg5Gm8G eAigRTTze2E9+YO7743Bds3XAx4f+uuI9vLmIFIt/RWVMYCwKj4EA49AcfCjr1Oe7VtU XQuQ== X-Gm-Message-State: ALoCoQn+ghoeYphgV7BOTt66+NEkAIvBWu7vBs2F3rEZhWHLT3LvHrnOZXFczeqFxpECBb3rJN8L MIME-Version: 1.0 X-Received: by 10.236.0.193 with SMTP id 41mr93295437yhb.146.1427057923089; Sun, 22 Mar 2015 13:58:43 -0700 (PDT) Received: by 10.170.104.86 with HTTP; Sun, 22 Mar 2015 13:58:43 -0700 (PDT) In-Reply-To: References: <20150321220246.GE14650@dft-labs.eu> <20150321232622.GF14650@dft-labs.eu> Date: Sun, 22 Mar 2015 21:58:43 +0100 Message-ID: Subject: Re: How to traverse kernel threads? From: Oliver Pinter To: Benjamin Kaduk Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , Mateusz Guzik , Yue Chen X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2015 20:58:51 -0000 Probably take a look at DDB: https://github.com/HardenedBSD/hardenedBSD/blob/hardened/current/master/sys/ddb/db_thread.c#L88 On Sun, Mar 22, 2015 at 9:45 PM, Benjamin Kaduk wrote: > On Sat, 21 Mar 2015, Mateusz Guzik wrote: > >> But once more the real question is what are you trying to do. I don't >> see any use for stack info of random threads. > > One thing that comes to mind is for live binary-patching the kernel, to > confirm that no thread is currently in a routine which would be patched. > > -Ben > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 00:56:11 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3342D9BB for ; Mon, 23 Mar 2015 00:56:11 +0000 (UTC) Received: from mail-wg0-x230.google.com (mail-wg0-x230.google.com [IPv6:2a00:1450:400c:c00::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B2193F95 for ; Mon, 23 Mar 2015 00:56:10 +0000 (UTC) Received: by wgra20 with SMTP id a20so133750175wgr.3 for ; Sun, 22 Mar 2015 17:56:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=LCopYcU5wEh48i6qo7X8JJv1tf63yY61h2CncGHnRd4=; b=dbgK6Y0unICaMd8C4zpIChoe+rYS5/yONWtDs3aCn2HUiNckqE7cix+8+f6ox0yo4P e5tkx5vhkVkXgJ3g7OByigKIY42Yn+aClE/A03tHKQ3NzrD8GSj35fcupgfrEln7RFAp qFWiw+oWstsJAhXQyY7Tex/94tsWb/U9X+sZI4/I0mQ9pfPRCr4pkOJwvwjuw8wPHGH9 ZV7m6XvmqtULYyGnTA0E8IsVpdwHP/Ehq2ZaTDdvzKCXPcwSwLRk4PYf2r/QxgHjOMLh +yOBMgqCJiOe/qbNLZUNmoNPVa94XrmOAPAOqh0AknIaXuROQraXjIoe9OaUTYArinXE OhMw== X-Received: by 10.180.198.110 with SMTP id jb14mr14946013wic.57.1427072169169; Sun, 22 Mar 2015 17:56:09 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id dm6sm8733281wib.22.2015.03.22.17.56.07 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Sun, 22 Mar 2015 17:56:07 -0700 (PDT) Date: Mon, 23 Mar 2015 01:56:05 +0100 From: Mateusz Guzik To: Konstantin Belousov Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150323005605.GA6798@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Konstantin Belousov , Tiwei Bie , freebsd-hackers@freebsd.org References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322102428.GZ2379@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150322102428.GZ2379@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@freebsd.org, Tiwei Bie X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 00:56:11 -0000 On Sun, Mar 22, 2015 at 12:24:28PM +0200, Konstantin Belousov wrote: > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > > > Sorry, I introduced a bug... allproc_lock could not be used to protect > > > the access to corefilename[]. > > > > > > > First off I committed the code, so the fault is on me. > > > > > Because, sysctl_kern_corefile() could be called very early: > > > > > [..] > > > That is to say, when the tunable `kern.corefile' is set in loader.conf, > > > sysctl_kern_corefile() will be called as the priority of (SI_SUB_KMEM, > > > SI_ORDER_FIRST). > > > > > > At this time, allproc_lock is not initialized. > > > > > > I couldn't find a proper existing lock for this task. Maybe a dedicated > > > lock needs to be created. And initialize it together with sysctlmemlock: > > > > > [..] > > > Or maybe sysctlmemlock could be used, which is only acuqired when > > > req.oldlen > PAGE_SIZE. > > > > > > > > > > I was somehow convinced that tunables are dealt with other code. > > > > If such sysctl handler is also called for tunables, the kernel should > > pass a flag or some other indicator so that the function knows it is > > dealing with a tunable and that would avoid locking and thus solve the > > problem. > > > > I'm wondering if we should go a little bit further and get rid of > > static char corefilename[MAXPATHLEN] > > > > and have a static char *corefilename instead. > Accessing the array through the pointer dereference is micro-pessimization, > as well as having to maintain metadata for the malloced memory, isn't it ? > Having this dynamically allocated opens up a way to set such path per-jail, which may be a desirable feature. Also gets rid of a 1024 bytes table. > > > > A dedicated sysinit func could fetch and validate the tunable, if any. > > If no tunable was provided it would alloc memory for the default. > > Or you could move initialization of the sx in question earlier. -- Mateusz Guzik From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 00:58:58 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C710EAA8 for ; Mon, 23 Mar 2015 00:58:58 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 488BCFA6 for ; Mon, 23 Mar 2015 00:58:58 +0000 (UTC) Received: by wibgn9 with SMTP id gn9so46616481wib.1 for ; Sun, 22 Mar 2015 17:58:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=w76GZe5XZZUGOnap0aCKJKb3nKWelXz7wuDn84rqDs4=; b=n83UaB4lDYKBQeD7VlYxN9zn9XqNQK5DQlDeXMmNcgyuSR98nPGhwEhR3xCeoD+q8G 73EFBmFRuN7mdMa76B/+rROtOKJufAMw1XEPNivae1ew+NEDdubzNzJS3t6eieN4la+D aH9Gxo7rRVSgQoIhT3c5fxEOpA2DJFj7CeQlvK3F17QUwswRkS3Oe8kHDXrt7oubVsF6 ygGwLprTwpB5Op2neCGYtHPjzUUL5/ROVrJGYQnPhU/TJq8SuQHsIjo/mVAuvPoL3j7J YcY9ETM1fvGMBJzOvVKNqk+rgDNAWgm6qVMZMw6Np9sjiNwjd1/HXu8QfZ3J1c63/xAZ vdKA== X-Received: by 10.180.102.73 with SMTP id fm9mr15284737wib.12.1427072335686; Sun, 22 Mar 2015 17:58:55 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id ga8sm8762022wib.11.2015.03.22.17.58.54 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Sun, 22 Mar 2015 17:58:54 -0700 (PDT) Date: Mon, 23 Mar 2015 01:58:52 +0100 From: Mateusz Guzik To: Tiwei Bie Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150323005852.GB6798@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Tiwei Bie , Konstantin Belousov , freebsd-hackers@freebsd.org References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> <20150322113822.GB2379@kib.kiev.ua> <20150322120655.GA59757@freebsd> <20150322131524.GA95795@freebsd> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150322131524.GA95795@freebsd> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 00:58:58 -0000 On Sun, Mar 22, 2015 at 09:15:24PM +0800, Tiwei Bie wrote: > On Sun, Mar 22, 2015 at 08:06:55PM +0800, Tiwei Bie wrote: > > On Sun, Mar 22, 2015 at 01:38:22PM +0200, Konstantin Belousov wrote: > > > On Sun, Mar 22, 2015 at 07:25:55PM +0800, Tiwei Bie wrote: > > > > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > > > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > > > [..] > > I will submit my new patch later. > > > > Here is my new patch: > > --- > sys/kern/kern_sig.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++------- > 1 file changed, 64 insertions(+), 9 deletions(-) > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > index 8410d9d..5162ab6 100644 > --- a/sys/kern/kern_sig.c > +++ b/sys/kern/kern_sig.c > @@ -3094,21 +3094,81 @@ static int compress_user_cores = 0; > */ > #define corefilename_lock allproc_lock > > -static char corefilename[MAXPATHLEN] = {"%N.core"}; > +static char corefilename[MAXPATHLEN] = "%N.core"; > + > +static bool > +corefilename_check(const char *format) > +{ > + int i, counti; > + > + counti = 0; > + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { > + if (format[i] == '%') { > + i++; > + switch (format[i]) { > + case 'I': > + counti++; > + if (counti > 1) > + return (false); > + case '%': > + case 'H': > + case 'N': > + case 'P': > + case 'U': > + break; > + default: > + return (false); > + } > + } > + } > + > + if (i == MAXPATHLEN) > + return (false); > + > + return (true); > +} > + > +static void > +corefilename_init(void *arg) > +{ > + char *format; > + > + format = kern_getenv("kern.corefile"); > + if (format != NULL && corefilename_check(format)) > + strcpy(corefilename, format); > +} > +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); > The user should know that the tunable got discarded (e.g. like with prinf removed below). > static int > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > { > + char *format; > int error; > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > + > + sx_slock(&corefilename_lock); > + strcpy(format, corefilename); > + sx_sunlock(&corefilename_lock); > + > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > + if (error || !req->newptr || strcmp(format, corefilename) == 0) > + goto out; > + Is this strcmp useful? Normally you don't replace the pattern with identical one. > + if (!corefilename_check(format)) { > + error = EINVAL; > + goto out; > + } > + > sx_xlock(&corefilename_lock); > - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), > - req); > + strcpy(corefilename, format); > sx_xunlock(&corefilename_lock); > > +out: > + free(format, M_TEMP); > return (error); > } > -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | > +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | > CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", > "Process corefile name format string"); > > @@ -3170,11 +3230,6 @@ corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, > case 'U': /* user id */ > sbuf_printf(&sb, "%u", uid); > break; > - default: > - log(LOG_ERR, > - "Unknown format character %c in " > - "corename `%s'\n", format[i], format); > - break; > } This case should be an assertion. E.g. KASSERT(0, "unknown format in character") > break; > default: > -- > 2.1.2 > > Best regards, > Tiwei Bie > -- Mateusz Guzik From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 02:03:46 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 84238786 for ; Mon, 23 Mar 2015 02:03:46 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id 9CD02813 for ; Mon, 23 Mar 2015 02:03:44 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygDXuRZ2dA9VPnPGAw--.50379S2; Mon, 23 Mar 2015 10:03:40 +0800 (CST) Date: Mon, 23 Mar 2015 10:03:14 +0800 From: Tiwei Bie To: Mateusz Guzik Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150323020314.GA30143@freebsd> References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> <20150322113822.GB2379@kib.kiev.ua> <20150322120655.GA59757@freebsd> <20150322131524.GA95795@freebsd> <20150323005852.GB6798@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150323005852.GB6798@dft-labs.eu> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygDXuRZ2dA9VPnPGAw--.50379S2 X-Coremail-Antispam: 1UD129KBjvJXoWxKFW3ur13Zw47Gr48Xr47Jwb_yoW3Aw13pF WrCrZ8Ars5Cr43Crsav3yDAa4Yyws5Jw4UW347X3WYkr1F9rykXF18Kr1F9Fn7GrWvgryD Za15Wr9xKry8XrJanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUkYb7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26F4UJVW0owA2z4x0Y4 vEx4A2jsIEc7CjxVAFwI0_GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xv F2IEw4CE5I8CrVC2j2WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r4j6F 4UMcvjeVCFs4IE7xkEbVWUJVW8JwACjcxG0xvEwIxGrwCY02Avz4vE14v_Gr1l42xK82IY c2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s 026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1Y6r17MIIYrxkI7VAKI48JMIIF 0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r1j6r4UMIIF0x vE42xK8VAvwI8IcIk0rVWrZr1j6s0DMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2 jsIEc7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU5AKs5UUUUU== X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUQAVQhl-peFAAesZ Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 02:03:46 -0000 On Mon, Mar 23, 2015 at 01:58:52AM +0100, Mateusz Guzik wrote: > On Sun, Mar 22, 2015 at 09:15:24PM +0800, Tiwei Bie wrote: > > On Sun, Mar 22, 2015 at 08:06:55PM +0800, Tiwei Bie wrote: > > > On Sun, Mar 22, 2015 at 01:38:22PM +0200, Konstantin Belousov wrote: > > > > On Sun, Mar 22, 2015 at 07:25:55PM +0800, Tiwei Bie wrote: > > > > > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > > > > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > > > > > [..] > > > I will submit my new patch later. > > > > > > > Here is my new patch: > > > > --- > > sys/kern/kern_sig.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++------- > > 1 file changed, 64 insertions(+), 9 deletions(-) > > > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > > index 8410d9d..5162ab6 100644 > > --- a/sys/kern/kern_sig.c > > +++ b/sys/kern/kern_sig.c > > @@ -3094,21 +3094,81 @@ static int compress_user_cores = 0; > > */ > > #define corefilename_lock allproc_lock > > > > -static char corefilename[MAXPATHLEN] = {"%N.core"}; > > +static char corefilename[MAXPATHLEN] = "%N.core"; > > + > > +static bool > > +corefilename_check(const char *format) > > +{ > > + int i, counti; > > + > > + counti = 0; > > + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { > > + if (format[i] == '%') { > > + i++; > > + switch (format[i]) { > > + case 'I': > > + counti++; > > + if (counti > 1) > > + return (false); > > + case '%': > > + case 'H': > > + case 'N': > > + case 'P': > > + case 'U': > > + break; > > + default: > > + return (false); > > + } > > + } > > + } > > + > > + if (i == MAXPATHLEN) > > + return (false); > > + > > + return (true); > > +} > > + > > +static void > > +corefilename_init(void *arg) > > +{ > > + char *format; > > + > > + format = kern_getenv("kern.corefile"); > > + if (format != NULL && corefilename_check(format)) > > + strcpy(corefilename, format); > > +} > > +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); > > > > The user should know that the tunable got discarded (e.g. like with > prinf removed below). > I don't know how to do this. We need to update the man-pages? And sorry again, maybe I was too tired yesterday. I just find a new bug in the codes above when reading your comments. Though it won't affect anything cause dynamic_kenv hasn't been enabled at this time, it is really a bug. `format' needs to be freed, sorry... What a terrible day. ;-( > > static int > > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > > { > > + char *format; > > int error; > > > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > > + > > + sx_slock(&corefilename_lock); > > + strcpy(format, corefilename); > > + sx_sunlock(&corefilename_lock); > > + > > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > > + if (error || !req->newptr || strcmp(format, corefilename) == 0) > > + goto out; > > + > > Is this strcmp useful? Normally you don't replace the pattern with > identical one. > I wrote these codes after consulting the codes in kern/kern_clocksource.c /* * Report or change the active event timers hardware. */ static int sysctl_kern_eventtimer_timer(SYSCTL_HANDLER_ARGS) { char buf[32]; struct eventtimer *et; int error; ET_LOCK(); et = timer; snprintf(buf, sizeof(buf), "%s", et->et_name); ET_UNLOCK(); error = sysctl_handle_string(oidp, buf, sizeof(buf), req); ET_LOCK(); et = timer; if (error != 0 || req->newptr == NULL || strcasecmp(buf, et->et_name) == 0) { ET_UNLOCK(); return (error); } et = et_find(buf, 0, 0); if (et == NULL) { ET_UNLOCK(); return (ENOENT); } configtimer(0); et_free(timer); if (et->et_flags & ET_FLAGS_C3STOP) cpu_disable_c3_sleep++; if (timer->et_flags & ET_FLAGS_C3STOP) cpu_disable_c3_sleep--; periodic = want_periodic; timer = et; et_init(timer, timercb, NULL, NULL); configtimer(1); ET_UNLOCK(); return (error); } SYSCTL_PROC(_kern_eventtimer, OID_AUTO, timer, CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, 0, 0, sysctl_kern_eventtimer_timer, "A", "Chosen event timer"); It also needs to validate the user's inputs. And it uses a strcasecmp() to check whether a different timer has been specified by the user. If the user is going to replace the pattern with identical one, we won't need to do any update, we just `goto out'. > > + if (!corefilename_check(format)) { > > + error = EINVAL; > > + goto out; > > + } > > + > > sx_xlock(&corefilename_lock); > > - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), > > - req); > > + strcpy(corefilename, format); > > sx_xunlock(&corefilename_lock); > > > > +out: > > + free(format, M_TEMP); > > return (error); > > } > > -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | > > +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | > > CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", > > "Process corefile name format string"); > > > > @@ -3170,11 +3230,6 @@ corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, > > case 'U': /* user id */ > > sbuf_printf(&sb, "%u", uid); > > break; > > - default: > > - log(LOG_ERR, > > - "Unknown format character %c in " > > - "corename `%s'\n", format[i], format); > > - break; > > } > > This case should be an assertion. E.g. KASSERT(0, "unknown format in > character") > I will update my patch. My new patch: --- sys/kern/kern_sig.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 70 insertions(+), 7 deletions(-) diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c index 8410d9d..0254517 100644 --- a/sys/kern/kern_sig.c +++ b/sys/kern/kern_sig.c @@ -3094,21 +3094,85 @@ static int compress_user_cores = 0; */ #define corefilename_lock allproc_lock -static char corefilename[MAXPATHLEN] = {"%N.core"}; +static char corefilename[MAXPATHLEN] = "%N.core"; + +static bool +corefilename_check(const char *format) +{ + int i, counti; + + counti = 0; + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { + if (format[i] == '%') { + i++; + switch (format[i]) { + case 'I': + counti++; + if (counti > 1) + return (false); + case '%': + case 'H': + case 'N': + case 'P': + case 'U': + break; + default: + return (false); + } + } + } + + if (i == MAXPATHLEN) + return (false); + + return (true); +} + +static void +corefilename_init(void *arg) +{ + char *format; + + format = kern_getenv("kern.corefile"); + + if (format != NULL) { + if (corefilename_check(format)) + strcpy(corefilename, format); + freeenv(format); + } +} +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); static int sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) { + char *format; int error; + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); + + sx_slock(&corefilename_lock); + strcpy(format, corefilename); + sx_sunlock(&corefilename_lock); + + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); + if (error || !req->newptr || strcmp(format, corefilename) == 0) + goto out; + + if (!corefilename_check(format)) { + error = EINVAL; + goto out; + } + sx_xlock(&corefilename_lock); - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), - req); + strcpy(corefilename, format); sx_xunlock(&corefilename_lock); +out: + free(format, M_TEMP); return (error); } -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", "Process corefile name format string"); @@ -3171,9 +3235,8 @@ corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, sbuf_printf(&sb, "%u", uid); break; default: - log(LOG_ERR, - "Unknown format character %c in " - "corename `%s'\n", format[i], format); + KASSERT(0, ("Unknown format character %c in " + "corename `%s'\n", format[i], format)); break; } break; -- 2.1.2 Best regards, Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 02:48:31 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8C25ADAC; Mon, 23 Mar 2015 02:48:31 +0000 (UTC) Received: from mail-ig0-x22c.google.com (mail-ig0-x22c.google.com [IPv6:2607:f8b0:4001:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 50D84BE6; Mon, 23 Mar 2015 02:48:31 +0000 (UTC) Received: by igcau2 with SMTP id au2so25029804igc.1; Sun, 22 Mar 2015 19:48:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Mu09Wfh6e7OFI9bqMqW9/Blas9TyGnkz6XpeVDCBVns=; b=ikWKCUuyUYuQBKnNNG2MgiEc/WOQx/2vJSzYXIbPURBsfjcmolCauiTuMgqxDn/FtE LLnH7im4LBQ6+PCbJrasi0L4XxnzQNdWZa2V8FZ9vcY0ynRhC8C4s5JulNzT8QCbJMqm PhG0u8q7/hTN+tCGTZn7YGOdhNafLB90bqQYAuLk9BoTDuThHkqF5Xm6H5tOy/UTQHCK zZtWLk3tF901kDAhnO+yMFGhtMH/aavnCACQh+Xp5tuIdU/p8vkHmjbkXvyrZLXQBdOG aJ4SpExy/niMQrWDw0MEQCyfUGyAy3B+bDZyRGgT8NckXphOg6Thz1xncYnbMCQUF6Dv go8Q== MIME-Version: 1.0 X-Received: by 10.107.12.150 with SMTP id 22mr2761623iom.71.1427078910737; Sun, 22 Mar 2015 19:48:30 -0700 (PDT) Received: by 10.107.143.149 with HTTP; Sun, 22 Mar 2015 19:48:30 -0700 (PDT) Date: Sun, 22 Mar 2015 21:48:30 -0500 Message-ID: Subject: [gsoc15][idea-proposal] Port FreeBSD to a smartphone [needs-feedback] From: Giovanny Andres Gongora Granada To: wkoszek@FreeBSD.org, freebsd-mobile@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 02:48:31 -0000 Hello, My name is Giovanny, I am an student from Universidad Cooperativa de Colombia planning to participate this year in the gsoc with FreeBSD organization. I reviewed the project list [1] for this year and I am interested in propose a project for Port FreeBSD to a smartphone (any maker and model) [2]. I will share with you my idea, I really appreciate have some feedback for my final proposal in the gsoc page. FreeBSD has a good compiling process for ARM [3] architecture and hardware. Taking advantage of this FreeBSD could run on a smartphone by three ways: 1. Using Android (based device and build process) we can compile FreeBSD kernel with support to the hardware of the device, then use a SD as system partition and configure it as a normal FreeBSD system. Flash an image generated for boot, equal to zImage in the block partition of the device using fastboot, so at reboot it will be running the FreeBSD system. After have the device running is more easy work on a display lib for the device. At the end if it can boot a device using this process and emulator could do this (changing the boot image for the one with FreeBSD) too, I use AOSP to define an environment and a set of tools to compile a light emulator with just the components necessaries to run the FreeBSD system. Mozilla does this to build Firefox OS with AOSP (devices and emulators) [4]. [This is not a image to chroot-jail] 2. Porting the idea of Gentoo for Android [5] to FreeBSD, this could be more difficult than expected due to port the concept of Gentoo RAP to FreeBSD could take a long time of research. Added to it, Gentoo RAP is made to work over Linux, so will take more time adapt the concept to FreeBSD and get something working with the low level layer of Android in the device. If by X reason the concept of Gentoo for Android it's done for FreeBSD could be possible think on port and use Webtop [6] as output for X servers how was done in Ubuntu for Android. 3. Running FreeBSD directly on an Android device using an application that emulates-wraps a terminal to boot a pre-compiled obb (just a tar package with a different name) package of a FreeBSD system. It will provide a shell to access all the programs in FreeBSD system but only text-mode will be supported. To interact from the smartphone with a graphical environment in FreeBSD there are to ways, first use other apps like a vnc client or embed a custom vnc/display client with own libs to display the enviroment. This method doesn't require root on the phone to run FreeBSD and the final apk of the application to run FreeBSD could be published in the Play store (if the idea is to publish it), so everyone could install it and run FreeBSD instantly after apk installation from Play Store. Three possible ways but personally I see 1 and 3 are the ones with best results. Number 1 is the way we more friendly for daily testing and with more percentage for future development. It will keep compatibility with latest changes in arm build process and after new CPU and boards support is added, will be easy build and test on more devices with this method. Number 3 it's more a user side and probably not offers a complete FreeBSD system experience as expected. I see that part of the gsoc is R&D, so I will be happy to research and develop until the end with all the ideas to get the best experience of FreeBSD on a phone. I know all this needs to be more detailed to get and idea if I am wrong or not, so feel free to quote where you think needs clarification in the process. Any feedback or opinions about this is really appreciated to my final proposal in the gsoc page. Thanks, Gio ---------------------------------------------------------------------- * About me: I am studying at University Cooperativa de Colombia, my future degree will be Systems Engineer. I had been contributing to open source project since 4 years ago, today I am an active Mozilla contributor [7], I work sending patches to the Firefox and Firefox OS code base [8]. Also I am part of the reviewer team at Firefox Marketplace (I review the apps that are sent to the Marketplace to be published) and I am a Mozilla Reps [9] too. I contribute to other open source projects too (Tor Project time ago as ES translator) different to Mozilla, you can see some of my contributions to other projects in my Github profile [10]. Most of my projects are open sourced in Github [11] and those projects are coded using different languages, some of them are C, Go, Python, Javascript, Ruby, Java and some shell scripts. * Previous experience with FreeBSD: I maintain some FreeBSD systems running in my University and a year ago I helped to run some FreeBSD servers with an IT team for a hosting provider company. ---------------------------------------------------------------------- [1]: https://wiki.freebsd.org/SummerOfCodeIdeas [2]: https://wiki.freebsd.org/SummerOfCodeIdeas#Port_FreeBSD_to_a_smartphone_.28any_maker_and_model.29 [3]: http://www.freebsd.org/platforms/arm.html [4]: https://github.com/mozilla-b2g/B2G [5]: https://wiki.gentoo.org/wiki/Project:Android [6]: http://sourceforge.net/motorola/motorola-webtop/home/Home/ [7]: https://mozillians.org/en-US/u/gioyik/ [8]: http://hg.mozilla.org/mozilla-central/log?rev=gioyik%40gmail.com [9]: https://reps.mozilla.org/u/gioyik/ [10]: https://github.com/Gioyik/ [11]: https://github.com/Gioyik?tab=repositories From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 06:50:07 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 533A5BE0; Mon, 23 Mar 2015 06:50:07 +0000 (UTC) Received: from mail-ob0-x22e.google.com (mail-ob0-x22e.google.com [IPv6:2607:f8b0:4003:c01::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 19493770; Mon, 23 Mar 2015 06:50:07 +0000 (UTC) Received: by obcjt1 with SMTP id jt1so96205104obc.2; Sun, 22 Mar 2015 23:50:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=jrYfHkJgE1QbFDLHC7d6bc64kpvn3SRr94Dxqd98LIU=; b=Z0ejQ3F6KcGbKKil9ZoAShaqMcfuHExZKyATjIWVin3moiYv3gZx5LticC77O8ZpAQ Sshq8MA4scwJa/0m3L3+fF8ftkjA+i7P3C6dUL27OX0zkMf0Gtx4tigOMwjWCJXrIXkb bLoa/Jm88pWrCn1YN+7nY/9SI9IbFmBcCA4Rm08bTK/ixtkdC9O6Qp2jcFjAl67gCU+U 4O9aqBnyKM9EpBf6lqJRp6lADn0wigj4WaW5yFJ36soCmIBfB8+oX7/t6H0V3WhbQjb7 DF6PU9bpkbRvnIx0o8C8hULKF+dnUayCdVzejzW8PJu4KiDzZg1yF1eEvfjPccpDK+LP 6yxg== MIME-Version: 1.0 X-Received: by 10.202.79.20 with SMTP id d20mr6331284oib.96.1427093406063; Sun, 22 Mar 2015 23:50:06 -0700 (PDT) Sender: pali.gabor@gmail.com Received: by 10.182.135.35 with HTTP; Sun, 22 Mar 2015 23:50:05 -0700 (PDT) In-Reply-To: References: Date: Mon, 23 Mar 2015 07:50:05 +0100 X-Google-Sender-Auth: YVeZL6t5sytIPq_0yBEmaCQbf90 Message-ID: Subject: Re: Self introduciton of Hao Sun and thoughts on LibNetstat From: Gabor Pali To: Hao Sun Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: George Neville-Neil , Robert Watson , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 06:50:07 -0000 Hi Hao, 2015-03-22 13:09 GMT+01:00 Hao Sun : > Following your guidance, I've cloned the FreeBSD mirror from GitHub and w= ill get down to > have an initial scratch with the latest version. That is great. I forgot to mention, but maybe it is also said on the introductory wiki pages you have also cited originally, that it is probably the best if you try to build the userland ("world") and the kernel on your VM and install it first. This should give you the basic feel for the development cycle and help to spot problems with the clean system itself before you start hacking on it. The development branch of FreeBSD (that is called "current") shall build and install just fine for most of the time, but do not be discouraged if not, ask for help. > On the project's wiki page [1], I think the target for GSoC > 2015 is to finish the tasks haven't been done in the following table. But= according to your > comments in the emails, it seems like I need to start the job from scratc= h. Thus the question > is should I keep the existing code and add new features to the previous v= ersion or just start > the project from the very beginning? I might have sounded a bit pessimistic, I do not necessarily insist on rewriting the entire library :-) I think it is just common sense: study the current implementation, take a look at the FreeBSD ecosystem and kernel, discuss the topic with the interested hackers, and work out your proposal based on your findings. You may find some of the old code base and concepts reusable, which is excellent, and you may decide to take another approach for the rest. It might be worthwhile to accommodate some "stretch goals" in your proposal if you accidentally completed your summer task too quickly :-) For making things a bit easier (hopefully) for you, I may also include George Neville-Neil in the conversation (see him CC'ed) who has shown some interest in driving this library into the base system in the past if I recall correctly. Along with Robert, he is also a high-profile src committer, with experience in networking and related areas. (And also a potential mentor for this project as well?) Cheers, G=C3=A1bor [1] https://wiki.freebsd.org/LibNetstat From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 06:57:03 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9F471E8E; Mon, 23 Mar 2015 06:57:03 +0000 (UTC) Received: from mail-ob0-x22e.google.com (mail-ob0-x22e.google.com [IPv6:2607:f8b0:4003:c01::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5D7D5841; Mon, 23 Mar 2015 06:57:03 +0000 (UTC) Received: by obbgg8 with SMTP id gg8so116475216obb.1; Sun, 22 Mar 2015 23:57:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=GPNxdzMpqmC/j3MQUTQt67Yh1tYUJFrx4KubX2O4rjc=; b=RN8FBwTudB0wumHt5UjkOquZY5x5i9RhXooo3ndn6nSCp5zttw7uEc1T6EYmkKjVJi 4glSvh+rudtGR/JGHaA4crG7OaYP4ESFHNC6e6OigNXbA4if7ZpvfaGOmQGbNuYxUjnY F0y2IDQuj4l+LrPXZTulYwJpHk6DHRCTSvzcj/xc4aPI++tzuH3bFtTb9E+O5aMhxiI6 NUZz1yk8gjm04aewE8MZ2GhMH+gmCtsxTG0/2thjDkCJuZHIMHuCBOteg+80mdIJdFHG zJdNJTOacWwfhaZX1MyhWg+vSMnIbk+8VwrqyHX953mzDU/pJ2rpJ1ovhTe5Tdn58vcJ zMTg== MIME-Version: 1.0 X-Received: by 10.182.33.98 with SMTP id q2mr69388093obi.79.1427093822765; Sun, 22 Mar 2015 23:57:02 -0700 (PDT) Sender: pali.gabor@gmail.com Received: by 10.182.135.35 with HTTP; Sun, 22 Mar 2015 23:57:02 -0700 (PDT) In-Reply-To: <826E80C6-55BE-4690-B35B-14670EE482AD@FreeBSD.org> References: <826E80C6-55BE-4690-B35B-14670EE482AD@FreeBSD.org> Date: Mon, 23 Mar 2015 07:57:02 +0100 X-Google-Sender-Auth: mysjihzCT8M0QK0XyPBNUQ_rric Message-ID: Subject: Re: Self introduciton of Hao Sun and thoughts on LibNetstat From: Gabor Pali To: "Robert N. M. Watson" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Hao Sun X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 06:57:03 -0000 Hi Robert: 2015-03-22 14:52 GMT+01:00 Robert N. M. Watson : > Tuning in slightly late: I=E2=80=99m not sure if the bsnmp=E2=80=99s plug= ins might also want > to learn about libnetstat and friends? Indeed that can be also a nice goal to achieve. However, based on my previous experiences, would not it be a better to set the "make this into the base system" as the main theme? It may not sound like a "true" project, but preparing the current code base for this can easily span over months, especially for someone who has not worked with the FreeBSD base system and kernel before. From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 08:47:44 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2A6E6610; Mon, 23 Mar 2015 08:47:44 +0000 (UTC) Received: from mail-yh0-x229.google.com (mail-yh0-x229.google.com [IPv6:2607:f8b0:4002:c01::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D2C53664; Mon, 23 Mar 2015 08:47:43 +0000 (UTC) Received: by yhjf44 with SMTP id f44so65192487yhj.3; Mon, 23 Mar 2015 01:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=BwA5VnHqK0W8TI/LUNELBlaZKLPlnjdBKb8z4jWgC8Y=; b=bEoRpXqpFSYMBSE/wBInSzqNj8NgGf0INK6ox96EoBfh/9Noo6NJ5fi8oVZcf/KPw0 qHESQgW+8IraBDQrQbrpHv6t//C6GEURVgEjkbU1ChMDShdx7e0OAnCP1TwZGYbIiAnX 2qYvVGzznlTf1D1EGvoDbxDAmqdyIUIFGI/yze8eJCoCmK8CNyUZGuM/5gncOp6msq94 hf5tqvfWiYmkYLzYa/0G2a9swrncu7huKqfSK2cM2MzkUsxRjmVptGVF8Ae0+EatfL/v aib9NC4lvvBzXdX0lWAbjfj7yDByMot2CNbjpvJkXsOU0hrGxqpiutRsdU7VbGRLQSDK LGTA== MIME-Version: 1.0 X-Received: by 10.236.228.2 with SMTP id e2mr93050759yhq.122.1427100462906; Mon, 23 Mar 2015 01:47:42 -0700 (PDT) Received: by 10.170.60.69 with HTTP; Mon, 23 Mar 2015 01:47:42 -0700 (PDT) In-Reply-To: References: Date: Mon, 23 Mar 2015 01:47:42 -0700 Message-ID: Subject: Re: Self introduciton of Hao Sun and thoughts on LibNetstat From: Mehmet Erol Sanliturk To: Gabor Pali Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: George Neville-Neil , Hao Sun , Robert Watson , FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 08:47:44 -0000 On Sun, Mar 22, 2015 at 11:50 PM, Gabor Pali wrote: > Hi Hao, > > 2015-03-22 13:09 GMT+01:00 Hao Sun : > > Following your guidance, I've cloned the FreeBSD mirror from GitHub and > will get down to > > have an initial scratch with the latest version. > > That is great. I forgot to mention, but maybe it is also said on the > introductory wiki pages you have also cited originally, that it is > probably the best if you try to build the userland ("world") and the > kernel on your VM and install it first. This should give you the > basic feel for the development cycle and help to spot problems with > the clean system itself before you start hacking on it. The > development branch of FreeBSD (that is called "current") shall build > and install just fine for most of the time, but do not be discouraged > if not, ask for help. > > > On the project's wiki page [1], I think the target for GSoC > > 2015 is to finish the tasks haven't been done in the following table. > But according to your > > comments in the emails, it seems like I need to start the job from > scratch. Thus the question > > is should I keep the existing code and add new features to the previous > version or just start > > the project from the very beginning? > > I might have sounded a bit pessimistic, I do not necessarily insist on > rewriting the entire library :-) I think it is just common sense: > study the current implementation, take a look at the FreeBSD ecosystem > and kernel, discuss the topic with the interested hackers, and work > out your proposal based on your findings. You may find some of the > old code base and concepts reusable, which is excellent, and you may > decide to take another approach for the rest. It might be worthwhile > to accommodate some "stretch goals" in your proposal if you > accidentally completed your summer task too quickly :-) > > For making things a bit easier (hopefully) for you, I may also include > George Neville-Neil in the conversation (see him CC'ed) who has shown > some interest in driving this library into the base system in the past > if I recall correctly. Along with Robert, he is also a high-profile > src committer, with experience in networking and related areas. (And > also a potential mentor for this project as well?) > > Cheers, > G=C3=A1bor > > [1] https://wiki.freebsd.org/LibNetstat > _______________________________________________ > > To improve the acceptance of "Testing" concepts , another good point may be to define a testing facility of the developed sources through use of Jenkins during the development : https://github.com/rodrigc/kyua/wiki/Quickstart-Guide https://wiki.freebsd.org/Jenkins Following the principles like defined in the following pages may be useful ( I think these are ignored grossly in many open ( I do not have any idea about closed ) source projects ) : http://en.wikipedia.org/wiki/Software_development_process http://en.wikipedia.org/wiki/Requirements_analysis http://en.wikipedia.org/wiki/Software_design http://en.wikipedia.org/wiki/Software_construction http://en.wikipedia.org/wiki/Software_testing http://en.wikipedia.org/wiki/Debugging http://en.wikipedia.org/wiki/Software_deployment http://en.wikipedia.org/wiki/Software_maintenance Following the above steps and producing reports by considering them , will make the generated software much more easily maintainable by several people and over time even by the initial developer . Thank you very much . Mehmet Erol Sanliturk From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 18:26:28 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F1E06522; Mon, 23 Mar 2015 18:26:27 +0000 (UTC) Received: from mail-ob0-x236.google.com (mail-ob0-x236.google.com [IPv6:2607:f8b0:4003:c01::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B38117E5; Mon, 23 Mar 2015 18:26:27 +0000 (UTC) Received: by obbgg8 with SMTP id gg8so129707154obb.1; Mon, 23 Mar 2015 11:26:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=+Z3HP7AkGyG3HEPharCgLTTNQ2HLsXjyt2Kmf0u5NAM=; b=L+km4hUnljVQk41hm5Dki1JFv+RwD1/lK1WUFt/En0BmkSSONn1TLT5ljKS0InUwLr eRc3y3jGCurUijZCoJJv2bR3lAWYhVGAJAu/dfrgXtYqMOSbDEUpvRMK/V6NmmXxn3uz 7ofGqimxRWLeGdbMxTPWf5YAvDN/y4vA20Cno43lYD2vmNKLxNqhaX6IYwUCpc/CWeu8 8qjf+XsNuqcKICOmX2+hhYpXq0iwW4l+yLLxPFQsIHnwjI4BdQBqolucjT8m7XiYhDXY b+A7OyfaFCIWMkEAT8en4VpkCHazRZUkcGm/a2sIA4v/f5qPEq2esBj6R1mP8CSZ+sWS gLnQ== MIME-Version: 1.0 X-Received: by 10.202.230.69 with SMTP id d66mr414847oih.8.1427135187005; Mon, 23 Mar 2015 11:26:27 -0700 (PDT) Received: by 10.182.248.164 with HTTP; Mon, 23 Mar 2015 11:26:26 -0700 (PDT) Date: Mon, 23 Mar 2015 14:26:26 -0400 Message-ID: Subject: Time to be real From: Joe Nosay To: freebsd-current , FreeBSD PowerPC ML , FreeBSD Hackers , FreeBSD Mailing List Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 18:26:28 -0000 Go to hell. Go fuck yourselves. From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 20:46:23 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 671151E8 for ; Mon, 23 Mar 2015 20:46:23 +0000 (UTC) Received: from mail-we0-x22c.google.com (mail-we0-x22c.google.com [IPv6:2a00:1450:400c:c03::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E9D22B58 for ; Mon, 23 Mar 2015 20:46:22 +0000 (UTC) Received: by wegp1 with SMTP id p1so147723327weg.1 for ; Mon, 23 Mar 2015 13:46:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6xL+trFmF+2H2314n8eynAZJnsfn8uQyQAIM34rWT5g=; b=eLvgnGmPMmzZUtryeBJ54VzA/OI7/jADPClhBdfsz4jSPZ5679GtxfGQbnlp/BI+VD 5KYLxb3FXTE7hKl+iDXPzNMX4oZHj7puAKOI1vadma5BfFNJ2k9puXOgViqU6YGaejM9 5P0mPhRvBwulCiLNhTH0quSIDu9Li5vlkqnylvpzvsf9nnKXLgJxXga1P+7M9i9cOfS4 itHOZILxDsnUx5E9FhIFIHdO7GO0rW/BoGUmmk1KyMPbgnc15zcbW9pRHvSqFPV/cjz/ loHH/ZhD1guLJ/nn5BJEjNT7XhiT9YtLje3dNpDiiGho4Yqlup4lucHhuNsGODhh+m7n EfcQ== MIME-Version: 1.0 X-Received: by 10.194.179.41 with SMTP id dd9mr1882866wjc.72.1427143580472; Mon, 23 Mar 2015 13:46:20 -0700 (PDT) Received: by 10.194.18.37 with HTTP; Mon, 23 Mar 2015 13:46:20 -0700 (PDT) In-Reply-To: <20150319013231.GR51048@funkthat.com> References: <20150319013231.GR51048@funkthat.com> Date: Mon, 23 Mar 2015 17:46:20 -0300 Message-ID: Subject: Re: GELI support on /boot folder From: Pedro Arthur To: John-Mark Gurney Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 20:46:23 -0000 Based on your idea the project could be merge the GPT boot stages into a single program with support for GELI and instead of having a binary for zfs and other for ufs we could have the boot program with support for both file systems so that we can choose to boot from any partition in the GPT being zfs or ufs. What I want to know is if it is a good idea or merging these boot stages has any drawback or infringes any design choices? 2015-03-18 22:32 GMT-03:00 John-Mark Gurney : > Pedro Arthur wrote this message on Wed, Mar 18, 2015 at 15:50 -0300: > > I was discussing with Kris Moore about adding support for GELI in > > bootloader as a GSoC project, > > thus the /boot folder could be encrypted. > > However the stage 2 boot program has a limit size of ~8 Kb which is > almost > > reached in the default > > HEAD src. > > Thus I would like to know your thoughts about this project, if it is > > viable, and what can be done to > > overcome these 8 Kb limit. > > One option is to not support MBR and only support GPT for this... w/ > GPT we do not have the 8k limitation (and actually the limit is 7.5k > as .5k has historically been used for MBR boot code/partition table > in the dangerously dedicated mode)... > > If we go thise route, I'd ask why we don't put loader into the gptboot > instead of using the existing shim to load loader... Then the project > would be to add GELI decryption to loader which could then be used > w/ MBR in the limited sense of loading kernel and modules, though > boot/loader would still have to be on an unencrypted partition... > > I hope others who know the boot process better will inform us why > this is a good or bad idea... > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 21:14:33 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 18447ED7; Mon, 23 Mar 2015 21:14:33 +0000 (UTC) Received: from udns.ultimatedns.net (unknown [IPv6:2602:d1:b4d6:e600:4261:86ff:fef6:aa2a]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1850E5E; Mon, 23 Mar 2015 21:14:32 +0000 (UTC) Received: from ultimatedns.net (localhost [127.0.0.1]) by udns.ultimatedns.net (8.14.9/8.14.9) with ESMTP id t2NL89Gp052297; Mon, 23 Mar 2015 14:08:54 -0700 (PDT) (envelope-from bsd-lists@bsdforge.com) To: freebsd-current , FreeBSD PowerPC ML , FreeBSD Hackers , FreeBSD Mailing List In-Reply-To: References: From: "Chris H" Subject: Re: Time to be real Date: Mon, 23 Mar 2015 14:08:54 -0700 Content-Type: text/plain; charset=UTF-8; format=fixed MIME-Version: 1.0 Message-id: Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 21:14:33 -0000 On Mon, 23 Mar 2015 14:26:26 -0400 Joe Nosay wrote _____________________________ /____________________________/| | ___ || | /__/| || | PLEASE | || || | | || || | DO NOT FEED | || || | |__|/ || | THE TROLLS ___ || | /__/| || | |__|/ || |____________________________|/ | || | ||/| |\|/\||/| ________/\// /\/ |_____________ > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 22:37:30 2015 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 55116579; Mon, 23 Mar 2015 22:37:30 +0000 (UTC) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DE991972; Mon, 23 Mar 2015 22:37:29 +0000 (UTC) Received: by wibdy8 with SMTP id dy8so60084902wib.0; Mon, 23 Mar 2015 15:37:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=ahRu5W1LwXPnUVcT7pMDVW/EcCMK/PBWDS2veKk8vt8=; b=x9hGIQ/gZbF5H678c3kCEC0n/A/F4FcckqUdf044u5OD01M3tJ/+oOH95kbn3WAw5J ZiRcyLWL7gGORQJGgjN5yitUe0RkBX7VNhztIdTQ90uOZzairpi3LmCXypAyAuiY12Vi V27zPzU8T5ltF1vEFSLoUWuY/4cfI3tdNp++4zP1ua++M5K8pJYAr8k8fwo4ipDTkoSr kEP45b9dH4AQceP2YSApjwfSIw30tGG6Tm2jR405tzRrEQTokEM0087tMgqXJET4FOba xaz31tkPkYRlYNt9UAYzN5bhj/t44QcpdXvFjOIoIbmk+RPDLQEQQA0F51lfNnvOsrw6 2+Rg== MIME-Version: 1.0 X-Received: by 10.194.108.137 with SMTP id hk9mr2339923wjb.112.1427150248269; Mon, 23 Mar 2015 15:37:28 -0700 (PDT) Received: by 10.194.58.162 with HTTP; Mon, 23 Mar 2015 15:37:28 -0700 (PDT) Date: Tue, 24 Mar 2015 00:37:28 +0200 Message-ID: Subject: ports errors classification From: umka ursa To: freebsd-ports@freebsd.org, hackers@freebsd.org X-Mailman-Approved-At: Mon, 23 Mar 2015 22:43:30 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 22:37:30 -0000 Hello comunity! Im looking for documentation about ports instalation error codes. I meen somthing like this: *** [install] Error code 1 *** [build-depends] Error code 1 Can not find it yet. Can you shere it please? From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 22:55:55 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 20F6CB01; Mon, 23 Mar 2015 22:55:55 +0000 (UTC) Received: from mail-out.smeets.im (mail-out.smeets.im [5.9.17.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C177BB81; Mon, 23 Mar 2015 22:55:54 +0000 (UTC) Received: from mail.smeets.im (mail.smeets.im [IPv6:2a01:4f8:160:918a::25:3]) by mail-out.smeets.im (Postfix) with ESMTP id 30D9613D0; Mon, 23 Mar 2015 23:55:46 +0100 (CET) Received: from amavis.smeets.im (amavis.smeets.im [IPv6:2a01:4f8:160:918a::aa:4]) by mail.smeets.im (Postfix) with ESMTP id B1144892AF; Mon, 23 Mar 2015 23:55:46 +0100 (CET) X-Virus-Scanned: amavisd-new at smeets.im Received: from mail.smeets.im ([IPv6:2a01:4f8:160:918a::25:3]) by amavis.smeets.im (amavis.smeets.im [IPv6:2a01:4f8:160:918a::aa:4]) (amavisd-new, port 10025) with ESMTP id IM0x-EBallLC; Mon, 23 Mar 2015 23:55:46 +0100 (CET) Received: from nibbler-wlan.home.lan (unknown [85.22.28.128]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.smeets.im (Postfix) with ESMTPSA id 0CFFF89287; Mon, 23 Mar 2015 23:55:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=smeets.im; s=default; t=1427151346; bh=wzjfAI8SnrISTLRGX19ytl4rEYHaZEODHnqpTr5Udto=; h=Subject:To:references:From:Date:in-reply-to; b=kutD6pJylQBUYXFARnuDnVu95CpEgm7ILbiBGFZC9HfDLOl4mGCJUfpc/GhNx3Yu3 ttYaF0BwfL7LXqYNZMvOM2/kHPP4SEjP1tkTvLtY/qherX/nmadmu1shgV5br08uEp aCctdQpZYc3moR7ncKovIU3e6PZhRs3/vBoPqNew= Subject: Re: Time to be real To: Joe Nosay , freebsd-current , FreeBSD PowerPC ML , FreeBSD Hackers , FreeBSD Mailing List , postmaster@FreeBSD.org references: From: Florian Smeets message-id: <551099EF.60606@smeets.im> Date: Mon, 23 Mar 2015 23:55:43 +0100 user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.0a2 mime-version: 1.0 in-reply-to: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="QiLcX1B4EWATpqjQXNOcBHiknGSwbaaap" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 22:55:55 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --QiLcX1B4EWATpqjQXNOcBHiknGSwbaaap Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi, Joe has indicated in the past that SPAM was sent from his account: https://lists.freebsd.org/pipermail/freebsd-ports/2014-September/095407.h= tml We (postmaster@) contacted Joe and are looking into the issue. Please do not reply to the thread anymore. Florian --QiLcX1B4EWATpqjQXNOcBHiknGSwbaaap Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQJ8BAEBCgBmBQJVEJnvXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNzAxMDMyMDNCQ0FCNDRBOThGRUM4NDRF NzA1M0RGOUZGODZGMDc2AAoJEOcFPfn/hvB2rvAQAItSKctjP+p57HkCDl/IEh80 fRIkU+z6NJwEC11kybV2md/tyFT+pKSLCFGPsUx7H8+X31Z0neabkNkJFUeeNn2+ irfnuLy3tw3jV9CC2CJcQoLLcVVWrLlOzXqW2O0uBx4vNAFAg+XF7s66T/6K19Lq c2WCN3F/2Qch27QiK0I5vCTUhEcb14Uk994KZnJx3pkTTnVTx4yQ1TSN1FPIDRlh 5As0GuD0M52ORAG1iZ5Fat4T9nit1omxV0cfPJqiqy3uM9cofoutvz8FjT8lTn+1 vKq2Ba23dOJQk1fxfIc4v51/21DFYSAqcV2c6SfsKB8Zicua/aZKz84vhLfbdYm9 sZ6Kdb8WL3LuboUc5Sj2utdpH8XouVJQA4F0ChCIHvcxsUNLhnl22CFumqc/RZ13 GE1bp/pGUpymlJDHSXZndyLSsZUvH+608eMTJQwYiUybro8BeMX0Nomc0iW2GCbN PRZH3ivRE27NsoONSa4TZRQUVL6Qs9DyEHDSjkLvD4b+J0pYncWRzV7EExAUzk5b bJbVdKC+OWZsSA20XJGl3RVQsLym8FFvmkLa7dVTCX258QPDP3W9YW5kudibirGv dfQCvMvyLykZjzlNYUTXTWSkzDoeGDn2FHlax6eBGAVPLvmdqSMd1EEYrnMW8LHR FK1HZijS3Qhwkrb3ypV/ =+A9w -----END PGP SIGNATURE----- --QiLcX1B4EWATpqjQXNOcBHiknGSwbaaap-- From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 23 23:25:37 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 143A454F for ; Mon, 23 Mar 2015 23:25:37 +0000 (UTC) Received: from mx1.scaleengine.net (beauharnois2.bhs1.scaleengine.net [142.4.218.15]) by mx1.freebsd.org (Postfix) with ESMTP id E0A80E71 for ; Mon, 23 Mar 2015 23:25:35 +0000 (UTC) Received: from [192.168.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 71B779F3F0 for ; Mon, 23 Mar 2015 23:25:34 +0000 (UTC) Message-ID: <5510A0F4.4090402@freebsd.org> Date: Mon, 23 Mar 2015 19:25:40 -0400 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: ports errors classification References: In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="osJuELwwQdMeE5tsElDSXCObrOiSdujli" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 23:25:37 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --osJuELwwQdMeE5tsElDSXCObrOiSdujli Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015-03-23 18:37, umka ursa wrote: > Hello comunity! > Im looking for documentation about ports instalation error codes. I mee= n > somthing like this: >=20 > *** [install] Error code 1 > *** [build-depends] Error code 1 >=20 > Can not find it yet. > Can you shere it please? > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.o= rg" >=20 Those error codes come from 'make', and are meaningless. Scroll up further to see the actual error. If you are at a terminal, you can use the scroll lock key to be able to scroll up with your arrow keys. --=20 Allan Jude --osJuELwwQdMeE5tsElDSXCObrOiSdujli Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJVEKD2AAoJEJrBFpNRJZKfIngP/31aT6IPNSmpnN+PqLMFy64b vWMmwn3PzcyZTk0rqiZ3WCCKDZhT7tiSohqcJGYh5tjFbX3KD4ptJl4ECz7l90dB MEJAmUULyzwNJrmSl0WbHeDQhnD6hLZvI3Ilxf2Fzc01M/xFQc1T0/cRzppvrvd9 67nXlQdU/QghPtfwWEEv9fHDpI6fZ18u+reStJ8+O+pPZjcgG2+JTFZvMVQGb5uS DfPB3KH5IHuqqmHcPo4jfv7wI+tGbH5aJij93uxxRuGm9cfW+QslbAnKhD1Td05L 5CpqQfxaVyAiKug0M4/vb2i3WvNoiVIX+xlExxLylg19xzR1TsRQVy0OYAFDodqU t5kIPClH6Wj0HqONbTvn26IjLBlEq8onB2oTCEUVovyqudziTnJENpsTZoX+7bXj +lB9OpCjHcCb8+JVpXYY40jHieFvvcJbluipGExxztfn+au3bMob8P7q1JYVUWIm vvkp2R/HqAIV+7Uog8JemVFnuwzjc4oEeUOrjSOnI2QmUZXWkzANlYFI3YHj2AAE OQFyH6HrPwVlcMO7OC+qr3GtggiUhpaa8kxTi8eCK5z/teiJtRfIydVxLY2FmTXG mZIDqOYodgUP1vwz93gMzbapqelXHBEXVzrM3qQ6mTn5iZYH+sgV4jL/PNweORti rizX3rMCvJaQkYt1BLRq =wy2Z -----END PGP SIGNATURE----- --osJuELwwQdMeE5tsElDSXCObrOiSdujli-- From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 24 03:27:45 2015 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BA2BA7CD; Tue, 24 Mar 2015 03:27:45 +0000 (UTC) Received: from mail-ob0-x22a.google.com (mail-ob0-x22a.google.com [IPv6:2607:f8b0:4003:c01::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78FD2CF4; Tue, 24 Mar 2015 03:27:45 +0000 (UTC) Received: by obdfc2 with SMTP id fc2so138393209obd.3; Mon, 23 Mar 2015 20:27:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=BGRTABMgehd4qvcsmgNZVpNVUUsBksCquvKIzLVQZIo=; b=wxrJhd5ItH/0IPc41w+ETjbqFRM1dpyDsd0U5DGjyDHkAY1FJ85y0jEzMqaIj1jYZi CP0QGlEu+U4C8MeeVCbLWxy8CaIxwM9fn5REzqw5UjoUNEXT9hBE+bW+OWQVX6w2OyHz G3gTcAiirsgPo8p6Iw5aFmH0kAYH46jh3UeKME5SirxfEdYTrUSFHaA9zRdp7uG0KV5i FOyndB6r9VzZmRYTxfc4yWooaXXl92GGiBqYy/lLL54WCXEBKHjXw+NI3+Ae+qCtOPmR RaGnUz3e3G0A31oXW+rwHiajOZL9CLFiRBV3HJnYXLtNiUlb6z27ejgWcXIZp5l3gcs1 7HBA== MIME-Version: 1.0 X-Received: by 10.182.22.167 with SMTP id e7mr1651833obf.31.1427167664761; Mon, 23 Mar 2015 20:27:44 -0700 (PDT) Sender: robbak@gmail.com Received: by 10.76.145.9 with HTTP; Mon, 23 Mar 2015 20:27:44 -0700 (PDT) In-Reply-To: References: Date: Tue, 24 Mar 2015 13:27:44 +1000 X-Google-Sender-Auth: _P-WCu2aMKitHFzbJAMG6dIiR8M Message-ID: Subject: Re: ports errors classification From: Robert Backhaus To: umka ursa X-Mailman-Approved-At: Tue, 24 Mar 2015 04:17:58 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: hackers@freebsd.org, Freebsd_mailinglist_PORTS X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2015 03:27:45 -0000 The code is just the value returned by the last process to run. Processes that complete successfully return 0, and if they fail, they return something else. So if you want to know what the codes mean, you have to look at the documentation - generally the man page - for the process, such as the compiler, or the 'cp ' command. That said, the number is generally 1 for any failure. On 24 March 2015 at 08:37, umka ursa wrote: > Hello comunity! > Im looking for documentation about ports instalation error codes. I meen > somthing like this: > > *** [install] Error code 1 > *** [build-depends] Error code 1 > > Can not find it yet. > Can you shere it please? > _______________________________________________ > freebsd-ports@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 24 12:35:23 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A7001D74 for ; Tue, 24 Mar 2015 12:35:23 +0000 (UTC) Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com [IPv6:2a00:1450:400c:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 305DAFBF for ; Tue, 24 Mar 2015 12:35:23 +0000 (UTC) Received: by wibdy8 with SMTP id dy8so73646486wib.0 for ; Tue, 24 Mar 2015 05:35:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=DDoKnqvkThLnADxkAJK6aVCACqKYToNyqyGigo0/48A=; b=bT9I1jG4E7ikMVO5ot62qJAmkSsiyV4m/UD0R0Jw2a9E2v3CMScgecXeG04ps6Ikqf e9T6Vq1LoyqEAiOTR9SbVCl+ZlLuw4wa62G8UMb+6goLqz8nkvjv0dpuaGxLt3GHUXg0 0vz14bDJ6TiEeBwBJ4NrELo/rgSk/rS51h1fITjdxIX5xTPB3u8lAmNzLLs6LZB0nzxX PSZpQ7GN56P1Ddo+YzbIR5TdW0r3VjSKp4hk+NnYo0h0JwYa/QZnYS/4/BMLET/UxEpn M3EG7DZozEB98kHdARb0t5UNBnmUskrsg5SUSixJW2MNFpT5ppDeLDW08fV/V0UoZ1xn WYWw== X-Received: by 10.194.24.103 with SMTP id t7mr7419906wjf.15.1427200521588; Tue, 24 Mar 2015 05:35:21 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id uo6sm5882647wjc.49.2015.03.24.05.35.19 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Tue, 24 Mar 2015 05:35:20 -0700 (PDT) Date: Tue, 24 Mar 2015 13:35:17 +0100 From: Mateusz Guzik To: Tiwei Bie Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150324123517.GA25678@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Tiwei Bie , Konstantin Belousov , freebsd-hackers@freebsd.org References: <1426946345-67889-1-git-send-email-btw@mail.ustc.edu.cn> <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> <20150322113822.GB2379@kib.kiev.ua> <20150322120655.GA59757@freebsd> <20150322131524.GA95795@freebsd> <20150323005852.GB6798@dft-labs.eu> <20150323020314.GA30143@freebsd> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150323020314.GA30143@freebsd> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2015 12:35:23 -0000 On Mon, Mar 23, 2015 at 10:03:14AM +0800, Tiwei Bie wrote: > On Mon, Mar 23, 2015 at 01:58:52AM +0100, Mateusz Guzik wrote: > > On Sun, Mar 22, 2015 at 09:15:24PM +0800, Tiwei Bie wrote: > > > On Sun, Mar 22, 2015 at 08:06:55PM +0800, Tiwei Bie wrote: > > > > On Sun, Mar 22, 2015 at 01:38:22PM +0200, Konstantin Belousov wrote: > > > > > On Sun, Mar 22, 2015 at 07:25:55PM +0800, Tiwei Bie wrote: > > > > > > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > > > > > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > > > > > > > [..] > > > > I will submit my new patch later. > > > > > > > > > > Here is my new patch: > > > > > > --- > > > sys/kern/kern_sig.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++------- > > > 1 file changed, 64 insertions(+), 9 deletions(-) > > > > > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > > > index 8410d9d..5162ab6 100644 > > > --- a/sys/kern/kern_sig.c > > > +++ b/sys/kern/kern_sig.c > > > @@ -3094,21 +3094,81 @@ static int compress_user_cores = 0; > > > */ > > > #define corefilename_lock allproc_lock > > > > > > -static char corefilename[MAXPATHLEN] = {"%N.core"}; > > > +static char corefilename[MAXPATHLEN] = "%N.core"; > > > + > > > +static bool > > > +corefilename_check(const char *format) > > > +{ > > > + int i, counti; > > > + > > > + counti = 0; > > > + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { > > > + if (format[i] == '%') { > > > + i++; > > > + switch (format[i]) { > > > + case 'I': > > > + counti++; > > > + if (counti > 1) > > > + return (false); > > > + case '%': > > > + case 'H': > > > + case 'N': > > > + case 'P': > > > + case 'U': > > > + break; > > > + default: > > > + return (false); > > > + } > > > + } > > > + } > > > + > > > + if (i == MAXPATHLEN) > > > + return (false); > > > + > > > + return (true); > > > +} > > > + > > > +static void > > > +corefilename_init(void *arg) > > > +{ > > > + char *format; > > > + > > > + format = kern_getenv("kern.corefile"); > > > + if (format != NULL && corefilename_check(format)) > > > + strcpy(corefilename, format); > > > +} > > > +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); > > > > > > > The user should know that the tunable got discarded (e.g. like with > > prinf removed below). > > > > I don't know how to do this. We need to update the man-pages? > > And sorry again, maybe I was too tired yesterday. I just find a new bug in > the codes above when reading your comments. Though it won't affect anything > cause dynamic_kenv hasn't been enabled at this time, it is really a bug. > > `format' needs to be freed, sorry... What a terrible day. ;-( > core(5). yeah, would be nice to mention all this stuff. > > > static int > > > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > > > { > > > + char *format; > > > int error; > > > > > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > > > + > > > + sx_slock(&corefilename_lock); > > > + strcpy(format, corefilename); > > > + sx_sunlock(&corefilename_lock); > > > + > > > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > > > + if (error || !req->newptr || strcmp(format, corefilename) == 0) > > > + goto out; > > > + > > > > Is this strcmp useful? Normally you don't replace the pattern with > > identical one. > > > > I wrote these codes after consulting the codes in kern/kern_clocksource.c > [..] > It also needs to validate the user's inputs. And it uses a strcasecmp() > to check whether a different timer has been specified by the user. > > If the user is going to replace the pattern with identical one, we won't > need to do any update, we just `goto out'. > Well, I presume original author did not want to play with timers when changing to the same timer. Corefile is safe to change, so I suggest just dropping that strcmp. Also see below. > My new patch: > > --- > sys/kern/kern_sig.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++----- > 1 file changed, 70 insertions(+), 7 deletions(-) > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > index 8410d9d..0254517 100644 > --- a/sys/kern/kern_sig.c > +++ b/sys/kern/kern_sig.c > @@ -3094,21 +3094,85 @@ static int compress_user_cores = 0; > */ > #define corefilename_lock allproc_lock > > -static char corefilename[MAXPATHLEN] = {"%N.core"}; > +static char corefilename[MAXPATHLEN] = "%N.core"; > + > +static bool > +corefilename_check(const char *format) > +{ > + int i, counti; > + > + counti = 0; > + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { > + if (format[i] == '%') { > + i++; > + switch (format[i]) { > + case 'I': > + counti++; > + if (counti > 1) > + return (false); countI? > + case '%': > + case 'H': > + case 'N': > + case 'P': > + case 'U': > + break; > + default: > + return (false); > + } > + } > + } > + > + if (i == MAXPATHLEN) > + return (false); > + > + return (true); > +} > + > +static void > +corefilename_init(void *arg) > +{ > + char *format; > + > + format = kern_getenv("kern.corefile"); > + > + if (format != NULL) { > + if (corefilename_check(format)) > + strcpy(corefilename, format); > + freeenv(format); > + } > +} printf something. "invalid format specified" or the like. Also the code may be nicer if you if (format == NULL) return; instead > +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); > > static int > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > { > + char *format; > int error; > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > + > + sx_slock(&corefilename_lock); > + strcpy(format, corefilename); > + sx_sunlock(&corefilename_lock); > + > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > + if (error || !req->newptr || strcmp(format, corefilename) == 0) > + goto out; > + this should be if (error != 0 || req->newptr == NULL). strmcp was covered earlier in the mail > + if (!corefilename_check(format)) { > + error = EINVAL; > + goto out; > + } > + > sx_xlock(&corefilename_lock); > - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), > - req); > + strcpy(corefilename, format); > sx_xunlock(&corefilename_lock); > > +out: > + free(format, M_TEMP); > return (error); > } > -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | > +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | > CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", > "Process corefile name format string"); > > @@ -3171,9 +3235,8 @@ corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, > sbuf_printf(&sb, "%u", uid); > break; > default: > - log(LOG_ERR, > - "Unknown format character %c in " > - "corename `%s'\n", format[i], format); > + KASSERT(0, ("Unknown format character %c in " > + "corename `%s'\n", format[i], format)); > break; > } > break; > -- > 2.1.2 > > Best regards, > Tiwei Bie > -- Mateusz Guzik From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 24 14:37:49 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2D697E42 for ; Tue, 24 Mar 2015 14:37:49 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id 481CDCB for ; Tue, 24 Mar 2015 14:37:47 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygAnL8uudhFVUxfRAw--.36362S2; Tue, 24 Mar 2015 22:37:43 +0800 (CST) Date: Tue, 24 Mar 2015 22:37:09 +0800 From: Tiwei Bie To: Mateusz Guzik Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150324143709.GA54065@freebsd> References: <20150321200500.GC14650@dft-labs.eu> <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> <20150322113822.GB2379@kib.kiev.ua> <20150322120655.GA59757@freebsd> <20150322131524.GA95795@freebsd> <20150323005852.GB6798@dft-labs.eu> <20150323020314.GA30143@freebsd> <20150324123517.GA25678@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150324123517.GA25678@dft-labs.eu> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygAnL8uudhFVUxfRAw--.36362S2 X-Coremail-Antispam: 1UD129KBjvJXoWxuw4DKF1DWryxXw18XrW3Awb_yoWxuryUpF W5CF9YyFs5Cr43CFnav3y8Za4Fy3s5Jr45W343Xr1Ykr1F9ry8XF10gw1F9F1kWrWvqF1q va15WFy7KrWUZ3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUkYb7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwV C2z280aVCY1x0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC 0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Cr0_Gr 1UMcvjeVCFs4IE7xkEbVWUJVW8JwACjcxG0xvEwIxGrwCY02Avz4vE14v_Xryl42xK82IY c2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s 026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1Y6r17MIIYrxkI7VAKI48JMIIF 0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r1j6r4UMIIF0x vE42xK8VAvwI8IcIk0rVWrZr1j6s0DMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2 jsIEc7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU5qdyUUUUUU== X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUSAVQhl-solQACsx Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2015 14:37:49 -0000 On Tue, Mar 24, 2015 at 01:35:17PM +0100, Mateusz Guzik wrote: > On Mon, Mar 23, 2015 at 10:03:14AM +0800, Tiwei Bie wrote: > > On Mon, Mar 23, 2015 at 01:58:52AM +0100, Mateusz Guzik wrote: > > > On Sun, Mar 22, 2015 at 09:15:24PM +0800, Tiwei Bie wrote: > > > > On Sun, Mar 22, 2015 at 08:06:55PM +0800, Tiwei Bie wrote: > > > > > On Sun, Mar 22, 2015 at 01:38:22PM +0200, Konstantin Belousov wrote: > > > > > > On Sun, Mar 22, 2015 at 07:25:55PM +0800, Tiwei Bie wrote: > > > > > > > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > > > > > > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > > [..] > > I don't know how to do this. We need to update the man-pages? > > > > And sorry again, maybe I was too tired yesterday. I just find a new bug in > > the codes above when reading your comments. Though it won't affect anything > > cause dynamic_kenv hasn't been enabled at this time, it is really a bug. > > > > `format' needs to be freed, sorry... What a terrible day. ;-( > > > > core(5). yeah, would be nice to mention all this stuff. Yeah, I will update the man page! > > I wrote these codes after consulting the codes in kern/kern_clocksource.c > > > [..] > > It also needs to validate the user's inputs. And it uses a strcasecmp() > > to check whether a different timer has been specified by the user. > > > > If the user is going to replace the pattern with identical one, we won't > > need to do any update, we just `goto out'. > > > > Well, I presume original author did not want to play with timers when > changing to the same timer. > > Corefile is safe to change, so I suggest just dropping that strcmp. > > Also see below. > Yeah, I will remove the strcmp! > > My new patch: > > > > --- > > sys/kern/kern_sig.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++----- > > 1 file changed, 70 insertions(+), 7 deletions(-) > > > > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > > index 8410d9d..0254517 100644 > > --- a/sys/kern/kern_sig.c > > +++ b/sys/kern/kern_sig.c > > @@ -3094,21 +3094,85 @@ static int compress_user_cores = 0; > > */ > > #define corefilename_lock allproc_lock > > > > -static char corefilename[MAXPATHLEN] = {"%N.core"}; > > +static char corefilename[MAXPATHLEN] = "%N.core"; > > + > > +static bool > > +corefilename_check(const char *format) > > +{ > > + int i, counti; > > + > > + counti = 0; > > + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { > > + if (format[i] == '%') { > > + i++; > > + switch (format[i]) { > > + case 'I': > > + counti++; > > + if (counti > 1) > > + return (false); > > countI? > I thought `countI' seems to be CamelCase, so I named it as counti. I will update it. > > + case '%': > > +} [..] > > +static void > > +corefilename_init(void *arg) > > +{ > > + char *format; > > + > > + format = kern_getenv("kern.corefile"); > > + > > + if (format != NULL) { > > + if (corefilename_check(format)) > > + strcpy(corefilename, format); > > + freeenv(format); > > + } > > +} > > printf something. "invalid format specified" or the like. > > Also the code may be nicer if you > if (format == NULL) > return; > > instead > > > +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); > > > > static int > > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > > { > > + char *format; > > int error; > > > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > > + > > + sx_slock(&corefilename_lock); > > + strcpy(format, corefilename); > > + sx_sunlock(&corefilename_lock); > > + > > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > > + if (error || !req->newptr || strcmp(format, corefilename) == 0) > > + goto out; > > + > > this should be if (error != 0 || req->newptr == NULL). > > strmcp was covered earlier in the mail > Thank you so much for your great patience! ^_^ This is my new patch: --- share/man/man5/core.5 | 4 +++ sys/kern/kern_sig.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 78 insertions(+), 7 deletions(-) diff --git a/share/man/man5/core.5 b/share/man/man5/core.5 index 3f54f89..17b1ef7 100644 --- a/share/man/man5/core.5 +++ b/share/man/man5/core.5 @@ -91,6 +91,10 @@ The name defaults to yielding the traditional .Fx behaviour. +When changing the name via the +.Va kern.corefile +sysctl, it will fail if the new name contains +unknown format specifiers, or its length is too long. .Pp By default, a process that changes user or group credentials whether real or effective will not create a corefile. diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c index 154c250..8cd3638 100644 --- a/sys/kern/kern_sig.c +++ b/sys/kern/kern_sig.c @@ -3094,21 +3094,89 @@ static int compress_user_cores = 0; */ #define corefilename_lock allproc_lock -static char corefilename[MAXPATHLEN] = {"%N.core"}; +static char corefilename[MAXPATHLEN] = "%N.core"; + +static bool +corefilename_check(const char *format) +{ + int i, countI; + + countI = 0; + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { + if (format[i] == '%') { + i++; + switch (format[i]) { + case 'I': + countI++; + if (countI > 1) + return (false); + case '%': + case 'H': + case 'N': + case 'P': + case 'U': + break; + default: + return (false); + } + } + } + + if (i == MAXPATHLEN) + return (false); + + return (true); +} + +static void +corefilename_init(void *arg) +{ + char *format; + + format = kern_getenv("kern.corefile"); + if (format == NULL) + return; + + if (corefilename_check(format)) + strcpy(corefilename, format); + else + printf("Invalid format character specified in " + "corename `%s'\n", format); + + freeenv(format); +} +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); static int sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) { + char *format; int error; + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); + + sx_slock(&corefilename_lock); + strcpy(format, corefilename); + sx_sunlock(&corefilename_lock); + + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); + if (error != 0 || req->newptr == NULL) + goto out; + + if (!corefilename_check(format)) { + error = EINVAL; + goto out; + } + sx_xlock(&corefilename_lock); - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), - req); + strcpy(corefilename, format); sx_xunlock(&corefilename_lock); +out: + free(format, M_TEMP); return (error); } -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", "Process corefile name format string"); @@ -3171,9 +3239,8 @@ corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, sbuf_printf(&sb, "%u", uid); break; default: - log(LOG_ERR, - "Unknown format character %c in " - "corename `%s'\n", format[i], format); + KASSERT(0, ("Unknown format character %c in " + "corename `%s'\n", format[i], format)); break; } break; -- 2.1.2 Best regards, Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 24 15:00:55 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B9F00826; Tue, 24 Mar 2015 15:00:55 +0000 (UTC) Received: from mail-we0-x236.google.com (mail-we0-x236.google.com [IPv6:2a00:1450:400c:c03::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 455DD347; Tue, 24 Mar 2015 15:00:55 +0000 (UTC) Received: by wetk59 with SMTP id k59so165702512wet.3; Tue, 24 Mar 2015 08:00:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=DJSuRHo8b65+RoL9S6C7K1klzfVrr0WO+Aprx95pDbw=; b=zNwKn5ncgUnQIIkacLAf9KVUv3hstSwhE4WVlQLR/1iEKFMnVFmkWSC9ZUJWoRgo5Z JBpwI2QaaS9YxDVvGA+bJ3Ao4wlh4poZIG5kl6E9qP/se3DtpBWdSN0/dsGu+wWEy1Vd gEJF0RLG98QqKd+3nd3KihtLdCXC9xukbR/NdWReSJEC2eiFsfV7UIeaZzJof7vNGShX TugKhj3m1k+tOfAMGp9Du4nO77f4MhrrPzhgMoo802nvC0lZIoGsTl4yxt98Vk2N0tGI 7uXNr5TPYguPjXqxEe4UShffhx5pYE6fanE3yED6r1iiEJo13E9n5VvEPzXRu1p2yxWh Z9Xw== MIME-Version: 1.0 X-Received: by 10.180.74.230 with SMTP id x6mr29362021wiv.58.1427209253661; Tue, 24 Mar 2015 08:00:53 -0700 (PDT) Received: by 10.28.98.70 with HTTP; Tue, 24 Mar 2015 08:00:53 -0700 (PDT) In-Reply-To: References: Date: Tue, 24 Mar 2015 23:00:53 +0800 Message-ID: Subject: Re: Self introduciton of Hao Sun and thoughts on LibNetstat From: Hao Sun To: Gabor Pali Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: George Neville-Neil , Robert Watson , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2015 15:00:55 -0000 Hi G=C3=A1bor, Thanks a lot for the advice and suggestions. Based on our discusstion and initial investigation, I roughly draw a route map for the LibNetstat of GSoC. I'd like to follow the following process to get down to more details from requirement to implementation. 0. Warm up. I'll build the FreeBSD, just as your suggestion, the "world" version, and then install the system together with setting up the dev environment. Hopefully after this step I'll get familar with the FreeBSD dev process and be well prepared for the next steps. 1. Requirement gathering. The main task of our project is to create wrapper libraries to support monitoring and management applications to avoid direct use of kvm(3) and sysctl(3) interfaces. In this step, I plan to read the kvm(3) and sysctl(3) implementations and filter out the interfaces and functions which are related to networks. Then I'll classify all the requirements in order to get different modules of the LibNetstat. After this step, I should have a vague idea about creating the abstactions for the modules. 2. Case study. In this step, I want to learn about the Libmemstat(3) and existing LibNetstat implementations in order to get inspirations and reuseable code snippets. After the case study, hopefully I'll be cleared how to make the abstraction. 3. Create abstraction. After Step #2 and #3, I'll start to write code from sketch and the interfaces should be well defined after this step. 4. Core implementation. Based on the kvm(3) and sysctl(3), I'll create the detailed implementation for the library. 5. Applications update preparation. Now our library is ready! In this step I'd like to read the implemation of netstat(1) and bsnmpd(1) and point out which part could be replaced with our new library. 6. Applications update. Update the netstat(1) and bsnmpd(1), also test the new library. 7. Documentation. Maybe we need to create a manual page for the library. The above are my initial ideas. I'd like to dive into each step during the next few days and add more details in my final proposal. Also I plan add some buffer time in case there is a unexpected delay and some "stretch goals" in case the abrove goals are done quickly :). I also feel excited to corporate and share my ideas with you, Robert and George. Any suggestions or supplements are really welcomed :) Thanks, Hao 2015-03-23 14:50 GMT+08:00 Gabor Pali : > Hi Hao, > > 2015-03-22 13:09 GMT+01:00 Hao Sun : > > Following your guidance, I've cloned the FreeBSD mirror from GitHub and > will get down to > > have an initial scratch with the latest version. > > That is great. I forgot to mention, but maybe it is also said on the > introductory wiki pages you have also cited originally, that it is > probably the best if you try to build the userland ("world") and the > kernel on your VM and install it first. This should give you the > basic feel for the development cycle and help to spot problems with > the clean system itself before you start hacking on it. The > development branch of FreeBSD (that is called "current") shall build > and install just fine for most of the time, but do not be discouraged > if not, ask for help. > > > On the project's wiki page [1], I think the target for GSoC > > 2015 is to finish the tasks haven't been done in the following table. > But according to your > > comments in the emails, it seems like I need to start the job from > scratch. Thus the question > > is should I keep the existing code and add new features to the previous > version or just start > > the project from the very beginning? > > I might have sounded a bit pessimistic, I do not necessarily insist on > rewriting the entire library :-) I think it is just common sense: > study the current implementation, take a look at the FreeBSD ecosystem > and kernel, discuss the topic with the interested hackers, and work > out your proposal based on your findings. You may find some of the > old code base and concepts reusable, which is excellent, and you may > decide to take another approach for the rest. It might be worthwhile > to accommodate some "stretch goals" in your proposal if you > accidentally completed your summer task too quickly :-) > > For making things a bit easier (hopefully) for you, I may also include > George Neville-Neil in the conversation (see him CC'ed) who has shown > some interest in driving this library into the base system in the past > if I recall correctly. Along with Robert, he is also a high-profile > src committer, with experience in networking and related areas. (And > also a potential mentor for this project as well?) > > Cheers, > G=C3=A1bor > > [1] https://wiki.freebsd.org/LibNetstat > From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 24 18:46:51 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DDFD5A88 for ; Tue, 24 Mar 2015 18:46:50 +0000 (UTC) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7F17D5FC for ; Tue, 24 Mar 2015 18:46:50 +0000 (UTC) Received: by wibdy8 with SMTP id dy8so82825178wib.0 for ; Tue, 24 Mar 2015 11:46:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=date:from:to:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=7eURfSdalSusRZRupuPyFWFXwomqC+l4Q71cJ/J4lVY=; b=Z1br/6NRg3M6LEHkkoct4c/vQBPjmGCGFI4p7da4yJma8E0jugesQz7qXSEDuP2KqH Anqb/XnCqn+TxdmAHA6TN4znIQivWaY3Cch5w3Am8vq5r0N76IXhvU/VtKz5hP9HZ/x/ 3ofulOVdX91c1Me4E6THRdnQI3i2dGCbh98YhTGDD+LVnQyhmIQLNZIFY9U9AlaaJQz6 zCGYXkEFrhuWYygENGwOt+3o+Ap97RVSJLdmhL6h1QE3XEQAXHd94KirPFNOhIJ6d9Nj aLmbonDIQqE0RGeM3d53xbW2IaJ52AulLOwuz/mgV6ysYfUxMg6vKTm5XcIweHNBTv2o ab/A== X-Received: by 10.180.171.35 with SMTP id ar3mr31685673wic.24.1427222808974; Tue, 24 Mar 2015 11:46:48 -0700 (PDT) Received: from bath (fnord.cryptophone.de. [62.220.7.20]) by mx.google.com with ESMTPSA id ax10sm137802wjc.26.2015.03.24.11.46.47 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Mar 2015 11:46:48 -0700 (PDT) Date: Tue, 24 Mar 2015 18:45:59 +0000 From: Stefan Grundmann To: freebsd-hackers@freebsd.org Subject: Re: GELI support on /boot folder Message-ID: <20150324184559.GA9056@bath> References: <20150319013231.GR51048@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2015 18:46:51 -0000 On Mon, Mar 23, 2015 at 05:46:20PM -0300, Pedro Arthur wrote: > Based on your idea the project could be merge the GPT boot stages into > a single program with support for GELI and instead of having a binary for > zfs and other for ufs we could have the boot program with support for both > file systems so that we can choose to boot from any partition in the GPT > being zfs or ufs. > > What I want to know is if it is a good idea or merging these boot stages has > any drawback or infringes any design choices? one technical problem when merging gptload and/or zfsload code into gptboot is, that gptldr.S has to be extended to be able to load more boot2 code than the currently supported 64k. about a year ago i wrote a TPM 1.2 static root of trust boot loader (chain): 1. sys/boot/i386/pmbr was extended to support TCG_CompactHashLogExtendEvent so that it will measure the freebsd-boot partion 2. sys/boot/i386/gptboot/gptldr.S was modified to support up to 256k boot2 code 3. functionality was added to sys/boot/i386/gptboot/* - get (PCR sealed) Key and config_data path from TPM - read and decrypt the config_data from UFS - use config data to set kernel environment, load disk keys, verify and load kernel + modules (from UFS) - boot the kernel The code was tested on ThinkPad (X60,X61,T410s,X230) notebooks and HP ProLiant DL360p servers. I expect it to work on any amd64 or x86 system that has working TPM 1.2 and is able to boot gpt schema disks from (legacy) BIOS. The code is in daylie use on my Thinkpads. However: man page content and high level documentation (blog post, article, ...) has not been written and a code review is also needed. All this was planed but was deprioritized. Today i got the o.k. from $work to release (BSD 2 clause) a preview which will consist of a patchset against FreeBSD 10.1-p8 (it will apply and work on any 10* and 11) and a minimal howto. I'm in the process of writing the minimal howto and will send the preview to this list today or tomorrow. best regards, Stefan Grundmann From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 24 21:58:46 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1A3B2B16; Tue, 24 Mar 2015 21:58:46 +0000 (UTC) Received: from dmz-mailsec-scanner-2.mit.edu (dmz-mailsec-scanner-2.mit.edu [18.9.25.13]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 82764E30; Tue, 24 Mar 2015 21:58:45 +0000 (UTC) X-AuditID: 1209190d-f79676d000000da0-20-5511dce0681e Received: from mailhub-auth-3.mit.edu ( [18.9.21.43]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-2.mit.edu (Symantec Messaging Gateway) with SMTP id E4.48.03488.0ECD1155; Tue, 24 Mar 2015 17:53:36 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id t2OLrZPi014389; Tue, 24 Mar 2015 17:53:35 -0400 Received: from multics.mit.edu (system-low-sipb.mit.edu [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id t2OLrXaL011893 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 24 Mar 2015 17:53:34 -0400 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id t2OLrX7P022806; Tue, 24 Mar 2015 17:53:33 -0400 (EDT) Date: Tue, 24 Mar 2015 17:53:32 -0400 (EDT) From: Benjamin Kaduk X-X-Sender: kaduk@multics.mit.edu To: freebsd-hackers@freebsd.org Subject: Re: Call for FreeBSD 2015Q1 (January-March) Status Reports In-Reply-To: Message-ID: References: User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrJIsWRmVeSWpSXmKPExsUixCmqrfvgjmCoQd82IYtd106zW8x584HJ Yvvmf4wOzB4zPs1nCWCM4rJJSc3JLEst0rdL4Mo4/eEmS8Ev7ooH65tYGhjvcXYxcnJICJhI fP/ykgXCFpO4cG89WxcjF4eQwGImida+dewQzkZGiQO/nzBCOIeYJD5vuwrlNDBKzGvsYgfp ZxHQljg5aw8biM0moCbxeG8zK8RcRYnNpyYxdzFycIgIyEssOG8PEmYGMv9fucwEYgsLOEss OPkFzOYUcJR42DgdbCQvkD1x+VYwW0jAQeJjxxmwGlEBHYnV+6ewQNQISpyc+YQFYqaWxPLp 21gmMArNQpKahSS1gJFpFaNsSm6Vbm5iZk5xarJucXJiXl5qka6RXm5miV5qSukmRnAIS/Lu YHx3UOkQowAHoxIPb8ASgVAh1sSy4srcQ4ySHExKorybtgmGCvEl5adUZiQWZ8QXleakFh9i lOBgVhLhfd4OlONNSaysSi3Kh0lJc7AoifNu+sEXIiSQnliSmp2aWpBaBJOV4eBQkuCtuw3U KFiUmp5akZaZU4KQZuLgBBnOAzR8EkgNb3FBYm5xZjpE/hSjopQ4bylIQgAkkVGaB9cLSzGv GMWBXhHmzQap4gGmJ7juV0CDmYAGn8vnAxlckoiQkmpgDFx2vHK/PMvmdn79vOTyrIKZUj+e m5b6NoRe7y6rXZah5KLe2Nl2yNhdqahxgv6JAw3hfLtteU19Qm8mdyt/9dm9ZO+b+Q1vPcpj P09yWfp/WR/DAll14Z6y/f2xW1xmMJdanHywRl5lk9s060maK68fe+ubxy52aAp/xIoU/R9F uZeXvL2pxFKckWioxVxUnAgAM1IVKgwDAAA= Cc: freebsd-current@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2015 21:58:46 -0000 Reminder: only two weeks remain in the submission period. -Ben On Sat, 7 Mar 2015, Benjamin Kaduk wrote: > Dear FreeBSD Community, > > The deadline for the next FreeBSD Quarterly Status update is April 7, > 2015, for work done in January through March. > > Status report submissions do not have to be very long. They may be > about anything happening in the FreeBSD project and community, and > provide a great way to inform FreeBSD users and developers about what > you're working on. Submission of reports is not restricted to > committers. Anyone doing anything interesting and FreeBSD-related > can -- and should -- write one! > > The preferred and easiest submission method is to use the XML > generator [1] with the results emailed to the status report team at > monthly at freebsd.org . There is also an XML template [2] which can be > filled out manually and attached if preferred. For the expected > content and style, please study our guidelines on how to write a good > status report [3]. You can also review previous issues [4][5] for > ideas on the style and format. > > We are looking forward to all of your 2015Q1 reports! > > Thanks, > Ben (on behalf of monthly@) > > > [1] http://www.freebsd.org/cgi/monthly.cgi > [2] http://www.freebsd.org/news/status/report-sample.xml > [3] http://www.freebsd.org/news/status/howto.html > [4] http://www.freebsd.org/news/status/report-2014-07-2014-09.html > [5] http://www.freebsd.org/news/status/report-2014-10-2014-12.html > > From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 24 22:15:44 2015 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E71A15D7 for ; Tue, 24 Mar 2015 22:15:44 +0000 (UTC) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id ACA46A0 for ; Tue, 24 Mar 2015 22:15:44 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 6DEEA3592F1; Tue, 24 Mar 2015 23:15:41 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 5C8BA28494; Tue, 24 Mar 2015 23:15:41 +0100 (CET) Date: Tue, 24 Mar 2015 23:15:41 +0100 From: Jilles Tjoelker To: Ivan Radovanovic Subject: Re: kevent behavior Message-ID: <20150324221541.GA67584@stack.nl> References: <550A6DA2.1070004@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <550A6DA2.1070004@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2015 22:15:45 -0000 On Thu, Mar 19, 2015 at 07:33:06AM +0100, Ivan Radovanovic wrote: > Is there defined (and guaranteed) behavior of kevent if kqueue FD is > being closed while blocking kevent call is in progress? > Possible scenario would be like this: > Thread 1: > ... > kfd = kqueue(); > ... > // create second thread afterwords > // and do blocking wait for events > result = kevent(kfd, changelist, nchanges, eventlist, nevents, NULL); > if (result == -1) > // check if there was request to stop listening for events > Thread 2: > // do something > // then close kqueue's fd > close(kfd); > I am asking this because file watcher implementation for mono is > implemented that way (which I find nicer than using timeout), but this > is apparently based on expected kevent behavior under Darwin, and I > can't find any mention that in FreeBSD kevent is going to behave the > same way (at least not on kqueue(2) manual page) This method is inherently unsafe, since you cannot be sure thread 1 has started blocking in kevent() when you close() in thread 2. If not, there might be a thread 3 creating a kqueue between thread 2's close and thread 1's kevent, and thread 1 will manipulate the new kqueue. Fortunately, EVFILT_USER provides an easy way to wake up a thread blocked in kevent(). If kevent() was implemented as a cancellation point, pthread_cancel() would be another option, but it is not. -- Jilles Tjoelker From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 24 22:28:52 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2E396A2C; Tue, 24 Mar 2015 22:28:52 +0000 (UTC) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E3A6F1DA; Tue, 24 Mar 2015 22:28:51 +0000 (UTC) Received: by igcau2 with SMTP id au2so85615966igc.0; Tue, 24 Mar 2015 15:28:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=yiGl/IGm8ZncHMVsMxqf3hSbnAix4743VR9+2t6ujeA=; b=xsej2G9v7yheEXDg/IT22/ZtuAPWo8L9EoFtadd7L85BT2Or+X/Xw502ad9ySNAEzB MEep7CAx7suu9R5FJ3BD6ucpLbtNp/OpocuktylpR+7qPps6C5d3TUEAC9ZN9EeItC4x jReKNZ/0rQ6T2KZd4Ie8rzMq86gh3ZJ3S9IpatnVfY9h3HvsrR7Earys2+ZdHiBIuRe6 yYAPHicTLQ2noDJtwwIMqE6SZQB+hUbEiIwXYThDylL4V+9RghUYxSMmfbTceWHlstoE wG1MJec550sroxvkhO3k141uZ3+DsunokVUId30wkH8Eb68OTfv7LetmFE5im2TO5uxd shTg== MIME-Version: 1.0 X-Received: by 10.42.166.1 with SMTP id m1mr29743004icy.10.1427236131117; Tue, 24 Mar 2015 15:28:51 -0700 (PDT) Received: by 10.107.143.149 with HTTP; Tue, 24 Mar 2015 15:28:51 -0700 (PDT) In-Reply-To: References: Date: Tue, 24 Mar 2015 17:28:51 -0500 Message-ID: Subject: Re: [gsoc15][idea-proposal] Port FreeBSD to a smartphone [needs-feedback] From: Giovanny Andres Gongora Granada To: freebsd-mobile@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2015 22:28:52 -0000 Hi, Someone had time to check my proposal? I will really appreciate some feedback to my finish project proposal in gsoc page. I had published a more detailed draft of my proposal here: www.google-melange.com/gsoc/proposal/public/google/gsoc2015/gioyik/5629499534213120 Thanks a lot, Gio 2015-03-22 21:48 GMT-05:00 Giovanny Andres Gongora Granada : > Hello, > > My name is Giovanny, I am an student from Universidad Cooperativa de > Colombia planning to participate this year in the gsoc with FreeBSD > organization. I reviewed the project list [1] for this year and I am > interested in propose a project for Port FreeBSD to a smartphone (any maker > and model) [2]. I will share with you my idea, I really appreciate have > some feedback for my final proposal in the gsoc page. > > FreeBSD has a good compiling process for ARM [3] architecture and > hardware. Taking advantage of this FreeBSD could run on a smartphone by > three ways: > > 1. Using Android (based device and build process) we can compile FreeBSD > kernel with support to the hardware of the device, then use a SD as system > partition and configure it as a normal FreeBSD system. Flash an image > generated for boot, equal to zImage in the block partition of the device > using fastboot, so at reboot it will be running the FreeBSD system. After > have the device running is more easy work on a display lib for the device. > At the end if it can boot a device using this process and emulator could do > this (changing the boot image for the one with FreeBSD) too, I use AOSP to > define an environment and a set of tools to compile a light emulator with > just the components necessaries to run the FreeBSD system. Mozilla does > this to build Firefox OS with AOSP (devices and emulators) [4]. [This is > not a image to chroot-jail] > > 2. Porting the idea of Gentoo for Android [5] to FreeBSD, this could be > more difficult than expected due to port the concept of Gentoo RAP to > FreeBSD could take a long time of research. Added to it, Gentoo RAP is made > to work over Linux, so will take more time adapt the concept to FreeBSD and > get something working with the low level layer of Android in the device. If > by X reason the concept of Gentoo for Android it's done for FreeBSD could > be possible think on port and use Webtop [6] as output for X servers how > was done in Ubuntu for Android. > > 3. Running FreeBSD directly on an Android device using an application that > emulates-wraps a terminal to boot a pre-compiled obb (just a tar package > with a different name) package of a FreeBSD system. It will provide a shell > to access all the programs in FreeBSD system but only text-mode will be > supported. To interact from the smartphone with a graphical environment in > FreeBSD there are to ways, first use other apps like a vnc client or embed > a custom vnc/display client with own libs to display the enviroment. This > method doesn't require root on the phone to run FreeBSD and the final apk > of the application to run FreeBSD could be published in the Play store (if > the idea is to publish it), so everyone could install it and run FreeBSD > instantly after apk installation from Play Store. > > Three possible ways but personally I see 1 and 3 are the ones with best > results. Number 1 is the way we more friendly for daily testing and with > more percentage for future development. It will keep compatibility with > latest changes in arm build process and after new CPU and boards support is > added, will be easy build and test on more devices with this method. Number > 3 it's more a user side and probably not offers a complete FreeBSD system > experience as expected. I see that part of the gsoc is R&D, so I will be > happy to research and develop until the end with all the ideas to get the > best experience of FreeBSD on a phone. > > I know all this needs to be more detailed to get and idea if I am wrong or > not, so feel free to quote where you think needs clarification in the > process. Any feedback or opinions about this is really appreciated to my > final proposal in the gsoc page. > > Thanks, > Gio > > ---------------------------------------------------------------------- > > * About me: > I am studying at University Cooperativa de Colombia, my future degree will > be Systems Engineer. I had been contributing to open source project since 4 > years ago, today I am an active Mozilla contributor [7], I work sending > patches to the Firefox and Firefox OS code base [8]. Also I am part of the > reviewer team at Firefox Marketplace (I review the apps that are sent to > the Marketplace to be published) and I am a Mozilla Reps [9] too. > > I contribute to other open source projects too (Tor Project time ago as ES > translator) different to Mozilla, you can see some of my contributions to > other projects in my Github profile [10]. Most of my projects are open > sourced in Github [11] and those projects are coded using different > languages, some of them are C, Go, Python, Javascript, Ruby, Java and some > shell scripts. > > * Previous experience with FreeBSD: > I maintain some FreeBSD systems running in my University and a year ago I > helped to run some FreeBSD servers with an IT team for a hosting provider > company. > > ---------------------------------------------------------------------- > > [1]: https://wiki.freebsd.org/SummerOfCodeIdeas > [2]: > https://wiki.freebsd.org/SummerOfCodeIdeas#Port_FreeBSD_to_a_smartphone_.28any_maker_and_model.29 > [3]: http://www.freebsd.org/platforms/arm.html > [4]: https://github.com/mozilla-b2g/B2G > [5]: https://wiki.gentoo.org/wiki/Project:Android > [6]: http://sourceforge.net/motorola/motorola-webtop/home/Home/ > [7]: https://mozillians.org/en-US/u/gioyik/ > [8]: http://hg.mozilla.org/mozilla-central/log?rev=gioyik%40gmail.com > [9]: https://reps.mozilla.org/u/gioyik/ > [10]: https://github.com/Gioyik/ > [11]: https://github.com/Gioyik?tab=repositories > -- Mozilla Hispano: http://mozilla-hispano.org Mentor en mozilla-hispano.org: Gioyik Mozilla Colombia: Miembro en MozillaColombia twitter: Gioyik Blog: gioyik.wordpress.com irc: gioyik From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 00:24:12 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3852B85A for ; Wed, 25 Mar 2015 00:24:12 +0000 (UTC) Received: from mail-ig0-x235.google.com (mail-ig0-x235.google.com [IPv6:2607:f8b0:4001:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F1FD4E8 for ; Wed, 25 Mar 2015 00:24:11 +0000 (UTC) Received: by igbqf9 with SMTP id qf9so12423768igb.1 for ; Tue, 24 Mar 2015 17:24:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=nbHc5osBlatsecjTEpzwCpvfFrQ9Ni5ecqsjWyKdc9g=; b=yB6aqN08o/8AizCIkx9+YUYJkCIFvBp8hFy+q0loTmlrOW1r6+Doh3Hta61JP9mK4X Ja2ePAWoy5Skjegj05XShg58glZw5/0p/3+8BVc3yfvPb002wsTCqJ6mFYudrraLwXPh krMZ9k6wzVL9iqsqL4AVmiK8kIXpoGemUkGg1sTjCCOe+dvCr/qHe5qpw/4FOKKEH1k7 rf9Sko082QcPC5LVEn98VWMy5MwlbNRUnUxmcJLQHhEXm5Qh1cA7v6LLQaA1hu01+96B zzjIlBWugdqgHTKxyt6OKAQ3RSjEH+0z/VmQKw1jB92wjo64k8X0wWEBDBnQYO8phCm4 1daw== MIME-Version: 1.0 X-Received: by 10.50.114.4 with SMTP id jc4mr26015063igb.14.1427243051347; Tue, 24 Mar 2015 17:24:11 -0700 (PDT) Received: by 10.64.73.7 with HTTP; Tue, 24 Mar 2015 17:24:11 -0700 (PDT) In-Reply-To: References: Date: Wed, 25 Mar 2015 05:54:11 +0530 Message-ID: Subject: Fwd: Port FreeBSD to a smartphone (any make and model) for GSoC15 From: Rakesh Sharma To: freebsd-hackers@freebsd.org X-Mailman-Approved-At: Wed, 25 Mar 2015 01:25:00 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 00:24:12 -0000 I am Rakesh Kumar Sharma, an engineering Student of NIT Karnataka. I Am Interested in working on porting Free BSD to a smartphone (any make or model).sorry for late writing, i wasnt aware of GSoC, my friend recently suggested me about it. I am not much into the field, but I believe i can pull it off as I am hardworking and have good technical skills. one idea is to dual boot the free bsd like ubuntu touch, we need to install the BSD to the memory of the device, and than loading the kernel from inside the android kernel. this process should kill all the android processes except the ones really needed. the BSD kernel will load the required files and overtake the android kernel. I Checked BSD have its ARM version support, so i think instruction porting wont be needed. The second idea is to entirely replace the original kernel and operating system, in this case i need to do some research, but i believe i should be able to modify the boot loader to boot the BSD RISC Kernel stored in my memory, with problems of course, then i can modify the kernel to fix the bugs. I idea is not to replace the entire operating system, as if something goes wrong i can return to my old os, This can be changed later. Please provide me suggestions, and comments on my idea. i am really looking forward to working on it, I have an old device which i can work on, and a new intel x86 phone, which stand more chance of success. Thank you From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 02:01:16 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F0CC8C35 for ; Wed, 25 Mar 2015 02:01:16 +0000 (UTC) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B5EDCD18 for ; Wed, 25 Mar 2015 02:01:16 +0000 (UTC) Received: from jre-mbp.elischer.org (ppp121-45-255-201.lns20.per4.internode.on.net [121.45.255.201]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t2P21BtE078678 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Tue, 24 Mar 2015 19:01:14 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <551216E2.9040604@freebsd.org> Date: Wed, 25 Mar 2015 10:01:06 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: GELI support on /boot folder References: <20150319013231.GR51048@funkthat.com> <20150324184559.GA9056@bath> In-Reply-To: <20150324184559.GA9056@bath> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 02:01:17 -0000 On 3/25/15 2:45 AM, Stefan Grundmann wrote: > Today i got the o.k. from $work to release (BSD 2 clause) a preview > which will consist of a patchset against FreeBSD 10.1-p8 (it will > apply and work on any 10* and 11) and a minimal howto. I'm in the > process of writing the minimal howto and will send the preview to > this list today or tomorrow. best regards, Stefan Grundmann > _______________________________________________ excellent From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 06:00:47 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BDCB4B68 for ; Wed, 25 Mar 2015 06:00:47 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id 15D3D876 for ; Wed, 25 Mar 2015 06:00:46 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygCnVzwHTxJVSnfVAw--.50550S2; Wed, 25 Mar 2015 14:00:43 +0800 (CST) Date: Wed, 25 Mar 2015 14:00:12 +0800 From: Tiwei Bie To: Mateusz Guzik Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150325060012.GA75674@freebsd> References: <20150322091853.GA89976@freebsd> <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> <20150322113822.GB2379@kib.kiev.ua> <20150322120655.GA59757@freebsd> <20150322131524.GA95795@freebsd> <20150323005852.GB6798@dft-labs.eu> <20150323020314.GA30143@freebsd> <20150324123517.GA25678@dft-labs.eu> <20150324143709.GA54065@freebsd> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150324143709.GA54065@freebsd> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygCnVzwHTxJVSnfVAw--.50550S2 X-Coremail-Antispam: 1UD129KBjvJXoWxKw18WF1Utry7ur1kKr43GFg_yoW7Ww18pF Wakr95AFs0kr43Cr9ay3yrZ34Yywn5Jay5X347Zw4akrWFgryDXr18Kw1YvFykGrWvq3s8 Ja15Wry2gryUZ3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUkYb7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Ar0_tr1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwV C2z280aVCY1x0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC 0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Cr0_Gr 1UMcvjeVCFs4IE7xkEbVWUJVW8JwACjcxG0xvEwIxGrwCY02Avz4vE14v_GF1l42xK82IY c2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s 026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1Y6r17MIIYrxkI7VAKI48JMIIF 0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r1j6r4UMIIF0x vE42xK8VAvwI8IcIk0rVWrZr1j6s0DMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2 jsIEc7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU59SdJUUUUU== X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUTAVQhl-tn9QAAsd Cc: Konstantin Belousov , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 06:00:47 -0000 On Tue, Mar 24, 2015 at 10:37:09PM +0800, Tiwei Bie wrote: > On Tue, Mar 24, 2015 at 01:35:17PM +0100, Mateusz Guzik wrote: > > On Mon, Mar 23, 2015 at 10:03:14AM +0800, Tiwei Bie wrote: > > > On Mon, Mar 23, 2015 at 01:58:52AM +0100, Mateusz Guzik wrote: > > > > On Sun, Mar 22, 2015 at 09:15:24PM +0800, Tiwei Bie wrote: > > > > > On Sun, Mar 22, 2015 at 08:06:55PM +0800, Tiwei Bie wrote: > > > > > > On Sun, Mar 22, 2015 at 01:38:22PM +0200, Konstantin Belousov wrote: > > > > > > > On Sun, Mar 22, 2015 at 07:25:55PM +0800, Tiwei Bie wrote: > > > > > > > > On Sun, Mar 22, 2015 at 11:14:01AM +0100, Mateusz Guzik wrote: > > > > > > > > > On Sun, Mar 22, 2015 at 05:19:40PM +0800, Tiwei Bie wrote: > > > [..] > +static void > +corefilename_init(void *arg) > +{ > + char *format; > + > + format = kern_getenv("kern.corefile"); > + if (format == NULL) > + return; > + > + if (corefilename_check(format)) > + strcpy(corefilename, format); > + else > + printf("Invalid format character specified in " > + "corename `%s'\n", format); > + The message printed in corefilename_init() is not proper, it doesn't cover the case that the length is too long, nor the case that %I is specified more than once. I extended the corefilename_check() to be able to return the reason (error code) if it fails. But I'm not sure whether it is worthwhile, cause it seems currently there is no way to specified a string longer than MAXPATHLEN via sysctl or kenv. I also removed the '\n' character from KASSERT() in corefile_open(), as what the other codes do. Here is my new patch: --- share/man/man5/core.5 | 7 ++++ sys/kern/kern_sig.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 93 insertions(+), 7 deletions(-) diff --git a/share/man/man5/core.5 b/share/man/man5/core.5 index 3f54f89..3047e0a 100644 --- a/share/man/man5/core.5 +++ b/share/man/man5/core.5 @@ -78,6 +78,7 @@ An index starting at zero until the sysctl is reached. This can be useful for limiting the number of corefiles generated by a particular process. +This specifier can only be specified at most once. .It Em \&%N process name. .It Em \&%P @@ -91,6 +92,12 @@ The name defaults to yielding the traditional .Fx behaviour. +When changing the name via the +.Va kern.corefile +sysctl, it will fail if the new name contains +unknown format specifiers, or +.Em \&%I +is specified more than once, or its length is too long. .Pp By default, a process that changes user or group credentials whether real or effective will not create a corefile. diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c index 154c250..7fdb079 100644 --- a/sys/kern/kern_sig.c +++ b/sys/kern/kern_sig.c @@ -3094,21 +3094,101 @@ static int compress_user_cores = 0; */ #define corefilename_lock allproc_lock -static char corefilename[MAXPATHLEN] = {"%N.core"}; +static char corefilename[MAXPATHLEN] = "%N.core"; + +static int +corefilename_check(const char *format) +{ + int i, countI; + + countI = 0; + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { + if (format[i] == '%') { + i++; + switch (format[i]) { + case 'I': + countI++; + if (countI > 1) + return (EINVAL); + case '%': + case 'H': + case 'N': + case 'P': + case 'U': + break; + default: + return (EINVAL); + } + } + } + + if (i == MAXPATHLEN) + return (ENAMETOOLONG); + + return (0); +} + +static void +corefilename_init(void *arg) +{ + char *format; + int error; + + format = kern_getenv("kern.corefile"); + if (format == NULL) + return; + + error = corefilename_check(format); + + switch (error) { + case 0: + strcpy(corefilename, format); + break; + case EINVAL: + printf("Invalid format specified for corename `%s'\n", format); + break; + case ENAMETOOLONG: + printf("The format specified for corename is too long\n"); + break; + default: + KASSERT(0, ("%s: unknown return value %d from " + "corefilename_check", __func__, error)); + break; + } + + freeenv(format); +} +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); static int sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) { + char *format; int error; + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); + + sx_slock(&corefilename_lock); + strcpy(format, corefilename); + sx_sunlock(&corefilename_lock); + + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); + if (error != 0 || req->newptr == NULL) + goto out; + + error = corefilename_check(format); + if (error != 0) + goto out; + sx_xlock(&corefilename_lock); - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), - req); + strcpy(corefilename, format); sx_xunlock(&corefilename_lock); +out: + free(format, M_TEMP); return (error); } -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", "Process corefile name format string"); @@ -3171,9 +3251,8 @@ corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, sbuf_printf(&sb, "%u", uid); break; default: - log(LOG_ERR, - "Unknown format character %c in " - "corename `%s'\n", format[i], format); + KASSERT(0, ("Unknown format character %c in " + "corename `%s'", format[i], format)); break; } break; -- 2.1.2 Best regards, Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 08:41:41 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3F33A729 for ; Wed, 25 Mar 2015 08:41:41 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D460DA13 for ; Wed, 25 Mar 2015 08:41:40 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t2P8fVkB013883 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 25 Mar 2015 10:41:31 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t2P8fVkB013883 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t2P8fUDq013882; Wed, 25 Mar 2015 10:41:30 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 25 Mar 2015 10:41:30 +0200 From: Konstantin Belousov To: Tiwei Bie Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150325084130.GX2379@kib.kiev.ua> References: <20150322101401.GH14650@dft-labs.eu> <20150322112555.GA44277@freebsd> <20150322113822.GB2379@kib.kiev.ua> <20150322120655.GA59757@freebsd> <20150322131524.GA95795@freebsd> <20150323005852.GB6798@dft-labs.eu> <20150323020314.GA30143@freebsd> <20150324123517.GA25678@dft-labs.eu> <20150324143709.GA54065@freebsd> <20150325060012.GA75674@freebsd> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150325060012.GA75674@freebsd> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-hackers@freebsd.org, Mateusz Guzik X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 08:41:41 -0000 On Wed, Mar 25, 2015 at 02:00:12PM +0800, Tiwei Bie wrote: > share/man/man5/core.5 | 7 ++++ > sys/kern/kern_sig.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++---- > 2 files changed, 93 insertions(+), 7 deletions(-) > > diff --git a/share/man/man5/core.5 b/share/man/man5/core.5 > index 3f54f89..3047e0a 100644 > --- a/share/man/man5/core.5 > +++ b/share/man/man5/core.5 Please bump date for the man page. > @@ -78,6 +78,7 @@ An index starting at zero until the sysctl > is reached. > This can be useful for limiting the number of corefiles > generated by a particular process. > +This specifier can only be specified at most once. > .It Em \&%N > process name. > .It Em \&%P > @@ -91,6 +92,12 @@ The name defaults to > yielding the traditional > .Fx > behaviour. > +When changing the name via the > +.Va kern.corefile > +sysctl, it will fail if the new name contains > +unknown format specifiers, or > +.Em \&%I > +is specified more than once, or its length is too long. > .Pp > By default, a process that changes user or group credentials whether > real or effective will not create a corefile. > diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c > index 154c250..7fdb079 100644 > --- a/sys/kern/kern_sig.c > +++ b/sys/kern/kern_sig.c > @@ -3094,21 +3094,101 @@ static int compress_user_cores = 0; > */ > #define corefilename_lock allproc_lock > > -static char corefilename[MAXPATHLEN] = {"%N.core"}; > +static char corefilename[MAXPATHLEN] = "%N.core"; > + > +static int > +corefilename_check(const char *format) > +{ > + int i, countI; > + > + countI = 0; > + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { Was it already suggested to use sizeof(corefilename) instead of MAXPATHLEN ? > + if (format[i] == '%') { > + i++; > + switch (format[i]) { > + case 'I': > + countI++; > + if (countI > 1) > + return (EINVAL); > + case '%': > + case 'H': > + case 'N': > + case 'P': > + case 'U': > + break; > + default: > + return (EINVAL); > + } > + } > + } > + > + if (i == MAXPATHLEN) > + return (ENAMETOOLONG); > + > + return (0); > +} > + > +static void > +corefilename_init(void *arg) > +{ > + char *format; > + int error; > + > + format = kern_getenv("kern.corefile"); > + if (format == NULL) > + return; > + > + error = corefilename_check(format); > + > + switch (error) { > + case 0: > + strcpy(corefilename, format); Use strlcpy(9). > + break; > + case EINVAL: > + printf("Invalid format specified for corename `%s'\n", format); > + break; > + case ENAMETOOLONG: > + printf("The format specified for corename is too long\n"); > + break; Sigh. Why kernel should decode the error codes ? Why should it print anything on the console, with non-zero chance of sysctl caller not seeing the text ? > + default: > + KASSERT(0, ("%s: unknown return value %d from " > + "corefilename_check", __func__, error)); > + break; > + } > + > + freeenv(format); > +} > +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); > > static int > sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) > { > + char *format; > int error; > > + format = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); > + > + sx_slock(&corefilename_lock); > + strcpy(format, corefilename); Again, use strlcpy. It works correctly, comparing with strncpy. > + sx_sunlock(&corefilename_lock); > + > + error = sysctl_handle_string(oidp, format, MAXPATHLEN, req); > + if (error != 0 || req->newptr == NULL) > + goto out; > + > + error = corefilename_check(format); > + if (error != 0) > + goto out; > + > sx_xlock(&corefilename_lock); > - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), > - req); > + strcpy(corefilename, format); And there. > sx_xunlock(&corefilename_lock); > > +out: > + free(format, M_TEMP); > return (error); > } > -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | > +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | > CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", > "Process corefile name format string"); > > @@ -3171,9 +3251,8 @@ corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, > sbuf_printf(&sb, "%u", uid); > break; > default: > - log(LOG_ERR, > - "Unknown format character %c in " > - "corename `%s'\n", format[i], format); > + KASSERT(0, ("Unknown format character %c in " > + "corename `%s'", format[i], format)); > break; > } > break; > -- > 2.1.2 > > Best regards, > Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 09:00:51 2015 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5A126C3D for ; Wed, 25 Mar 2015 09:00:51 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D8329BB1 for ; Wed, 25 Mar 2015 09:00:50 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t2P90fDU018059 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 25 Mar 2015 11:00:41 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t2P90fDU018059 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t2P90fQb018055; Wed, 25 Mar 2015 11:00:41 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 25 Mar 2015 11:00:41 +0200 From: Konstantin Belousov To: Jilles Tjoelker Subject: Re: kevent behavior Message-ID: <20150325090041.GY2379@kib.kiev.ua> References: <550A6DA2.1070004@gmail.com> <20150324221541.GA67584@stack.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150324221541.GA67584@stack.nl> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-hackers@FreeBSD.org, Ivan Radovanovic X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 09:00:51 -0000 On Tue, Mar 24, 2015 at 11:15:41PM +0100, Jilles Tjoelker wrote: > Fortunately, EVFILT_USER provides an easy way to wake up a thread > blocked in kevent(). > > If kevent() was implemented as a cancellation point, pthread_cancel() > would be another option, but it is not. Do you consider it is possible to make kqueue(2) cancellation point ? Susv4 is very explicit about functions it allows to handle cancellation. Might be, it is time for kqueue2(3), which would take flags. In addition to allowing cancellation, it could take close-on-exec flag. It is somewhat non-trivial to implement, at least I do not want to make kevent2(). From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 09:04:38 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DE573D6C for ; Wed, 25 Mar 2015 09:04:38 +0000 (UTC) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43B9CC6F for ; Wed, 25 Mar 2015 09:04:38 +0000 (UTC) Received: by wibg7 with SMTP id g7so101186738wib.1 for ; Wed, 25 Mar 2015 02:04:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=date:from:to:subject:message-id:mime-version:content-type :content-disposition:user-agent; bh=CtcxspuAyz1B+AQ13GmnazPLIY7bjH6g9vLNaMmfmTw=; b=tYvjTPoWEG3oLsWl+dtATpJeJS1qvpkaMg7bAfn+6FqMtlhh9yeH0JDEpN6BjWVf5o oMbAsv5Io8AuM+UfpVoVBgiEOWx+QRJMWDtWueHjO0HNvmEVM4XKwozVznyAMNrmNhlY VUmGaRdWPJbgUNaQQDINxWKxWT3+ej0oDXvpjyd/nwcWQOYLH6b1n60+XDFwhVis/Wvt BBYiVXwyBzsf9NetSdWnXLAk1ar/bOvZK67c4qmhLbm53/f8NUQcldDC9r3NQT23ZtIX wB0GZTGT7/aprBYBX8Id2p0CfPC/YSPw+hBHXjzFP9VtPsiyYBusIPTlQmll25axoQA9 ULyg== X-Received: by 10.180.82.9 with SMTP id e9mr35904692wiy.88.1427274276659; Wed, 25 Mar 2015 02:04:36 -0700 (PDT) Received: from bath (fnord.cryptophone.de. [62.220.7.20]) by mx.google.com with ESMTPSA id hj10sm2629589wjc.48.2015.03.25.02.04.35 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Mar 2015 02:04:35 -0700 (PDT) Date: Wed, 25 Mar 2015 09:03:48 +0000 From: Stefan Grundmann To: freebsd-hackers@freebsd.org Subject: trusted gptboot preview [patch] Message-ID: <20150325090348.GA17422@bath> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="EeQfGwPcQSOJBaQU" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 09:04:39 -0000 --EeQfGwPcQSOJBaQU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline hi, the promised minimal howto got a bit longer than planed. best regards Stefan Grundmann -------------------------------------------------------------------------- SRTM (Static Root Of Trust) FreeBSD boot * Motivation Anti Evil Maid http://theinvisiblethings.blogspot.de/2011/09/anti-evil-maid.html * Existing SRTM solutions ** TrustedGRUB - http://trustedgrub.sourceforge.net/ - ported by BSSSD - GNU GRUB 1.* based - no GPT support - insufficient UFS support - no easy way to decrypt disk keys with secrets from TPM ** GRUB-IMA - http://sourceforge.net/projects/trousers/files/Grub-IMA/ - same limitations as TrustedGRUB plus never ported to FreeBSD * Deliverables The attached patch should work on any FreeBSD-10 or FreeBSD-11 amd64 or x86 system. TPM 1.2 is required. ** sys/boot/i386/tpmbr - this is sys/boot/i386/pmbr plus TCG (Trusted Computing Group) support - had to sacrifice most of the error messages to make space for the TCG_CompactHashLogExtendEvent code - the entire 'freebsd-boot' partition is measured (up to 545k) ** sys/boot/i386/tgptboot - gptboot(8) extended with parts of loader(8) and TCG support - get Key and config data path from TPM - read and decrypt the config data from UFS - use config data to set kernel environment, load disk keys, verify and load kernel + modules (from UFS) - boot the kernel ** sbin/tgptbootdata - create tgptboot config_data files * Use Case For a system that has TPM 1.2 enabled and configured to have a KEY in TPM NVRAM. The system performs a (legacy) BIOS boot from a GTP disk that has tpmbr as bootcode and tgptboot as partcode in the 'freebsd-boot' partition. boot sequence, see [TCG] section 3.2: 1. S-CRTM (Static Core Root of Trust) - TPM Startup - load and measure BIOS - execute BIOS 2. BIOS - measure configuration - load and measure PMBR (Protective Master Boot Record) - execute PMBR 3. PMBR - locate 'freebsd-boot' partition - load and measure 'freebsd-boot' partition (extend PCR9) - transfer control to tgptboot 4. tgptboot - select UFS partition (take gptboot(8) attributes into account) - get KEY from TPM NVRAM - get DATA_PATH from TPM NVRAM - read and decrypt DATA at DATA_PATH using KEY - set kernel environment (if specified in DATA) - load and checksum kernel and modules (path, size and hash in DATA) - load disk keys (if specified in DATA) - execute kernel Since each measurement modifies TPM PCRs (Platform Configuration Registers), the TPM NVRAM index that contains the KEY can not be read if the current PCRs do not match the PCRs at write time (for the set of PCRs the TPM NVRAM index write was sealed under). If tgptboot can not contiune because of an error, the next GPT UFS partion is selected (gptboot(8) behaviour). kernel tpm(4) support is only needed for provisioning. changes to anything measured into the PCRs included in the TPM NVRAM index write for the KEY (BIOS update, changes to tpmbr or tgptboot, ...) require reprovisioning. kernel update does not require tpm interaction. [TCG] TCG PC Client Specific Implementaion Specification for Conventional BIOS Specification Version 1.21 Errata Revision 1.00 February 24th, 2012 For TPM Family 1.2; Level 2 * Setup (step by step tutorial) SRTM on an ThinkPad X60 ** preparations *** buildsystem - FreeBSD-10.1-p8 amd64 - r280275 in /usr/src - results of successfull buildworld and buildkernel (GENERIC) for r280275 in /usr/obj *** tpm-tools 1.3.8 - the security/tpm-tools in the ports tree is too old (missing nvram write support) - get the tpm-tools 1.3.8 port from https://github.com/sg2342/freebsd_port-tpm-tools.git - build packages for tpm-tools and its runtime dependencies ** build deliverables *** apply 0001-trusted-gptboot-preview.patch to /usr/src # cd /usr/src && patch -p0 < path_to/0001_trusted-gptboot-preview.patch *** build tpmbr # make -C sys/boot/i386/tpmbr clean obj # make -C sys/boot/i386/tpmbr depend all *** build tgptboot # make -C sys/boot/i386/tgptboot clean obj # make -C sys/boot/i386/tgptboot depend all *** build tgptbootdata # make -C sbin/tgptbootdata clean obj # make -C sbin/tgptbootdata depend all ** create provisioning USB stick - usb stick is da0 in the buildsystem - usb stick size is at least 2GB *** create GPT schema and partitions # gpart destroy -F da0 # gpart create -s GPT da0 # gpart bootcode -b /usr/obj/usr/src/sys/boot/i386/tpmbr/tpmbr da0 # gpart add -b 40 -s 384 -t freebsd-boot -l tgptboot -i 1 da0 # dd if=/dev/zero of=/dev/da0p1 # gpart bootcode -p /usr/obj/usr/src/sys/boot/i386/tgptboot/tgptboot -i 1 da0 # gpart add -a 4k -s 131072 -t freebsd-ufs -l bootA -i 2 da0 # gpart add -a 4k -s 131072 -t freebsd-ufs -l bootB -i 3 da0 # gpart add -a 4k -s 1G -t freebsd-ufs -l toolRoot -i 4 da0 *** newfs # newfs -m 0 -L bootA /dev/gpt/bootA # newfs -m 0 -L bootB /dev/gpt/bootB # newfs -m 0 -L toolRoot /dev/gpt/toolRoot *** installworld and kernel in toolRoot # mkdir -p /tmp/toolRoot && mount -o noatime /dev/ufs/toolRoot /tmp/toolRoot # cd /usr/src # make installworld DESTDIR=/tmp/toolRoot # make distribution DESTDIR=/tmp/toolRoot # make installkernel DESTDIR=/tmp/toolRoot *** install tpm-tools in toolRoot # mkdir -p /tmp/toolRoot/tmp/pkg # mount_nullfs -o ro path_to_package_directory /tmp/toolRoot/tmp/pkg # pkg -c /tmp/toolRoot add /tmp/pkg/All/tpm-tools-1.3.8.txz # umount /tmp/toolRoot/tmp/pkg install a the tgptbootdata tool as well # cp /usr/obj/usr/src/sbin/tgptbootdata/tgptbootdata /tmp/toolRoot/sbin/ *** fix fstab in toolRoot # printf "/dev/ufs/toolRoot\t/\tufs\trw\t1 1\n" > /tmp/toolRoot/etc/fstab *** install kernel in bootA # mkdir -p /tmp/bootA && mount -o noatime /dev/ufs/bootA /tmp/bootA # mkdir -p /tmp/bootA/boot/kernel # cp /tmp/toolRoot/boot/kernel/kernel /tmp/bootA/boot/kernel *** create tgptbootdata file in bootA there is no key and data_path in the TPM NVRAM yet, we force tgptboot to use key and data_file from well known override locations # mkdir -p /tmp/bootA/boot/magic - existence of /boot/magic/key on UFS boot partition will make tgptboot to use the (all zero in this case) key to decrypt the data file # dd if=/dev/zero of=/tmp/bootA/boot/magic/key bs=32 count=1 - existence of /boot/magic/data_force will make tgptboot to use the tgptboot data_file at /boot/magic/data # touch /tmp/bootA/boot/magic/data_force - create minimal tgptbootdata config file: # cat >> /tmp/magic_toolRoot.conf << EOF {provision,[ {env,[{"vfs.root.mountfrom","ufs:/dev/ufs/toolRoot"}]}, {load, [{"/tmp/bootA/boot/kernel/kernel", "/boot/kernel/kernel",""}]}, {key, []}]}. EOF - create the data_file # /usr/obj/10.1/usr/src/10.1/sbin/tgptbootdata/tgptbootdata \ -f /tmp/magic_toolRoot.conf -k /tmp/bootA/boot/magic/key \ -o /tmp/bootA/boot/magic/data *** set bootme on bootA # gpart set -a bootme -i 2 da0 *** umount file systems, eject usb stic # umount /tmp/toolRoot # umount /tmp/bootA # camcontrol eject da0 ** test the provisioning USB stick - the USB stick should boot the target system regardless of TPM configuration in the ThinkPad BIOS - actually the USB stick should boot anything amd64 system that supports BIOS legacy boot with GENERIC kernel and empty kernel environment ** setup target system TPM in BIOS - poweroff the ThinkPad X66 - press power button while holding F1 (TPM physical presence indicator in Thinkpad BIOS, required to clear TPM) until the BIOS Setup Utility is shown - in > Security > Security Chip - set 'Security Chip' to 'Active' - clear security chip - in > Security > Security Chip > Security Reporting Options several Reporting Options can be set which will together with the choice of PCR set used to seal the KEY determine which changes to BIOS options and system configuration will require TPM reprovisioning. selection of the right PCR selection is out of scope, we will use only PCR9 in this tutorial. ** provision target system TPM - boot ThinkPad with the provisioning USB stick - log in as root - start tcsd # kldload tpm # service tcsd onestart - take tpm ownership # tpm_takeownership -y -z - define NVRAM area for the AES key, read selection PCR 9 - 0x020002323 is the index tgptboot will use when looking for the KEY - the READ_STCLEAR permission will make this nvram index unreadable after size 0 read (which is done by tgptboot right before executing the kernel) # tpm_nvdefine -i 0x020002323 -s 32 -y -r 9 -p "OWNERWRITE|READ_STCLEAR" - create AES key and write NV RAM # tpm_nvwrite -i 0x020002323 -z -s 32 -d "012345678901234567890123456789012345" - define NV RAM area for the tgptboot data file path - 0x020004242 is the index tgptboot will use when looking for the data file path # tpm_nvdefine -i 0x020004242 -s 128 -y -p OWNERWRITE - fill the NV RAM area for the tgptboot data file path with zeros # tpm_nvwrite -i 0x020004242 -z -s 128 -m 0x00 - write the path "/boot/x60" # tpm_nvwrite -i 0x020004242 -z -d "/boot/x60" ** install kernel, and one modules and tgptboot data file to /dev/gpt/bootB # mkdir -p /tmp/bootB # mount -o noatime /dev/ufs/bootB /tmp/bootB # mkdir -p /tmp/bootB/boot/kernel # cp /boot/kernel/kernel /tmp/bootB/boot/kernel/ # cp /boot/kernel/tpm.ko /tmp/bootB/boot/kernel/ - create tgptbootdata config file, kern + one module, no disk key, only vsf.root.mountfrom in env: # cat >> /tmp/tgptbootdata.conf << EOF {provision,[ {env,[{"vfs.root.mountfrom","ufs:/dev/ufs/toolRoot"}]}, {load, [{"/tmp/bootB/boot/kernel/kernel", "/boot/kernel/kernel",""}, {"/tmp/bootB/boot/kernel/tmp.ko", "/boot/kernel/tpm.ko",undefined}]} {key, []}]}. EOF - create the data_file # echo -n "012345678901234567890123456789012345" > /tmp/aes.key # tgptbootdata/tgptbootdata \ -f /tmp/tgptbootdata.conf -k /tpm/aes.key \ -o /tmp/bootB/boot/x60 - set gptboot(8) attribute bootme to bootB remove it from bootA # gpart set -a bootme -i 3 da0 # gpart unset -a bootme -i 2 da0 ** reboot target system will now boot using key and path from TPM nvram ------------------------------------------------------------------------------------------- --EeQfGwPcQSOJBaQU Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-trusted-gptboot-preview.patch" diff --git sbin/tgptbootdata/Makefile sbin/tgptbootdata/Makefile new file mode 100644 index 0000000..1b03ebe --- /dev/null +++ sbin/tgptbootdata/Makefile @@ -0,0 +1,22 @@ +# $FreeBSD$ + +PROG= tgptbootdata +SRCS= parse.y scan.l y.tab.h tgptbootdata.h tgptbootdata.c +SRCS+= rijndael-alg-fst.c rijndael-api-fst.c +CFLAGS+=-I. -I${.CURDIR} +CFLAGS+=-DYY_NO_UNPUT -DYY_NO_INPUT +CFLAGS+= -I${.CURDIR}/../../sys + +NO_WCAST_ALIGN= +NO_WMISSING_VARIABLE_DECLARATIONS= + +.PATH: ${.CURDIR}/../../sys/crypto/rijndael + +LDADD= -ll -lmd +DPADD= ${LIBL} ${LIBMD} + +MAN= tgptbootdata.8 + +WARN?= 3 + +.include diff --git sbin/tgptbootdata/parse.y sbin/tgptbootdata/parse.y new file mode 100644 index 0000000..6ac5894 --- /dev/null +++ sbin/tgptbootdata/parse.y @@ -0,0 +1,191 @@ +%{ +/*- + * + * Copyright (c) 2014, GSMK mbh + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#include +#include +#include +#include +#include +#include +#include +#include "tgptbootdata.h" + +static void +add_env(const char *key, const char *val) +{ + struct q_env *qq; + + if (NULL == (qq = (struct q_env *)malloc(sizeof(struct q_env))) || + NULL == (qq->key = strdup(key)) || + NULL == (qq->val = strdup(val))) + errx(1, "malloc failed"); + STAILQ_INSERT_TAIL(&q_env_h, qq, link); +} + +static void +add_load(const char *module_file, const char *path, const char *flags) +{ + struct q_load *qq; + + if (NULL == (qq = (struct q_load *)malloc(sizeof(struct q_load))) || + NULL == (qq->module_file = strdup(module_file)) || + NULL == (qq->path = strdup(path))) + errx(1, "malloc failed"); + if (NULL == flags || 0 == *flags) + qq->flags = NULL; + else if (NULL == (qq->flags = strdup(flags))) + errx(1, "malloc failed"); + STAILQ_INSERT_TAIL(&q_load_h, qq, link); +} + + +static void +add_key(const char *key_name, const char *prov, const char *key_file) +{ + struct q_key *qq; + + if (NULL == (qq = (struct q_key *)malloc(sizeof(struct q_key))) || + NULL == (qq->key_name = strdup(key_name)) || + NULL == (qq->prov = strdup(prov)) || + NULL == (qq->key_file = strdup(key_file))) + errx(1, "malloc failed"); + STAILQ_INSERT_TAIL(&q_key_h, qq, link); +} +%} + +%union { + char *str; +} + +%token COMMA DOT BEGINTUPLE ENDTUPLE +%token BEGINLIST ENDLIST +%token SEMICOLON BEGINBLOCK ENDBLOCK COMMA +%token STRING +%token KEY ENV LOAD PROVISION UNDEFINED + + +%% + +config_file + : provision_t + | + ; + +provision_t + : BEGINTUPLE PROVISION COMMA provision_l ENDTUPLE DOT + ; + +provision_l + : BEGINLIST provision_i_ ENDLIST + ; + +provision_i_ + : provision_i provision_i__ + ; + +provision_i__ + : provision_i__ COMMA provision_i + | + ; + +provision_i + : env_t + | load_t + | key_t + ; + +env_t + : BEGINTUPLE ENV COMMA env_l ENDTUPLE + ; + +env_l + : BEGINLIST env_i_ ENDLIST + | BEGINLIST ENDLIST + ; + +env_i_ + : env_i env_i__ + ; + +env_i__ + : env_i__ COMMA env_i + | + ; + +env_i + : BEGINTUPLE STRING COMMA STRING ENDTUPLE { add_env($2,$4); } + ; + +load_t + : BEGINTUPLE LOAD COMMA load_l ENDTUPLE + ; + +load_l + : BEGINLIST load_i_ ENDLIST + | BEGINLIST ENDLIST + ; + +load_i_ + : load_i load_i__ + ; + +load_i__ + : load_i__ COMMA load_i + | + ; + +load_i + : BEGINTUPLE STRING COMMA STRING COMMA STRING ENDTUPLE { add_load($2,$4,$6); } + | BEGINTUPLE STRING COMMA STRING COMMA UNDEFINED ENDTUPLE { add_load($2,$4,NULL); } + ; + +key_t + : BEGINTUPLE KEY COMMA key_l ENDTUPLE + ; + +key_l + : BEGINLIST key_i_ ENDLIST + | BEGINLIST ENDLIST + ; + +key_i_ + : key_i key_i__ + ; + +key_i__ + : key_i__ COMMA key_i + | + ; + +key_i + : BEGINTUPLE STRING COMMA STRING COMMA STRING ENDTUPLE { add_key($2,$4,$6); } + ; + +%% diff --git sbin/tgptbootdata/scan.l sbin/tgptbootdata/scan.l new file mode 100644 index 0000000..8bf4d46 --- /dev/null +++ sbin/tgptbootdata/scan.l @@ -0,0 +1,102 @@ +%{ +/*- + * + * Copyright (c) 2014, GSMK mbh + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#include +#include +#include +#include +#include "y.tab.h" + +int lineno = 1; + +void yyerror(const char *); + +int yylex(void); + +static void +update_lineno(const char *cp) +{ + while (*cp) + if (*cp++ == '\n') + lineno++; +} + +%} + +%option nounput +%option noinput + +%% + +[ \t]+ ; +\n lineno++; +, { return COMMA; } +\. { return DOT; } +\/\*([^*]|(\*+([^*\/])))*\*+\/ { update_lineno(yytext); } +\{ { return BEGINTUPLE; } +\} { return ENDTUPLE; } +\[ { return BEGINLIST; } +\] { return ENDLIST; } +\"[^"]*\" { + size_t len = strlen(yytext) - 2; + char *walker; + unsigned int i; + update_lineno(yytext); + if ((yylval.str = (char *) malloc(len + 1)) == NULL) + goto out; + walker = yylval.str; + for (i = 1; i <= len; i++) { + if (yytext[i] == '\\' && + yytext[i + 1] == '\n') { + i += 2; + while(isspace(yytext[i])) + i++; + } + *walker++ = yytext[i]; + } + *walker++ = '\0'; + out:; + return STRING; + } +\%.*$ ; + +key { return KEY; } +env { return ENV; } +load { return LOAD; } +provision { return PROVISION; } +undefined { return UNDEFINED; } + +%% + +void +yyerror(const char *s) +{ + fprintf(stderr, "line %d: %s%s %s.\n", lineno, yytext, yytext?":":"", s); +} diff --git sbin/tgptbootdata/tgptbootdata.8 sbin/tgptbootdata/tgptbootdata.8 new file mode 100644 index 0000000..f34c7f9 --- /dev/null +++ sbin/tgptbootdata/tgptbootdata.8 @@ -0,0 +1,46 @@ +.\" Copyright (c) 2014, GSMK mbh +.\" The Regents of the University of California. All rights reserved. +.\" +.\" This code is derived from software contributed to Berkeley by +.\" the Institute of Electrical and Electronics Engineers, Inc. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" 4. Neither the name of the University nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd April 6, 2014 +.Dt TGPTBOOTDATA 8 +.Os +.Sh NAME +.Nm tgptbootdata +.Nd create tgptboot data files +.Sh SYNOPSIS +.Nm +.Sh DESCRIPTION +TODO +.Sh SEE ALSO +.Xr tgptboot 8 +.Sh AUTHORS +Stefan Grundmann diff --git sbin/tgptbootdata/tgptbootdata.c sbin/tgptbootdata/tgptbootdata.c new file mode 100644 index 0000000..41837f4 --- /dev/null +++ sbin/tgptbootdata/tgptbootdata.c @@ -0,0 +1,434 @@ +/*- + * Copyright (c) 2014, GSMK mbh + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "tgptbootdata.h" + +/* yacc leaves us with these */ +extern FILE *yyin; +extern int lineno; +extern int yparse(void); + + +struct q_env_h q_env_h = STAILQ_HEAD_INITIALIZER(q_env_h); +struct q_load_h q_load_h = STAILQ_HEAD_INITIALIZER(q_load_h); +struct q_key_h q_key_h = STAILQ_HEAD_INITIALIZER(q_key_h); + +enum { + C_LEN = 4, /* four bytes (Big Endian) length(s) */ + C_TAG = 1, /* one byte TAG */ + C_TERM = 1, /* \0 terminated strings */ + C_SIZE = C_LEN, + C_STR = C_TAG + C_LEN + C_TERM, +}; + +enum { + AES_KEY_SIZE = 32, + AES_KEY_BITS = (AES_KEY_SIZE * 8), + AES_BLOCK_SIZE = 16, + AES_IV_SIZE = AES_BLOCK_SIZE, + SHA512_DIGEST_SIZE = 64, +}; + +enum { + T_env = 8, + T_env_Key = 9, + T_env_Val = 10, + T_key = 16, + T_key_Prov = 17, + T_key_Key = 18, + T_key_Name = 19, + T_load = 32, + T_load_Path = 33, + T_load_Size = 34, + T_load_Args = 35, + T_load_Hash = 36, +}; + + +static inline void +st32BE(uint8_t *p, uint32_t x) +{ + p[0] = x >> 24; + p[1] = (x >> 16) & 0xff; + p[2] = (x >> 8) & 0xff; + p[3] = x & 0xff; +} + +static void +usage(void) +{ + fprintf(stderr, "usage: %s -f configfile [-k keyfile] [-o outfile]\n", + getprogname()); + exit(1); +} + +static void +parse_config(const char *configfile) +{ + if (NULL == (yyin = fopen(configfile, "r"))) + errx(EX_OSFILE, "Cannot open config file %s", configfile); + + lineno = 1; + + if (0 != yyparse()) + errx(EX_CONFIG, "Cannot parse %s at line %d", configfile, lineno); + if (0 != fclose(yyin)) + errx(EX_OSERR, "fclose failed; %s", strerror(errno)); +} + +static uint32_t +file_size(const char *path) +{ + struct stat sb; + + if (0 != stat(path, &sb)) + errx(EX_OSFILE, "coud not stat \"%s\" : %s", path, strerror(errno)); + if (sb.st_size > UINT32_MAX) + errx(EX_DATAERR, "%s too large", path); + return (uint32_t)sb.st_size; +} + +static void +file_sha512(const char *module_path, uint8_t *hash) +{ + FILE *fp; + SHA512_CTX sha512; + size_t cnt; + const size_t buf_size = 1 << 12;/* 4K */ + uint8_t *buf; + + + if (NULL == (fp = fopen(module_path, "rb"))) + errx(EX_OSFILE, "Cannot open module file %s: %s", module_path, strerror(errno)); + + if (NULL == (buf = (uint8_t *)malloc(buf_size))) + errx(EX_OSERR, "malloc failed"); + + SHA512_Init(&sha512); + while (0 != (cnt = fread(buf, 1, buf_size, fp))) + SHA512_Update(&sha512, buf, cnt); + + SHA512_Final(hash, &sha512); + if (0 != fclose(fp)) + errx(EX_OSERR, "fclose failed: %s", strerror(errno)); + free(buf); +} + +static void +encode_load(uint8_t **buf, uint32_t *buf_size) +{ + struct q_load *qq; + uint8_t *p; + + for (qq = STAILQ_FIRST(&q_load_h), *buf_size = 0; NULL != qq; (qq = STAILQ_NEXT(qq, link))) { + *buf_size += C_TAG + C_LEN + + C_STR + strlen(qq->path) + + C_TAG + C_LEN + C_SIZE + + ((NULL == qq->flags) ? C_TAG + C_LEN : C_STR + (strlen(qq->flags))) + + C_TAG + C_LEN + SHA512_DIGEST_SIZE; + } + if (NULL == (*buf = (uint8_t *)malloc(*buf_size))) + errx(EX_OSERR, "malloc failed"); + + p = *buf; + memset(p, 0, *buf_size); + + for (qq = STAILQ_FIRST(&q_load_h); NULL != qq; (qq = STAILQ_NEXT(qq, link))) { + uint8_t sha[SHA512_DIGEST_SIZE]; + + const uint32_t sz = file_size(qq->module_file); + const uint32_t pl = strlen(qq->path); + const uint32_t fl = (NULL == qq->flags) ? 0 : (strlen(qq->flags)); + + file_sha512(qq->module_file, sha); + + /* start load */ + *p++ = T_load; + st32BE(p, C_STR + pl + C_TAG + C_LEN + C_SIZE + (fl ? C_STR + fl : C_TAG + C_LEN) + + C_TAG + C_LEN + SHA512_DIGEST_SIZE); + p += C_LEN; + + /* store path */ + *p++ = T_load_Path; + st32BE(p, pl + C_TERM); + p += C_LEN; + memcpy(p, qq->path, pl); + p += pl + C_TERM; /* string + \0 (from memset) */ + + /* store size */ + *p++ = T_load_Size; + st32BE(p, C_SIZE), + p += C_LEN; + st32BE(p, sz); + p += C_SIZE; + + /* store args/flags */ + *p++ = T_load_Args; + st32BE(p, fl ? fl + C_TERM : 0); + p += C_LEN; + if (fl) { + memcpy(p, qq->flags, fl); + p += fl + C_TERM; /* string + \0 (from memset) */ + } + /* store hash */ + *p++ = T_load_Hash; + st32BE(p, SHA512_DIGEST_SIZE); + p += C_LEN; + memcpy(p, sha, SHA512_DIGEST_SIZE); + p += SHA512_DIGEST_SIZE; + } +} + +static void +encode_env(uint8_t **buf, uint32_t *buf_size) +{ + struct q_env *qq; + uint8_t *p; + + for (qq = STAILQ_FIRST(&q_env_h), *buf_size = 0; NULL != qq; (qq = STAILQ_NEXT(qq, link))) + *buf_size += C_TAG + C_LEN + + C_STR + strlen(qq->key) + + C_STR + strlen(qq->val); + if (NULL == (*buf = (uint8_t *)malloc(*buf_size))) + errx(EX_OSERR, "malloc failed"); + + p = *buf; + memset(p, 0, *buf_size); + + for (qq = STAILQ_FIRST(&q_env_h); NULL != qq; (qq = STAILQ_NEXT(qq, link))) { + const uint32_t kl = strlen(qq->key); + const uint32_t vl = strlen(qq->val); + + /* start env */ + *p++ = T_env; + st32BE(p, kl + vl + 2 * C_STR); + p += C_LEN; + + /* store key */ + *p++ = T_env_Key; + st32BE(p, kl + C_TERM); + p += C_LEN; + memcpy(p, qq->key, kl); + p += kl + C_TERM; /* string + \0 (from memset) */ + + /* store value */ + *p++ = T_env_Val; + st32BE(p, vl + C_TERM); + p += C_LEN; + memcpy(p, qq->val, vl); + p += vl + C_TERM; /* string + \0 (from memset) */ + } +} + +static void +encode_gelikey(uint8_t **buf, uint32_t *buf_size) +{ + struct q_key *qq; + uint8_t *p; + + for (qq = STAILQ_FIRST(&q_key_h), *buf_size = 0; NULL != qq; (qq = STAILQ_NEXT(qq, link))) + *buf_size += C_TAG + C_LEN + + C_STR + strlen(qq->key_name) + + C_STR + strlen(qq->prov) + + C_TAG + C_LEN + file_size(qq->key_file); + if (NULL == (*buf = (uint8_t *)malloc(*buf_size))) + errx(EX_OSERR, "malloc failed"); + + p = *buf; + memset(p, 0, *buf_size); + + for (qq = STAILQ_FIRST(&q_key_h); NULL != qq; (qq = STAILQ_NEXT(qq, link))) { + FILE *fp; + const uint32_t nl = strlen(qq->key_name); + const uint32_t pl = strlen(qq->prov); + const uint32_t sz = file_size(qq->key_file); + + /* start key */ + *p++ = T_key; + st32BE(p, C_STR + nl + C_STR + pl + C_TAG + C_LEN + sz); + p += C_LEN; + + /* store name */ + *p++ = T_key_Name; + st32BE(p, nl + C_TERM); + p += C_LEN; + memcpy(p, qq->key_name, nl); + p += nl + C_TERM; /* string + \0 (from memset) */ + + /* store provider */ + *p++ = T_key_Prov; + st32BE(p, pl + C_TERM); + p += C_LEN; + memcpy(p, qq->prov, pl); + p += pl + C_TERM; /* string + \0 (from memset) */ + + /* store the key */ + *p++ = T_key_Key; + st32BE(p, sz); + p += C_LEN; + if (NULL == (fp = fopen(qq->key_file, "rb"))) + errx(EX_OSFILE, "Cannot open key file %s: %s", qq->key_file, strerror(errno)); + if (0 == fread(p, sz, 1, fp)) + errx(EX_OSFILE, "Error reading %s: %s", qq->key_file, strerror(errno)); + if (0 != fclose(fp)) + errx(EX_OSERR, "fclose failed: %s", strerror(errno)); + p += sz; + } +} + +static void +encode_magic(uint8_t **buf, uint32_t *buf_size) +{ + uint8_t *env, *load, *key, *p; + uint32_t env_size, load_size, key_size, pt_size, pad_size; + SHA512_CTX sha512; + + encode_env(&env, &env_size); + encode_load(&load, &load_size); + encode_gelikey(&key, &key_size); + + pt_size = env_size + load_size + key_size; + pad_size = AES_BLOCK_SIZE - (pt_size % AES_BLOCK_SIZE); + *buf_size = SHA512_DIGEST_SIZE + AES_BLOCK_SIZE + pt_size + pad_size; + + if (NULL == (*buf = (uint8_t *)malloc(*buf_size))) + errx(EX_OSERR, "malloc failed"); + + + p = *buf; + memset(p, 0, *buf_size); + + p += SHA512_DIGEST_SIZE; /* skip 64 bytes for the hash */ + st32BE(p, pt_size); /* encode PT length */ + p += AES_BLOCK_SIZE; /* leave the rest of the block at \0 */ + + memcpy(p, env, env_size); + p += env_size; + free(env); + memcpy(p, load, load_size); + p += load_size; + free(load); + memcpy(p, key, key_size); + p += key_size; + free(key); + + SHA512_Init(&sha512); + SHA512_Update(&sha512, *buf + SHA512_DIGEST_SIZE, pt_size + AES_BLOCK_SIZE); + SHA512_Final(*buf, &sha512); + +} + +static void +encrypt_magic(const char *keyfile, const char *outfile, uint8_t *buf, uint32_t buf_size) +{ + uint8_t key[AES_KEY_SIZE]; + keyInstance ki; + cipherInstance ci; + int error; + FILE *fp; + + /* key: 32 bytes from keyfile or stdin */ + if (NULL != keyfile) { + if (NULL == (fp = fopen(keyfile, "r"))) + errx(EX_OSFILE, "Cannot open key file %s: %s", keyfile, strerror(errno)); + if (0 == fread(key, AES_KEY_SIZE, 1, fp)) + errx(EX_OSFILE, "Error reading %s:%s", keyfile, strerror(errno)); + if (0 != fclose(fp)) + errx(EX_OSERR, "fclose failed: %s", strerror(errno)); + } else + fread(key, AES_KEY_SIZE, 1, stdin); + + if (NULL == outfile) + fp = stdout; + else if (NULL == (fp = fopen(outfile, "wb"))) + errx(EX_OSFILE, "Cannot open output file %s: %s", outfile, strerror(errno)); + + /* encrypt PT buffer in place */ + if (0 >= (error = rijndael_cipherInit(&ci, MODE_CBC, NULL))) + errx(1, "rijndael_cipherInit=%d", error); + if (0 >= (error = rijndael_makeKey(&ki, DIR_ENCRYPT, AES_KEY_BITS, (char *)key))) + errx(1, "rijndael_makeKey=%d", error); + if (0 >= (error = rijndael_blockEncrypt(&ci, &ki, buf, 8 * buf_size, buf))) + errx(1, "rijndael_blockEncrypt=%d", error); + + /* write CT */ + if (0 == fwrite(buf, buf_size, 1, fp)) + errx(EX_OSFILE, "Error writing %s", (NULL == outfile) ? "" : outfile); + if (0 != fclose(fp)) + errx(EX_OSERR, "fclose failed; %s", strerror(errno)); +} + +int +main(int argc, char **argv) +{ + char *configfile = NULL; + char *keyfile = NULL; + char *outfile = NULL; + uint8_t *buf; + uint32_t buf_size; + int ch; + + while ((ch = getopt(argc, argv, "f:k:o:")) != -1) { + switch (ch) { + case 'f': + configfile = optarg; + break; + case 'k': + keyfile = optarg; + break; + case 'o': + outfile = optarg; + break; + default: + usage(); + } + } + if (NULL == configfile) + usage(); + + parse_config(configfile); + encode_magic(&buf, &buf_size); + encrypt_magic(keyfile, outfile, buf, buf_size); + return 0; +} diff --git sbin/tgptbootdata/tgptbootdata.h sbin/tgptbootdata/tgptbootdata.h new file mode 100644 index 0000000..ce84c33 --- /dev/null +++ sbin/tgptbootdata/tgptbootdata.h @@ -0,0 +1,69 @@ +/*- + * Copyright (c) 2014, GSMK mbh + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef TGPTBOOTDATA_H +#define TGPTBOOTDATA_H + + +STAILQ_HEAD (q_env_h, q_env); + +struct q_env { + char *key; + char *val; + STAILQ_ENTRY (q_env) link; +}; + + +STAILQ_HEAD (q_load_h, q_load); + +struct q_load { + char *module_file; + char *path; + char *flags; + STAILQ_ENTRY (q_load) link; +}; + + +STAILQ_HEAD (q_key_h, q_key); + +struct q_key { + char *key_name; + char *prov; + char *key_file; + STAILQ_ENTRY (q_key) link; +}; + +extern struct q_env_h q_env_h; +extern struct q_load_h q_load_h; +extern struct q_key_h q_key_h; + +void yyerror(const char *s); +int yylex(void); +int yyparse(void); + +#endif diff --git sys/boot/i386/tgptboot/Makefile sys/boot/i386/tgptboot/Makefile new file mode 100644 index 0000000..e9fbf1b --- /dev/null +++ sys/boot/i386/tgptboot/Makefile @@ -0,0 +1,77 @@ +# $FreeBSD$ + +.PATH: ${.CURDIR}/../common \ + ${.CURDIR}/../../common ${SYSDIR}/crypto/sha2 \ + ${SYSDIR}/crypto/rijndael +FILES= tgptboot + +ORG1= 0x7c00 +ORG2= 0x0 + +CFLAGS= -DBOOTPROG=\"tgptboot\" \ + -O1 \ + -DGPT \ + -DTLOADER_VERBOSE \ + -DLOADER_MBR_SUPPORT \ + -DLOADER_GPT_SUPPORT \ + -I${.CURDIR}/.. \ + -I${SYSDIR}/crypto/sha2 \ + -I${SYSDIR}/crypto/rijndael \ + -I${.CURDIR}/../../common \ + -I${.CURDIR}/../common \ + -I${.CURDIR}/../btx/lib -I. \ + -I${.CURDIR}/../../.. \ + -Wall -Waggregate-return -Wbad-function-cast \ + -Wmissing-declarations -Wmissing-prototypes -Wnested-externs \ + -Wpointer-arith -Wshadow -Wstrict-prototypes -Wwrite-strings \ + -Winline + +CFLAGS.gcc+= --param max-inline-insns-single=100 + +LIBI386= ${.OBJDIR}/../libi386/libi386.a +LD_FLAGS= -static -N --gc-sections +LIBSTAND= ${.OBJDIR}/../../libstand32/libstand.a + +# Pick up ../Makefile.inc early. +.include + +CLEANFILES= tgptboot + +tgptboot: gptldr.bin tgptboot.bin ${BTXKERN} + btxld -v -E ${ORG2} -f bin -b ${BTXKERN} -l gptldr.bin \ + -o ${.TARGET} tgptboot.bin + +gptldr.bin: gptldr.out + objcopy -S -O binary gptldr.out ${.TARGET} + +gptldr.out: gptldr.o + ${LD} ${LD_FLAGS} -e start -Ttext ${ORG1} -o ${.TARGET} gptldr.o + +CLEANFILES+= gptldr.bin gptldr.out gptldr.o + +tgptboot.bin: tgptboot.out + objcopy -S -O binary tgptboot.out ${.TARGET} + +TGPTOBJS= bcache.o conf.o console.o crc32.o devopen.o disk.o drv.o gpt.o \ + helper.o isapnp.o load_elf32.o load_elf32_obj.o load_elf64.o \ + load_elf64_obj.o misc.o panic.o part.o pnp.o reloc_elf32.o \ + reloc_elf64.o rijndael-alg-fst.o rijndael-api-fst.o sha2.o \ + tgptboot.o tloader.o tpm.o + +tgptboot.out: ${BTXCRT} ${TGPTOBJS} + ${LD} ${LD_FLAGS} -Ttext ${ORG2} -o ${.TARGET} ${.ALLSRC} ${LIBI386} ${LIBSTAND} + +CLEANFILES+= ${TGPTOBJS} tgptboot.out tgptboot.bin + +.if ${MACHINE_CPUARCH} == "amd64" +beforedepend tgptboot.o: machine +CLEANFILES+= machine +machine: + ln -sf ${.CURDIR}/../../../i386/include machine +.endif + +.include + +# XXX: clang integrated-as doesn't grok .codeNN directives yet +CFLAGS.gptldr.S= ${CLANG_NO_IAS} +CFLAGS+= ${CFLAGS.${.IMPSRC:T}} diff --git sys/boot/i386/tgptboot/conf.c sys/boot/i386/tgptboot/conf.c new file mode 100644 index 0000000..abae79f --- /dev/null +++ sys/boot/i386/tgptboot/conf.c @@ -0,0 +1,162 @@ +/*- + * Copyright (c) 1998 Michael Smith + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include "libi386/libi386.h" +#if defined(LOADER_ZFS_SUPPORT) +#include "../zfs/libzfs.h" +#endif + +/* + * We could use linker sets for some or all of these, but + * then we would have to control what ended up linked into + * the bootstrap. So it's easier to conditionalise things + * here. + * + * XXX rename these arrays to be consistent and less namespace-hostile + * + * XXX as libi386 and biosboot merge, some of these can become linker sets. + */ + +#if defined(LOADER_NFS_SUPPORT) && defined(LOADER_TFTP_SUPPORT) +#error "Cannot have both tftp and nfs support yet." +#endif + +#if defined(LOADER_FIREWIRE_SUPPORT) +extern struct devsw fwohci; + +#endif + +/* Exported for libstand */ +struct devsw *devsw[] = { + &biosdisk, +#if defined(LOADER_NFS_SUPPORT) || defined(LOADER_TFTP_SUPPORT) + &pxedisk, +#endif +#if defined(LOADER_FIREWIRE_SUPPORT) + &fwohci, +#endif +#if defined(LOADER_ZFS_SUPPORT) + &zfs_dev, +#endif + NULL +}; + +struct fs_ops *file_system[] = { +#if defined(LOADER_ZFS_SUPPORT) + &zfs_fsops, +#endif + &ufs_fsops, + &ext2fs_fsops, + &dosfs_fsops, + &cd9660_fsops, +#if defined(LOADER_NANDFS_SUPPORT) + &nandfs_fsops, +#endif +#ifdef LOADER_SPLIT_SUPPORT + &splitfs_fsops, +#endif +#ifdef LOADER_GZIP_SUPPORT + &gzipfs_fsops, +#endif +#ifdef LOADER_BZIP2_SUPPORT + &bzipfs_fsops, +#endif +#ifdef LOADER_NFS_SUPPORT + &nfs_fsops, +#endif +#ifdef LOADER_TFTP_SUPPORT + &tftp_fsops, +#endif + NULL +}; + +/* Exported for i386 only */ +/* + * Sort formats so that those that can detect based on arguments + * rather than reading the file go first. + */ +extern struct file_format i386_elf; +extern struct file_format i386_elf_obj; +extern struct file_format amd64_elf; +extern struct file_format amd64_elf_obj; + +struct file_format *file_formats[] = { +#ifdef LOADER_PREFER_AMD64 + &amd64_elf, + &amd64_elf_obj, +#endif + &i386_elf, + &i386_elf_obj, +#ifndef LOADER_PREFER_AMD64 + &amd64_elf, + &amd64_elf_obj, +#endif + NULL +}; + +/* + * Consoles + * + * We don't prototype these in libi386.h because they require + * data structures from bootstrap.h as well. + */ +extern struct console vidconsole; +extern struct console comconsole; + +#if defined(LOADER_FIREWIRE_SUPPORT) +extern struct console dconsole; + +#endif +extern struct console nullconsole; +extern struct console spinconsole; + +struct console *consoles[] = { + &vidconsole, + &comconsole, +#if defined(LOADER_FIREWIRE_SUPPORT) + &dconsole, +#endif + &nullconsole, + &spinconsole, + NULL +}; + +extern struct pnphandler isapnphandler; +extern struct pnphandler biospnphandler; +extern struct pnphandler biospcihandler; + +struct pnphandler *pnphandlers[] = { + &biospnphandler, /* should go first, as it may set + * isapnp_readport */ + &isapnphandler, + &biospcihandler, + NULL +}; diff --git sys/boot/i386/tgptboot/gptldr.S sys/boot/i386/tgptboot/gptldr.S new file mode 100644 index 0000000..b951c79 --- /dev/null +++ sys/boot/i386/tgptboot/gptldr.S @@ -0,0 +1,142 @@ +/*- + * Copyright (c) 2007 Yahoo!, Inc. + * All rights reserved. + * Written by: John Baldwin + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the author nor the names of any co-contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + * + * Partly from: src/sys/boot/i386/boot2/boot1.S 1.31 + * + * extended to handle up to 256k boot2 code (GSMK mbh 2014) + */ + +/* Memory Locations */ + .set MEM_REL,0x700 # Relocation address + .set MEM_ARG,0x900 # Arguments + .set MEM_ORG,0x7c00 # Origin + .set MEM_BUF,0x8cec # Load area + .set MEM_BTX,0x9000 # BTX start + .set MEM_JMP,0x9010 # BTX entry point + .set MEM_USR,0xa000 # Client start + .set BDA_BOOT,0x472 # Boot howto flag + +/* Misc. Constants */ + .set SIZ_PAG,0x1000 # Page size + .set SIZ_SEC,0x200 # Sector size + + .globl start + .code16 + +/* + * Copy BTX and boot2 to the right locations and start it all up. + */ + +/* + * Setup the segment registers to flat addressing (segment 0) and setup the + * stack to end just below the start of our code. + */ +start: xor %cx,%cx # Zero + mov %cx,%es # Address + mov %cx,%ds # data + mov %cx,%ss # Set up + mov $start,%sp # stack +/* + * BTX is right after us at 'end'. We read the length of BTX out of + * its header to find boot2. We need to copy BTX to MEM_BTX and boot2 + * to MEM_USR. We aren't sure exactly how long boot2 is, but we assume + * it can't be longer than 256k, so we just always copy 256k. + */ + mov $end,%bx # BTX + + cld # Copy forwards + + mov 0xa(%bx),%cx # Get BTX length and set + mov %bx,%si # %si to start of BTX + mov $MEM_BTX,%di # start of BTX at MEM_BTX + rep movsb # Move BTX + + mov 0xa(%bx),%si # Get BTX length and set + add %bx,%si # %si to start of boot2 + mov %si,%ax # Align %ds:%si on a + shr $4,%ax # paragraph boundary + and $0xf,%si # with the smallest + mov %ax,%ds # possible %si + + mov %si,%bp # Save the 0..15 offset in %bp + + xor %di,%di + mov $MEM_USR/16,%ax + mov %ax,%es + + mov $4,%dh # Repeat 4 times ie. copy 256k + +copy64k: sub %si,%cx # First, copy 64k-%si bytes + rep movsb # Now %si=64k and %di=64k-%bp + + mov %ds,%ax + add $0x1000,%ax # Actually %si=0 + mov %ax,%ds # so move %ds 64k ahead + + mov %bp,%cx # Next, copy %bp bytes + rep movsb # Now %si=%bp and %di=64k + + mov %es,%ax + add $0x1000,%ax # Actually %di=0 + mov %ax,%es # so move %es 64k ahead + + dec %dh + jnz copy64k + mov $0,%ax # zero segment register + mov %ax,%ds +/* + * Enable A20 so we can access memory above 1 meg. + * Use the zero-valued %cx as a timeout for embedded hardware which do not + * have a keyboard controller. + */ +seta20: cli # Disable interrupts +seta20.1: dec %cx # Timeout? + jz seta20.3 # Yes + inb $0x64,%al # Get status + testb $0x2,%al # Busy? + jnz seta20.1 # Yes + movb $0xd1,%al # Command: Write + outb %al,$0x64 # output port +seta20.2: inb $0x64,%al # Get status + testb $0x2,%al # Busy? + jnz seta20.2 # Yes + movb $0xdf,%al # Enable + outb %al,$0x60 # A20 +seta20.3: sti # Enable interrupts + +/* + * Save drive number from BIOS so boot2 can see it and start BTX. + */ + movb %dl,MEM_ARG + jmp MEM_JMP # Start BTX + +endorg: .org MEM_USR-MEM_ORG-(endorg-start) +end: diff --git sys/boot/i386/tgptboot/helper.c sys/boot/i386/tgptboot/helper.c new file mode 100644 index 0000000..115d7a2 --- /dev/null +++ sys/boot/i386/tgptboot/helper.c @@ -0,0 +1,249 @@ +/*- + * derived from /sys/boot/common/boot.c: + * Copyright (c) 1998 Michael Smith + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include + + +#include "bootstrap.h" + + +vm_offset_t loadaddr = 0; +struct preloaded_file *preloaded_files = NULL; + +static struct kernel_module *file_findmodule(struct preloaded_file *fp, char *modname, struct mod_depend *verinfo); +void file_insert_tail(struct preloaded_file *fp); +int file_load(char *filename, vm_offset_t dest, struct preloaded_file **result); + + +int +getrootmount(char *ignored) +{ + if (NULL == getenv("vfs.root.mountfrom")) + return 1; + return 0; +} + +int +file_load(char *filename, vm_offset_t dest, struct preloaded_file **result) +{ + static int last_file_format = 0; + struct preloaded_file *fp; + int error; + int i; + + if (archsw.arch_loadaddr != NULL) + dest = archsw.arch_loadaddr(LOAD_RAW, filename, dest); + + error = EFTYPE; + for (i = last_file_format, fp = NULL; + file_formats[i] && fp == NULL; i++) { + error = (file_formats[i]->l_load) (filename, dest, &fp); + if (error == 0) { + fp->f_loader = last_file_format = i; /* remember the loader */ + *result = fp; + break; + } else if (last_file_format == i && i != 0) { + /* Restart from the beginning */ + i = -1; + last_file_format = 0; + fp = NULL; + continue; + } + if (error == EFTYPE) + continue; /* Unknown to this handler? */ + if (error) { + printf("can't load file '%s': %s", + filename, strerror(error)); + break; + } + } + return (error); +} + +void +file_insert_tail(struct preloaded_file *fp) +{ + struct preloaded_file *cm; + + /* Append to list of loaded file */ + fp->f_next = NULL; + if (preloaded_files == NULL) { + preloaded_files = fp; + } else { + for (cm = preloaded_files; cm->f_next != NULL; cm = cm->f_next); + cm->f_next = fp; + } +} + +struct preloaded_file * +file_alloc(void) +{ + struct preloaded_file *fp; + + if ((fp = malloc(sizeof(struct preloaded_file))) != NULL) { + bzero(fp, sizeof(struct preloaded_file)); + } + return (fp); +} + +void +file_discard(struct preloaded_file *fp) +{ + struct file_metadata *md, *md1; + struct kernel_module *mp, *mp1; + + if (fp == NULL) + return; + md = fp->f_metadata; + while (md) { + md1 = md; + md = md->md_next; + free(md1); + } + mp = fp->f_modules; + while (mp) { + if (mp->m_name) + free(mp->m_name); + mp1 = mp; + mp = mp->m_next; + free(mp1); + } + if (fp->f_name != NULL) + free(fp->f_name); + if (fp->f_type != NULL) + free(fp->f_type); + if (fp->f_args != NULL) + free(fp->f_args); + free(fp); +} + +struct preloaded_file * +file_findfile(char *name, char *type) +{ + struct preloaded_file *fp; + + for (fp = preloaded_files; fp != NULL; fp = fp->f_next) { + if (((name == NULL) || !strcmp(name, fp->f_name)) && + ((type == NULL) || !strcmp(type, fp->f_type))) + break; + } + return (fp); +} + +void +file_addmetadata(struct preloaded_file *fp, int type, size_t size, void *p) +{ + struct file_metadata *md; + + md = malloc(sizeof(struct file_metadata) - sizeof(md->md_data) + size); + md->md_size = size; + md->md_type = type; + bcopy(p, md->md_data, size); + md->md_next = fp->f_metadata; + fp->f_metadata = md; +} + +struct kernel_module * +file_findmodule(struct preloaded_file *fp, char *modname, + struct mod_depend *verinfo) +{ + struct kernel_module *mp, *best; + int bestver, mver; + + if (fp == NULL) { + for (fp = preloaded_files; fp; fp = fp->f_next) { + mp = file_findmodule(fp, modname, verinfo); + if (mp) + return (mp); + } + return (NULL); + } + best = NULL; + bestver = 0; + for (mp = fp->f_modules; mp; mp = mp->m_next) { + if (strcmp(modname, mp->m_name) == 0) { + if (verinfo == NULL) + return (mp); + mver = mp->m_version; + if (mver == verinfo->md_ver_preferred) + return (mp); + if (mver >= verinfo->md_ver_minimum && + mver <= verinfo->md_ver_maximum && + mver > bestver) { + best = mp; + bestver = mver; + } + } + } + return (best); +} + +int +file_addmodule(struct preloaded_file *fp, char *modname, int version, + struct kernel_module **newmp) +{ + struct kernel_module *mp; + struct mod_depend mdepend; + + bzero(&mdepend, sizeof(mdepend)); + mdepend.md_ver_preferred = version; + mp = file_findmodule(fp, modname, &mdepend); + if (mp) + return (EEXIST); + mp = malloc(sizeof(struct kernel_module)); + if (mp == NULL) + return (ENOMEM); + bzero(mp, sizeof(struct kernel_module)); + mp->m_name = strdup(modname); + mp->m_version = version; + mp->m_fp = fp; + mp->m_next = fp->f_modules; + fp->f_modules = mp; + if (newmp) + *newmp = mp; + return (0); +} + +struct file_metadata * +file_findmetadata(struct preloaded_file *fp, int type) +{ + struct file_metadata *md; + + for (md = fp->f_metadata; md != NULL; md = md->md_next) + if (md->md_type == type) + break; + return (md); +} diff --git sys/boot/i386/tgptboot/tgptboot.8 sys/boot/i386/tgptboot/tgptboot.8 new file mode 100644 index 0000000..a6db09e --- /dev/null +++ sys/boot/i386/tgptboot/tgptboot.8 @@ -0,0 +1,49 @@ +.\" Copyright (c) 2014, GSMK mbh +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd April 23, 2014 +.Dt TGPTBOOT 8 +.Os +.Sh NAME +.Nm tgptboot +.Nd TPM enabled GPT bootcode and UFS loader for BIOS-based computers +.Sh DESCRIPTION +.Nm +is used on BIOS-based computers to boot from a UFS partition on a +GPT-partitioned disk. +.Nm +is installed in a +.Cm freebsd-boot +partition with +.Xr gpart 8 . +.Sh IMPLEMENTATION NOTES +TODO + +.Sh SEE ALSO +.Xr tgptbootdata 8 , +.Xr loader 8 , +.Xr tpmbr 8 , +.Xr gptboot 8 diff --git sys/boot/i386/tgptboot/tgptboot.c sys/boot/i386/tgptboot/tgptboot.c new file mode 100644 index 0000000..89bc6f5 --- /dev/null +++ sys/boot/i386/tgptboot/tgptboot.c @@ -0,0 +1,310 @@ +/*- + * based on /sys/boot/i386/gptboot.c: Copyright (c) 1998 Robert Nordier + * and /sys/boot/i386/loader/main.c: Copyright (c) 1998 Michael Smith + * All rights reserved. + * + * Redistribution and use in source and binary forms are freely + * permitted provided that the above copyright notice and this + * paragraph and the following disclaimer are duplicated in all + * such forms. + * + * This software is provided "AS IS" and without any express or + * implied warranties, including, without limitation, the implied + * warranties of merchantability and fitness for a particular + * purpose. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include + + +#include +#include +#include +#include +#include + +#include "bootstrap.h" +#include "common/bootargs.h" +#include "libi386/libi386.h" + +#include "gpt.h" + +#define ARGS 0x900 +#define NDEV 3 +#define MEM_BASE 0x12 +#define MEM_EXT 0x15 + +#define DRV_HARD 0x80 +#define DRV_MASK 0x7f + +#define TYPE_AD 0 +#define TYPE_DA 1 +#define TYPE_MAXHARD TYPE_DA +#define TYPE_FD 2 + +extern uint32_t _end; + +static const uuid_t freebsd_ufs_uuid = GPT_ENT_TYPE_FREEBSD_UFS; +uint32_t opts; /* drv.c needs this */ + +static const unsigned char dev_maj[NDEV] = {30, 4, 2}; + +static struct dsk dsk; +struct bootinfo bootinfo; + +static u_int32_t initial_howto; + +struct arch_switch archsw; /* MI/MD interface boundary */ + +extern char bootprog_name[], bootprog_rev[], bootprog_date[], bootprog_maker[]; + +static int load(struct bootinfo *initial_bootinfo, u_int32_t initial_bootdev); + +static uint32_t memsize(void); + +static int isa_inb(int port); +static void isa_outb(int port, int value); +static void extract_currdev(struct bootinfo *, u_int32_t); +void exit(int code); +int main(void); + +extern int tloader(void); + +static char dma_secbuf[DEV_BSIZE]; +static uint8_t dsk_meta; + +static inline uint32_t +memsize(void) +{ + + v86.addr = MEM_EXT; + v86.eax = 0x8800; + v86int(); + return (v86.eax); +} + +static inline void +clear_screen(void) +{ + v86.addr = 0x10; + v86.eax = 0x0700; /* scroll down, entire window */ + v86.ecx = 0x0; /* 0,0 as upper left corner */ + v86.edx = 0x184f; /* 24,79 as lower right corner */ + v86.ebx = 0x0700; /* normal attributes */ + v86int(); + v86.eax = 0x0200; /* set cursor position */ + v86.edx = 0x0; /* cursor at 0,0 */ + v86.ebx = 0x0; /* video page 0 */ + v86int(); +} + +static int +gptinit(void) +{ + if (gptread(&freebsd_ufs_uuid, &dsk, dma_secbuf) == -1) { + printf("%s: unable to load GPT\n", BOOTPROG); + return (-1); + } + if (gptfind(&freebsd_ufs_uuid, &dsk, dsk.part) == -1) { + printf("%s: no UFS partition was found\n", BOOTPROG); + return (-1); + } + dsk_meta = 0; + return (0); +} + +int +main(void) +{ + clear_screen(); + v86.ctl = V86_FLAGS; + v86.efl = PSL_RESERVED_DEFAULT | PSL_I; + dsk.drive = *(uint8_t *)PTOV(ARGS); + dsk.type = dsk.drive & DRV_HARD ? TYPE_AD : TYPE_FD; + dsk.unit = dsk.drive & DRV_MASK; + dsk.part = -1; + dsk.start = 0; + bootinfo.bi_version = BOOTINFO_VERSION; + bootinfo.bi_size = sizeof(bootinfo); + bootinfo.bi_basemem = 0; /* XXX will be filled by loader or + * kernel */ + bootinfo.bi_extmem = memsize(); + bootinfo.bi_memsizes_valid++; + + if (gptinit() != 0) + return (-1); + + for (;;) { + bootinfo.bi_bios_dev = dsk.drive; + load(&bootinfo, MAKEBOOTDEV(dev_maj[dsk.type], dsk.part + 1, dsk.unit, 0xff)); + gptbootfailed(&dsk); + if (gptfind(&freebsd_ufs_uuid, &dsk, -1) == -1) + break; + dsk_meta = 0; + } + panic("out of bootable UFS filesystems"); +} + +/* provide this for panic, as it's not in the startup code */ +void +exit(int code) +{ + __exit(code); +} + + +/* ISA bus access functions for PnP. */ +static int +isa_inb(int port) +{ + return (inb(port)); +} + +static void +isa_outb(int port, int value) +{ + outb(port, value); +} + +/* + * Set the 'current device' by (if possible) recovering the boot device as + * supplied by the initial bootstrap. + * + */ +static void +extract_currdev(struct bootinfo *initial_bootinfo, u_int32_t initial_bootdev) +{ + struct i386_devdesc new_currdev; + int biosdev = -1; + + /* Assume we are booting from a BIOS disk by default */ + new_currdev.d_dev = &biosdisk; + + /* new-style boot loaders such as pxeldr and cdldr */ + if ((initial_bootdev & B_MAGICMASK) != B_DEVMAGIC) { + /* The passed-in boot device is bad */ + new_currdev.d_kind.biosdisk.slice = -1; + new_currdev.d_kind.biosdisk.partition = 0; + biosdev = -1; + } else { + new_currdev.d_kind.biosdisk.slice = B_SLICE(initial_bootdev) - 1; + new_currdev.d_kind.biosdisk.partition = B_PARTITION(initial_bootdev); + biosdev = initial_bootinfo->bi_bios_dev; + + /* + * If we are booted by an old bootstrap, we have to guess at the BIOS + * unit number. We will lose if there is more than one disk type + * and we are not booting from the lowest-numbered disk type + * (ie. SCSI when IDE also exists). + */ + if ((biosdev == 0) && (B_TYPE(initial_bootdev) != 2)) /* biosdev doesn't match + * major */ + biosdev = 0x80 + B_UNIT(initial_bootdev); /* assume harddisk */ + } + new_currdev.d_type = new_currdev.d_dev->dv_type; + + /* + * If we are booting off of a BIOS disk and we didn't succeed in determining + * which one we booted off of, just use disk0: as a reasonable default. + */ + if ((new_currdev.d_type == biosdisk.dv_type) && + ((new_currdev.d_unit = bd_bios2unit(biosdev)) == -1)) { + printf("Can't work out which disk we are booting from.\n" + "Guessed BIOS device 0x%x not found by probes, defaulting to disk0:\n", biosdev); + new_currdev.d_unit = 0; + + } + env_setenv("currdev", EV_VOLATILE, i386_fmtdev(&new_currdev), + i386_setcurrdev, env_nounset); + env_setenv("loaddev", EV_VOLATILE, i386_fmtdev(&new_currdev), env_noset, + env_nounset); +} + +int +load(struct bootinfo *initial_bootinfo, u_int32_t initial_bootdev) +{ + + + static void *heap_top; + static void *heap_bottom; + int i; + int error; + + initial_howto = 0x80000000; + + /* + * Initialise the heap as early as possible. Once this is done, malloc() is usable. + */ + bios_getmem(); + + if (high_heap_size > 0) { + heap_top = PTOV(high_heap_base + high_heap_size); + heap_bottom = PTOV(high_heap_base); + if (high_heap_base < memtop_copyin) + memtop_copyin = high_heap_base; + } else { + heap_top = (void *)PTOV(bios_basemem); + heap_bottom = (void *)_end; + } + setheap(heap_bottom, heap_top); + + bi_setboothowto(initial_howto); + + if (initial_howto & RB_MULTIPLE) { + if (initial_howto & RB_SERIAL) + setenv("console", "comconsole vidconsole", 1); + else + setenv("console", "vidconsole comconsole", 1); + } else if (initial_howto & RB_SERIAL) + setenv("console", "comconsole", 1); + else if (initial_howto & RB_MUTE) + setenv("console", "nullconsole", 1); + cons_probe(); + + /* + * Initialise the block cache + */ + bcache_init(32, 512); /* 16k cache XXX tune this */ + + archsw.arch_autoload = i386_autoload; + archsw.arch_getdev = i386_getdev; + archsw.arch_copyin = i386_copyin; + archsw.arch_copyout = i386_copyout; + archsw.arch_readin = i386_readin; + archsw.arch_isainb = isa_inb; + archsw.arch_isaoutb = isa_outb; + + /* + * March through the device switch probing for things. + */ + for (i = 0; devsw[i] != NULL; i++) + if (devsw[i]->dv_init != NULL) + (devsw[i]->dv_init) (); + printf("BIOS %dkB/%dkB available memory\n", bios_basemem / 1024, bios_extmem / 1024); + if (initial_bootinfo != NULL) { + initial_bootinfo->bi_basemem = bios_basemem / 1024; + initial_bootinfo->bi_extmem = bios_extmem / 1024; + } + /* detect ACPI for future reference */ + biosacpi_detect(); + + /* detect SMBIOS for future reference */ + smbios_detect(); + + printf("\n"); + extract_currdev(initial_bootinfo, initial_bootdev); /* set $currdev and * + * $loaddev */ + setenv("LINES", "24", 1); /* optional */ + bios_getsmap(); + + error = tloader(); + printf("tloader returned %d\n", error); + return (error); +} diff --git sys/boot/i386/tgptboot/tloader.c sys/boot/i386/tgptboot/tloader.c new file mode 100644 index 0000000..a096de4 --- /dev/null +++ sys/boot/i386/tgptboot/tloader.c @@ -0,0 +1,645 @@ +/*- + * Copyright (c) 2014, GSMK mbh + * All rights reserved. + * + * Redistribution and use in source and binary forms are freely + * permitted provided that the above copyright notice and this + * paragraph and the following disclaimer are duplicated in all + * such forms. + * + * This software is provided "AS IS" and without any express or + * implied warranties, including, without limitation, the implied + * warranties of merchantability and fitness for a particular + * purpose. + */ + +/* + * get key and data path from nvram (or fallback) then + * decrypt, checksum and process the data (set environment, + * check and load kernel+modules, load disk keys), boot. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include + +#include "bootstrap.h" +#include "rijndael-api-fst.h" +#include "sha2.h" + +#include "tpm.h" + +#ifdef TLOADER_VERBOSE +#define v_printf printf +#else +#define v_printf(...) while(0) +#endif + +#define FALLBACK_KEY "/boot/magic/key" +#define FALLBACK_DATA "/boot/magic/data" +#define FALLBACK_FORCE_DATA "/boot/magic/data_force" + +/* Error Codes */ +enum { + E_TPM_STATUS = -7, + E_TPM_NV_READ = -8, + E_OPEN = -9, + E_READ = -10, + E_STAT = -11, + E_NO_KEY = -12, + E_NO_DATA = -13, + E_MALLOC = -14, + E_DIGEST_MISMATCH = -15, + E_PARSE = -16, + E_UNSETENV = -17, + E_SETENV = -18, + E_SIZE_MISMATCH = -19, + E_FILE_LOAD = -20, + E_NO_KERNEL = -21, +}; + +extern vm_offset_t loadaddr; +extern void file_insert_tail(struct preloaded_file *mp); +extern int file_load(char *, vm_offset_t, struct preloaded_file **); + +int tloader(void); + +/* data tags */ +enum { + T_env = 8, + T_env_Key = 9, + T_env_Val = 10, + T_key = 16, + T_key_Prov = 17, + T_key_Key = 18, + T_key_Name = 19, + T_load = 32, + T_load_Path = 33, + T_load_Size = 34, + T_load_Args = 35, + T_load_Hash = 36, +}; + +enum { + C_LEN = 4, /* four bytes (Big Endian) length(s) */ + C_TAG = 1, /* one byte TAG */ +}; + +/* cipher/hash constants */ +enum { + AES_KEY_SIZE = 32, + AES_KEY_BITS = (AES_KEY_SIZE * 8), + AES_BLOCK_SIZE = 16, + AES_BLOCK_BITS = (AES_BLOCK_SIZE * 8), + SHA512_DIGEST_SIZE = 64, +}; + +typedef struct { + uint8_t tag; + uint32_t len; + const uint8_t *val; +} tlv_t; + +typedef struct saved_env { + char *name; + char *value; + struct saved_env *next; +} saved_env_t; + +typedef struct unset_env { + char *name; + struct unset_env *next; +} unset_env_t; + +static inline uint32_t +get_32be(const uint8_t *p) +{ + return (uint32_t)p[0] << 24 | + (uint32_t)p[1] << 16 | + (uint32_t)p[2] << 8 | + (uint32_t)p[3]; +} + +static int +get_key_tpm(uint8_t *key) +{ + uint32_t r; + + if (0 != tpm_status()) { + v_printf("key from nv ram failed: tpm_status\n"); + return E_TPM_STATUS; + } + if (0 != (r = key_from_tpm_nvram(key))) { + v_printf("key from nv ram failed: 0x%08x\n", r); + return E_TPM_NV_READ; + } + v_printf("key from nv ram.\n"); + return 0; +} + +static int +get_key(uint8_t *key) +{ + int fd; + struct stat sb; + + if (0 == get_key_tpm(key)) + return 0; + if (-1 == (fd = open(FALLBACK_KEY, O_RDONLY))) { + v_printf("key from fallback failed: open\n"); + return E_OPEN; + } + if (AES_KEY_SIZE != read(fd, key, AES_KEY_SIZE)) { + close(fd); + v_printf("key from fallback failed: read\n"); + return E_READ; + } + close(fd); + v_printf("key from fallback.\n"); + return 0; +} + +static int +get_path_tpm(char **path) +{ + uint32_t r; + struct stat sb; + + if (0 != tpm_status()) { + v_printf("path from nv ram failed: tpm_status\n"); + return E_TPM_STATUS; + } + if (0 != (r = path_from_tpm_nvram((uint8_t **)path))) { + v_printf("path from nv ram failed: 0x%08x\n", r); + return E_TPM_NV_READ; + } + if (0 != stat(*path, &sb)) { + v_printf("path from nv ram failed: stat\n"); + return E_STAT; + } + v_printf("path from nv ram: %s\n", *path); + return 0; +} + +static int +get_path(char **path) +{ + struct stat sb; + + if (0 != stat(FALLBACK_FORCE_DATA, &sb)) + if (0 == get_path_tpm(path)) + return 0; + if (0 != stat(FALLBACK_DATA, &sb)) { + v_printf("path from fallback failed: stat\n"); + return E_STAT; + } + *path = strdup(FALLBACK_DATA); + v_printf("path from fallback: %s\n", *path); + return 0; +} + +static int +decrypt_data(uint8_t **buf, size_t *buf_size, const char *path, const uint8_t *key) +{ + int fd, cnt, r; + uint8_t ctbuf[AES_BLOCK_SIZE], ptbuf[AES_BLOCK_SIZE]; + uint8_t sha_result[SHA512_DIGEST_SIZE], checksum[SHA512_DIGEST_SIZE]; + uint8_t *p; + SHA512_CTX sha; + keyInstance ki; + cipherInstance ci; + + if (-1 == (fd = open(path, O_RDONLY))) + return E_OPEN; + + rijndael_cipherInit(&ci, MODE_CBC, NULL); + rijndael_makeKey(&ki, DIR_DECRYPT, AES_KEY_BITS, (char *)key); + + /* read and decrypt checksum */ + cnt = SHA512_DIGEST_SIZE / AES_BLOCK_SIZE; + p = checksum; + while (cnt) { + if (AES_BLOCK_SIZE != read(fd, ctbuf, AES_BLOCK_SIZE)) { + r = E_READ; + goto fail; + } + r = rijndael_blockDecrypt(&ci, &ki, ctbuf, AES_BLOCK_BITS, p); + if (0 > r) + goto fail; + rijndael_cipherInit(&ci, MODE_CBC, (char *)ctbuf); + p += AES_BLOCK_SIZE; + cnt--; + } + SHA512_Init(&sha); + + /* read and decrypt buf_size, checksum the block */ + if (AES_BLOCK_SIZE != read(fd, ctbuf, AES_BLOCK_SIZE)) { + r = E_READ; + goto fail; + } + rijndael_blockDecrypt(&ci, &ki, ctbuf, AES_BLOCK_BITS, ptbuf); + rijndael_cipherInit(&ci, MODE_CBC, (char *)ctbuf); + *buf_size = get_32be(ptbuf); + SHA512_Update(&sha, ptbuf, AES_BLOCK_SIZE); + + /* alloc buf */ + if (NULL == (*buf = (uint8_t *)malloc(*buf_size))) { + r = E_MALLOC; + goto fail; + } + cnt = *buf_size; + p = *buf; + + /* read and decrypt the rest int buf */ + while (cnt >= AES_BLOCK_SIZE) { + if (AES_BLOCK_SIZE != read(fd, ctbuf, AES_BLOCK_SIZE)) { + r = E_READ; + goto fail; + } + rijndael_blockDecrypt(&ci, &ki, ctbuf, AES_BLOCK_BITS, p); + rijndael_cipherInit(&ci, MODE_CBC, (char *)ctbuf); + p += AES_BLOCK_SIZE; + cnt -= AES_BLOCK_SIZE; + } + if (cnt) { + if (AES_BLOCK_SIZE != read(fd, ctbuf, AES_BLOCK_SIZE)) { + r = E_READ; + goto fail; + } + rijndael_blockDecrypt(&ci, &ki, ctbuf, AES_BLOCK_BITS, ptbuf); + memcpy(p, ptbuf, cnt); + } + close(fd); + bzero(&ki, sizeof(ki)); + bzero(&ci, sizeof(ci)); + + /* check sha512 digest */ + SHA512_Update(&sha, *buf, *buf_size); + SHA512_Final(sha_result, &sha); + if (memcmp(sha_result, checksum, SHA512_DIGEST_SIZE)) { + free(*buf); + *buf = NULL; + return E_DIGEST_MISMATCH; + } + return 0; +fail: + close(fd); + bzero(&ci, sizeof(ci)); + bzero(&ki, sizeof(ki)); + if (NULL != *buf) { + bzero(buf, *buf_size); + free(*buf); + *buf = NULL; + } + return r; +} + +static int +find_tlv(tlv_t *res, const uint8_t *buf, size_t buf_size) +{ + uint8_t tag; + uint32_t len; + + while (buf_size >= (C_TAG + C_LEN)) { + tag = *buf++; + buf_size--; + len = get_32be(buf); + buf += C_LEN; + buf_size -= C_LEN; + if (buf_size < len) + break; + if (tag == res->tag) { + res->len = len; + res->val = buf; + return 0; + } + buf += len; + buf_size -= len; + } + return 1; +} + +static int +process_env(tlv_t *x) +{ + tlv_t name; + tlv_t value; + + name.tag = T_env_Key; + value.tag = T_env_Val; + if (0 != find_tlv(&name, x->val, x->len)) + return E_PARSE; + if (0 != find_tlv(&value, x->val, x->len)) + return E_PARSE; + if (0 == value.len) { + if (0 != unsetenv((const char *)name.val)) + return E_UNSETENV; + } else { + if (0 != setenv((const char *)name.val, + (const char *)value.val, 1)) + return E_SETENV; + } + return 0; +} + +static int +hash_and_size(const char *path, const uint8_t *checksum, uint32_t fsize) +{ + SHA512_CTX sha; + uint8_t buf[4096]; + uint8_t sha_result[SHA512_DIGEST_SIZE]; + uint32_t cnt = 0; + int fd; + + if (-1 == (fd = open(path, O_RDONLY))) { + v_printf("size and digest (\"%s\") failed: open\n", path); + return E_OPEN; + } + SHA512_Init(&sha); + + for (;;) { + ssize_t res; + + if (0 > (res = read(fd, buf, sizeof(buf)))) { + v_printf("size and digest (\"%s\") failed: read\n", path); + return E_READ; + } else if (0 == res) { + close(fd); + if (cnt == fsize) + break; + else { + v_printf("size and digest (\"%s\") failed: size\n", path); + return E_SIZE_MISMATCH; + } + } else { + SHA512_Update(&sha, buf, res); + cnt += res; + } + } + + SHA512_Final(sha_result, &sha); + if (memcmp(sha_result, checksum, SHA512_DIGEST_SIZE)) { + v_printf("size and digest (\"%s\") failed: digest\n", path); + return E_DIGEST_MISMATCH; + } + v_printf("size and digest ok: %s\n", path); + return 0; +} + +static void +unload(void) +{ + struct preloaded_file *fp; + + while (preloaded_files != NULL) { + fp = preloaded_files; + preloaded_files = preloaded_files->f_next; + bzero((void *)fp->f_addr, fp->f_size); + file_discard(fp); + } + loadaddr = 0; + unsetenv("kernelname"); +} + +static int +process_load(const tlv_t *x) +{ + tlv_t path, size, args, hash; + uint32_t fsize; + int r; + struct preloaded_file *fp; + + path.tag = T_load_Path; + size.tag = T_load_Size; + args.tag = T_load_Args; + hash.tag = T_load_Hash; + if (0 != find_tlv(&path, x->val, x->len)) + return E_PARSE; + if (0 != find_tlv(&size, x->val, x->len)) + return E_PARSE; + if (0 != find_tlv(&args, x->val, x->len)) + return E_PARSE; + if (0 != find_tlv(&hash, x->val, x->len)) + return E_PARSE; + + fsize = get_32be(size.val); + if (0 != (r = hash_and_size((const char *)path.val, hash.val, fsize))) + return r; + + if (0 != (r = file_load((char *)path.val, loadaddr, &fp))) { + if (fp) + file_discard(fp); + return E_FILE_LOAD; + } + if (0 != args.len) + fp->f_args = strdup((char *)args.val); + else + fp->f_args = NULL; + loadaddr = fp->f_addr + fp->f_size; + file_insert_tail(fp); + return 0; +} + +static int +process_disk_key(const tlv_t *x) +{ + tlv_t prov, name, key; + struct preloaded_file *fp; + vm_offset_t laddr; + + prov.tag = T_key_Prov; + key.tag = T_key_Key; + name.tag = T_key_Name; + if (0 != find_tlv(&prov, x->val, x->len)) + return E_PARSE; + if (0 != find_tlv(&key, x->val, x->len)) + return E_PARSE; + if (0 != find_tlv(&name, x->val, x->len)) + return E_PARSE; + + if (0 == loadaddr) + return E_NO_KERNEL; + + laddr = loadaddr; + laddr = archsw.arch_copyin((void *)key.val, laddr, key.len); + fp = file_alloc(); + fp->f_name = strdup((char *)name.val); + fp->f_type = strdup((char *)prov.val); + fp->f_args = NULL; + fp->f_metadata = NULL; + fp->f_loader = -1; + fp->f_addr = loadaddr; + fp->f_size = key.len; + loadaddr = laddr; + file_insert_tail(fp); + + v_printf("disk key loaded: %s\n", prov.val); + return 0; +} + +static int +process_data(const uint8_t *buf, size_t buf_size) +{ + size_t pz = buf_size; + tlv_t x; + int r; + + while (pz >= (C_TAG + C_LEN)) { + x.tag = *buf++; + pz--; + x.len = get_32be(buf); + buf += C_LEN; + pz -= C_LEN; + if (pz < x.len) + return E_PARSE; + x.val = buf; + switch (x.tag) { + case T_env: + if (0 != (r = process_env(&x))) + return r; + break; + case T_load: + if (0 != (r = process_load(&x))) + return r; + break; + case T_key: + if (0 != (r = process_disk_key(&x))) + return r; + break; + default: + return E_PARSE; + } + buf += x.len; + pz -= x.len; + } + if (0 == loadaddr) + return E_NO_KERNEL; + return 0; +} + +static saved_env_t * +save_env(void) +{ + saved_env_t *start = NULL, *p1 = NULL, *p; + + struct env_var *ev = environ; + + for (; NULL != ev; ev = ev->ev_next) { + p = malloc(sizeof(saved_env_t)), + p->name = strdup(ev->ev_name); + if (NULL != ev->ev_value) + p->value = strdup(ev->ev_value); + else + p->value = NULL; + p->next = NULL; + if (NULL == p1) + start = p; + else + p1->next = p; + p1 = p; + } + return start; +} + +static void +free_saved_env(saved_env_t *p) +{ + saved_env_t *p1; + + while (NULL != p) { + p1 = p; + p = p->next; + if (NULL != p1->value) + free(p1->value); + if (NULL != p1->name) + free(p1->name); + free(p1); + } +} + +static void +restore_env(saved_env_t *saved) +{ + saved_env_t *p; + unset_env_t *unset = NULL, *up1 = NULL, *up; + struct env_var *ev = environ; + + for (; NULL != ev; ev = ev->ev_next) { + for (p = saved; NULL != p; p = p->next) { + if (!strcmp(ev->ev_name, p->name)) { + if (strcmp(ev->ev_value, p->value)) + setenv(p->name, p->value, 1); + break; + } + } + if (NULL == p) { + up = malloc(sizeof(unset_env_t)); + up->name = strdup(ev->ev_name); + up->next = NULL; + if (NULL == unset) + unset = up; + else + up1->next = up; + up1 = up; + } + } + while (NULL != unset) { + up = unset; + unset = unset->next; + unsetenv(up->name); + free(up->name); + free(up); + } +} + +int +tloader(void) +{ + int r; + size_t buf_size = 0; + uint8_t *buf = NULL; + uint8_t key[AES_KEY_SIZE]; + char *path; + saved_env_t *env = save_env(); + + if (0 != get_key(key)) { + v_printf("no key.\n"); + return E_NO_KEY; + } + if (0 != get_path(&path)) { + v_printf("no data path.\n"); + return E_NO_DATA; + } + r = decrypt_data(&buf, &buf_size, path, key); + bzero(key, AES_KEY_SIZE); + free(path); + + if (0 != r) { + v_printf("decrypt_data failed (%d)\n", r); + return r; + } + if (0 != (r = process_data(buf, buf_size))) { + restore_env(env); + bzero(buf, buf_size); + free(buf); + unload(); + v_printf("process_data failed (%d).\n", r); + return r; + } + free_saved_env(env); + bzero(buf, buf_size); + free(buf); + + if (0 == tpm_status()) { + uint32_t r = key_read_stclear(); + v_printf("key_read_stclear: (%d)\n", r); + } + + /* does not return: */ + file_formats[preloaded_files->f_loader]->l_exec(preloaded_files); + return 0; +} diff --git sys/boot/i386/tgptboot/tpm.c sys/boot/i386/tgptboot/tpm.c new file mode 100644 index 0000000..0bfc535 --- /dev/null +++ sys/boot/i386/tgptboot/tpm.c @@ -0,0 +1,426 @@ +/*- + * Copyright (c) 2014, GSMK mbh + * All rights reserved. + * + * Redistribution and use in source and binary forms are freely + * permitted provided that the above copyright notice and this + * paragraph and the following disclaimer are duplicated in all + * such forms. + * + * This software is provided "AS IS" and without any express or + * implied warranties, including, without limitation, the implied + * warranties of merchantability and fitness for a particular + * purpose. + */ + +/* + * TPM support for tgptboot: + * + * -> check TPM status + * -> get tgptboot data key from NV area + * -> get tgptboot data path from NV area + * + * Implements INT 1A TCG Functions TCG_StatusCheck ([4] 13.7) and + * TCG_PassThroughToTPMOutput ([4] 13.9). The pass-through function + * provides access for TPM_NV_ReadValue ([3] 20.4) and + * TPM_GetCapability ([3] 7.1). TPM_GetCapability is used to get the + * dataSize of an NV area and TPM_NV_ReadValue to get its content. + * + * [2] TPM Main + * Part 2 TPM Structures + * Specification version 1.2 + * Level 2 Revision 161 + * March 2011 + * [3] TPM Main + * Part 3 Commands + * Specification Version 1.2 + * Level 2 Revision 116 + * 1 March 2011 + * [4] TCG PC Client Specific + * Implementaion Specification for Conventional BIOS + * Specification Version 1.21 Errata + * Revision 1.00 + * February 24th, 2012 + * For TPM Family 1.2; Level 2 + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +#include "tpm.h" + +/* [4] 13.3 Return Codes */ +#define TCG_PC_OK 0x0000 + +/* [4] 13.Application Level Interface - INT 1A TCG Functions */ +#define TCG_TCPA 0x41504354L + +/* [2] 6. TPM_TAG (Command and Response Tags) */ +#define TPM_TAG_RSP_COMMAND 0x00c4 +#define TPM_TAG_RQU_COMMAND 0x00c1 + +/* [2] 3.1 TPM_STRUCTURE_TAG */ +#define TPM_TAG_NV_DATA_PUBLIC 0x0018 + +/* [2] 17. Ordinals */ +#define TPM_ORD_NV_ReadValue 0x000000cfL +#define TPM_ORD_GetCapability 0x00000065L + +/* [2] 21.1 TPM_CAPABILITY_AREA for TPM_GetCapability */ +#define TPM_CAP_NV_LIST 0x0000000dL +#define TPM_CAP_NV_INDEX 0x00000011L + +/* [2] 16. Return Codes */ +#define TPM_SUCCESS 0x00000000L + + +/* TCG_PassThroughToTPM input and output buffer, + * accessed in real mode by btx v86, must be located + * below 1MB + */ +uint8_t TCG_IPB[512]; +uint8_t TCG_OPB[1024]; + +/* [4] 13.9.1 TCG_PassThroughToTPMInput Parameter Block */ +enum { + IPB_IPBLength = 0, + IPB_RESERVED1 = 2, + IPB_OPBLength = 4, + IPB_RESERVED2 = 6, + IPB_TPMOperandIn = 8 +}; + +/* [4] 13.9.2 TCG_PassThroughToTPMOutput Parameter Block */ +enum { + OPB_OPBLength = 0, + OPB_RESERVED = 2, + OPB_TPMOperandOut = 4, +}; + +/* Outgoing Parameters + * [3] 7.1 TPM_GetCapability + * 20.4 TPM_NV_ReadValue + */ +enum { + OpOut_tag = 0, + OpOut_paramSize = 2, + OpOut_returnCode = 6, + OpOut_respSize = 10, + OpOut_resp = 14, +}; + +const size_t MAX_respSize = sizeof(TCG_OPB) - (OpOut_resp + OPB_TPMOperandOut); + +static inline void +ld32BE(const uint8_t *p, uint32_t *x) +{ + *x = (p[0] << 24) | (p[1] << 16) | (p[2] << 8) | p[3]; +} + +static inline void +st32BE(uint8_t *p, uint32_t x) +{ + p[0] = x >> 24; + p[1] = (x >> 16) & 0xff; + p[2] = (x >> 8) & 0xff; + p[3] = x & 0xff; +} + +static inline void +ld16BE(const uint8_t *p, uint16_t *x) +{ + *x = (p[0] << 8) | p[1]; +} + +static inline void +st16BE(uint8_t *p, uint16_t x) +{ + p[0] = (x >> 8) & 0xff; + p[1] = x & 0xff; +} + +static inline void +st16LE(uint8_t *p, uint16_t x) +{ + p[0] = x & 0xff; + p[1] = (x >> 8) & 0xff; +} + +/* [4] 13.9 TCG_PassThroughToTPM */ +static uint32_t +TCG_PassThroughToTPM(uint8_t **TPMOperandOut, const uint8_t *TPMOperandIn) +{ + uint32_t paramSize = 0, IPBLength, ret; + + ld32BE(TPMOperandIn + OpOut_paramSize, ¶mSize); + IPBLength = paramSize + IPB_TPMOperandIn; + if (IPBLength > sizeof(TCG_IPB)) + return 0xffff0001; + + st16LE(TCG_IPB + IPB_IPBLength, IPBLength); + st16LE(TCG_IPB + IPB_RESERVED1, 0); + st16LE(TCG_IPB + IPB_OPBLength, sizeof(TCG_OPB)); + st16LE(TCG_IPB + IPB_RESERVED2, 0); + memcpy(TCG_IPB + IPB_TPMOperandIn, TPMOperandIn, paramSize); + + v86.ctl = 0; + v86.addr = 0x1a; + v86.eax = 0xbb02; + v86.ebx = TCG_TCPA; + v86.ecx = 0; + v86.edx = 0; + v86.es = VTOPSEG(TCG_IPB); + v86.edi = VTOPOFF(TCG_IPB); + v86.ds = VTOPSEG(TCG_OPB); + v86.esi = VTOPOFF(TCG_OPB); + v86int(); + + if (TCG_PC_OK != (ret = v86.eax)) + goto fail; + + ld32BE(TCG_OPB + OPB_TPMOperandOut + OpOut_paramSize, ¶mSize); + if (paramSize > (sizeof(TCG_OPB) - OPB_TPMOperandOut)) { + ret = 0xffff0002; + goto fail; + } + if (NULL == (*TPMOperandOut = (uint8_t *)malloc(paramSize))) { + ret = 0xffff0003; + goto fail; + } + memcpy(*TPMOperandOut, TCG_OPB + OPB_TPMOperandOut, paramSize); + +fail: + bzero(TCG_IPB, IPBLength); + bzero(TCG_OPB, sizeof(TCG_OPB)); + return ret; +} + +static uint32_t +TPM_NV_ReadValue_1(uint8_t *dst, uint32_t nvIndex, uint32_t offset, + uint32_t size) +{ + uint16_t tag; + uint32_t ret, returnCode, respSize, paramSize; + uint8_t OpIn[22]; + uint8_t *OpOut; + + /* [3] 20.4 Incoming Operands and Sizes */ + enum { + OpIn_tag = 0, + OpIn_paramSize = 2, + OpIn_ordinal = 6, + OpIn_nvIndex = 10, + OpIn_offset = 14, + OpIn_dataSize = 18, + }; + + st16BE(OpIn + OpIn_tag, TPM_TAG_RQU_COMMAND); + st32BE(OpIn + OpIn_paramSize, sizeof(OpIn)); + st32BE(OpIn + OpIn_ordinal, TPM_ORD_NV_ReadValue); + st32BE(OpIn + OpIn_nvIndex, nvIndex); + st32BE(OpIn + OpIn_offset, offset); + st32BE(OpIn + OpIn_dataSize, size); + + if (0 != (ret = TCG_PassThroughToTPM(&OpOut, OpIn))) + return ret; + + ld32BE(OpOut + OpOut_paramSize, ¶mSize); + ld32BE(OpOut + OpOut_returnCode, &returnCode); + if (0 != (ret = returnCode)) + goto fail; + + ld16BE(OpOut + OpOut_tag, &tag); + ld32BE(OpOut + OpOut_respSize, &respSize); + if (TPM_TAG_RSP_COMMAND != tag || + size != respSize) { + ret = 0xffff0004; + goto fail; + } + if (size) + memcpy(dst, OpOut + OpOut_resp, size); + +fail: + bzero(OpOut, paramSize); + free(OpOut); + return ret; +} + +/* [3] 20.4 TPM_NV_ReadValue */ +static uint32_t +TPM_NV_ReadValue(uint8_t *dst, uint32_t nvIndex, uint32_t offset, uint32_t size) +{ + uint32_t ret; + + while (MAX_respSize <= size) { + if (0 != (ret = TPM_NV_ReadValue_1(dst, nvIndex, offset, MAX_respSize))) + return ret; + size -= MAX_respSize; + offset += MAX_respSize; + dst += MAX_respSize; + } + return TPM_NV_ReadValue_1(dst, nvIndex, offset, size); +} + +/* [4] 13.7 TCG_StatusCheck */ +static uint32_t +TCG_StatusCheck(uint8_t *major, uint8_t *minor, uint8_t **event_log) +{ + v86.ctl = 0; + v86.addr = 0x1a; + v86.eax = 0xbb00; + v86int(); + + if (v86.eax) + return (v86.eax); + + if (v86.ebx != TCG_TCPA) + return 0xffff0005; + + if (NULL != major && NULL != minor) { + *major = (uint8_t)((v86.ecx & 0xff00) >> 8); + *minor = (uint8_t)(v86.ecx & 0xff); + } + if (NULL != event_log) + *event_log = (uint8_t *)PTOV(v86.esi); + + return 0; +} + +/* [3] 7.1 TPM_GetCapability */ +static uint32_t +TPM_GetCapability(uint32_t *respSize, uint8_t **resp, uint32_t capArea, + uint32_t subCapSize, uint8_t *subCap) +{ + uint16_t tag; + uint32_t ret, returnCode, paramSize; + uint8_t *OpIn; + uint8_t *OpOut; + + /* [3] 7.1 Incoming Parameters and Sizes */ + enum { + OpIn_tag = 0, + OpIn_paramSize = 2, + OpIn_ordinal = 6, + OpIn_capArea = 10, + OpIn_subCapSize = 14, + OpIn_subCap = 18, + }; + + paramSize = subCapSize + OpIn_subCap; + if (NULL == (OpIn = (uint8_t *)malloc(paramSize))) + return 0xffff0006; + + st16BE(OpIn + OpIn_tag, TPM_TAG_RQU_COMMAND); + st32BE(OpIn + OpIn_paramSize, paramSize); + st32BE(OpIn + OpIn_ordinal, TPM_ORD_GetCapability); + st32BE(OpIn + OpIn_capArea, capArea); + st32BE(OpIn + OpIn_subCapSize, subCapSize); + if (0 != subCapSize) + memcpy(OpIn + OpIn_subCap, subCap, subCapSize); + + ret = TCG_PassThroughToTPM(&OpOut, OpIn); + + free(OpIn); + + if (TCG_PC_OK != ret) + return ret; + + ld32BE(OpOut + OpOut_returnCode, &returnCode); + if (0 != (ret = returnCode)) + goto fail; + + ld32BE(OpOut + OpOut_paramSize, ¶mSize); + if (paramSize < OpOut_respSize) { + ret = 0xffff0007; + goto fail; + } + ld32BE(OpOut + OpOut_respSize, respSize); + ld16BE(OpOut + OpOut_tag, &tag); + if (TPM_TAG_RSP_COMMAND != tag || + paramSize != *respSize + OpOut_resp) { + ret = 0xffff0008; + goto fail; + } + memmove(OpOut, OpOut + OpOut_resp, *respSize); + *resp = OpOut; + return ret; + +fail: + free(OpOut); + return ret; +} + +static uint32_t +NV_dataSize(uint32_t nvIndex, uint32_t *dataSize) +{ + uint8_t subCap[4]; + uint8_t *resp; + uint32_t ret, respSize; + + st32BE(subCap, nvIndex); + ret = TPM_GetCapability(&respSize, &resp, TPM_CAP_NV_INDEX, + sizeof(subCap), subCap); + if (0 != ret) + return ret; + + /* + * [2] 19.3 TPM_NV_DATA_PUBLIC: we are only interested in the + * 'dataSize' element (last four bytes of the response) + */ + ld32BE(resp + (respSize - 4), dataSize); + + free(resp); + return ret; +} + +uint32_t +tpm_status() +{ + uint32_t ret; + uint8_t major, minor; + + if (0 != (ret = TCG_StatusCheck(&major, &minor, NULL))) + return ret; + + if (major != 1 || minor != 2) + return 0xffff0009; + + return 0; +} + +uint32_t +key_read_stclear() +{ + uint8_t *dummy = NULL; + return TPM_NV_ReadValue(dummy, NV_KEY_INDEX, NV_KEY_OFFSET, 0); +} + +uint32_t +key_from_tpm_nvram(uint8_t *key) +{ + return TPM_NV_ReadValue(key, NV_KEY_INDEX, NV_KEY_OFFSET, NV_KEY_SIZE); +} + +uint32_t +path_from_tpm_nvram(uint8_t **path) +{ + uint32_t ret; + uint32_t dataSize; + + if (0 != (ret = NV_dataSize(NV_PATH_INDEX, &dataSize))) + return ret; + + if (NULL == (*path = (uint8_t *)malloc(dataSize + 1))) + return 0xffff000a; + + *path[dataSize] = 0; + if (0 != (ret = TPM_NV_ReadValue(*path, NV_PATH_INDEX, 0, dataSize))) + free(*path); + else + *path[dataSize] = 0; + + return ret; +} diff --git sys/boot/i386/tgptboot/tpm.h sys/boot/i386/tgptboot/tpm.h new file mode 100644 index 0000000..36fc9c8 --- /dev/null +++ sys/boot/i386/tgptboot/tpm.h @@ -0,0 +1,34 @@ +#ifndef _TPM_H_ +#define _TPM_H_ + + +/* see TPM MainPart 2 TPM Structures Specification version 1.2 + * 19.1 TPM_NV_INDEX + */ +#ifndef NV_KEY_INDEX +#define NV_KEY_INDEX 0x20002323L +#endif + +#ifndef NV_PATH_INDEX +#define NV_PATH_INDEX 0x20004242L +#endif + +#ifndef NV_KEY_OFFSET +#define NV_KEY_OFFSET 0x00000000L +#endif + +#ifndef NV_KEY_SIZE +#define NV_KEY_SIZE 0x00000020 +#endif + +uint32_t key_from_tpm_nvram(uint8_t *key); +uint32_t key_read_stclear(void); + +/* success: return 0, *path points to null terminated data + * failure: return != 0 + */ +uint32_t path_from_tpm_nvram(uint8_t **path); + +uint32_t tpm_status(void); + +#endif diff --git sys/boot/i386/tpmbr/Makefile sys/boot/i386/tpmbr/Makefile new file mode 100644 index 0000000..c9754ea --- /dev/null +++ sys/boot/i386/tpmbr/Makefile @@ -0,0 +1,16 @@ +# $FreeBSD$ + +PROG= tpmbr +STRIP= +BINMODE=${NOBINMODE} +NO_MAN= +SRCS= ${PROG}.s + +ORG= 0x600 + +TPMBR_PCR_INDEX?= 0x09 + +AFLAGS+=--defsym PCR_INDEX=${TPMBR_PCR_INDEX} +LDFLAGS=-e start -Ttext ${ORG} -Wl,-N,-S,--oformat,binary + +.include diff --git sys/boot/i386/tpmbr/tpmbr.s sys/boot/i386/tpmbr/tpmbr.s new file mode 100644 index 0000000..bc62a71 --- /dev/null +++ sys/boot/i386/tpmbr/tpmbr.s @@ -0,0 +1,297 @@ +#- +# Copyright (c) 2007 Yahoo!, Inc. +# All rights reserved. +# Written by: John Baldwin +# TCG Support added by: Stefan Grundmann (GSMK mbh) +# +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# 3. Neither the name of the author nor the names of any co-contributors +# may be used to endorse or promote products derived from this software +# without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# +# $FreeBSD$ +# +# Partly from: src/sys/boot/i386/mbr/mbr.s 1.7 + +# A 512 byte PMBR boot manager that looks for a FreeBSD boot GPT partition +# and boots it. + + .set LOAD,0x7c00 # Load address + .set EXEC,0x600 # Execution address + .set MAGIC,0xaa55 # Magic: bootable + .set SECSIZE,0x200 # Size of a single disk sector + .set DISKSIG,440 # Disk signature offset + .set STACK,EXEC+SECSIZE*4 # Stack address + .set GPT_ADDR,STACK # GPT header address + .set GPT_SIG,0 + .set GPT_SIG_0,0x20494645 # "EFI " + .set GPT_SIG_1,0x54524150 # "PART" + .set GPT_MYLBA,24 + .set GPT_PART_LBA,72 + .set GPT_NPART,80 + .set GPT_PART_SIZE,84 + .set PART_ADDR,GPT_ADDR+SECSIZE # GPT partition array address + .set PART_TYPE,0 + .set PART_START_LBA,32 + .set PART_END_LBA,40 + .set DPBUF,PART_ADDR+SECSIZE + .set DPBUF_SEC,0x10 # Number of sectors + + .set NHRDRV,0x475 # Number of hard drives + + # TCG related constants: +# .set PCR_INDEX,0x09 # The PCR number to which the hashed boot_code is to be extended +# # set via --defsym + .set EVENT_INFO,0x00002342 # The information value to be placed into the event field + + .globl start # Entry point + .code16 + +# +# Setup the segment registers for flat addressing and setup the stack. +# +start: cld # String ops inc + xorw %ax,%ax # Zero + movw %ax,%es # Address + movw %ax,%ds # data + movw %ax,%ss # Set up + movw $STACK,%sp # stack +# +# Relocate ourself to a lower address so that we have more room to load +# other sectors. +# + movw $main-EXEC+LOAD,%si # Source + movw $main,%di # Destination + movw $SECSIZE-(main-start),%cx # Byte count + rep # Relocate + movsb # code +# +# Jump to the relocated code. +# + jmp main-LOAD+EXEC # To relocated code +# +# Validate drive number in %dl. +# +main: cmpb $0x80,%dl # Drive valid? + jb main.1 # No + movb NHRDRV,%dh # Calculate the highest + addb $0x80,%dh # drive number available + cmpb %dh,%dl # Within range? + jb main.2 # Yes +main.1: movb $0x80,%dl # Assume drive 0x80 +# +# Load the GPT header and verify signature. Try LBA 1 for the primary one and +# the last LBA for the backup if it is broken. +# +main.2: call getdrvparams # Read drive parameters + movb $1,%dh # %dh := 1 (reading primary) +main.2a: movw $GPT_ADDR,%bx + movw $lba,%si + call read # Read header and check GPT sig + cmpl $GPT_SIG_0,GPT_ADDR+GPT_SIG + jnz main.2b + cmpl $GPT_SIG_1,GPT_ADDR+GPT_SIG+4 + jnz main.2b + jmp load_part +main.2b: cmpb $1,%dh # Reading primary? + jne err_pt # If no - invalid table found +# +# Try alternative LBAs from the last sector for the GPT header. +# +main.3: movb $0,%dh # %dh := 0 (reading backup) + movw $DPBUF+DPBUF_SEC,%si # %si = last sector + 1 + movw $lba,%di # %di = $lba +main.3a: decl (%si) # 0x0(%si) = last sec (0-31) + movw $2,%cx + rep + movsw # $lastsec--, copy it to $lba + jmp main.2a # Read the next sector +# +# Load a partition table sector from disk and look for a FreeBSD boot +# partition. +# +load_part: movw $GPT_ADDR+GPT_PART_LBA,%si + movw $PART_ADDR,%bx + call read +scan: movw %bx,%si # Compare partition UUID + movw $boot_uuid,%di # with FreeBSD boot UUID + movb $0x10,%cl + repe cmpsb + jnz next_part # Didn't match, next partition +# +# We found a boot partition. Load it into RAM starting at 0x7c00. +# + movw %bx,%di # Save partition pointer in %di + leaw PART_START_LBA(%di),%si + movw $LOAD/16,%bx + movw %bx,%es + xorw %bx,%bx +# +#TCG_StatusCheck, set tpm to 0x01 if successful +# + pushal + movw $0xbb00,%ax + int $0x1a + test %eax,%eax + jnz tcg_status.0 + cmpl tpm_sig,%ebx # "TCPA" + jnz tcg_status.0 + cmp $0x0102,%cx # we need 1.2 + jnz tcg_status.0 + movb $0x01,tpm +tcg_status.0: popal +load_boot: push %si # Save %si + call read + pop %si # Restore +# +# TCG_CompactHashLogExtendEvent the sector at %es:%bx with EVENT_INFO into PCR_INDEX +# + cmpb $0x01,tpm # only if we have a working + jnz load_boot.1 # tpm + pushal + mov $0xe2e,%ax # + movw $0x7,%bx # put . + int $0x10 # + movw $0xbb07,%ax + mov %bx,%di + movl $EVENT_INFO,%esi + movl tpm_sig,%ebx + movl $SECSIZE,%ecx + movl $PCR_INDEX,%edx + int $0x1a + test %eax,%eax + jnz err_tpm + popal +load_boot.1: movl PART_END_LBA(%di),%eax # See if this was the last LBA + cmpl (%si),%eax + jnz next_boot + movl PART_END_LBA+4(%di),%eax + cmpl 4(%si),%eax + jnz next_boot + mov %bx,%es # Reset %es to zero + jmp LOAD # Jump to boot code +next_boot: incl (%si) # Next LBA + adcl $0,4(%si) + mov %es,%ax # Adjust segment for next + addw $SECSIZE/16,%ax # sector + cmp $0x9000,%ax # Don't load past 0x90000, + jae err_big # 545k should be enough for + mov %ax,%es # any boot code. :) + jmp load_boot +# +# Move to the next partition. If we walk off the end of the sector, load +# the next sector. We assume that partition entries are smaller than 64k +# and that they won't span a sector boundary. +# +next_part: decl GPT_ADDR+GPT_NPART # Was this the last partition? + jnz next_part.1 + int $0x18 # Fail, -> ROM BASIC via int 0x18 +next_part.1: movw GPT_ADDR+GPT_PART_SIZE,%ax + addw %ax,%bx # Next partition + cmpw $PART_ADDR+0x200,%bx # Still in sector? + jb scan + incl GPT_ADDR+GPT_PART_LBA # Next sector + adcl $0,GPT_ADDR+GPT_PART_LBA+4 + jmp load_part +# +# Load a sector (64-bit LBA at %si) from disk %dl into %es:%bx by creating +# a EDD packet on the stack and passing it to the BIOS. Trashes %ax and %si. +# +read: pushl 0x4(%si) # Set the LBA + pushl 0x0(%si) # address + pushw %es # Set the address of + pushw %bx # the transfer buffer + pushw $0x1 # Read 1 sector + pushw $0x10 # Packet length + movw %sp,%si # Packer pointer + movw $0x4200,%ax # BIOS: LBA Read from disk + int $0x13 # Call the BIOS + add $0x10,%sp # Restore stack + jc err_rd # If error + ret + +# +# Check the number of LBAs on the drive index %dx. Trashes %ax and %si. +# +getdrvparams: + movw $DPBUF,%si # Set the address of result buf + movw $0x001e,(%si) # len + movw $0x4800,%ax # BIOS: Read Drive Parameters + int $0x13 # Call the BIOS + jc err_rd # "I/O error" if error + ret +# +# Various error message entry points. +# +err_tpm: movw $msg_tpm,%si # "TPM error" + jmp putstr + +err_big: movw $msg_big,%si # "Boot loader too + jmp putstr # large" + +err_pt: movw $msg_pt,%si # "Invalid partition + jmp putstr # table" + +err_rd: movw $msg_rd,%si # "I/O error loading + jmp putstr # boot loader" + +# +# Output an ASCIZ string to the console via the BIOS. +# +putstr.0: movw $0x7,%bx # Page:attribute + movb $0xe,%ah # BIOS: Display + int $0x10 # character +putstr: lodsb # Get character + testb %al,%al # End of string? + jnz putstr.0 # No +putstr.1: jmp putstr.1 # Await reset + +msg_tpm: .asciz "!tpm" +msg_big: .asciz "!big" +msg_pt: .asciz "!PT" +msg_rd: .asciz "!I/O" + +lba: .quad 1 # LBA of GPT header + +tpm_sig: .long 0x41504354 + +tpm: .byte 0x00 + +boot_uuid: .long 0x83bd6b9d + .word 0x7f41 + .word 0x11dc + .byte 0xbe + .byte 0x0b + .byte 0x00 + .byte 0x15 + .byte 0x60 + .byte 0xb8 + .byte 0x4f + .byte 0x0f + + .org DISKSIG,0x90 +sig: .long 0 # OS Disk Signature + .word 0 # "Unknown" in PMBR + +partbl: .fill 0x10,0x4,0x0 # Partition table + .word MAGIC # Magic number --EeQfGwPcQSOJBaQU-- From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 10:47:44 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 35DC37FA for ; Wed, 25 Mar 2015 10:47:44 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id 56A47AAB for ; Wed, 25 Mar 2015 10:47:42 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygCHaRZGkhJVwErXAw--.38117S2; Wed, 25 Mar 2015 18:47:39 +0800 (CST) Date: Wed, 25 Mar 2015 18:47:07 +0800 From: Tiwei Bie To: Konstantin Belousov Subject: Re: [PATCH] Finish the task 'Validate coredump format string' Message-ID: <20150325104707.GA25729@freebsd> References: <20150322112555.GA44277@freebsd> <20150322113822.GB2379@kib.kiev.ua> <20150322120655.GA59757@freebsd> <20150322131524.GA95795@freebsd> <20150323005852.GB6798@dft-labs.eu> <20150323020314.GA30143@freebsd> <20150324123517.GA25678@dft-labs.eu> <20150324143709.GA54065@freebsd> <20150325060012.GA75674@freebsd> <20150325084130.GX2379@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150325084130.GX2379@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygCHaRZGkhJVwErXAw--.38117S2 X-Coremail-Antispam: 1UD129KBjvJXoW3Wr4fXw4UtF47XFyDXFW5ZFb_yoW7WF1UpF yak3s0yrs5Cr43Cr1fZ3yrA34Yywn5Ja15W347Zw1YkryFgry8XF1Fg34FvFykGryvqryD Xa15XFy7KryDZ3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUk0b7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Ar0_tr1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Cr0_Gr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I 8E87Iv6xkF7I0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI 64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r106r15McIj6I8E87Iv67AKxVWxJVW8Jr 1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0Y48IcVAKI48JMxkIecxEwVAFwVW8twCF04k20xvY 0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I 0E7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jrv_JF1lIxkGc2Ij64vIr41lIxAI cVC0I7IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Jr0_Gr1lIxAIcV CF04k26cxKx2IYs7xG6rW3Jr0E3s1lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280 aVCY1x0267AKxVWUJVW8JbIYCTnIWIevJa73UjIFyTuYvjxUq4rAUUUUU X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUTAVQhl-t+-gAFsK Cc: freebsd-hackers@freebsd.org, Mateusz Guzik X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 10:47:44 -0000 On Wed, Mar 25, 2015 at 10:41:30AM +0200, Konstantin Belousov wrote: > On Wed, Mar 25, 2015 at 02:00:12PM +0800, Tiwei Bie wrote: > > share/man/man5/core.5 | 7 ++++ > > sys/kern/kern_sig.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++---- > > 2 files changed, 93 insertions(+), 7 deletions(-) > > > > diff --git a/share/man/man5/core.5 b/share/man/man5/core.5 > > index 3f54f89..3047e0a 100644 > > --- a/share/man/man5/core.5 > > +++ b/share/man/man5/core.5 > Please bump date for the man page. > Yeah, I will! > > @@ -78,6 +78,7 @@ An index starting at zero until the sysctl > > + [..] > > +static int > > +corefilename_check(const char *format) > > +{ > > + int i, countI; > > + > > + countI = 0; > > + for (i = 0; i < MAXPATHLEN && format[i] != '\0'; i++) { > Was it already suggested to use sizeof(corefilename) instead of MAXPATHLEN ? > Sorry, I didn't notice this. > > + if (format[i] == '%') { > > + [..] > > + format = kern_getenv("kern.corefile"); > > + if (format == NULL) > > + return; > > + > > + error = corefilename_check(format); > > + > > + switch (error) { > > + case 0: > > + strcpy(corefilename, format); > Use strlcpy(9). > Got it, thanks! > > + break; > > + case EINVAL: > > + printf("Invalid format specified for corename `%s'\n", format); > > + break; > > + case ENAMETOOLONG: > > + printf("The format specified for corename is too long\n"); > > + break; > Sigh. Why kernel should decode the error codes ? Why should it print > anything on the console, with non-zero chance of sysctl caller not seeing > the text ? > I will remove these prints. My new patch: --- share/man/man5/core.5 | 9 +++++- sys/kern/kern_sig.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 78 insertions(+), 8 deletions(-) diff --git a/share/man/man5/core.5 b/share/man/man5/core.5 index 3f54f89..cfbcb1c 100644 --- a/share/man/man5/core.5 +++ b/share/man/man5/core.5 @@ -28,7 +28,7 @@ .\" @(#)core.5 8.3 (Berkeley) 12/11/93 .\" $FreeBSD$ .\" -.Dd March 8, 2015 +.Dd March 25, 2015 .Dt CORE 5 .Os .Sh NAME @@ -78,6 +78,7 @@ An index starting at zero until the sysctl is reached. This can be useful for limiting the number of corefiles generated by a particular process. +This specifier can only be specified at most once. .It Em \&%N process name. .It Em \&%P @@ -91,6 +92,12 @@ The name defaults to yielding the traditional .Fx behaviour. +When changing the name via the +.Va kern.corefile +sysctl, it will fail if the new name contains +unknown format specifiers, or +.Em \&%I +is specified more than once, or its length is too long. .Pp By default, a process that changes user or group credentials whether real or effective will not create a corefile. diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c index 154c250..611d138 100644 --- a/sys/kern/kern_sig.c +++ b/sys/kern/kern_sig.c @@ -3094,21 +3094,85 @@ static int compress_user_cores = 0; */ #define corefilename_lock allproc_lock -static char corefilename[MAXPATHLEN] = {"%N.core"}; +static char corefilename[MAXPATHLEN] = "%N.core"; + +static int +corefilename_check(const char *format) +{ + int i, countI; + + countI = 0; + for (i = 0; i < sizeof(corefilename) && format[i] != '\0'; i++) { + if (format[i] == '%') { + i++; + switch (format[i]) { + case 'I': + countI++; + if (countI > 1) + return (EINVAL); + case '%': + case 'H': + case 'N': + case 'P': + case 'U': + break; + default: + return (EINVAL); + } + } + } + + if (i == sizeof(corefilename)) + return (ENAMETOOLONG); + + return (0); +} + +static void +corefilename_init(void *arg) +{ + char *format; + + format = kern_getenv("kern.corefile"); + if (format == NULL) + return; + + if (corefilename_check(format) == 0) + strlcpy(corefilename, format, sizeof(corefilename)); + + freeenv(format); +} +SYSINIT(corefilename, SI_SUB_KMEM, SI_ORDER_FIRST, corefilename_init, 0); static int sysctl_kern_corefile(SYSCTL_HANDLER_ARGS) { + char *format; int error; + format = malloc(sizeof(corefilename), M_TEMP, M_WAITOK); + + sx_slock(&corefilename_lock); + strlcpy(format, corefilename, sizeof(corefilename)); + sx_sunlock(&corefilename_lock); + + error = sysctl_handle_string(oidp, format, sizeof(corefilename), req); + if (error != 0 || req->newptr == NULL) + goto out; + + error = corefilename_check(format); + if (error != 0) + goto out; + sx_xlock(&corefilename_lock); - error = sysctl_handle_string(oidp, corefilename, sizeof(corefilename), - req); + strlcpy(corefilename, format, sizeof(corefilename)); sx_xunlock(&corefilename_lock); +out: + free(format, M_TEMP); return (error); } -SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RWTUN | +SYSCTL_PROC(_kern, OID_AUTO, corefile, CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, 0, 0, sysctl_kern_corefile, "A", "Process corefile name format string"); @@ -3171,9 +3235,8 @@ corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, sbuf_printf(&sb, "%u", uid); break; default: - log(LOG_ERR, - "Unknown format character %c in " - "corename `%s'\n", format[i], format); + KASSERT(0, ("Unknown format character %c in " + "corename `%s'", format[i], format)); break; } break; -- 2.1.2 Best regards, Tiwei Bie From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 12:57:47 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 912C71DF for ; Wed, 25 Mar 2015 12:57:47 +0000 (UTC) Received: from mail-ig0-x231.google.com (mail-ig0-x231.google.com [IPv6:2607:f8b0:4001:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 53821D1C for ; Wed, 25 Mar 2015 12:57:47 +0000 (UTC) Received: by igbud6 with SMTP id ud6so100234724igb.1 for ; Wed, 25 Mar 2015 05:57:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=7T+LgGAbFQ3UmWy1bfEZrVmr5eYAyPMrGbudsCbwg9w=; b=ecSSK+itNtWDXNXht8Y2jOMTJcXm6kYTFqBs7zXtpGVkHg9Kl2Eff5lPOxK9g6QrKS RbIRDL3/YW2/0L1eFkJhLRf87xhlx6q5mp+ujC2QuZRiqrHf5k/7dlwkn1tJGa5QJdTr c8KwTtGdK91hnwEcbbBtgVv1lmNMW+12/nreUiangdGBGlroefIz6QGE782MWjde4Cuz 1xwZc2H0x2dR4F0Hwv+57FGH/a/2ThWN8Gu0GnWtpiMomka00K9PPXtxF+MSu0CtOONY kdJ3hXqF3I5YcsBMHC51CF+YZjP538LlqJUkDr2s1IfhBchfhFdlgZFYjNsiSbVhp1OR vVNw== MIME-Version: 1.0 X-Received: by 10.107.135.212 with SMTP id r81mr14048222ioi.38.1427288266845; Wed, 25 Mar 2015 05:57:46 -0700 (PDT) Received: by 10.64.73.7 with HTTP; Wed, 25 Mar 2015 05:57:46 -0700 (PDT) In-Reply-To: References: Date: Wed, 25 Mar 2015 18:27:46 +0530 Message-ID: Subject: Re: Port FreeBSD to a smartphone (any make and model) for GSoC15 From: Rakesh Sharma To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 12:57:47 -0000 Respected sir, I ve forwarded it to freebsd-hackers mailing list. Is their any other mailing list where i should send it ? please let me know. i am really interested to work on this project Thank You Rakesh Kumar Sharma On Wed, Mar 25, 2015 at 5:54 AM, Rakesh Sharma wrote: > > I am Rakesh Kumar Sharma, an engineering Student of NIT Karnataka. > > I Am Interested in working on porting Free BSD to a smartphone (any make > or model).sorry for late writing, i wasnt aware of GSoC, my friend recently > suggested me about it. I am not much into the field, but I believe i can > pull it off as I am hardworking and have good technical skills. > > one idea is to dual boot the free bsd like ubuntu touch, we need to > install the BSD to the memory of the device, and than loading the kernel > from inside the android kernel. this process should kill all the android > processes except the ones really needed. the BSD kernel will load the > required files and overtake the android kernel. I Checked BSD have its ARM > version support, so i think instruction porting wont be needed. > > The second idea is to entirely replace the original kernel and operating > system, in this case i need to do some research, but i believe i should be > able to modify the boot loader to boot the BSD RISC Kernel stored in my > memory, with problems of course, then i can modify the kernel to fix the > bugs. I idea is not to replace the entire operating system, as if something > goes wrong i can return to my old os, This can be changed later. > > Please provide me suggestions, and comments on my idea. i am really > looking forward to working on it, I have an old device which i can work on, > and a new intel x86 phone, which stand more chance of success. > > Thank you > > From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 14:38:04 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D8C40A80 for ; Wed, 25 Mar 2015 14:38:04 +0000 (UTC) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 782FDD6C for ; Wed, 25 Mar 2015 14:38:04 +0000 (UTC) Received: by wixw10 with SMTP id w10so41679771wix.0 for ; Wed, 25 Mar 2015 07:38:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=Sd7HSvFU1FQkMrDhpPwSCVmKLTDGSY1xxHty7EQ205g=; b=r1deeQTZuuqRwScCwFx176cAH3xKy/q7zeqpuLl3Ap29V20T8nbshpnSIL/4isNupU Qqala19TPgXBdZ3Rmhe/18WDUXOqRKrQI0+5LK/jZFg7mSRRXTuX79ftWCFdtweDwqrd V75so8bAfaJGe+dkf8DSk2roWOdkxr5ze3LUeKH7Yqta0sGOklytFkX7q7ILkBD7BAvE oxHbyKuWdEfI6BQQ67JB2eTuNmr+71fvupIc5b4Niu0ztP7dlY6uNtD8ixAmZR8k18Dx HceC7d5JmbYcf7tWdqJGhQYVVFtF3UjrVg57AjpqKEbVzlvMb/hmFONEPuVOj/vxJ06u sG/A== X-Received: by 10.180.198.162 with SMTP id jd2mr37741979wic.21.1427294264933; Wed, 25 Mar 2015 07:37:44 -0700 (PDT) Received: from azdaja.softwarehood.com ([95.180.18.169]) by mx.google.com with ESMTPSA id ka1sm4043920wjc.2.2015.03.25.07.37.43 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Mar 2015 07:37:43 -0700 (PDT) Message-ID: <5512C835.7040207@gmail.com> Date: Wed, 25 Mar 2015 15:37:41 +0100 From: Ivan Radovanovic User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Jilles Tjoelker Subject: Re: kevent behavior References: <550A6DA2.1070004@gmail.com> <20150324221541.GA67584@stack.nl> In-Reply-To: <20150324221541.GA67584@stack.nl> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 14:38:05 -0000 On 03/24/2015 23:15, Jilles Tjoelker wrote: > On Thu, Mar 19, 2015 at 07:33:06AM +0100, Ivan Radovanovic wrote: >> Is there defined (and guaranteed) behavior of kevent if kqueue FD is >> being closed while blocking kevent call is in progress? > >> Possible scenario would be like this: > >> Thread 1: >> ... >> kfd = kqueue(); >> ... >> // create second thread afterwords >> // and do blocking wait for events >> result = kevent(kfd, changelist, nchanges, eventlist, nevents, NULL); >> if (result == -1) >> // check if there was request to stop listening for events > >> Thread 2: >> // do something >> // then close kqueue's fd >> close(kfd); > >> I am asking this because file watcher implementation for mono is >> implemented that way (which I find nicer than using timeout), but this >> is apparently based on expected kevent behavior under Darwin, and I >> can't find any mention that in FreeBSD kevent is going to behave the >> same way (at least not on kqueue(2) manual page) > > This method is inherently unsafe, since you cannot be sure thread 1 has > started blocking in kevent() when you close() in thread 2. If not, there > might be a thread 3 creating a kqueue between thread 2's close and > thread 1's kevent, and thread 1 will manipulate the new kqueue. > > Fortunately, EVFILT_USER provides an easy way to wake up a thread > blocked in kevent(). > > If kevent() was implemented as a cancellation point, pthread_cancel() > would be another option, but it is not. > It is not really created in the same way as I put there, I just wanted to illustrate that we have 2 threads and that one is blocked in kevent and the other one is trying to close its kqueue descriptor. I saw this EVFILT_USER thing, and it looked to me like it could be used to do much smarter unblocking of kevent but I believe manual page is relatively unclear there - what I do not understand is: * how you trigger this user event (I would say with kevent if corresponding structure's NOTE_TRIGGER flag is set, but that doesn't sound logical since in my understanding kevent is basically like select(2) (ie call which checks if something happened without triggering things = call which reports about status change in system without causing change itself)? * how does this work process-wise - is one ident restricted to process which created corresponding kqueue, or it is system wise (other process could trigger my kevent if same ident value is used)? From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 22:35:34 2015 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D4D33B8E for ; Wed, 25 Mar 2015 22:35:34 +0000 (UTC) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 99084D42 for ; Wed, 25 Mar 2015 22:35:34 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 747313592EF; Wed, 25 Mar 2015 23:35:30 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 4BC1428494; Wed, 25 Mar 2015 23:35:30 +0100 (CET) Date: Wed, 25 Mar 2015 23:35:30 +0100 From: Jilles Tjoelker To: Konstantin Belousov Subject: Re: kevent behavior Message-ID: <20150325223530.GA79065@stack.nl> References: <550A6DA2.1070004@gmail.com> <20150324221541.GA67584@stack.nl> <20150325090041.GY2379@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150325090041.GY2379@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@FreeBSD.org, Ivan Radovanovic X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 22:35:34 -0000 On Wed, Mar 25, 2015 at 11:00:41AM +0200, Konstantin Belousov wrote: > On Tue, Mar 24, 2015 at 11:15:41PM +0100, Jilles Tjoelker wrote: > > Fortunately, EVFILT_USER provides an easy way to wake up a thread > > blocked in kevent(). > > If kevent() was implemented as a cancellation point, pthread_cancel() > > would be another option, but it is not. > Do you consider it is possible to make kqueue(2) cancellation point ? > Susv4 is very explicit about functions it allows to handle cancellation. > Might be, it is time for kqueue2(3), which would take flags. > In addition to allowing cancellation, it could take close-on-exec flag. > It is somewhat non-trivial to implement, at least I do not want to make > kevent2(). I take it you mean making kevent() a cancellation point, not kqueue(). The latter would just annoy application writers for no benefit. POSIX does not say anything about kevent(), including whether it should be a cancellation point or not. Given that kevent() may block for a long time, making it a cancellation point seems to make sense. The kevent() cancellation point implementation would be much like most other cancellation points such as pselect(), with the difference that the call may have committed even if it fails, in the case that nchanges > nevents and in the case that nchanges > 0 && errno == EINTR. If cancellation is requested while blocking in the latter case, libthr will have to return -1/EINTR to indicate to the application that the changes were applied successfully while allowing the thread to be cancelled at the next cancellation point, even though there may not be any signal handler. If nevents == 0, kevent() does not block and need not be a cancellation point. This special case may simplify things slightly for application programmers. A kqueue() flag for cancellation does not seem very useful: since such a flag would be a kernel thing and pthread cancellation is a libthr thing, requesting the flag from the kernel would slow down all kevent() calls. -- Jilles Tjoelker From owner-freebsd-hackers@FreeBSD.ORG Wed Mar 25 22:42:53 2015 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 755B2DED for ; Wed, 25 Mar 2015 22:42:53 +0000 (UTC) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 37A5DE2C for ; Wed, 25 Mar 2015 22:42:53 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 7C0E43592EF; Wed, 25 Mar 2015 23:42:51 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 5319C28494; Wed, 25 Mar 2015 23:42:51 +0100 (CET) Date: Wed, 25 Mar 2015 23:42:51 +0100 From: Jilles Tjoelker To: Ivan Radovanovic Subject: Re: kevent behavior Message-ID: <20150325224251.GB79065@stack.nl> References: <550A6DA2.1070004@gmail.com> <20150324221541.GA67584@stack.nl> <5512C835.7040207@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5512C835.7040207@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 22:42:53 -0000 On Wed, Mar 25, 2015 at 03:37:41PM +0100, Ivan Radovanovic wrote: > I saw this EVFILT_USER thing, and it looked to me like it could be used > to do much smarter unblocking of kevent but I believe manual page is > relatively unclear there - what I do not understand is: > * how you trigger this user event (I would say with kevent if > corresponding structure's NOTE_TRIGGER flag is set, but that doesn't > sound logical since in my understanding kevent is basically like > select(2) (ie call which checks if something happened without triggering > things = call which reports about status change in system without > causing change itself)? The changelist passed to kevent(2) causes changes to the kqueue. One kevent(2) call can both change what is monitored and check whether something has happened. An example of EVFILT_USER is in tools/regression/kqueue/user.c. > * how does this work process-wise - is one ident restricted to process > which created corresponding kqueue, or it is system wise (other process > could trigger my kevent if same ident value is used)? The ident is per kqueue. -- Jilles Tjoelker From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 14:04:38 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EC5EFA00 for ; Thu, 26 Mar 2015 14:04:38 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 72DF4CB4 for ; Thu, 26 Mar 2015 14:04:37 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2QDZwkn078504 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 26 Mar 2015 14:35:58 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2QAQXPj001423 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 26 Mar 2015 11:26:33 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2QAQRHs001420 for ; Thu, 26 Mar 2015 11:26:28 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Thu, 26 Mar 2015 11:26:27 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: freebsd-hackers@freebsd.org Subject: Seagate Archive HDD Message-ID: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Thu, 26 Mar 2015 14:35:58 +0100 (CET) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 14:04:39 -0000 http://www.storagereview.com/seagate_archive_hdd_review_8tb i want to buy 2 such drives for backup server. This drives use shingled recording. Are anyone using them and can confirm they are compatible on software level with other disks? I understand average random write time would be 5-10 times slower than normal drive because of the need of rewrite few full tracks worth of data, but otherwise will then be compatible and can i use it as usual? From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 14:16:18 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A11EC483 for ; Thu, 26 Mar 2015 14:16:18 +0000 (UTC) Received: from mail-oi0-x231.google.com (mail-oi0-x231.google.com [IPv6:2607:f8b0:4003:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 656BBE68 for ; Thu, 26 Mar 2015 14:16:18 +0000 (UTC) Received: by oigz129 with SMTP id z129so5231165oig.1 for ; Thu, 26 Mar 2015 07:16:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=6LKy66qcws3ZwGyxIMMAha+8s38xSOVPOERVIVMtDWY=; b=tvJnFHLm5zU0p+YmqS9ge02Ayj2gAeQ8v5AplQT8DuFQREOY98tXLzwKjGoA/FuvhA lhFY/X09W9EorQ2rs9V1EK26/qXP8sAXeGl5znWauqn4RK6oxZj/NYAkpK487Jfjkvyp Ja2278We3krKQKYDehVXx2n8lCCzlxqBcH94BThcLNZ4Klrk0DjqNHmXBILLQVH9EmeS khVPTI6N5atYM11ne1dDBEYGdACqdpJhq2NcnWOCQ4pOLOJg+2vzZkEpTTZdOnVTJMXZ 4x+Y/IY7jJui31Kz8NetH5pPMbGeI2DPzPEoNcY4BG8pQtDv2psH78h8JllkHWut/Fe2 NqqQ== X-Received: by 10.60.47.80 with SMTP id b16mr12358606oen.10.1427379377597; Thu, 26 Mar 2015 07:16:17 -0700 (PDT) Received: from [21.227.54.213] ([66.87.120.213]) by mx.google.com with ESMTPSA id ku2sm4455123obc.12.2015.03.26.07.16.16 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 26 Mar 2015 07:16:16 -0700 (PDT) References: Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: X-Mailer: iPhone Mail (12B440) From: "B. Estrade" Subject: Re: Seagate Archive HDD Date: Thu, 26 Mar 2015 09:16:12 -0500 To: Wojciech Puchar Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 14:16:18 -0000 Interesting. The write up says it's "drive managed" and the host OS need not= be be aware of the on disk SMR. Short of a real testimonial, it seems like i= t's just be seen as a normal drive. Brett Sent from my iPhone > On Mar 26, 2015, at 5:26 AM, Wojciech Puchar wrote: >=20 > http://www.storagereview.com/seagate_archive_hdd_review_8tb >=20 > i want to buy 2 such drives for backup server. >=20 > This drives use shingled recording. >=20 > Are anyone using them and can confirm they are compatible on software leve= l with other disks? I understand average random write time would be 5-10 tim= es slower than normal drive because of the need of rewrite few full tracks w= orth of data, but otherwise will then be compatible and can i use it as usua= l? > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"= From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 14:23:57 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8590682F for ; Thu, 26 Mar 2015 14:23:57 +0000 (UTC) Received: from mail-oi0-x22c.google.com (mail-oi0-x22c.google.com [IPv6:2607:f8b0:4003:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 445AEF99 for ; Thu, 26 Mar 2015 14:23:57 +0000 (UTC) Received: by oicf142 with SMTP id f142so41017998oic.3 for ; Thu, 26 Mar 2015 07:23:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=hq9T+cAwStFyhPk1B/N49X4W6Gz711kJCdB862V4Q8c=; b=QXXq4l7UlopCzSs6jLGx/aAgnSs/SHHrWnfYdMBZFEj6yb3y3MwtvWfEFayssWTVWd GOUVeUJmdnIz+jhoEQ0lq4Le3uviL7/ZLh8wKrhi0/uBIY2/7/wjV8J4Gc+ZrTFS2PJ5 IQkdl1b/jZufaeO00s3OoQuDY3DzEE75BZuBTycc4sKId+XX1ApjhWVMByFcS0DWxrWm 0c9Tsh1ajQ1PBN5G0GpIlNE3tIREgKnulFD28XCtzSpRshEr94qz7oxRf0gLKZ8DcOmm TcRkKkTXgpGwo76QHKHGge8+YtzH2EpeVpf22ZpaoZ0SXxDbUjBdFoOThTshkQsyy0/p cvSQ== X-Received: by 10.182.72.200 with SMTP id f8mr11790304obv.51.1427379836574; Thu, 26 Mar 2015 07:23:56 -0700 (PDT) Received: from [21.227.54.213] ([66.87.120.213]) by mx.google.com with ESMTPSA id y6sm4435430oes.9.2015.03.26.07.23.55 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 26 Mar 2015 07:23:55 -0700 (PDT) References: Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: X-Mailer: iPhone Mail (12B440) From: "B. Estrade" Subject: Re: Seagate Archive HDD Date: Thu, 26 Mar 2015 09:23:52 -0500 To: Wojciech Puchar Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 14:23:57 -0000 Sorry for the double reply. "Use As normal" sounds fine as long it's not in a= ny kind of NAS application or RAID configuration - software (eg, ZFS) or oth= erwise, per the article's caveats further down. Brett > On Mar 26, 2015, at 5:26 AM, Wojciech Puchar wrote: >=20 > http://www.storagereview.com/seagate_archive_hdd_review_8tb >=20 > i want to buy 2 such drives for backup server. >=20 > This drives use shingled recording. >=20 > Are anyone using them and can confirm they are compatible on software leve= l with other disks? I understand average random write time would be 5-10 tim= es slower than normal drive because of the need of rewrite few full tracks w= orth of data, but otherwise will then be compatible and can i use it as usua= l? > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"= From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 14:44:35 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 65AF4361 for ; Thu, 26 Mar 2015 14:44:35 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C2D9E26F for ; Thu, 26 Mar 2015 14:44:34 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2QEiVXu081817 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 26 Mar 2015 15:44:32 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2QEiSMB001241; Thu, 26 Mar 2015 15:44:28 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2QEiNdZ001238; Thu, 26 Mar 2015 15:44:23 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Thu, 26 Mar 2015 15:44:23 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: "B. Estrade" Subject: Re: Seagate Archive HDD In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Thu, 26 Mar 2015 15:44:32 +0100 (CET) Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 14:44:35 -0000 that's fine.. i think firmware reserve SMALL part of media in non-shingled format as a log to keep small writes, and then in periods of low load it writes it for real. the chance of coalescing multiple writes gets high this way. seems fine for drive that is just used to do backups over net with rsync On Thu, 26 Mar 2015, B. Estrade wrote: > Interesting. The write up says it's "drive managed" and the host OS need not be be aware of the on disk SMR. Short of a real testimonial, it seems like it's just be seen as a normal drive. > > Brett > > Sent from my iPhone > >> On Mar 26, 2015, at 5:26 AM, Wojciech Puchar wrote: >> >> http://www.storagereview.com/seagate_archive_hdd_review_8tb >> >> i want to buy 2 such drives for backup server. >> >> This drives use shingled recording. >> >> Are anyone using them and can confirm they are compatible on software level with other disks? I understand average random write time would be 5-10 times slower than normal drive because of the need of rewrite few full tracks worth of data, but otherwise will then be compatible and can i use it as usual? >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 18:28:44 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82148C2D for ; Thu, 26 Mar 2015 18:28:44 +0000 (UTC) Received: from zeeb.org (zeeb.org [178.63.54.19]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3B0EF3BB for ; Thu, 26 Mar 2015 18:28:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zeeb.org; s=z1; h=Sender:In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:To:From:Date; bh=dvMVBn24BQBh/ghJoiNvtyq8UJnroZeWeBSsUzytbq0=; b=T2jbduM0Zh6mSkRxy61M6oYJBLAMBOb8kHvrB4zNNvW/Ou7b86ToBUSkc73Hxo84r5TCt87QVGg1GEVOaiEE5esaBNdjFORmkh8DeG2iEZT9xcSqg2bb/9mjZTwT8qdLzEXmB5DnRTB8EbbfxTxRy0NY4YxMXqA6lrpMkmF1MrZzeb6cYstLRRQO1QF6Fzld+to1pXwYdg6gMi7BdO4djgEDyFwkuhDF+D18s1rec9s242EId8KNyPOpgmOd8vN3QW5TlOKg3sr+XfNywikdWcv0vgFZXBpyfpuRqWjDsKUrZ820kxac02pvr6W2HzKU2X0paKvfR2tmORhm/VFTeQ==; Received: from mwest by zeeb.org with local (Exim 4.82_1-5b7a7c0-XX (FreeBSD)) (envelope-from ) id 1YbCA9-0000ji-EI for freebsd-hackers@freebsd.org; Thu, 26 Mar 2015 20:05:49 +0200 Date: Thu, 26 Mar 2015 20:05:49 +0200 From: Matthew West To: freebsd-hackers@freebsd.org Subject: Re: Seagate Archive HDD Message-ID: <20150326180549.GA145@zeeb.org> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Sender: Matthew West X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 18:28:44 -0000 Hi Wojciech, On Thu, Mar 26, 2015 at 11:26:27AM +0100, Wojciech Puchar wrote: > http://www.storagereview.com/seagate_archive_hdd_review_8tb > > i want to buy 2 such drives for backup server. > > This drives use shingled recording. > > Are anyone using them and can confirm they are compatible on software > level with other disks? I understand average random write time would be > 5-10 times slower than normal drive because of the need of rewrite few > full tracks worth of data, but otherwise will then be compatible and can i > use it as usual? I don't have access to these drives myself, but I've been following a thread about them here: http://www.reddit.com/r/DataHoarder/comments/2zy0uj/just_installed_2x8tb_seagate_archives/ It does seem that the drives should simply appear as a standard SATA drive to the OS. Obviously, running them in any configuration other than standalone (ie, in a RAID, etc.) has some nasty caveats. Another interesting write-up here: http://www.technikaffe.de/anleitung-263-seagate_archive_hdd_v2_mit_8tb_im_test Matthew From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 19:39:56 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B83425F5 for ; Thu, 26 Mar 2015 19:39:56 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 25464D97 for ; Thu, 26 Mar 2015 19:39:55 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2QJdp8W091756 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 26 Mar 2015 20:39:51 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2QJdlx5000880; Thu, 26 Mar 2015 20:39:48 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2QJdgif000877; Thu, 26 Mar 2015 20:39:42 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Thu, 26 Mar 2015 20:39:42 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: Matthew West Subject: Re: Seagate Archive HDD In-Reply-To: <20150326180549.GA145@zeeb.org> Message-ID: References: <20150326180549.GA145@zeeb.org> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Thu, 26 Mar 2015 20:39:51 +0100 (CET) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 19:39:56 -0000 thanks for all info. I don't use any RAIDs, just multiple separate drives with one UFS filesystem on each (+geli encryption but it doesn't matter). I will buy them tomorrow On Thu, 26 Mar 2015, Matthew West wrote: > Hi Wojciech, > > On Thu, Mar 26, 2015 at 11:26:27AM +0100, Wojciech Puchar wrote: >> http://www.storagereview.com/seagate_archive_hdd_review_8tb >> >> i want to buy 2 such drives for backup server. >> >> This drives use shingled recording. >> >> Are anyone using them and can confirm they are compatible on software >> level with other disks? I understand average random write time would be >> 5-10 times slower than normal drive because of the need of rewrite few >> full tracks worth of data, but otherwise will then be compatible and can i >> use it as usual? > > I don't have access to these drives myself, but I've been following a > thread about them here: > > http://www.reddit.com/r/DataHoarder/comments/2zy0uj/just_installed_2x8tb_seagate_archives/ > > It does seem that the drives should simply appear as a standard SATA > drive to the OS. > > Obviously, running them in any configuration other than standalone (ie, > in a RAID, etc.) has some nasty caveats. > > Another interesting write-up here: > > http://www.technikaffe.de/anleitung-263-seagate_archive_hdd_v2_mit_8tb_im_test > > Matthew > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 20:30:08 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21DDA889 for ; Thu, 26 Mar 2015 20:30:08 +0000 (UTC) Received: from mail-yh0-x22b.google.com (mail-yh0-x22b.google.com [IPv6:2607:f8b0:4002:c01::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CD7513C4 for ; Thu, 26 Mar 2015 20:30:07 +0000 (UTC) Received: by yhjf44 with SMTP id f44so31739191yhj.3 for ; Thu, 26 Mar 2015 13:30:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=hpoGOumC5Ty9dLJITlJOR8+5QQT4kXIYBqLF10KfBk4=; b=VcaQYgz+XGORLoS4SVcy+rnu0s5ewCpaYGIVDy83uZSdp8cBH0fE0LApsw4fnU//uk M+JoUn3JZ1e6CHS7fmAOxxYIJZpQidh1dB0aUbiRdJXJfXd2DYb3cFsaZeVgjQ7zipGA k/bCWqPh82g/aD7P6M9wW/ovPcUYmY1If3ar0VRO8vhKew4mPKqyPMFC31msOI4Ufv4A e5z9dz8zaq/uZf5qRllH+hY+mkoxuCAuTTsUYCgOBahBxFkQ3fU31eFZhr5deoAGOkA2 A9y5zS68gozeURWAcbAPZ8t1dmoU0EogY6nTILWmjY0A5Fdi+rqiLkF1MudZTS9zAs9A gfmw== MIME-Version: 1.0 X-Received: by 10.236.26.143 with SMTP id c15mr16307982yha.192.1427401806978; Thu, 26 Mar 2015 13:30:06 -0700 (PDT) Received: by 10.170.60.69 with HTTP; Thu, 26 Mar 2015 13:30:06 -0700 (PDT) In-Reply-To: References: <20150326180549.GA145@zeeb.org> Date: Thu, 26 Mar 2015 13:30:06 -0700 Message-ID: Subject: Re: Seagate Archive HDD From: Mehmet Erol Sanliturk To: Wojciech Puchar Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 20:30:08 -0000 On Thu, Mar 26, 2015 at 12:39 PM, Wojciech Puchar wrote: > thanks for all info. I don't use any RAIDs, just multiple separate drives > with one UFS filesystem on each (+geli encryption but it doesn't matter). > > I will buy them tomorrow > > On Thu, 26 Mar 2015, Matthew West wrote: > > Hi Wojciech, >> >> On Thu, Mar 26, 2015 at 11:26:27AM +0100, Wojciech Puchar wrote: >> >>> http://www.storagereview.com/seagate_archive_hdd_review_8tb >>> >>> i want to buy 2 such drives for backup server. >>> >>> This drives use shingled recording. >>> >>> Are anyone using them and can confirm they are compatible on software >>> level with other disks? I understand average random write time would be >>> 5-10 times slower than normal drive because of the need of rewrite few >>> full tracks worth of data, but otherwise will then be compatible and can >>> i >>> use it as usual? >>> >> >> I don't have access to these drives myself, but I've been following a >> thread about them here: >> >> http://www.reddit.com/r/DataHoarder/comments/2zy0uj/ >> just_installed_2x8tb_seagate_archives/ >> >> It does seem that the drives should simply appear as a standard SATA >> drive to the OS. >> >> Obviously, running them in any configuration other than standalone (ie, >> in a RAID, etc.) has some nasty caveats. >> >> Another interesting write-up here: >> >> http://www.technikaffe.de/anleitung-263-seagate_archive_ >> hdd_v2_mit_8tb_im_test >> >> Matthew >> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org >> " >> >> >> _______________________________________________ > > There is a customer review in the following page : http://www.amazon.com/Seagate-Archive-HDD-ST8000AS0002-drive/dp/B00QX0ZGO6 Mehmet Erol Sanliturk From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 21:45:57 2015 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CA5FC540 for ; Thu, 26 Mar 2015 21:45:57 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6C594E6E for ; Thu, 26 Mar 2015 21:45:57 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t2QLjpQj094242 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 26 Mar 2015 23:45:51 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t2QLjpQj094242 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t2QLjpq7094229; Thu, 26 Mar 2015 23:45:51 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 26 Mar 2015 23:45:51 +0200 From: Konstantin Belousov To: Jilles Tjoelker Subject: Re: kevent behavior Message-ID: <20150326214551.GG2379@kib.kiev.ua> References: <550A6DA2.1070004@gmail.com> <20150324221541.GA67584@stack.nl> <20150325090041.GY2379@kib.kiev.ua> <20150325223530.GA79065@stack.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150325223530.GA79065@stack.nl> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-hackers@FreeBSD.org, Ivan Radovanovic X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 21:45:58 -0000 On Wed, Mar 25, 2015 at 11:35:30PM +0100, Jilles Tjoelker wrote: > On Wed, Mar 25, 2015 at 11:00:41AM +0200, Konstantin Belousov wrote: > > On Tue, Mar 24, 2015 at 11:15:41PM +0100, Jilles Tjoelker wrote: > > > Fortunately, EVFILT_USER provides an easy way to wake up a thread > > > blocked in kevent(). > > > > If kevent() was implemented as a cancellation point, pthread_cancel() > > > would be another option, but it is not. > > > Do you consider it is possible to make kqueue(2) cancellation point ? > > Susv4 is very explicit about functions it allows to handle cancellation. > > Might be, it is time for kqueue2(3), which would take flags. > > In addition to allowing cancellation, it could take close-on-exec flag. > > > It is somewhat non-trivial to implement, at least I do not want to make > > kevent2(). > > I take it you mean making kevent() a cancellation point, not kqueue(). > The latter would just annoy application writers for no benefit. Um, sorry, yes. > > POSIX does not say anything about kevent(), including whether it should > be a cancellation point or not. Given that kevent() may block for a long > time, making it a cancellation point seems to make sense. But POSIX language is very explicit for its own set of interfaces, and that makes sense. This is why I think that cancellability, after the 15+ years of kqueue existence, should be opt in. > > The kevent() cancellation point implementation would be much like most > other cancellation points such as pselect(), with the difference that > the call may have committed even if it fails, in the case that > nchanges > nevents and in the case that nchanges > 0 && errno == EINTR. > If cancellation is requested while blocking in the latter case, libthr > will have to return -1/EINTR to indicate to the application that the > changes were applied successfully while allowing the thread to be > cancelled at the next cancellation point, even though there may not be > any signal handler. > > If nevents == 0, kevent() does not block and need not be a cancellation > point. This special case may simplify things slightly for application > programmers. > > A kqueue() flag for cancellation does not seem very useful: since such a > flag would be a kernel thing and pthread cancellation is a libthr thing, > requesting the flag from the kernel would slow down all kevent() calls. Yes, I thought about adding some (small) userspace table which would track cancellable kqueues. But if we make the cancellation state per-call, and not a state for kqueue descriptor, we might indicate the cancellability by some event type, which is not passed to the kernel at all. The libthr wrapper for kevent(2) would need to scan the changelist and zero-out the cancellation indicator. From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 22:33:11 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 170B0198 for ; Thu, 26 Mar 2015 22:33:11 +0000 (UTC) Received: from mail-ig0-x235.google.com (mail-ig0-x235.google.com [IPv6:2607:f8b0:4001:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D063C3D2 for ; Thu, 26 Mar 2015 22:33:10 +0000 (UTC) Received: by igbud6 with SMTP id ud6so5394382igb.1 for ; Thu, 26 Mar 2015 15:33:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=RDNmxhVD3MyvjgFZHujkitsyoHyxK2Jkutt9DyisB4o=; b=wDHG5At3Gz4ecizh84s8Fjw2OtYk/47xRd/3Ps8nnW0FJx4HFIMgme2ZAg+hDZvdSW 7X5/lyUakG6mdku0JknIFb+npTT7LJe7OH24xepiV9kVykCAxw/1UKzj8wjnl42kVv49 WjElrVSBZ1X9S280W7o0x0gnx+cimafDTVDsZFUQHi3Q4FFzmOtHDcdD1S2jagFj1xIN gLherP8J7nWkH208DQPongqjFq5rSMiTjOv3mdRr12Yvzum2JTvLG7ZAKUgAIXG1fuwI f93/FCuS2ccTbFuCoSdfDeGb7lupqEKZ8ukRfc3yR3UaNpL6+9NEUIFH+qzNgPf99wir CnPg== MIME-Version: 1.0 X-Received: by 10.42.93.83 with SMTP id w19mr41049045icm.37.1427409190310; Thu, 26 Mar 2015 15:33:10 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Thu, 26 Mar 2015 15:33:10 -0700 (PDT) In-Reply-To: References: Date: Thu, 26 Mar 2015 15:33:10 -0700 X-Google-Sender-Auth: qtpqxiJoJo635z0XcJK8A6qbg2I Message-ID: Subject: Re: Seagate Archive HDD From: Adrian Chadd To: Wojciech Puchar Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 22:33:11 -0000 Hm, what's required to get the stripe size for doing IO or whatever the under-the-hood thing is for the Sharding? -a From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 22:37:19 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 990D8640; Thu, 26 Mar 2015 22:37:19 +0000 (UTC) Received: from mail-oi0-x22c.google.com (mail-oi0-x22c.google.com [IPv6:2607:f8b0:4003:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5A07D5EE; Thu, 26 Mar 2015 22:37:19 +0000 (UTC) Received: by oiag65 with SMTP id g65so62092411oia.2; Thu, 26 Mar 2015 15:37:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=qfVoZWlVsDjfJceGKykSEUjSBENvkfsbafH4o3jTOfE=; b=gu3s7Cg4i6rdXerlPqHktCcLr7d5pK9imrNd6kz0FxN6jv71rRN3KAC2EhtCYBMOIg FBUAfukAd4rQBbLwa+SzIe2lQMeaAP4Pz/l1nEVaShGA27B9E0b/Z7T1lch5SUQDGuxA 2uYogPQk/158kSmZpvWwLkbPnBSQiK27rq1kVi3XjPfp2+rkknsdmfpj4HvDo8hMzWPd 9LklWFJSBSihxjZEm1z2nxZii7muXcnFtbFCtfk+4hZyUm0oak1TiYDpH1zasPYPhR9r FwUwp3zw0jRj53RLpTiAeqJx/aEkDzOIU7EGqlVqalkp0OjyH8ZKhDkPnkkUEm/qHdRG J9iA== MIME-Version: 1.0 X-Received: by 10.202.187.5 with SMTP id l5mr143944oif.51.1427409438288; Thu, 26 Mar 2015 15:37:18 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.215.7 with HTTP; Thu, 26 Mar 2015 15:37:18 -0700 (PDT) In-Reply-To: References: Date: Thu, 26 Mar 2015 16:37:18 -0600 X-Google-Sender-Auth: 1EgO2QCGU0jwDhv5eDpcE1HIlhk Message-ID: Subject: Re: Seagate Archive HDD From: Alan Somers To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 22:37:19 -0000 On Thu, Mar 26, 2015 at 4:33 PM, Adrian Chadd wrote: > Hm, what's required to get the stripe size for doing IO or whatever > the under-the-hood thing is for the Sharding? > > -a My understanding is that it's impossible; the drive-managed SMR drives present themselves as ordinary disks. But, if their design is similar to the upcoming host-aware SMR drives, then the band size will be on the order of a few hundred megabytes. -Alan From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 22:55:00 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E1E86C8C for ; Thu, 26 Mar 2015 22:54:59 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C621A833 for ; Thu, 26 Mar 2015 22:54:59 +0000 (UTC) Received: from zeta.ixsystems.com (unknown [12.229.62.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 502A912D01; Thu, 26 Mar 2015 15:54:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1427410499; x=1427424899; bh=M5Nq7nTdz3UysTGcFuXV4vah+6vzlEJlG3M6kd+yBBQ=; h=Date:From:Reply-To:To:Subject:References:In-Reply-To; b=zKbhgg1tnrMOGffsMujh8vjtCBgV4a0VStsnMp1tBCA5zgDyqewQ+MTYvpFHRMq4w Inwqqrz++4ZCGMJgoQmZkR/9ouEfGEvgXxJkOfbM00bQs2OoR7jwUbMfviGVKYlcuH fCmhap7Er8jZitJv6B8zhSB7jmBZFCAd/vVGpYDE= Message-ID: <55148E42.80708@delphij.net> Date: Thu, 26 Mar 2015 15:54:58 -0700 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: Seagate Archive HDD References: In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 22:55:00 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03/26/15 03:26, Wojciech Puchar wrote: > http://www.storagereview.com/seagate_archive_hdd_review_8tb > > i want to buy 2 such drives for backup server. > > This drives use shingled recording. > > Are anyone using them and can confirm they are compatible on > software level with other disks? I understand average random write > time would be 5-10 times slower than normal drive because of the > need of rewrite few full tracks worth of data, but otherwise will > then be compatible and can i use it as usual? My understanding is that the SMR drives are actually different class of storage device. The "drive managed" drives as shipped now tries to emulate normal hard drive's behavior but they present unique risks: for instance, a rewrite of a small block may end up in a read-modify-write of a much larger area, so we must refrain from doing such operations for critical file system data structure, probably by reorganizing the on-disk format to satisfy the need. That's said, if the backup is mostly something where you can group writes together and not being written often, the drives would be just fine working with today's OS and FS because the risk is not very big for that type of operation. If the backup is something "warm" (i.e. a replicated database that receives live update stream from a master one), or the power supply is not very reliable, it's a good idea to avoid these drives for now until we have SMR aware drivers and filesystems for the new type of disk (and I would expect the disk used with these be host managed ones, or we would probably need some quirks to tell the OS to avoid doing writes in bad way). Cheers, - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.1.2 (FreeBSD) iQIcBAEBCgAGBQJVFI4/AAoJEJW2GBstM+nskW0P/jtbgja1GUZpBtqsqiYd4dQO EvmwY1Xvzk5tfQXTiYNH5GgEvNK3BoFz6GpYZgFPEgETA6v5WuuoKHnFQpFmf2Eq QK7QmwaQmon1NPzo9LAkCF21WIoq7ED/oRHk/djEHMNgwzhpdFNLno0YH20T0pG1 nQ+cJolZVD1ndtqRcKR1b9sRdnCFE+PRyr40DR9AsfXcOxUR7dKswzeaAs+Y/FTH UKvHSFWqzNcScATBrn5C/sATbWZblXai18HnoRB3j/S1v6uUonkzGu4aNsU+9Fg9 9hVnKvO75Z12UbxZZaOJZBJara8nZYgVDSQXK/o/Bm1bSEslT2sj4nqC3ZCOu8d8 HWj41qlnP2x6/FXei/kLxeX7KxunX2zKHtQeCGyiOQ5cAlXnE3BmkwBxz116hku2 i6SXT8+3wMHoqpSu0r8Z2es0+eoucP+WuHoHdzCHgPIZH5BGyoUALyH43ihLHV3u AixU6fTOvXz+BHiFXRGMFxlnThXkkbetHMtxWY/gaw8UMjrALLxQstV6CPXMTMpa 8vTe6WfBL5yEIeEL+ND5sFyaLwwkuoVNGjWyPxnNtU2S2p/wTvDiMEOz3wXJurDv VHf1Bz/dsDghEAeM9al4lJ8N3I3ubk0e7HmJxEac1GfQaNEfInCfOjmNfXKWk4Eb N3yEo3ChS7h9zsk73zOh =pImh -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 22:56:18 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 432ECD78 for ; Thu, 26 Mar 2015 22:56:18 +0000 (UTC) Received: from mail-qc0-x233.google.com (mail-qc0-x233.google.com [IPv6:2607:f8b0:400d:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB9A284B for ; Thu, 26 Mar 2015 22:56:17 +0000 (UTC) Received: by qcay5 with SMTP id y5so3510102qca.1 for ; Thu, 26 Mar 2015 15:56:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=4IWZlgqnIKLFljLKznPJkbsd+VathOKUYWD99mSTQWE=; b=e1L18CtryRxnSZ407Tq9NPa7YxgB0PDHjp7wYMv8nHl/cs4jbmmu/Gj4xWMy2m5k+4 5uuVbS7j27X4zrM2vkR7XGFTL2QXy6KV8QvoC2jYOic7qqclzKgCXGOK9adrnb54M65Z fOk5rKK3bJbg+ftJAxjtYru4ANxf4+5m32YGvfmpnflTXdPO2jXPpTQqp+MSkUoNp/OP WjNaTXFn7ELM+N9rqfldR1oAcxwg5a7aVEo1VoSZkys749KOZDJ1Nvf6DrpMEN24NK3I nxIMKtNH+UOL6SeDL6ehMvi6e+kKNyBLaUFGnon1EunEoF8V0maO6uEsBZiLidg/EdWK B4gg== X-Received: by 10.229.65.8 with SMTP id g8mr21843066qci.15.1427410577056; Thu, 26 Mar 2015 15:56:17 -0700 (PDT) MIME-Version: 1.0 Sender: mozolevsky@gmail.com Received: by 10.140.91.112 with HTTP; Thu, 26 Mar 2015 15:55:36 -0700 (PDT) In-Reply-To: References: From: Igor Mozolevsky Date: Thu, 26 Mar 2015 22:55:36 +0000 X-Google-Sender-Auth: f9x8meWl2OMew2t7-896FrSyiow Message-ID: Subject: Re: Seagate Archive HDD To: Wojciech Puchar Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Hackers freeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 22:56:18 -0000 On 26 March 2015 at 10:26, Wojciech Puchar wrote: > > Are anyone using them and can confirm they are compatible on software > level with other disks? I understand average random write time would be > 5-10 times slower than normal drive because of the need of rewrite few fu= ll > tracks worth of data, but otherwise will then be compatible and can i use > it as usual? I suspect so, but the drives are designed to be used as "tape-on-disk" backup (hence "archive"), i. e. long sequential writes. Using these drives for anything else seems as insane as using the "video recording"-grade drives for data, in my opinion=E2=80=A6 --=20 Igor M. From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 22:58:30 2015 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 47B92F50 for ; Thu, 26 Mar 2015 22:58:30 +0000 (UTC) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0827F869 for ; Thu, 26 Mar 2015 22:58:30 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 3E84A358C5A; Thu, 26 Mar 2015 23:58:27 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 22E7228494; Thu, 26 Mar 2015 23:58:27 +0100 (CET) Date: Thu, 26 Mar 2015 23:58:27 +0100 From: Jilles Tjoelker To: Konstantin Belousov Subject: Re: kevent behavior Message-ID: <20150326225826.GA97319@stack.nl> References: <550A6DA2.1070004@gmail.com> <20150324221541.GA67584@stack.nl> <20150325090041.GY2379@kib.kiev.ua> <20150325223530.GA79065@stack.nl> <20150326214551.GG2379@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150326214551.GG2379@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@FreeBSD.org, Ivan Radovanovic X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 22:58:30 -0000 On Thu, Mar 26, 2015 at 11:45:51PM +0200, Konstantin Belousov wrote: > On Wed, Mar 25, 2015 at 11:35:30PM +0100, Jilles Tjoelker wrote: > > POSIX does not say anything about kevent(), including whether it should > > be a cancellation point or not. Given that kevent() may block for a long > > time, making it a cancellation point seems to make sense. > But POSIX language is very explicit for its own set of interfaces, and > that makes sense. This is why I think that cancellability, after the > 15+ years of kqueue existence, should be opt in. Hmm, I think most code written is cancel-unsafe. It is unlikely to have cancel-safe code using kevent() and relying on it not being a cancellation point, but still possible. This also means we shouldn't wait too long with adding missing cancellation points like ppoll(). > > The kevent() cancellation point implementation would be much like most > > other cancellation points such as pselect(), with the difference that > > the call may have committed even if it fails, in the case that > > nchanges > nevents and in the case that nchanges > 0 && errno == EINTR. > > If cancellation is requested while blocking in the latter case, libthr > > will have to return -1/EINTR to indicate to the application that the > > changes were applied successfully while allowing the thread to be > > cancelled at the next cancellation point, even though there may not be > > any signal handler. > > If nevents == 0, kevent() does not block and need not be a cancellation > > point. This special case may simplify things slightly for application > > programmers. > > A kqueue() flag for cancellation does not seem very useful: since such a > > flag would be a kernel thing and pthread cancellation is a libthr thing, > > requesting the flag from the kernel would slow down all kevent() calls. > Yes, I thought about adding some (small) userspace table which would > track cancellable kqueues. I think the interaction with dup() and similar calls and with exec makes this too nasty. > But if we make the cancellation state per-call, and not a state for kqueue > descriptor, we might indicate the cancellability by some event type, which > is not passed to the kernel at all. The libthr wrapper for kevent(2) would > need to scan the changelist and zero-out the cancellation indicator. This seems a bit hackish. However, enabling cancellation per-call seems better than per-kqueue. -- Jilles Tjoelker From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 23:29:08 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7ACC0864; Thu, 26 Mar 2015 23:29:08 +0000 (UTC) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3BEB2B69; Thu, 26 Mar 2015 23:29:08 +0000 (UTC) Received: by iecvj10 with SMTP id vj10so59214619iec.0; Thu, 26 Mar 2015 16:29:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=HDF8/2oDa15/2vebzs7AHvKcC8IcTLkV5794DTJ4XCY=; b=q8/IFJr1x/OYT6E3wsy8p/0lBAA7+RVHLz8bckqEqLjfIaTeLC8eK75MjLMS4i6OF4 vZepz7MzcxG0yLt/JU7amBUdPSyfczohK1Fqjbqp4UYYm0vmaGXSrIcwZ3Cb8eTMmLAE hsSFBdsk8lq4Non5XpPiOZAsPiUrCRgyYxIQPgS1IAg5KXPz9T8hapm218lDkX4ieyx2 AON6hVHi7RUsd7wHg1YM3+XuUGnpx199SK2kD2n/hFXstxuBW2M8UMIPJLcUYgv3EW5k AnHbYD2DXsp7EyM8fafrNzInd8y91mXzi9hJsnuOtCxl9/8JH6/gIICdJKAhqPBqz+5k Fqog== MIME-Version: 1.0 X-Received: by 10.107.5.131 with SMTP id 125mr24381043iof.88.1427412547500; Thu, 26 Mar 2015 16:29:07 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Thu, 26 Mar 2015 16:29:07 -0700 (PDT) In-Reply-To: References: Date: Thu, 26 Mar 2015 16:29:07 -0700 X-Google-Sender-Auth: QnjXo_ZE-i1CUgvTuKMsXB2VeJM Message-ID: Subject: Re: Seagate Archive HDD From: Adrian Chadd To: Alan Somers Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 23:29:08 -0000 On 26 March 2015 at 15:37, Alan Somers wrote: > On Thu, Mar 26, 2015 at 4:33 PM, Adrian Chadd wrote: >> Hm, what's required to get the stripe size for doing IO or whatever >> the under-the-hood thing is for the Sharding? >> >> -a > > My understanding is that it's impossible; the drive-managed SMR drives > present themselves as ordinary disks. But, if their design is similar > to the upcoming host-aware SMR drives, then the band size will be on > the order of a few hundred megabytes. Does anyone have anything that we can expose to userland / GEOM? There's a bunch of useful userland storage stuff that would benefit from knowing this kind of thing. -adrian From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 26 23:58:12 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1EF12784 for ; Thu, 26 Mar 2015 23:58:12 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [64.62.153.212]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0302EEC0 for ; Thu, 26 Mar 2015 23:58:11 +0000 (UTC) Received: from zeta.ixsystems.com (unknown [12.229.62.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 539CCE0AD; Thu, 26 Mar 2015 16:58:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1427414291; x=1427428691; bh=e25IvTiFdkgbzBWaTNaRTMnMNtfmEq49Uk0964+Z5Fw=; h=Date:From:Reply-To:To:Subject:References:In-Reply-To; b=AmQRpUwYRuisI5ZdiniQo2II0nhjYxIK73wxuSu0akzyB9dN3Pi5EVDogcIsWNE6n d190/Ao4Lh920uh/uE4T3OVWv7SEAwBHxTersOxUwYqVPhyalJ/XuUkBdpAcTQqWdE soW6QxniTO4/dvRYf0gi+tE5mZ95+2nrOwPdvvmE= Message-ID: <55149D12.6070602@delphij.net> Date: Thu, 26 Mar 2015 16:58:10 -0700 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: Pedro Arthur , "" Subject: Re: GELI support on /boot folder References: In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 23:58:12 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03/18/15 11:50, Pedro Arthur wrote: > Hi, I was discussing with Kris Moore about adding support for GELI > in bootloader as a GSoC project, thus the /boot folder could be > encrypted. What's the benefit of encrypting /boot? If it's encrypted, will the system use only passphrase to unlock the storage, instead of the combination of both passphrase and a key file? (Use passphrase only is a bad idea because that would mean we essentially encrypt different data with the same key, if two encrypted providers both use the same passphrase. This is probably not a big deal for single disk systems like a laptop system but could have bad consequence if it's used in larger scale). Cheers, - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.1.2 (FreeBSD) iQIcBAEBCgAGBQJVFJ0QAAoJEJW2GBstM+nsiPAP/icF4pbVf5yosxErQKQHd2nm /0Kf0uurpcYCvQgzU6F4Y1vDonGEZgQV6Iyl0Xq5NgvKWYowp2gl6289/Qndnk0m QPDSEAfrmkoNSoTUV6KLuV/3q2aBZdZFKSWhQdJLbPHWpqVGrisbqmFmPaNafhoC 1Y62gvaA+u1r+IxGMVTedz7PErA2//jEF1PcnMKM6L5K7m9YIf5L8UgVkqWBXDtR nc6pDryvdLvm9m+T8b1CN0EYeqYGQrpql0SCZR7GkTfWwev1X/23oZYmCgtUuft8 FIj0loQc9A2jZ0cZ9aYhpAFxiLVJjB19b3cCkEtbc+9U+wcyTo1BayC3G9t8ZI5g VIx/I9lssZ07onVUu50C70IGbI+tB9Hjo+pA/76RIhzg93JfNpCJ1/Lqfcq3IzqB UUYgqnPf3+oV+OipmTUfG0/gYsBk5TpYEmAv3wpoatNRNwrofFMAGQLeVvLEGJ2a 5n8v8kZfSPXtI/8WE8EZKEP8iGPYiOHm0vujkVMJStWYwS8H9Z+y1zWGodcnO5Jq 7J+K1776WGWDGVetL3S3Ma+UjtCd9K3y+HJWHjYRhpFJsKi38Ur9o9p5U6F7r5M1 PbGnf6dZNVUAZJYBQgKXMpHbWmCuU9KLXUX1NtCfGqhx4BVXmU7e0/EmPFafbeGm FD5SwR3r9khIEoNFSgrK =+O3x -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 00:00:25 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5AEA287E for ; Fri, 27 Mar 2015 00:00:25 +0000 (UTC) Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by mx1.freebsd.org (Postfix) with ESMTP id E0A4EED9 for ; Fri, 27 Mar 2015 00:00:24 +0000 (UTC) Received: from ppp121-45-104-240.lns20.adl6.internode.on.net (HELO midget.dons.net.au) ([121.45.104.240]) by ipmail06.adl6.internode.on.net with ESMTP; 27 Mar 2015 10:25:13 +1030 Received: from [10.0.2.26] ([10.0.2.26]) (authenticated bits=0) by midget.dons.net.au (8.15.1/8.14.9) with ESMTPSA id t2QNt5ek020669 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 27 Mar 2015 10:25:10 +1030 (CST) (envelope-from darius@dons.net.au) Subject: Re: Seagate Archive HDD Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Content-Type: text/plain; charset=us-ascii From: "O'Connor, Daniel" In-Reply-To: <55148E42.80708@delphij.net> Date: Fri, 27 Mar 2015 10:25:05 +1030 Content-Transfer-Encoding: quoted-printable Message-Id: <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> References: <55148E42.80708@delphij.net> To: d@delphij.net X-Mailer: Apple Mail (2.2070.6) X-Spam-Score: -2.899 () ALL_TRUSTED,BAYES_00,URIBL_BLOCKED X-Scanned-By: MIMEDefang 2.75 on 10.0.2.1 Cc: freebsd-hackers@freebsd.org, Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 00:00:25 -0000 > On 27 Mar 2015, at 09:24, Xin Li wrote: > On 03/26/15 03:26, Wojciech Puchar wrote: >> http://www.storagereview.com/seagate_archive_hdd_review_8tb >>=20 >> i want to buy 2 such drives for backup server. >>=20 >> This drives use shingled recording. >>=20 >> Are anyone using them and can confirm they are compatible on >> software level with other disks? I understand average random write >> time would be 5-10 times slower than normal drive because of the >> need of rewrite few full tracks worth of data, but otherwise will >> then be compatible and can i use it as usual? >=20 > My understanding is that the SMR drives are actually different class > of storage device. >=20 > The "drive managed" drives as shipped now tries to emulate normal hard > drive's behavior but they present unique risks: for instance, a > rewrite of a small block may end up in a read-modify-write of a much > larger area, so we must refrain from doing such operations for > critical file system data structure, probably by reorganizing the > on-disk format to satisfy the need. I don't think this is necessarily true - I watched a very informative = presentation about SMR drives - = https://www.usenix.org/conference/fast15/technical-sessions/presentation/a= ghayev The drive has many regions with guard bands so it doesn't have to = rewrite the entire disk for certain writes (it may have to rewrite a = whole band though) It also has a log section which it writes to and then back fills into = the shingled area later. The upshot is that you get very weird latencies, but generally writing = sequentially is OK. Random read is OK, random write is utterly abysmal = (duh). One thing I would like to know is if the drive supports TRIM so it can = avoid rewriting a band or not. -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 00:04:01 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 57E759E1 for ; Fri, 27 Mar 2015 00:04:01 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B720FA1 for ; Fri, 27 Mar 2015 00:04:01 +0000 (UTC) Received: from zeta.ixsystems.com (unknown [12.229.62.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id EBF70E1BF; Thu, 26 Mar 2015 17:04:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1427414641; x=1427429041; bh=N4KIZt8vHQLLHOhC30bDNsYqpJKL3GjvyErTdR19nzY=; h=Date:From:Reply-To:To:CC:Subject:References:In-Reply-To; b=m9/d4pqmxM+heNHQ8fd8726MCSWxu2R/FMCgFSzvog3PpKeAvi76Mn1AqpD9gKRv0 FSIWEzBVJ0xV6azLEMbF5LkulBUDrHh9UFsnjEnEoDI7+DNeMm6Yl9svA9UicHJlmZ yljzYiZXVRhy9A/ijN2UH9k0I0fZBcvgkgNuHYAw= Message-ID: <55149E70.30608@delphij.net> Date: Thu, 26 Mar 2015 17:04:00 -0700 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: John-Mark Gurney , Pedro Arthur Subject: Re: GELI support on /boot folder References: <20150319013231.GR51048@funkthat.com> In-Reply-To: <20150319013231.GR51048@funkthat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: "" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 00:04:01 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03/18/15 18:32, John-Mark Gurney wrote: > If we go thise route, I'd ask why we don't put loader into the > gptboot instead of using the existing shim to load loader... Then > the project would be to add GELI decryption to loader which could > then be used w/ MBR in the limited sense of loading kernel and > modules, though boot/loader would still have to be on an > unencrypted partition... > > I hope others who know the boot process better will inform us why > this is a good or bad idea... If we make changes to loader more often, it could be a bad idea because merging both parties would make it harder for those who develop loader changes. Additionally, it may be desirable to keep different copies of loaders in different "boot environment" datasets, it's more convenient for debugging: let's say one developer decided to make some changes to ZFS support of loader, and that's installed to a new boot environment, then they can try it out without making a usable boot disk at hand before hand. Once the zfsloader is proven to be working (we still have zfsloader.old or a different boot environment available), we would have much more confident that the system will boot after a gptzfsboot update because they share the same code. Cheers, - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.1.2 (FreeBSD) iQIcBAEBCgAGBQJVFJ5rAAoJEJW2GBstM+ns2o8P/jdBS5VfnD2N+sKXyE7d7cmE nHzJVLA1TgkH7EdEwyzxpi/wv6LDmnsXDKAeS2iYU9T5v888XWpltJYo6Iq43h7j 5m7Y+BlMaLUlZl+IbmI07z8qrc4eYjsDKfzRiDuVTXMuW6AY5yfi+Ainmi0TbpyY KmQF5Xk/iQMUaK2S6Br3dckPnffxbaABaUUOLDwEVGwlorjsjw6pM2ckHWoSxzV6 ITE5mwhuAhE3JL/YUS0zhXD4y6ya62V4WOUbxQvivw0NHoqZ0RZhSistC5bPP5+Z JiNMVJx7NIrBYTXOpuUztpCs05QS88NF+AnMo2jtwZ78DRQFAvZAlKIV5+wAF2ZO pSTRFVir+MXM9mS4sLtg/0CViQ5V7VMPXeXP/9fHErWrSrGcM3sa4cUxI4/vfIeh cfu2MEV6+7G0anxu4x4El8epGOrK0r+oOyF2/LiiZ5fvsGLisTD8JgJvHk3g2Dh2 62ud004lTaq/ZamlLDq4gjO013MoVDVdLltfE526Fl2nL1y+loHSEnV3xflkFbfO INevkg39Oo2/Nl7d0vkJJwp3p53jhmQHKC7XBYZ4Taz5GWjJws9MUCZlD7IzlpfR ZS7Eomcu9S1bQdVxJX4kdaJEyWpmHrvLc5gye7wTM1E3evRjTinoNLDhIk2amUwb Y5kuXgSGxaJhqpjxaDal =5WOr -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 00:08:47 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EDA3BCEF for ; Fri, 27 Mar 2015 00:08:47 +0000 (UTC) Received: from mail-qc0-x22a.google.com (mail-qc0-x22a.google.com [IPv6:2607:f8b0:400d:c01::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A0720FEC for ; Fri, 27 Mar 2015 00:08:47 +0000 (UTC) Received: by qcbkw5 with SMTP id kw5so4685988qcb.2 for ; Thu, 26 Mar 2015 17:08:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=bJDKziWqX8mvODFZ18XoVSxfdsOw75tJ+T6S75SiCHs=; b=qeZjX13WDkTwTRvdle7wOuWuDCIfIW8td1uVUdBGF8VDmFMutgIB0CkVFtHOKP1PtV UDH/QOOinKBlxrQnlfWDHOl1kLGFq3bRRdW0K3WZl/G3MD2xl+FzNTWgLZyRX2YjMsQy SNa1d7JKOK5OG/2qlmY7//xgXDFA+HfAlgm97xANHnEKRfey8li93Nd4+ZFUCTu1eKrE IkQYNtwOBGWyMoh+cSsHpALBmn+fFPZsNSOygXcOh7PmdeN2bEftmm2z4Ks3ZPUc09Zk z5zpVLUs2HxtBVUMT0q9bEkPGqOVjZSoOJ6SxANiws/cuqvCz2PvSANKLQRf4WlHUsl8 BZJA== X-Received: by 10.55.26.91 with SMTP id a88mr35383952qka.0.1427414926793; Thu, 26 Mar 2015 17:08:46 -0700 (PDT) MIME-Version: 1.0 Sender: mozolevsky@gmail.com Received: by 10.140.91.112 with HTTP; Thu, 26 Mar 2015 17:08:06 -0700 (PDT) In-Reply-To: <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> References: <55148E42.80708@delphij.net> <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> From: Igor Mozolevsky Date: Fri, 27 Mar 2015 00:08:06 +0000 X-Google-Sender-Auth: 5ZzGG4w5YRKF_MqIFEUszIT8F9A Message-ID: Subject: Re: Seagate Archive HDD To: "O'Connor, Daniel" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Hackers freeBSD , Xin LI , Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 00:08:48 -0000 On 26 March 2015 at 23:55, O'Connor, Daniel wrote: One thing I would like to know is if the drive supports TRIM so it can > avoid rewriting a band or not. > Seagate's manual for the Archive HDD series doesn't mention support for TRIM, but to be fair, the manual hasn't been updated since the 5TB version= =E2=80=A6 --=20 Igor M. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 00:15:49 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DA58FF8E for ; Fri, 27 Mar 2015 00:15:48 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [64.62.153.212]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BDB86127 for ; Fri, 27 Mar 2015 00:15:48 +0000 (UTC) Received: from zeta.ixsystems.com (unknown [12.229.62.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 09B31E2FB; Thu, 26 Mar 2015 17:15:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1427415348; x=1427429748; bh=EsENsIAupIRjop/Uc8Vg4qVqyAev8oidmtIpILpypAY=; h=Date:From:Reply-To:To:CC:Subject:References:In-Reply-To; b=sN8YYwxF89/3M/NSQUmmQAQE4HYgmMc/24EJw5BvQzMl93jPrjiIY+0l6YLUagOJl Xkpbh+ve26sJDdqcx+Vb3EFdG9A1sziMmO7buk30/gfc+uS3qx770fGKuFUj90hiCu mRbjvw+YMElcEUtB2q9+4eQkiViFLHc4KAL3Kf2s= Message-ID: <5514A133.8060409@delphij.net> Date: Thu, 26 Mar 2015 17:15:47 -0700 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: "O'Connor, Daniel" , d@delphij.net Subject: Re: Seagate Archive HDD References: <55148E42.80708@delphij.net> <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> In-Reply-To: <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org, Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 00:15:49 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03/26/15 16:55, O'Connor, Daniel wrote: >> On 27 Mar 2015, at 09:24, Xin Li wrote: The >> "drive managed" drives as shipped now tries to emulate normal >> hard drive's behavior but they present unique risks: for >> instance, a rewrite of a small block may end up in a >> read-modify-write of a much larger area, so we must refrain from >> doing such operations for critical file system data structure, >> probably by reorganizing the on-disk format to satisfy the need. > > I don't think this is necessarily true - I watched a very > informative presentation about SMR drives - > https://www.usenix.org/conference/fast15/technical-sessions/presentation/aghayev > > The drive has many regions with guard bands so it doesn't have to > rewrite the entire disk for certain writes (it may have to rewrite > a whole band though) That really depends on whether we can afford to lose the whole band of (meta)data. File systems guard against loss of critical data structure by storing replicas of the data in distant places, but these may not have been distant enough to survive band sized loss, or let's say loss of a few cylinders. That's is why I am not that optimistic about what we do nowadays, we would need to revisit the on-disk formats with the new drives and see if adjustments would be necessary. Cheers, - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.1.2 (FreeBSD) iQIcBAEBCgAGBQJVFKEwAAoJEJW2GBstM+nsMtIP/j0I4w1gHhKoE6cFSLXZBwgF LosPyr3w3m1NbkFkmOsibqactliuDOUmxKD4OkGnYED6QFW3Xs0S4xuDcjKfbMDy hT4H8e3HEn8AV7DlzCdMUDtKs2zTHTnfiUxfYHfGeYaK2Vr6HZD2oMUgnzJXEKms SbTxrbY1GCHgrlDqlA6ThHToxKT/KQxCDVYdYnLKnUaj+ZdYU42iPiWIx7RWTpQJ ti8xd0Q1gZ2u2/xcVJ6mJMKGASEkQP3yVp/oplkuFqhS+GSePfvoKLaPU2VOpZCU 4RZ2/fF8zGa1OhcgSRPbSTbKKHN1TRwM847uPDeL9i/G82K1ju+7kG9E92lKFUTE p2WXhM6eqVbPx6ZTWtthB9hVlJzJkOekLbCwjgy2ShWLDmtJ2MGsmqha6nWudj26 4V5HEtEmgYPhJ+MdvnqdqcHut82AtNvbLoH9p1IFwKLhqaE+wRA4aqNjBauZRSZ+ Tg8GoR1IURHGmoKM2NVDuwWJujcJTO5bqRm1IPwZSRrBcxrULnmO3o/pmxwnSJhS gpYVAPNLhtCHrUHRKGxnJ+0upPko2h+JV0nZYvbgf4FVk79JnDEKbUQeIVljk4VP mvyTLKCOxGWswZkLGCX2I7pMvrJEXbfxCnHRvPg+LFFJow1VLOFQDzdiQOS6zdS7 lN1g5HJyAQ9gFH2rAN1e =p51q -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 00:18:49 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C7CA5108 for ; Fri, 27 Mar 2015 00:18:49 +0000 (UTC) Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by mx1.freebsd.org (Postfix) with ESMTP id 59A0B137 for ; Fri, 27 Mar 2015 00:18:48 +0000 (UTC) Received: from ppp121-45-104-240.lns20.adl6.internode.on.net (HELO midget.dons.net.au) ([121.45.104.240]) by ipmail05.adl6.internode.on.net with ESMTP; 27 Mar 2015 10:48:16 +1030 Received: from [10.0.2.26] ([10.0.2.26]) (authenticated bits=0) by midget.dons.net.au (8.15.1/8.14.9) with ESMTPSA id t2R0IAX2036049 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 27 Mar 2015 10:48:15 +1030 (CST) (envelope-from darius@dons.net.au) Subject: Re: Seagate Archive HDD Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Content-Type: text/plain; charset=windows-1252 From: "O'Connor, Daniel" In-Reply-To: <5514A133.8060409@delphij.net> Date: Fri, 27 Mar 2015 10:48:10 +1030 Content-Transfer-Encoding: quoted-printable Message-Id: <88DE4F68-B05E-4E4D-8C4A-DED8147172E7@dons.net.au> References: <55148E42.80708@delphij.net> <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> <5514A133.8060409@delphij.net> To: d@delphij.net X-Mailer: Apple Mail (2.2070.6) X-Spam-Score: -2.899 () ALL_TRUSTED,BAYES_00,URIBL_BLOCKED X-Scanned-By: MIMEDefang 2.75 on 10.0.2.1 Cc: freebsd-hackers@freebsd.org, Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 00:18:49 -0000 > On 27 Mar 2015, at 10:45, Xin Li wrote: > On 03/26/15 16:55, O'Connor, Daniel wrote: >>=20 >> The drive has many regions with guard bands so it doesn't have to >> rewrite the entire disk for certain writes (it may have to rewrite >> a whole band though) >=20 > That really depends on whether we can afford to lose the whole band of > (meta)data. I don't think it will lose a whole band, it journals writes, i.e. copies = the band to the journal then writes out the whole thing (modified). > That's is why I am not that optimistic about what we do nowadays, we > would need to revisit the on-disk formats with the new drives and see > if adjustments would be necessary. I agree, but it does look like they have gone to reasonable steps to = make it survive power issues. -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 00:58:14 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 64B30EA1 for ; Fri, 27 Mar 2015 00:58:14 +0000 (UTC) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E9A2E779 for ; Fri, 27 Mar 2015 00:58:13 +0000 (UTC) Received: by wibbg6 with SMTP id bg6so9218306wib.0 for ; Thu, 26 Mar 2015 17:58:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=AGvmacpMetNYCf4cVrrH0zscEAhz9ASnC8nhrTQc+TA=; b=o1RuFwwdfXC7AmXvAYw9yOE+HgWxBWN7hnwgGc52nPa4nVOvu/PVNtLgYelDc16kPA +UEpAsYFZ8KbEPzwrV5OZdCGrLH4bYFuCdFqOjBkiuoljXUsPwpWWxlotKAVdAr3QWFe QZ44jVkYd7BytJPd5Z4jS67tyt7RBrLc/zg5iIabCMuHtqp3GFAUVsQX+fkmBhlsqesg HZlS/sZ+s2WIglXMtoQ3xmA5czb7mya3razTNEVG95LBpH2rosy79BxKL80RrBw8Kf4f SWjk0Y5HcTQiOmnR2JaQbc0hg8wJg8mfz7Jb67/VygKrTwS++fuvVJ3u9NQ6qaExhrrr cpfw== MIME-Version: 1.0 X-Received: by 10.194.48.12 with SMTP id h12mr34072632wjn.74.1427417892375; Thu, 26 Mar 2015 17:58:12 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.180.40.14 with HTTP; Thu, 26 Mar 2015 17:58:12 -0700 (PDT) In-Reply-To: <88DE4F68-B05E-4E4D-8C4A-DED8147172E7@dons.net.au> References: <55148E42.80708@delphij.net> <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> <5514A133.8060409@delphij.net> <88DE4F68-B05E-4E4D-8C4A-DED8147172E7@dons.net.au> Date: Thu, 26 Mar 2015 17:58:12 -0700 X-Google-Sender-Auth: VWWYmYoWCIPLHFkfCYygEjUrMjs Message-ID: Subject: Re: Seagate Archive HDD From: Adrian Chadd To: "O'Connor, Daniel" Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , Xin LI , Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 00:58:14 -0000 ... all the talk about this stuff, and yet noone's talked about how ZFS is supposed to be append only, and boy wouldn't that be awesome. :) So - is anyone planning on working on the minimum set of stuff to expose the write topology for these shingled disks, so experiments can start being made? Even just knowing the shingle size(s) for writes would be enough to do useful amounts of work from userspace. -adrian From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 01:34:12 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 733F2778; Fri, 27 Mar 2015 01:34:12 +0000 (UTC) Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by mx1.freebsd.org (Postfix) with ESMTP id C7B80AD9; Fri, 27 Mar 2015 01:34:11 +0000 (UTC) Received: from ppp121-45-104-240.lns20.adl6.internode.on.net (HELO midget.dons.net.au) ([121.45.104.240]) by ipmail05.adl6.internode.on.net with ESMTP; 27 Mar 2015 12:03:54 +1030 Received: from [10.0.2.26] ([10.0.2.26]) (authenticated bits=0) by midget.dons.net.au (8.15.1/8.14.9) with ESMTPSA id t2R1XlJH084838 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 27 Mar 2015 12:03:53 +1030 (CST) (envelope-from darius@dons.net.au) Subject: Re: Seagate Archive HDD Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Content-Type: text/plain; charset=us-ascii From: "O'Connor, Daniel" In-Reply-To: Date: Fri, 27 Mar 2015 12:03:47 +1030 Content-Transfer-Encoding: quoted-printable Message-Id: <882C8337-923B-40CF-A4BE-69B20B0B3DE3@dons.net.au> References: <55148E42.80708@delphij.net> <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> <5514A133.8060409@delphij.net> <88DE4F68-B05E-4E4D-8C4A-DED8147172E7@dons.net.au> To: Adrian Chadd X-Mailer: Apple Mail (2.2070.6) X-Spam-Score: -2.9 () ALL_TRUSTED,BAYES_00 X-Scanned-By: MIMEDefang 2.75 on 10.0.2.1 Cc: "freebsd-hackers@freebsd.org" , Xin LI , Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 01:34:12 -0000 > On 27 Mar 2015, at 11:28, Adrian Chadd wrote: > So - is anyone planning on working on the minimum set of stuff to > expose the write topology for these shingled disks, so experiments can > start being made? >=20 > Even just knowing the shingle size(s) for writes would be enough to do > useful amounts of work from userspace. The presentation suggests it's all hidden and there's no public way to = find it out. So like the whole 4k farrago again :( -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 02:56:04 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E868FD9B for ; Fri, 27 Mar 2015 02:56:03 +0000 (UTC) Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com [IPv6:2a00:1450:400c:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7A73029A for ; Fri, 27 Mar 2015 02:56:03 +0000 (UTC) Received: by wibg7 with SMTP id g7so11139683wib.1 for ; Thu, 26 Mar 2015 19:56:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=49YYqO9P4qfq9ztyRhCMemcEGfxL1vX8+cNC6lKtCYM=; b=QVOHtc3ALu9RaZCQTyi1GGGFsDWiZrIjI84YAczh29w8R1euUpzUmQCv2mv7rKSYzp j82M2IasRL2l3MHvGi4yweY/arm9r1LIUFjovumpN9waZHbuNOA/YUGgZ5FtOXpLjlLa zIFgZvtsIHl6eieBwsUTJl9Ht7229HQa5Hltbpm7bhXWdUhCtWcYTmDVDlNSHRtK4guk dd9/g93Z4uyDK4mUK6CA6LgmozmcKG+3fqwrs5nja+g1twPL+ljPfG0k/LTF409QugRG iW/L1CVVR9mOiz6wJAUsIHnPH7BrZtKkToTcK2WEX4J5OCnW5zBI9EO5+M7I3OJd/8P6 bBhQ== MIME-Version: 1.0 X-Received: by 10.180.38.15 with SMTP id c15mr53069107wik.74.1427424960392; Thu, 26 Mar 2015 19:56:00 -0700 (PDT) Received: by 10.194.18.37 with HTTP; Thu, 26 Mar 2015 19:56:00 -0700 (PDT) In-Reply-To: <55149E70.30608@delphij.net> References: <20150319013231.GR51048@funkthat.com> <55149E70.30608@delphij.net> Date: Thu, 26 Mar 2015 23:56:00 -0300 Message-ID: Subject: Re: GELI support on /boot folder From: Pedro Arthur To: d@delphij.net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "" , John-Mark Gurney X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 02:56:04 -0000 I think that encrypting the boot folder will protect the boot configurations, kernel and kernel modules from being changed. > If we make changes to loader more often, it could be a bad idea > because merging both parties would make it harder for those who > develop loader changes. > > Additionally, it may be desirable to keep different copies of loaders > in different "boot environment" datasets, it's more convenient for > debugging: let's say one developer decided to make some changes to ZFS > support of loader, and that's installed to a new boot environment, > then they can try it out without making a usable boot disk at hand > before hand. Once the zfsloader is proven to be working (we still > have zfsloader.old or a different boot environment available), we > would have much more confident that the system will boot after a > gptzfsboot update because they share the same code. > > I agree with you, but the boot2 has already reached its size limit.For example if you try to compile the boot2 with clang < 3.5 (>=3.5 uses the enable-gvn flag) you will get an error saying boot2 exceeded its max size by ~20 bytes. I can't see other way to do it without merging. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 03:38:37 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 41A66C2B; Fri, 27 Mar 2015 03:38:37 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2111E9E3; Fri, 27 Mar 2015 03:38:37 +0000 (UTC) Received: from Xins-MBP.home.us.delphij.net (c-71-202-112-39.hsd1.ca.comcast.net [71.202.112.39]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id DD841EBCB; Thu, 26 Mar 2015 20:38:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1427427516; x=1427441916; bh=+d9p7mCRq+VZRQ+MtsZDx5ToWfBdgOx8jBSacGj5cK8=; h=Date:From:To:CC:Subject:References:In-Reply-To; b=JCrIE1BgKAIDxe/3qX0yvcT0VM/OF+9wuST6z4wyKESC4ik7NO8e5/z/Q6feBv2Fi FECz/2a1Y2y+SZPWh/5v0ZRdLBC8en1F+DwoXe5Bo9cinu+5g2udPqyGYxCEXJj7NG kvgyCnnX8QpBAyZtJWgTgRP2a7gMByoBIFYFLeSM= Message-ID: <5514D0A9.1040106@delphij.net> Date: Thu, 26 Mar 2015 20:38:17 -0700 From: Xin Li User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Adrian Chadd , "O'Connor, Daniel" Subject: Re: Seagate Archive HDD References: <55148E42.80708@delphij.net> <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> <5514A133.8060409@delphij.net> <88DE4F68-B05E-4E4D-8C4A-DED8147172E7@dons.net.au> In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Xin LI , Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 03:38:37 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 3/26/15 17:58, Adrian Chadd wrote: > ... all the talk about this stuff, and yet noone's talked about > how ZFS is supposed to be append only, and boy wouldn't that be > awesome. :) It doesn't overwrite existing data with the assumption that a, let's say, 4K-native drive doesn't have a 1MB "band" when writing. I hope the "journal" would be helpful for this assuming it is on non-volatile medium... Cheers, -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVFNCnAAoJEJW2GBstM+nsBQQQAIfR1JdSFYe+IsWDmnVDmKjy 35D0gsXKwNh37NTnvKiT2jzoM7BqwdJsQa6yWn+++ufzYkmMbcVHmlL52M8vk6ho UlJvkVONrYkZn2VVEZx01XulbX3hFq7DBCxTYspbBUdZcsZfiRQdMDPtsv+EW/ct nalBP3pHcHiK/CvgLVMELyQJZzMV1pb7gBaXSgRk7h0lyFhffIKwcD7qCt2cfV1W gbyYsQvC/envn9Yn5IDxOdddnyXihkDot/n/uX1kbUy5ZVfL+Wx1LidInwoc+/wY Jgh2zpXsNuwGTHodYj95unLhZnw177fy99n2w+7e8TuwPqZlLcqUxVvpnIn3d30e NyLeWW80yRjjVhWvl7zHQrWx2BCWuGHOvCExa3NuyXJWB5iJrn+izPUE6dXqkVzW YU1abrHNrMwJpLowm/Rjnlf9XWBdDhYUBIzibVXGhmdNKFcx7UslY3dWO4XckDdP hYmQVOAJrqK7cHNuzQCFrSKh4cVbo9t0gckD7lYJEk91BxAg+sBNt/iO9kR0AUvV 59BSBnremJAHNUQUDfJooWu2yKczGUASuhT20etKXPx/Y7qdhmA4i+McITwGZVJj tCIFUkysQocXY+XRX+6XkFd8j3MgzzkSzEvyqNdovczHACGRyVedNuyVmsGZ7ciL CUWJcAHv7ojKBXyDDlOa =e2je -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 03:44:23 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8AE0BD38 for ; Fri, 27 Mar 2015 03:44:23 +0000 (UTC) Received: from mx1.scaleengine.net (beauharnois2.bhs1.scaleengine.net [142.4.218.15]) by mx1.freebsd.org (Postfix) with ESMTP id 606F6A95 for ; Fri, 27 Mar 2015 03:44:22 +0000 (UTC) Received: from [192.168.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 6E164850F6 for ; Fri, 27 Mar 2015 03:44:21 +0000 (UTC) Message-ID: <5514D211.406@freebsd.org> Date: Thu, 26 Mar 2015 23:44:17 -0400 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: Seagate Archive HDD References: <55148E42.80708@delphij.net> <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> <5514A133.8060409@delphij.net> <88DE4F68-B05E-4E4D-8C4A-DED8147172E7@dons.net.au> <5514D0A9.1040106@delphij.net> In-Reply-To: <5514D0A9.1040106@delphij.net> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6Ghx9WbxclRj4PV7eBU6xaLLiXXSc1PId" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 03:44:23 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --6Ghx9WbxclRj4PV7eBU6xaLLiXXSc1PId Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015-03-26 23:38, Xin Li wrote: >=20 >=20 > On 3/26/15 17:58, Adrian Chadd wrote: >> ... all the talk about this stuff, and yet noone's talked about >> how ZFS is supposed to be append only, and boy wouldn't that be >> awesome. :) >=20 > It doesn't overwrite existing data with the assumption that a, let's > say, 4K-native drive doesn't have a 1MB "band" when writing. >=20 > I hope the "journal" would be helpful for this assuming it is on > non-volatile medium... >=20 > Cheers, > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.o= rg" >=20 Also, ZFS does write to the vdev label (256kb each, 2x at the front and 2x at the end of the drive) frequently (every transaction group, so at a minimum, every 5 seconds by default). The vdev label has a ring of uberblocks, that then point to the most recent metadata. If there was some special place on disk these could be located to avoid being part of a large band, and avoid the write amplification, that would likely make a big difference. --=20 Allan Jude --6Ghx9WbxclRj4PV7eBU6xaLLiXXSc1PId Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJVFNIVAAoJEJrBFpNRJZKfPDcP/i8h1Iu4vJO7JQ4orZ7TIspj BuzVZIfJlufQwRbYNIEy1ttaHsiQG9ISqwgL/9uyG39UpWvCvXnMlWiWtwiF8Psn ++y04DPgoaY68NsPpsslhiHqCBlV5sHdUrfBlRHSAABqOtPWbsjQH/u1xOaMhD9T 4nFtZFwvdikMS20P823W0sBa42uKBceg+wRaBlRnvuxes6GCTABmsCgjhDk7Xtii LVZg3wMh2kWScOGlHFzsQ8PsZfYaFdMnRQsyuowD3a5dkCyDt3ixDJwUjzYQ4B9N Kr8EzuvxKNkVRyF/+f9JpM1AxJKXl8SitdczTEhLFsFwIsihBW9PZ+MEcTwiO3e0 XnTCW/NsTnW7Vx9eaKtQaYN9p8RPlShrLbcwpoCmAZLc7T0RqZTXJojvt+6pVcz+ +MwTZpixgxIDTXCG2a45fOKriUiCE/4Hp034A4k8kIJ27qiThgCG4413wpOvd/hJ TsVOgHijAQHbVQWFzvSuPBbaSuFEBWyPskwxpZxaalQ14UkTClosattHpvMEae0L ap4V6AYo4PX269eoQjxOA1ASlpwM2ch5oBFiCE+7e5ejoNY1jdGDMl2eVmCyTJmf j432qTn5J+ciXX+IONIgCg5wY5cqd3RaA+qMFIb7tPQ49zrEyV3oa+MiDXIfKE2R EnozFMLCwV8lVzzoFOy/ =8p9X -----END PGP SIGNATURE----- --6Ghx9WbxclRj4PV7eBU6xaLLiXXSc1PId-- From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 03:46:38 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5FD05E1D; Fri, 27 Mar 2015 03:46:38 +0000 (UTC) Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by mx1.freebsd.org (Postfix) with ESMTP id B8E7BAA8; Fri, 27 Mar 2015 03:46:37 +0000 (UTC) Received: from ppp121-45-104-240.lns20.adl6.internode.on.net (HELO midget.dons.net.au) ([121.45.104.240]) by ipmail05.adl6.internode.on.net with ESMTP; 27 Mar 2015 14:12:41 +1030 Received: from [10.0.2.26] ([10.0.2.26]) (authenticated bits=0) by midget.dons.net.au (8.15.1/8.14.9) with ESMTPSA id t2R3gZGF069382 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 27 Mar 2015 14:12:40 +1030 (CST) (envelope-from darius@dons.net.au) Subject: Re: Seagate Archive HDD Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Content-Type: text/plain; charset=windows-1252 From: "O'Connor, Daniel" In-Reply-To: <5514D0A9.1040106@delphij.net> Date: Fri, 27 Mar 2015 14:12:34 +1030 Content-Transfer-Encoding: quoted-printable Message-Id: References: <55148E42.80708@delphij.net> <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> <5514A133.8060409@delphij.net> <88DE4F68-B05E-4E4D-8C4A-DED8147172E7@dons.net.au> <5514D0A9.1040106@delphij.net> To: Xin Li X-Mailer: Apple Mail (2.2070.6) X-Spam-Score: -2.899 () ALL_TRUSTED,BAYES_00,URIBL_BLOCKED X-Scanned-By: MIMEDefang 2.75 on 10.0.2.1 Cc: "freebsd-hackers@freebsd.org" , Adrian Chadd , Xin LI , Wojciech Puchar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 03:46:38 -0000 > On 27 Mar 2015, at 14:08, Xin Li wrote: > On 3/26/15 17:58, Adrian Chadd wrote: >> ... all the talk about this stuff, and yet noone's talked about >> how ZFS is supposed to be append only, and boy wouldn't that be >> awesome. :) >=20 > It doesn't overwrite existing data with the assumption that a, let's > say, 4K-native drive doesn't have a 1MB "band" when writing. >=20 > I hope the "journal" would be helpful for this assuming it is on > non-volatile medium... The journal for the existing drives appears to be on the actual platter = (but not shingled obviously). This gives from pretty suboptimal write latency as you can probably = imagine.. -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 08:32:36 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D117A37C for ; Fri, 27 Mar 2015 08:32:36 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 57C6096C for ; Fri, 27 Mar 2015 08:32:35 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2R8WUvX017059 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 27 Mar 2015 09:32:31 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2R8WR0i000718 for ; Fri, 27 Mar 2015 09:32:27 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2R8WM2W000715 for ; Fri, 27 Mar 2015 09:32:22 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Fri, 27 Mar 2015 09:32:22 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: freebsd-hackers@freebsd.org Subject: FreeBSD/cubieboard (to be exact - banana pi) Message-ID: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Fri, 27 Mar 2015 09:32:31 +0100 (CET) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 08:32:36 -0000 is there any progress on it? i found ahci, ethernet and mmc commented out in CUBIEBOARD2 in both FreeBSD-10 and current. NetBSD seems to support it completely but for many reasons i would prefer FreeBSD working on it. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 08:34:21 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1550B555; Fri, 27 Mar 2015 08:34:21 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8E66F983; Fri, 27 Mar 2015 08:34:20 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2R8YIvq017099 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Mar 2015 09:34:18 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2R8YFwE000729; Fri, 27 Mar 2015 09:34:15 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2R8YAi3000726; Fri, 27 Mar 2015 09:34:10 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Fri, 27 Mar 2015 09:34:10 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: Adrian Chadd Subject: Re: Seagate Archive HDD In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Fri, 27 Mar 2015 09:34:18 +0100 (CET) Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 08:34:21 -0000 > Hm, what's required to get the stripe size for doing IO or whatever > the under-the-hood thing is for the Sharding? > i found that SATA protocol got enhanced for OS to be able to know what sector ranges are for each stripe. So SMR optimized filesystem would certainly be nice. (but not ZFS please). If your workload is not very write intensive, you may just ignore it and allow hard disk to simulate things. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 08:34:46 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 61381637; Fri, 27 Mar 2015 08:34:46 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D62E198C; Fri, 27 Mar 2015 08:34:45 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2R8YhZi017115 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Mar 2015 09:34:43 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2R8YeFI000735; Fri, 27 Mar 2015 09:34:40 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2R8YZja000732; Fri, 27 Mar 2015 09:34:35 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Fri, 27 Mar 2015 09:34:35 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: Alan Somers Subject: Re: Seagate Archive HDD In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Fri, 27 Mar 2015 09:34:43 +0100 (CET) Cc: "freebsd-hackers@freebsd.org" , Adrian Chadd X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 08:34:46 -0000 > > My understanding is that it's impossible; the drive-managed SMR drives > present themselves as ordinary disks. But, if their design is similar > to the upcoming host-aware SMR drives, then the band size will be on > the order of a few hundred megabytes. > i think it is few tracks at most now. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 08:36:21 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0528A742 for ; Fri, 27 Mar 2015 08:36:21 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7F1869A7 for ; Fri, 27 Mar 2015 08:36:20 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2R8a55S017163 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Mar 2015 09:36:06 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2R8a3st000743; Fri, 27 Mar 2015 09:36:03 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2R8ZvJj000740; Fri, 27 Mar 2015 09:35:58 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Fri, 27 Mar 2015 09:35:57 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: d@delphij.net Subject: Re: Seagate Archive HDD In-Reply-To: <55148E42.80708@delphij.net> Message-ID: References: <55148E42.80708@delphij.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Fri, 27 Mar 2015 09:36:06 +0100 (CET) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 08:36:21 -0000 >> need of rewrite few full tracks worth of data, but otherwise will >> then be compatible and can i use it as usual? > > My understanding is that the SMR drives are actually different class > of storage device. > > The "drive managed" drives as shipped now tries to emulate normal hard > drive's behavior but they present unique risks: for instance, a > rewrite of a small block may end up in a read-modify-write of a much > larger area, so we must refrain from doing such operations for small part (like 10GB) of storage space is formatted without SMR and used as log. writes that doesn't fit full SMR stripe are first written to log then (after possibly very long delay and some coalescing) - read/modify/writted to normal space From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 08:36:49 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CA259828 for ; Fri, 27 Mar 2015 08:36:49 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DA2B9AF for ; Fri, 27 Mar 2015 08:36:48 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2R8akB5017190 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Mar 2015 09:36:47 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2R8aiwq000749; Fri, 27 Mar 2015 09:36:44 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2R8aceE000746; Fri, 27 Mar 2015 09:36:39 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Fri, 27 Mar 2015 09:36:38 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: Igor Mozolevsky Subject: Re: Seagate Archive HDD In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Fri, 27 Mar 2015 09:36:47 +0100 (CET) Cc: Hackers freeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 08:36:49 -0000 > > > I suspect so, but the drives are designed to be used as "tape-on-disk" backup (hence "archive"), i. e. long sequential writes. Using > these drives for anything else seems as insane as using the "video recording"-grade drives for data, in my opinion? what is a difference between "video recording" and normal SATA drives? except pricing of course. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 08:39:02 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 24E7591E; Fri, 27 Mar 2015 08:39:02 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9B55C9C2; Fri, 27 Mar 2015 08:39:01 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2R8cxiq017331 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Mar 2015 09:38:59 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2R8cuTr000757; Fri, 27 Mar 2015 09:38:56 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2R8cpC0000754; Fri, 27 Mar 2015 09:38:51 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Fri, 27 Mar 2015 09:38:51 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: Adrian Chadd Subject: Re: Seagate Archive HDD In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Fri, 27 Mar 2015 09:38:59 +0100 (CET) Cc: "freebsd-hackers@freebsd.org" , Alan Somers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 08:39:02 -0000 > > Does anyone have anything that we can expose to userland / GEOM? there are some SATA protocol extension proposed that would just show what are stripe ranges. I thing nandfs is the best starting point to implement filesystem on them. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 08:40:27 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 08959A13 for ; Fri, 27 Mar 2015 08:40:27 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 813ED9D1 for ; Fri, 27 Mar 2015 08:40:26 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2R8eB1Z017410 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Mar 2015 09:40:12 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2R8e81u000765; Fri, 27 Mar 2015 09:40:09 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2R8e32Y000762; Fri, 27 Mar 2015 09:40:03 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Fri, 27 Mar 2015 09:40:03 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: d@delphij.net Subject: Re: GELI support on /boot folder In-Reply-To: <55149D12.6070602@delphij.net> Message-ID: References: <55149D12.6070602@delphij.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Fri, 27 Mar 2015 09:40:12 +0100 (CET) Cc: "" , Pedro Arthur X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 08:40:27 -0000 >> in bootloader as a GSoC project, thus the /boot folder could be >> encrypted. > > What's the benefit of encrypting /boot? If it's encrypted, will the exactly none. > (Use passphrase only is a bad idea because that would mean we > essentially encrypt different data with the same key, if two encrypted > providers both use the same passphrase. This is probably not a big i use passphrase for root filesystem, put keyfiles generated from /dev/urandom on it and use for other filesystems. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 08:45:33 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0CCB52F9 for ; Fri, 27 Mar 2015 08:45:33 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 84CE3AD2 for ; Fri, 27 Mar 2015 08:45:32 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2R8j4vr017673 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Mar 2015 09:45:04 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2R8j1K0000815; Fri, 27 Mar 2015 09:45:01 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2R8isc0000812; Fri, 27 Mar 2015 09:44:54 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Fri, 27 Mar 2015 09:44:54 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: d@delphij.net Subject: Re: Seagate Archive HDD In-Reply-To: <5514A133.8060409@delphij.net> Message-ID: References: <55148E42.80708@delphij.net> <4F65B315-5FFE-4184-91FD-C05A40E0A26E@dons.net.au> <5514A133.8060409@delphij.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Fri, 27 Mar 2015 09:45:04 +0100 (CET) Cc: freebsd-hackers@freebsd.org, "O'Connor, Daniel" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 08:45:33 -0000 to support this drives really efficiently you need 1) new filesystem - to use it as usual drive 2) replacement for rsync or other backup tools - to write new data sequentially when doing backups, and other tool to (from time to time) select data to be removed to free space, and reorganizing disk. Written data can have geom_uzip&cd9660 (or ffs) compatible layout to be readable by just anything. I maybe would do some work on 2 as it will be my basic usage for it - incremental backup storage. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 08:57:32 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 48FA86C8 for ; Fri, 27 Mar 2015 08:57:32 +0000 (UTC) Received: from mail-qc0-x233.google.com (mail-qc0-x233.google.com [IPv6:2607:f8b0:400d:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F0302C83 for ; Fri, 27 Mar 2015 08:57:31 +0000 (UTC) Received: by qcbjx9 with SMTP id jx9so11222044qcb.0 for ; Fri, 27 Mar 2015 01:57:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=TUf6HzjE3jCCN4dAMAeBBRWMI94NO2S/6th9uqBLqo4=; b=XuuOYgBY8v1qJ/6u9GpTBfmLYFnz/u7eFHRzcOlaTUvYQ7qI9WPTeOZEdRXgrPyNNM L4ZATBCmQ5w/gDT0cb6WwIWrjYiu4Op6hlkZitP73pSUBuWLmMt4cxWmWbzEHH//kmSw izhQA13NlEqSePWGDsrzJ7D2wYYg/a6yMem8uXgRCnbyR+9goAQwnjaNG8irBXtmUVCQ zN32EMUtfZfo8ldzUaMYfBcsTFeufHlkC+i/jOOq8K197IwC9raGMWza3AcE2HmSQEhh 0d6zm5mKXXSiwj9utK5s1XuJTtua/eRTHfolHhXK+tQwv1l5R7j7OBgl5NbKJzgLqG9O Xs0Q== X-Received: by 10.55.22.32 with SMTP id g32mr36943987qkh.4.1427446651189; Fri, 27 Mar 2015 01:57:31 -0700 (PDT) MIME-Version: 1.0 Sender: mozolevsky@gmail.com Received: by 10.140.91.112 with HTTP; Fri, 27 Mar 2015 01:56:50 -0700 (PDT) In-Reply-To: References: From: Igor Mozolevsky Date: Fri, 27 Mar 2015 08:56:50 +0000 X-Google-Sender-Auth: nka0yGcpeZ2A_8b9NSwUWI1o1aY Message-ID: Subject: Re: Seagate Archive HDD To: Wojciech Puchar Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Hackers freeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 08:57:32 -0000 On 27 March 2015 at 08:36, Wojciech Puchar wrote: > >> >> I suspect so, but the drives are designed to be used as "tape-on-disk" >> backup (hence "archive"), i. e. long sequential writes. Using >> these drives for anything else seems as insane as using the "video >> recording"-grade drives for data, in my opinion? >> > > what is a difference between "video recording" and normal SATA drives? > except pricing of course. > The "video recording" HDDs have no, among other things, internal "long recovery" mechanisms (hence the price) because unlike "data", "video" doesn't really care if small part of a frame gets corrupted on disk=E2=80= =A6 --=20 Igor M. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 09:00:45 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4D291C37 for ; Fri, 27 Mar 2015 09:00:45 +0000 (UTC) Received: from mail-oi0-x229.google.com (mail-oi0-x229.google.com [IPv6:2607:f8b0:4003:c06::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0D04ECB2 for ; Fri, 27 Mar 2015 09:00:45 +0000 (UTC) Received: by oiag65 with SMTP id g65so70745820oia.2 for ; Fri, 27 Mar 2015 02:00:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=13BLGTUDrTK208FUYKeVG+V+3iVo1Z3w3Xh/5NeW+sM=; b=PQCkSnt1Y6Rb7cdoCFZa/EqAqYVLDV1ecMvV6YL8XgSjOBC8hPNzBtM/uSooCYUAFl A2Siy5uXiqLGSJtFhjwYAbl4SOE306EVHSqntC5T/7++mZTjoo63ih7TMwJbT8P9UGEf OxHfBAKCj9YMoTD0vYqjMR21YDM3KKeXIWT75OIZtugUbHrj/1OsvIj4HajcHtwWotkf V0Os4CnnDM1y2pKsP5Q6uyO0k5i2SxgYLsBQgueZ1tm6NgYsDUM54R+ymff8Ggmb0eyk kZ7mbpJC79izkHxpLQKPPPw3eHkooGaXnQOyhDCBwhaVZ6PnY55U0Xfg5m9Tc3dx0BHF 0DHg== MIME-Version: 1.0 X-Received: by 10.182.213.38 with SMTP id np6mr15288209obc.34.1427446844369; Fri, 27 Mar 2015 02:00:44 -0700 (PDT) Received: by 10.182.204.9 with HTTP; Fri, 27 Mar 2015 02:00:44 -0700 (PDT) In-Reply-To: References: Date: Fri, 27 Mar 2015 17:00:44 +0800 Message-ID: Subject: Re: FreeBSD/cubieboard (to be exact - banana pi) From: Ganbold Tsagaankhuu To: Wojciech Puchar Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 09:00:45 -0000 On Fri, Mar 27, 2015 at 4:32 PM, Wojciech Puchar wrote: > is there any progress on it? > > i found ahci, ethernet and mmc commented out in CUBIEBOARD2 in both > FreeBSD-10 and current. > ahci and mmc drivers exist in someway, but iirc those are not tested well yet. As for dwc (gmac) driver someone has to make existing dwc driver work for BananaPI. Ganbold > NetBSD seems to support it completely but for many reasons i would prefer > FreeBSD working on it. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 09:55:24 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1B26D77C for ; Fri, 27 Mar 2015 09:55:24 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A5553DC for ; Fri, 27 Mar 2015 09:55:23 +0000 (UTC) Received: from scdbackup.webframe.org ([79.192.89.235]) by mail.gmx.com (mrgmx103) with ESMTPSA (Nemesis) id 0MRGvT-1Z3EsZ1puR-00Ue03; Fri, 27 Mar 2015 10:55:20 +0100 Date: Fri, 27 Mar 2015 10:55:02 +0100 From: "Thomas Schmitt" To: freebsd-hackers@freebsd.org Subject: Re: Seagate Archive HDD Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit References: In-Reply-To: Message-Id: <24923565183434755291@scdbackup.webframe.org> X-Provags-ID: V03:K0:DxbGL0xjlo0EqP6Kpmyfa5EsuiOilGHNciEHUcyaogXwmRPkwJs VNfmBNuW+npYApoDHNDQDr4DBN1SKnwS7t8EVHkgDzKNHcnVnF33fpFoLF5ahrpHd6UAzuF CMMWTbIvQw5uFeB+8cMMgJ8noqr8hOOY/Nb0qviFkE3r+SdPrQv6U2arQo5PZZFawgav9he mf+v343KF9bz9ulF8PnQQ== X-UI-Out-Filterresults: notjunk:1; Cc: wojtek@puchar.net X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 09:55:24 -0000 Hi, Wojciech Puchar wrote: > 2) replacement for rsync or other backup tools [...] > Written data can have geom_uzip&cd9660 (or ffs) compatible > layout to be readable by just anything. The visible hardware constraints resemble those of BD-RE or DVD-RAM media (but on a much larger scale, of course). So multi-session ISO 9660 + Rock Ridge would be well suitable. A write run consists of one sequential tree-and-data write and one small update write of the PVD (superblock). One would either use the raw device resp. a partition or write the ISO into some data file of a file system. My own ISO 9660 backup program xorriso would bail out at 4 TiB, because of some signed 32 bit integers in the APIs which it uses. One would have to organize the backup into several smaller areas. Or one could simply make a new base backup when the file with the ISO filesystem grows near to 4 TiB. Incremental ISO 9660 works by adding a new file tree and the whole data content of changed files. So the storage efficiency is worse than rsync, especially with large data files which change frequently. On the other hand ISO 9660 multi-session by xorriso allows to retrieve the older backup states by mount_cd9660 option -s. xorriso can tell the necessary block number. But the "cd9660" reader in FreeBSD is very restricted in respect to location of directory trees (not above 4 GiB - 1) and to file size (not more than 4 GiB - 1 bytes). The tree restriction is because the byte location of directory records is used as 32 bit inode number which then is used again to address the directory record. The file size restriction is because multiple data extents per file are not supported. NetBSD offers an escape path from the tree problem by its generous 64-bit inode numbers. Linux obviously does not use the inode number for addressing purposes and thus can afford to work with byte address divided by 32. This results in a 128 GiB window for trees, where inode numbers are unique. Not a totally general solution, but sufficient for any imaginable real purpose. You can put a lot of file names and meta data into 128 GiB. xorriso can retrieve what it wrote. So if the inconvenience of a random-access archive backup is acceptable, then 4 TiB filesystem size is supposed to be the only restriction. Have a nice day :) Thomas From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 11:06:02 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0B7B9FCE for ; Fri, 27 Mar 2015 11:06:02 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ABFAFF20 for ; Fri, 27 Mar 2015 11:06:01 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3lD0jK1w1zz1PP for ; Fri, 27 Mar 2015 12:05:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1427454354; x=1430046355; bh=BVE foLeEV4TdjD3/FL4dv4m5jW6W+nWCPz4hVsW290c=; b=TbRkOEtjGfu/7D0BB9l ufb+fi1MV96YSs3zBPu4sRBNyfr6BziQCxuaLzc9WE6MMrCcxg8vteZ9FeGDCbQh cTameZPEaJ7A2R5BfgDALQ+iDFV/rMCEfoZIcNGhm+mxtHKqWB0PDsjpXVN+Qmm8 8iaSS8tI0n1aDjhIT/YZyi7w= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id acsTSoYnmN0l for ; Fri, 27 Mar 2015 12:05:54 +0100 (CET) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP for ; Fri, 27 Mar 2015 12:05:53 +0100 (CET) Received: from neli.ijs.si (neli.ijs.si [IPv6:2001:1470:ff80:88:21c:c0ff:feb1:8c91]) by mildred.ijs.si (Postfix) with ESMTP id 3lD0jF318zzF6 for ; Fri, 27 Mar 2015 12:05:53 +0100 (CET) Received: from sleepy.ijs.si ([2001:1470:ff80:e001::1:1]) by neli.ijs.si with HTTP (HTTP/1.1 POST); Fri, 27 Mar 2015 12:05:53 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Date: Fri, 27 Mar 2015 12:05:53 +0100 From: Mark Martinec To: Hackers freeBSD Subject: Re: Seagate Archive HDD Organization: J. Stefan Institute In-Reply-To: References: Message-ID: X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.1.0 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 11:06:02 -0000 2015-03-27 09:56, Igor Mozolevsky wrote: > On 27 March 2015 at 08:36, Wojciech Puchar wrote: >> what is a difference between "video recording" and normal SATA drives? >> except pricing of course. >=20 > The "video recording" HDDs have no, among other things, internal "long > recovery" mechanisms (hence the price) because unlike "data", "video" > doesn't really care if small part of a frame gets corrupted on disk=E2=80= =A6 AV disks support ATA streaming command set, are designed to last in high temperature always-on streaming digital AV environments, are silent, with Preemptive Wear Leveling (PWL) (the drive arm frequently sweeps across the disk to reduce uneven wear on the drive surface common to audio video streaming applications) (paraphrased from WD docs) Mark From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 11:59:15 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D4AE3C26 for ; Fri, 27 Mar 2015 11:59:15 +0000 (UTC) Received: from mail-qc0-x233.google.com (mail-qc0-x233.google.com [IPv6:2607:f8b0:400d:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B71374E for ; Fri, 27 Mar 2015 11:59:15 +0000 (UTC) Received: by qcbjx9 with SMTP id jx9so14023324qcb.0 for ; Fri, 27 Mar 2015 04:59:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=uOPrhCGoyX+y/WWn5U8SOmO6TnbNnClpDLH/5nAdyss=; b=QGJveNk9uPMD3l7RZ5mMb7ddzvEDJwIfW61ektxfsVwSD5+S0z+iCMf3yxTxNstQY0 hv/Wfzbu3gLIwxCJcowyc6XkyIJzooqXXF5XvERrQC6W5e6Q5T8dR4o/z591gT8U4whz FjZZVMEO55GwXSNFBcsNnmtvKxkU44+uVMj29ZnWCIvjME7utm2UaosBvpnoNtgiMPkb dKWID8YOdqdYuK0IZzZLs5VmOM9ngtqyQtzlB4A43ytBKAnX3SUnnVU/MbsmBARsuKBC MwxN+cTQ1fsN716GaVV2CLi1mYCRp5MeOyX6YPG31yiv9ClPTFDpACZG24yCOomGPDJ7 w/aw== X-Received: by 10.55.42.85 with SMTP id q82mr2118946qkh.48.1427457554755; Fri, 27 Mar 2015 04:59:14 -0700 (PDT) MIME-Version: 1.0 Sender: mozolevsky@gmail.com Received: by 10.140.91.112 with HTTP; Fri, 27 Mar 2015 04:58:34 -0700 (PDT) In-Reply-To: References: From: Igor Mozolevsky Date: Fri, 27 Mar 2015 11:58:34 +0000 X-Google-Sender-Auth: 7kFE3hGcHq4E2yETPyfsNaPyi8E Message-ID: Subject: Re: Seagate Archive HDD To: Mark Martinec Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Hackers freeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 11:59:16 -0000 On 27 March 2015 at 11:05, Mark Martinec wrote: > 2015-03-27 09:56, Igor Mozolevsky wrote: > >> On 27 March 2015 at 08:36, Wojciech Puchar wrote: >> >>> what is a difference between "video recording" and normal SATA drives? >>> except pricing of course. >>> >> >> The "video recording" HDDs have no, among other things, internal "long >> recovery" mechanisms (hence the price) because unlike "data", "video" >> doesn't really care if small part of a frame gets corrupted on disk=E2= =80=A6 >> > > > AV disks support ATA streaming command set, are designed to last > in high temperature always-on streaming digital AV environments, are > silent, with Preemptive Wear Leveling (PWL) (the drive arm frequently > sweeps across the disk to reduce uneven wear on the drive surface > common to audio video streaming applications) > > (paraphrased from WD docs) I was saying why the AV disks are not a good idea for general purpose data storage, not what the general idea was=E2=80=A6 HSGT have a whitepaper "for the masses" on the topic: "=E2=80=A6 In AV applications, it may be better to have some small segment of incorrect data delivered in the stream than to have a long delay. Short delays may result in the loss of only a few pixels*. A long delay in the data stream would result in the loss of a larger block of data, which would be noticeable to a viewer. A new Streaming Command Set has been developed for ATA drives, which allows AV products to change drive behavior to meet AV system requirements=E2=80=A6" [1] 1. http://www.hgst.com/tech/techlib.nsf/techdocs/FEF3B52BFE9A054586256E66005AA= 389/$file/WP_AV_25March.pdf --=20 Igor M. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 12:41:03 2015 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9FB6B996 for ; Fri, 27 Mar 2015 12:41:03 +0000 (UTC) Received: from mail-wg0-x230.google.com (mail-wg0-x230.google.com [IPv6:2a00:1450:400c:c00::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3148DCAC for ; Fri, 27 Mar 2015 12:41:03 +0000 (UTC) Received: by wgdm6 with SMTP id m6so98051185wgd.2 for ; Fri, 27 Mar 2015 05:41:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:subject:message-id:mime-version:content-type :content-disposition:user-agent; bh=xz6/kTZ1RUCplMEWBGrF4DisF+CBHwEQOLAWbBXAp/I=; b=cozT639RhVABnRrxLfVUPpUfFuAaMTXCuK6MNQhLG0IuNcb+d6CC3xFsIOnJGmJAwu nCZPQpKMFxb/x04lq8e2xRCniGruuY6d6GbjkLiNBX1EXhEU+R6ddPKMR+Vwi3O1cZrs AEiFAcTZO6/9VlyxhXAwLsWAK1GCfxMQJncpZIFlqtFqb5g+9f7ljriFSdOUEIN8uSqz S5WZeB7aI++8tHWnF/BGdAlmneh5/0TfaTDBd+z5jRrQ8BUjr1l27ctJKe1MVOhh777P 1Z4euzv32voizCOmPPakL9zNux+IcRF4L7LzsPU5aBjiEzRFWMlTOhnJn4QQi9b1JwsD N8hQ== X-Received: by 10.194.21.193 with SMTP id x1mr36882912wje.144.1427460061459; Fri, 27 Mar 2015 05:41:01 -0700 (PDT) Received: from brick.home (acys181.neoplus.adsl.tpnet.pl. [83.11.202.181]) by mx.google.com with ESMTPSA id i10sm2670777wja.40.2015.03.27.05.41.00 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Mar 2015 05:41:00 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Date: Fri, 27 Mar 2015 13:40:58 +0100 From: Edward Tomasz =?utf-8?Q?Napiera=C5=82a?= To: hackers@FreeBSD.org Subject: Linux core under FreeBSD. Message-ID: <20150327124058.GA3991@brick.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 12:41:03 -0000 Is there a way to debug a core generated by Linux process running under Linuxulator on 11-CURRENT? Thanks! From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 13:27:01 2015 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5BBB8BE7 for ; Fri, 27 Mar 2015 13:27:01 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F15D0182 for ; Fri, 27 Mar 2015 13:27:00 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t2RDQsvK010264 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 27 Mar 2015 15:26:54 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t2RDQsvK010264 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t2RDQsYP010263; Fri, 27 Mar 2015 15:26:54 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 27 Mar 2015 15:26:54 +0200 From: Konstantin Belousov To: Jilles Tjoelker Subject: Re: kevent behavior Message-ID: <20150327132654.GJ2379@kib.kiev.ua> References: <550A6DA2.1070004@gmail.com> <20150324221541.GA67584@stack.nl> <20150325090041.GY2379@kib.kiev.ua> <20150325223530.GA79065@stack.nl> <20150326214551.GG2379@kib.kiev.ua> <20150326225826.GA97319@stack.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150326225826.GA97319@stack.nl> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-hackers@FreeBSD.org, Ivan Radovanovic X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 13:27:01 -0000 On Thu, Mar 26, 2015 at 11:58:27PM +0100, Jilles Tjoelker wrote: > On Thu, Mar 26, 2015 at 11:45:51PM +0200, Konstantin Belousov wrote: > > On Wed, Mar 25, 2015 at 11:35:30PM +0100, Jilles Tjoelker wrote: > > > > POSIX does not say anything about kevent(), including whether it should > > > be a cancellation point or not. Given that kevent() may block for a long > > > time, making it a cancellation point seems to make sense. > > > But POSIX language is very explicit for its own set of interfaces, and > > that makes sense. This is why I think that cancellability, after the > > 15+ years of kqueue existence, should be opt in. > > Hmm, I think most code written is cancel-unsafe. It is unlikely to have > cancel-safe code using kevent() and relying on it not being a > cancellation point, but still possible. Ok, I re-considered my opinion after re-reading your responses. > > This also means we shouldn't wait too long with adding missing > cancellation points like ppoll(). > > > > The kevent() cancellation point implementation would be much like most > > > other cancellation points such as pselect(), with the difference that > > > the call may have committed even if it fails, in the case that > > > nchanges > nevents and in the case that nchanges > 0 && errno == EINTR. > > > If cancellation is requested while blocking in the latter case, libthr > > > will have to return -1/EINTR to indicate to the application that the > > > changes were applied successfully while allowing the thread to be > > > cancelled at the next cancellation point, even though there may not be > > > any signal handler. I do not quite understand this. If any error (except things like EFAULT) occured while processing the changelist, kevent(2) returns error count, and not the -1, regarless of the length of eventlist. In this case, we do not cancel. Syscall cannot return n/EINTR, where n >= 0. If -1/EINTR is returned, this means that the call blocked and sleep was interrupted. The changes from the changelist were applied, do you suggested that we must not cancel if nchanges > 0 ? > > > > If nevents == 0, kevent() does not block and need not be a cancellation > > > point. This special case may simplify things slightly for application > > > programmers. > > > > A kqueue() flag for cancellation does not seem very useful: since such a > > > flag would be a kernel thing and pthread cancellation is a libthr thing, > > > requesting the flag from the kernel would slow down all kevent() calls. > > > Yes, I thought about adding some (small) userspace table which would > > track cancellable kqueues. > > I think the interaction with dup() and similar calls and with exec makes > this too nasty. Hm, yes. > > > But if we make the cancellation state per-call, and not a state for kqueue > > descriptor, we might indicate the cancellability by some event type, which > > is not passed to the kernel at all. The libthr wrapper for kevent(2) would > > need to scan the changelist and zero-out the cancellation indicator. > > This seems a bit hackish. However, enabling cancellation per-call seems > better than per-kqueue. Patch below always considers kevent(2) as cancellable, unless nevents == 0. I will correct the call to thr_cancel_leave() if I misunderstood you. diff --git a/lib/libc/include/libc_private.h b/lib/libc/include/libc_private.h index e4bf4a6..ceaa2a3 100644 --- a/lib/libc/include/libc_private.h +++ b/lib/libc/include/libc_private.h @@ -221,6 +221,7 @@ enum { INTERPOS__pthread_mutex_init_calloc_cb, INTERPOS_spinlock, INTERPOS_spinunlock, + INTERPOS_kevent, INTERPOS_MAX }; @@ -293,6 +294,7 @@ void * __sys_freebsd6_mmap(void *, __size_t, int, int, int, int, __off_t); struct aiocb; struct fd_set; struct iovec; +struct kevent; struct msghdr; struct pollfd; struct rusage; @@ -315,6 +317,8 @@ int __sys_fsync(int); __pid_t __sys_fork(void); int __sys_ftruncate(int, __off_t); int __sys_gettimeofday(struct timeval *, struct timezone *); +int __sys_kevent(int, const struct kevent *, int, struct kevent *, + int, const struct timespec *); __off_t __sys_lseek(int, __off_t, int); void *__sys_mmap(void *, __size_t, int, int, int, __off_t); int __sys_msync(void *, __size_t, int); diff --git a/lib/libc/sys/Makefile.inc b/lib/libc/sys/Makefile.inc index 0edf644b..7745b2a 100644 --- a/lib/libc/sys/Makefile.inc +++ b/lib/libc/sys/Makefile.inc @@ -51,6 +51,7 @@ INTERPOSED = \ fcntl \ fsync \ fork \ + kevent \ msync \ nanosleep \ open \ diff --git a/lib/libc/sys/interposing_table.c b/lib/libc/sys/interposing_table.c index 0fd6c75..4290bc6 100644 --- a/lib/libc/sys/interposing_table.c +++ b/lib/libc/sys/interposing_table.c @@ -75,6 +75,7 @@ interpos_func_t __libc_interposing[INTERPOS_MAX] = { SLOT(_pthread_mutex_init_calloc_cb, _pthread_mutex_init_calloc_cb_stub), SLOT(spinlock, __libc_spinlock_stub), SLOT(spinunlock, __libc_spinunlock_stub), + SLOT(kevent, __sys_kevent), }; #undef SLOT diff --git a/lib/libc/sys/kevent.c b/lib/libc/sys/kevent.c new file mode 100644 index 0000000..5f84ef8 --- /dev/null +++ b/lib/libc/sys/kevent.c @@ -0,0 +1,53 @@ +/* + * Copyright (c) 2015 The FreeBSD Foundation. + * All rights reserved. + * + * Portions of this software were developed by Konstantin Belousov + * under sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice(s), this list of conditions and the following disclaimer as + * the first lines of this file unmodified other than the possible + * addition of one or more copyright notices. + * 2. Redistributions in binary form must reproduce the above copyright + * notice(s), this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, + * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE + * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, + * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include "libc_private.h" + +__weak_reference(__sys_kevent, __kevent); + +#pragma weak kevent +int +kevent(int kq, const struct kevent *changelist, int nchanges, + struct kevent *eventlist, int nevents, const struct timespec *timeout) +{ + + return (((int (*)(int, const struct kevent *, int, + struct kevent *, int, const struct timespec *)) + __libc_interposing[INTERPOS_kevent])(kq, changelist, nchanges, + eventlist, nevents, timeout)); +} diff --git a/lib/libc/sys/kqueue.2 b/lib/libc/sys/kqueue.2 index 93223f1..fa76c54 100644 --- a/lib/libc/sys/kqueue.2 +++ b/lib/libc/sys/kqueue.2 @@ -24,7 +24,7 @@ .\" .\" $FreeBSD$ .\" -.Dd July 18, 2014 +.Dd March 27, 2015 .Dt KQUEUE 2 .Os .Sh NAME @@ -41,7 +41,7 @@ .Fn kqueue "void" .Ft int .Fn kevent "int kq" "const struct kevent *changelist" "int nchanges" "struct kevent *eventlist" "int nevents" "const struct timespec *timeout" -.Fn EV_SET "&kev" ident filter flags fflags data udata +.Fn EV_SET "kev" ident filter flags fflags data udata .Sh DESCRIPTION The .Fn kqueue @@ -550,6 +550,16 @@ On return, .Va fflags contains the users defined flags in the lower 24 bits. .El +.Sh CANCELLATION BEHAVIOUR +If +.Fa nevents +is non-zero, i.e. the function is potentially blocking, the call +is the cancellation point. +Otherwise, i.e. if +.Fa nevents +is zero, the call is not cancellable. +Cancellation can only occur when the call was blocked and no changes +to the queue were requested. .Sh RETURN VALUES The .Fn kqueue diff --git a/lib/libthr/thread/thr_syscalls.c b/lib/libthr/thread/thr_syscalls.c index 10fbad4..e71bf4a 100644 --- a/lib/libthr/thread/thr_syscalls.c +++ b/lib/libthr/thread/thr_syscalls.c @@ -341,6 +341,29 @@ __thr_pselect(int count, fd_set *rfds, fd_set *wfds, fd_set *efds, return (ret); } +static int +__thr_kevent(int kq, const struct kevent *changelist, int nchanges, + struct kevent *eventlist, int nevents, const struct timespec *timeout) +{ + struct pthread *curthread; + int ret; + + if (nevents == 0) { + /* + * No blocking, do not make the call cancellable. + */ + return (__sys_kevent(kq, changelist, nchanges, eventlist, + nevents, timeout)); + } + curthread = _get_curthread(); + _thr_cancel_enter(curthread); + ret = __sys_kevent(kq, changelist, nchanges, eventlist, nevents, + timeout); + _thr_cancel_leave(curthread, ret == -1 && nchanges == 0); + + return (ret); +} + /* * Cancellation behavior: * Thread may be canceled at start, but if the system call got some data, @@ -599,6 +622,7 @@ __thr_interpose_libc(void) SLOT(writev); SLOT(spinlock); SLOT(spinunlock); + SLOT(kevent); #undef SLOT *(__libc_interposing_slot( INTERPOS__pthread_mutex_init_calloc_cb)) = From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 14:20:04 2015 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D7C51D6E for ; Fri, 27 Mar 2015 14:20:04 +0000 (UTC) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E317A74 for ; Fri, 27 Mar 2015 14:20:04 +0000 (UTC) Received: by wibbg6 with SMTP id bg6so28436740wib.0 for ; Fri, 27 Mar 2015 07:20:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:subject:message-id:mime-version:content-type :content-transfer-encoding; bh=vM4GfTI8CG3AttL75eaC82cfFse+95SzG689cSPt7eE=; b=IbRJ1Un/pCxUp0OJ1eLnFBBQV7ZfgtwpPboV2JZa6cmogUQBn6FlBmq/XbcojwqvVQ 7CnQBCxhe9+ZQUC4srOBKnVwX8AgwfeNtqCVRzN+Uz8OPibBmLxBbErhx29lhr7pRlnw OWSaa5C4WWyJIMyE7AX9lOUtKpuhaNE3K2h7zmKMPacw7upEvQJvYHYdWC1glXujcgXc IIavuRDIK0jH/FzgnWhBSz2t09EcLkrq3t3FUml9zLGRBeUkckV/qQKx4KX7u6A5EFN3 MCqfSCznNgFgkKWJYVI2t+WT3qxgIhe2cqZubcPmVHH64V4y3gMo53MYnSGqmsYFBtot xcRA== X-Received: by 10.194.63.172 with SMTP id h12mr37710962wjs.48.1427466002926; Fri, 27 Mar 2015 07:20:02 -0700 (PDT) Received: from localhost ([217.14.212.217]) by mx.google.com with ESMTPSA id wo10sm3041610wjb.35.2015.03.27.07.20.01 for (version=SSLv3 cipher=RC4-SHA bits=128/128); Fri, 27 Mar 2015 07:20:02 -0700 (PDT) Date: Fri, 27 Mar 2015 15:20:00 +0100 From: To: hackers@FreeBSD.org Subject: ports-mgmt/pkg (still) fails to install in DESTDIR Message-ID: <20150327152000.000015e6@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 14:20:04 -0000 # make -C /usr/ports/ports-mgmt/pkg DESTDIR=3D/poligon/freebsd install clean Fails at: ------------------ =3D=3D=3D=3D> Compressing man pages (compress-man) =3D=3D=3D> Installing for pkg-1.4.12 =3D=3D=3D> Checking if pkg already installed =3D=3D=3D> Registering installation for pkg-1.4.12 pkg-static: Cannot open "/var/run/ld-elf.so.hints": No such file or directo= ry *** Error code 1 Stop. make[1]: stopped in /tmp/mountpoint.w4WP0q/ports-mgmt/pkg *** Error code 1 Stop. make: stopped in /tmp/mountpoint.w4WP0q/ports-mgmt/pkg =3D=3D=3D> Chrooted make in /poligon/freebsd failed =3D=3D=3D> Cleaning up... *** Error code 1 Stop. make: stopped in /usr/ports/ports-mgmt/pkg ------------------ I've found this, but it was 2 years ago and still not implemented/fixed: http://lists.freebsd.org/pipermail/svn-src-stable-10/2013-December/0004= 95.html My fix vas to chroot in DESTDIR and simply issue: # pkg -v Which started bootstrapp ... Domagoj Smol=C4=8Di=C4=87 From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 14:26:57 2015 Return-Path: Delivered-To: hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3531DAB for ; Fri, 27 Mar 2015 14:26:57 +0000 (UTC) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.fagskolen.gjovik.no", Issuer "Fagskolen i Gj??vik" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9B82FBDE for ; Fri, 27 Mar 2015 14:26:55 +0000 (UTC) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.14.9/8.14.9) with ESMTP id t2REQoBv095769 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Mar 2015 15:26:50 +0100 (CET) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.14.9/8.14.9/Submit) with ESMTP id t2REQnHM095766; Fri, 27 Mar 2015 15:26:50 +0100 (CET) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Fri, 27 Mar 2015 15:26:49 +0100 (CET) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: rank1seeker@gmail.com Subject: Re: ports-mgmt/pkg (still) fails to install in DESTDIR In-Reply-To: <20150327152000.000015e6@gmail.com> Message-ID: References: <20150327152000.000015e6@gmail.com> User-Agent: Alpine 2.11 (BSF 23 2013-08-11) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, AWL autolearn=ham autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mail.fig.ol.no Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 14:26:57 -0000 On Fri, 27 Mar 2015 15:20+0100, rank1seeker@gmail.com wrote: > # make -C /usr/ports/ports-mgmt/pkg DESTDIR=/poligon/freebsd install clean > Fails at: > ------------------ > ====> Compressing man pages (compress-man) > ===> Installing for pkg-1.4.12 > ===> Checking if pkg already installed > ===> Registering installation for pkg-1.4.12 > pkg-static: Cannot open "/var/run/ld-elf.so.hints": No such file or directory I've seen this from time to time. I resolved it in two steps: rm /var/run/ld-elf*.so.hints /etc/rc.d/ldconfig start In your case, this should be done from within the chroot env. > *** Error code 1 > > Stop. > make[1]: stopped in /tmp/mountpoint.w4WP0q/ports-mgmt/pkg > *** Error code 1 > > Stop. > make: stopped in /tmp/mountpoint.w4WP0q/ports-mgmt/pkg > ===> Chrooted make in /poligon/freebsd failed > ===> Cleaning up... > *** Error code 1 > > Stop. > make: stopped in /usr/ports/ports-mgmt/pkg > ------------------ > > I've found this, but it was 2 years ago and still not implemented/fixed: > http://lists.freebsd.org/pipermail/svn-src-stable-10/2013-December/000495.html > > My fix vas to chroot in DESTDIR and simply issue: > # pkg -v > Which started bootstrapp ... -- +-------------------------------+------------------------------------+ | Vennlig hilsen, | Best regards, | | Trond Endrestøl, | Trond Endrestøl, | | IT-ansvarlig, | System administrator, | | Fagskolen Innlandet, | Gjøvik Technical College, Norway, | | tlf. mob. 952 62 567, | Cellular...: +47 952 62 567, | | sentralbord 61 14 54 00. | Switchboard: +47 61 14 54 00. | +-------------------------------+------------------------------------+ From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 15:55:34 2015 Return-Path: Delivered-To: hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8CF5CA55; Fri, 27 Mar 2015 15:55:34 +0000 (UTC) Received: from dchagin.static.corbina.net (dchagin.static.corbina.ru [78.107.232.239]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "dchagin.static.corbina.net", Issuer "dchagin.static.corbina.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E7638942; Fri, 27 Mar 2015 15:55:32 +0000 (UTC) Received: from dchagin.static.corbina.net (localhost [127.0.0.1]) by dchagin.static.corbina.net (8.14.9/8.14.9) with ESMTP id t2RFhmK6090886 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 27 Mar 2015 18:43:48 +0300 (MSK) (envelope-from dchagin@dchagin.static.corbina.net) Received: (from dchagin@localhost) by dchagin.static.corbina.net (8.14.9/8.14.9/Submit) id t2RFhmUt090885; Fri, 27 Mar 2015 18:43:48 +0300 (MSK) (envelope-from dchagin) Date: Fri, 27 Mar 2015 18:43:48 +0300 From: Chagin Dmitry To: Edward Tomasz Napierala Subject: Re: Linux core under FreeBSD. Message-ID: <20150327154348.GA90856@dchagin.static.corbina.net> References: <20150327124058.GA3991@brick.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150327124058.GA3991@brick.home> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 15:55:34 -0000 On Fri, Mar 27, 2015 at 01:40:58PM +0100, Edward Tomasz Napierala wrote: > Is there a way to debug a core generated by Linux process running > under Linuxulator on 11-CURRENT? Thanks! > ptrace implemented only for i386, so ktrace/kdump and machdep.uprintf_signal=1 -- Have fun! chd From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 18:09:53 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 513AE93E for ; Fri, 27 Mar 2015 18:09:53 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3143ED63 for ; Fri, 27 Mar 2015 18:09:53 +0000 (UTC) Received: from zeta.ixsystems.com (unknown [12.229.62.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id A6580114D3; Fri, 27 Mar 2015 11:09:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1427479792; x=1427494192; bh=qOVO0Vx6VQ2LZJw2z3eULsNEV9GxdhggJn2St6xXSN8=; h=Date:From:Reply-To:To:CC:Subject:References:In-Reply-To; b=EKzOSAnCFwTkjI0qyMN553myLUOjeOY/umn69ggTIerpPsFIWc2aBlSxJhEaQY28U 9eHpM3Rr3Ztmq6BwvYP2JMOWwRn+YClc8PIXJwheAkoU6VdgtihN56E+PGNJiAfFc4 hJ9zZ0kbAsC8lDDwLG1NDIQGyQnF/6Uo53tS2dms= Message-ID: <55159CF0.9060700@delphij.net> Date: Fri, 27 Mar 2015 11:09:52 -0700 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: Pedro Arthur , d@delphij.net Subject: Re: GELI support on /boot folder References: <20150319013231.GR51048@funkthat.com> <55149E70.30608@delphij.net> In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Cc: "" , John-Mark Gurney X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 18:09:53 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03/26/15 19:56, Pedro Arthur wrote: > I think that encrypting the boot folder will protect the boot > configurations, kernel and kernel modules from being changed. I see... Have you considered other approaches for this goal, for instance verifying signature? (But to make it useful, we still need something in the BIOS/EFI to enforce the integrity of the boot code itself). >> If we make changes to loader more often, it could be a bad idea >> because merging both parties would make it harder for those who >> develop loader changes. >> >> Additionally, it may be desirable to keep different copies of >> loaders in different "boot environment" datasets, it's more >> convenient for debugging: let's say one developer decided to make >> some changes to ZFS support of loader, and that's installed to a >> new boot environment, then they can try it out without making a >> usable boot disk at hand before hand. Once the zfsloader is >> proven to be working (we still have zfsloader.old or a different >> boot environment available), we would have much more confident >> that the system will boot after a gptzfsboot update because they >> share the same code. >> >> I agree with you, but the boot2 has already reached its size >> limit.For > example if you try to compile the boot2 with clang < 3.5 (>=3.5 > uses the enable-gvn flag) you will get an error saying boot2 > exceeded its max size by ~20 bytes. I can't see other way to do it > without merging. Hmm I don't quite follow -- we were discussing about whether to merge gpt[zfs]boot with [zfs]loader, right? (I don't have strong opinion on whether we merge or not merge the two, just would like to point out the pro/cons). Cheers, - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.1.2 (FreeBSD) iQIcBAEBCgAGBQJVFZztAAoJEJW2GBstM+nsBLUP/RuzlcrJ6+WW3h5vUF0gNwb+ zEv/WAPtiH6pZIgcUmUkL2F4icKEiEknoTgPhObpgARGPx4xrm7pYHZ4Zsule/MS KYE3Sys8eLwIONHSBl1sHJ3WV8K/Jv+buwRDWXsmwtjH8e7C5yxrmuytp4XJ4Lxp pRIqNJmfdPJOI1bNMJCI4sPNHo/1pnxQGNTN2vxJAdjSzgh9FvIiH00CyHm+r23z ZCQn1aAGded2Rnv4boG0EPklKQA38GG8kHdtQVaLySDZL13BvHFbF0P09b/1m0I7 TXypho3xgHEI2vVDiLPPIgFdnFm95AJ2ibVu5UP3g+4iqiGMSwtq7XYZRnDIGVJ5 MxZdgTgf1c7tvmf8SoQLFnfDVi8RfVzh+CpmbWr7+KotuW3BMfOgd20V2z/ItDhF 9ptZDPUILrqEUL127HwSMENX8mwLmMDo14lPzRtan7YfoIgNMgAh0B0ZwP5Ow0yO txsJ8/YQYgcCOP3sQinyu+OV3OD91qlK0OBIePrqX8eP5jI1paflXElikWhEYjvi pNO2c+HenFm09OGGaW5PrHvIk4fjknkpq0ndwS2a8dSQS2zFaEvfzvKvoCr2x7Lh KtTzdGrORXdYelHndeMg8dh9LXDWEQgNdWBNP+xKnL23xaXcVWo8qgWpLM4RIc72 uGDJqiUysU9rDEf3oq7z =H1bs -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 18:36:26 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7BFA31CC; Fri, 27 Mar 2015 18:36:26 +0000 (UTC) Received: from mail-pd0-x22a.google.com (mail-pd0-x22a.google.com [IPv6:2607:f8b0:400e:c02::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44285113; Fri, 27 Mar 2015 18:36:26 +0000 (UTC) Received: by pdbop1 with SMTP id op1so104154301pdb.2; Fri, 27 Mar 2015 11:36:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=RPRj5sdwLf91eFy29HUeTFiIjV0iPrDGLt2iVfS623I=; b=Lii9Kij7vr+j+hzgAUNIa1inGmNUEl3jAafL4O4ck6wucYBVsIbTTN90ZI9szgexkg H8ieHw0BsmdtuItcg2V8Qtkk352H2nZhZDrCPqWH8WRx8Rsx4R4B63RYQXJ5JKUL6zRK CIPlnbbxBsI+CNj/gmfRvaNXu9vpEFoiCnZNZldg9ikLdRHPmIZDInjFfWaHjMLUgpNb x6LNqPROqCqRbWEnMM+0UGUUS+pBeyA6q9In1C54fZZUYfbVCig5oSeRdWvMU9gykEEP 2IM+YVKdcMPhnz7TPIkswauVH3r0euq8OHHbJaHtcLUnLZoaK/Xc11clc1vluXZEOamz ExEg== X-Received: by 10.66.234.2 with SMTP id ua2mr37426234pac.137.1427481385794; Fri, 27 Mar 2015 11:36:25 -0700 (PDT) MIME-Version: 1.0 Received: by 10.66.179.203 with HTTP; Fri, 27 Mar 2015 11:35:55 -0700 (PDT) In-Reply-To: References: <20150321220246.GE14650@dft-labs.eu> <20150321232622.GF14650@dft-labs.eu> From: Yue Chen Date: Fri, 27 Mar 2015 14:35:55 -0400 Message-ID: Subject: Re: How to traverse kernel threads? To: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Benjamin Kaduk , Mateusz Guzik , Oliver Pinter X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 18:36:26 -0000 When using the following code on kernel module loading: ------------------------------------------------------------------------------------------ struct thread *td = kdb_thr_first(); td = kdb_thr_next(td); ------------------------------------------------------------------------------------------ The kernel panics. And when printing all threads in proc0 (all kernel threads?): ------------------------------------------------------------------------------------------ struct proc *p = pfind(0); FOREACH_THREAD_IN_PROC(p, td) { uprintf("td: %x\n", td); } td = curthread; uprintf("cur td: %x\n", td); ------------------------------------------------------------------------------------------ The ``curthread'' (from this kernel module running the above code) is not in the 0 process group. On Sun, Mar 22, 2015 at 4:58 PM, Oliver Pinter < oliver.pinter@hardenedbsd.org> wrote: > Probably take a look at DDB: > > https://github.com/HardenedBSD/hardenedBSD/blob/hardened/current/master/sys/ddb/db_thread.c#L88 > > On Sun, Mar 22, 2015 at 9:45 PM, Benjamin Kaduk wrote: > > On Sat, 21 Mar 2015, Mateusz Guzik wrote: > > > >> But once more the real question is what are you trying to do. I don't > >> see any use for stack info of random threads. > > > > One thing that comes to mind is for live binary-patching the kernel, to > > confirm that no thread is currently in a routine which would be patched. > > > > -Ben > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 19:13:12 2015 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 46887946 for ; Fri, 27 Mar 2015 19:13:12 +0000 (UTC) Received: from mail-wi0-x230.google.com (mail-wi0-x230.google.com [IPv6:2a00:1450:400c:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D87077FD for ; Fri, 27 Mar 2015 19:13:11 +0000 (UTC) Received: by wibg7 with SMTP id g7so37259699wib.1 for ; Fri, 27 Mar 2015 12:13:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:subject:message-id:mime-version:content-type :content-transfer-encoding; bh=z9cZy9MSFBSy336nLlx2yohDPOykwjYvUcC1NCfmnlY=; b=G1Kr2UXcljKP+DVSd8Edwzvt1eg03le9RhX/N6l7QC7QHhIta9jdzur+y70C4dztFP xmxrL7t0Xll1IjouDLKF+AgHtuP+JzKBW+gUfl/WdX0LTT3IBuy/oXWpjrGajzNKBXFc rvTuKcy9FXqPociPxKAp0b0brvNMWG7Htm8UpGeGsWy5fRV6SznzpbzymcZ4S4eFwvba /w81Cb7aoC7iXjkzcf/kvVqDT6ElNWccVaQmDBnzhd0/13o3UJtDQRhbncxvVYJbmn7x /a5nnPhvUGq4hYy72zFO4choLc0P94PJEFhaJNOeyLXeBHs9KWJR9Id6qXazzQeT8JNa 7kkg== X-Received: by 10.194.62.52 with SMTP id v20mr40579199wjr.137.1427483590371; Fri, 27 Mar 2015 12:13:10 -0700 (PDT) Received: from localhost ([193.198.56.245]) by mx.google.com with ESMTPSA id e18sm4063983wjz.27.2015.03.27.12.13.08 for (version=SSLv3 cipher=RC4-SHA bits=128/128); Fri, 27 Mar 2015 12:13:09 -0700 (PDT) Date: Fri, 27 Mar 2015 20:13:04 +0100 From: To: hackers@FreeBSD.org Subject: Port's make knobs in DESTDIR Message-ID: <20150327201304.00005886@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 19:13:12 -0000 # make VAR_EXISTING_IN_PORTS_MAKEFILE=value install clean OK => Overrides port's var from Makefile with value # make VAR_EXISTING_IN_PORTS_MAKEFILE=value DESTDIR=/path/ install clean FAIL => Fails to override port's var from Makefile when DESTDIR is set Why is it so? Domagoj S. From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 19:49:25 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CE12B5A6; Fri, 27 Mar 2015 19:49:25 +0000 (UTC) Received: from mail-wg0-x234.google.com (mail-wg0-x234.google.com [IPv6:2a00:1450:400c:c00::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 67967C5E; Fri, 27 Mar 2015 19:49:25 +0000 (UTC) Received: by wgbdm7 with SMTP id dm7so4145673wgb.1; Fri, 27 Mar 2015 12:49:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=k/fJN1V0AD/WTksuybNU9Z5V6kNTBiwPb1v+IyYPVFE=; b=slqzHjGSOBs8QjTwaQMqJRRm7YY58569e0rM9LDMDmdn9tUzVKWPg/fikHEt+tg+pP GkQ6KuTj8GxAuVfBS0q2TmYbsaQpLU2Ee0HIpv7+5PUsjV8FkwkyrzTl+ypDOnLmttZs LAB5QLKI4MV3xJSoAiQCPS1xmVKL0LaTTmT1KdBOG3SgdS/aAmrERAXPmFUJpfmzBepw cwYSURq0pcsCJoBmyiee3hm/fyrxvNYTMjs3Uul67SkaUGBZmaDSTtU4zMJkSmIst63Y BZiodiOa5HwAkFX+gtkvSL09g8Vv+5Ceejho5WBp4RJGB3/PBsedbJAuqEQlC4znawA5 4ZQA== X-Received: by 10.180.85.103 with SMTP id g7mr708737wiz.19.1427485763892; Fri, 27 Mar 2015 12:49:23 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id m9sm4764532wiz.24.2015.03.27.12.49.22 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Fri, 27 Mar 2015 12:49:22 -0700 (PDT) Date: Fri, 27 Mar 2015 20:49:20 +0100 From: Mateusz Guzik To: Yue Chen Subject: Re: How to traverse kernel threads? Message-ID: <20150327194920.GB18158@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Yue Chen , "freebsd-hackers@freebsd.org" , Benjamin Kaduk , Oliver Pinter References: <20150321220246.GE14650@dft-labs.eu> <20150321232622.GF14650@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "freebsd-hackers@freebsd.org" , Oliver Pinter , Benjamin Kaduk X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 19:49:26 -0000 On Fri, Mar 27, 2015 at 02:35:55PM -0400, Yue Chen wrote: > When using the following code on kernel module loading: > ------------------------------------------------------------------------------------------ > struct thread *td = kdb_thr_first(); > td = kdb_thr_next(td); > ------------------------------------------------------------------------------------------ > The kernel panics. > Panics how? Also you can easily see these functions don't lock anything, so it would be assumed you took appropriate locks. Except it seems there routines are supposed to be only used when execution is 'frozen' (e.g. when escaped to the debugger). > > And when printing all threads in proc0 (all kernel threads?): > ------------------------------------------------------------------------------------------ > struct proc *p = pfind(0); > FOREACH_THREAD_IN_PROC(p, td) { > uprintf("td: %x\n", td); > } > proc0 is an exported symbol, no need to pfind. > td = curthread; > uprintf("cur td: %x\n", td); > ------------------------------------------------------------------------------------------ > The ``curthread'' (from this kernel module running the above code) is not > in the 0 process group. > There is no 'curthread from kernel module'. My guess is you do this work from module initializator, and in that case curthread is the thread which loads the module, and such a thread is definitely not linked into proc0. Still nobody knows what you are trying to do. -- Mateusz Guzik From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 20:55:33 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CF350931; Fri, 27 Mar 2015 20:55:33 +0000 (UTC) Received: from mail-pa0-x22a.google.com (mail-pa0-x22a.google.com [IPv6:2607:f8b0:400e:c03::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 92EEB620; Fri, 27 Mar 2015 20:55:33 +0000 (UTC) Received: by pacwz10 with SMTP id wz10so54982341pac.2; Fri, 27 Mar 2015 13:55:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=FWgXUXSqTWjHehonf2bhxu/hxkvIzBnssSnWWWywk1s=; b=oi2KX+Q9qA1kt/Z7mWmsJhK4RuvhW6b1pKOFIaVZsfx2/wP9MpC4ej/GdAxADw1DJ+ aALEZ68Hj655Qx2TKWbhK0g+cEyFF6zfdnywXvIP5GeMXhjvf7ds/iXplgWXZWJBZiH0 U+zb4+s4YaYmb2sKDNJJWZh5WNW22KnNcC85sJh/VwEUWrJuXQkll1w5KrVhRcdXy1rk Is2ITCzvhnoAWFGKejHGQZI7zakm78j8BcoIUIXTgcpMYPv8aZWbGEI3JHYDQXEe/Fg2 tz57UfUbYIRbBIFeq3FEzz5sZu1xXtb72kjtd7gmfqUTtHFD3FMDl9Ig0DU6fwvPVJhq Fwjw== X-Received: by 10.66.118.198 with SMTP id ko6mr39056417pab.16.1427489733103; Fri, 27 Mar 2015 13:55:33 -0700 (PDT) MIME-Version: 1.0 Received: by 10.66.179.203 with HTTP; Fri, 27 Mar 2015 13:55:02 -0700 (PDT) In-Reply-To: <20150327194920.GB18158@dft-labs.eu> References: <20150321220246.GE14650@dft-labs.eu> <20150321232622.GF14650@dft-labs.eu> <20150327194920.GB18158@dft-labs.eu> From: Yue Chen Date: Fri, 27 Mar 2015 16:55:02 -0400 Message-ID: Subject: Re: How to traverse kernel threads? To: Mateusz Guzik , Oliver Pinter , Benjamin Kaduk , Yue Chen , "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 20:55:33 -0000 > Except it seems there routines are supposed to be only used when > execution is 'frozen' (e.g. when escaped to the debugger). It means probably we can run the code in ``smp_rendezvous()'' function, right? > Still nobody knows what you are trying to do. We are trying to enhance FreeBSD's security by randomizing kernel code basic blocks periodically at runtime, to mitigate the attacks like return-oriented programming (ROP). It is basically a stronger form of ASLR. After each randomization procedure, the function return addresses saved in the stack are the ``old'' addresses before randomization, so we need to update them to the new addresses. That's why we need to get all the stack ranges to find those addresses. Also, in kernel, we believe that not only the return addresses in stacks need to be updated, there may exist other ``old'' saved instruction (PC) addresses in memory. Like in exception handling (maybe, do not know), debugging-purpose code and restartable atomic sequences (RAS) implementation. That's why I asked how to traverse all the kernel pages and get their virtual addresses here: https://lists.freebsd.org/pipermail/freebsd-hackers/2015-March/047336.html Now we found that it seems needed to traverse the ``pv_entry'' structure for x86_64 MMU. Another problem is that we do not know if FreeBSD has any form of special encodings or offset form for 64-bit instruction addresses (e.g., saved %RIP) on X86_64, instead of hard-coded addresses. For example, using a 32-bit offset instead of the 64-bit full address; and doing what glibc does for the setjmp/longjmp jmp_buf (special encodings (PTR_MANGLE) for the saved register values). Any suggestion or correction are highly appreciated. Best, Yue On Fri, Mar 27, 2015 at 3:49 PM, Mateusz Guzik wrote: > On Fri, Mar 27, 2015 at 02:35:55PM -0400, Yue Chen wrote: > > When using the following code on kernel module loading: > > > ------------------------------------------------------------------------------------------ > > struct thread *td = kdb_thr_first(); > > td = kdb_thr_next(td); > > > ------------------------------------------------------------------------------------------ > > The kernel panics. > > > > Panics how? > > Also you can easily see these functions don't lock anything, so it would > be assumed you took appropriate locks. > > Except it seems there routines are supposed to be only used when > execution is 'frozen' (e.g. when escaped to the debugger). > > > > > And when printing all threads in proc0 (all kernel threads?): > > > ------------------------------------------------------------------------------------------ > > struct proc *p = pfind(0); > > FOREACH_THREAD_IN_PROC(p, td) { > > uprintf("td: %x\n", td); > > } > > > > proc0 is an exported symbol, no need to pfind. > > > td = curthread; > > uprintf("cur td: %x\n", td); > > > ------------------------------------------------------------------------------------------ > > The ``curthread'' (from this kernel module running the above code) is not > > in the 0 process group. > > > > There is no 'curthread from kernel module'. > > My guess is you do this work from module initializator, and in that case > curthread is the thread which loads the module, and such a thread is > definitely not linked into proc0. > > Still nobody knows what you are trying to do. > > -- > Mateusz Guzik > From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 20:57:10 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ADD22A52; Fri, 27 Mar 2015 20:57:10 +0000 (UTC) Received: from mail-wi0-x233.google.com (mail-wi0-x233.google.com [IPv6:2a00:1450:400c:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3FC0F63F; Fri, 27 Mar 2015 20:57:10 +0000 (UTC) Received: by wiaa2 with SMTP id a2so46902954wia.0; Fri, 27 Mar 2015 13:57:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=iCyvXgsqub+pq79CCi4sabBy0JgyZ76LZkvRyrl1PX8=; b=O1HjpK3BnBmCDPDp+DdnklS7PTPu4aHaWOTFdXG3UVKgbayw3qkGuAqagW7LaO0eEi pMFmpWm1OWnmTJiXWRQ5xx8pdvxIBy/IImW2ShfU6W6Hw9yMKRnGdz6kFwVyD88BLzud 6UypCB259UztJk4LgSWEA2K49UkIavi57yzkHqlAA1HX/KQUIW/bsCL9csHRwyPFXV2v FwgZ6p1qrovq11EdP78QNS+iENpxSoBEqsVXTqKtQMfVo0nvPv3Ob/Bjgb/ITAuT4N6M NNPxcn0rzZR7mV3gLZBVGRvNm8sa7g+TaBcKZyb6bO9zARBkT4h5lugEJnXJnMKPY4if jGPQ== X-Received: by 10.180.89.163 with SMTP id bp3mr1141313wib.88.1427489828595; Fri, 27 Mar 2015 13:57:08 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id bd1sm5001614wib.13.2015.03.27.13.57.07 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Mar 2015 13:57:07 -0700 (PDT) Sender: Alexander Motin Message-ID: <5515C421.4040703@FreeBSD.org> Date: Fri, 27 Mar 2015 22:57:05 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org, "freebsd-hackers@freebsd.org" Subject: MAXBSIZE increase Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 20:57:10 -0000 Hi. Experimenting with NFS and ZFS I found an inter-operation issue: ZFS by default uses block of 128KB, while FreeBSD NFS (both client and server) is limited to 64KB requests by the value of MAXBSIZE. On file rewrite that limitation makes ZFS to do slow read-modify-write cycles for every write operation, instead of just writing the new data. Trivial iozone test show major difference between initial write and rewrite speeds because of this issue. Looking through the sources I've found and in r280347 fixed number of improper MAXBSIZE use cases in device drivers. After that I see no any reason why MAXBSIZE can not be increased to at least 128KB to match ZFS default (ZFS now supports block up to 1MB, but that is not default and so far rare). I've made a test build and also successfully created UFS file system with 128KB block -- not sure it is needed, but seems it survives this change well too. Is there anything I am missing, or it is safe to rise this limit now? -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 22:20:41 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5648833F for ; Fri, 27 Mar 2015 22:20:41 +0000 (UTC) Received: from mail-ob0-f179.google.com (mail-ob0-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1A0CAFA7 for ; Fri, 27 Mar 2015 22:20:41 +0000 (UTC) Received: by obcjt1 with SMTP id jt1so82086015obc.2 for ; Fri, 27 Mar 2015 15:20:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=+M104RUp5/nVxx+g5xzimIJVKe3D/vRmIDEOS6J+M4o=; b=YJvUA5aRWhKKT0ho36n43fzcEic49dqa8V5fnzjEhW9D9MwDdjQQZ09dG7EEb1VU/a +UCX2nDrYs4TkXZJvja6Zcr49f4DZeixuNyocsFAHH+IrsyOMVsaFp07+He7bcnVVucQ tma/Yq7Dw5pUUHXFOeHzPNlPRhzanhpG0okXcqzkgA30Be4OvTY7izakBZ2Kbnow7RT1 cw5krhvPvBCiCx3f27++PZ/RQfd6sD6WAIUSOFcDkh/GGPo8WuYvnGaz3AIXgeotV+H2 Q5COmZstuRgjiSQJOjlKoemOBpEnjLuVOSpqo+ZMPs8ayDM7c8wYrz1W6dBP6OyUezJV A2RA== X-Gm-Message-State: ALoCoQno3kUnG4ZcIWBTwqR3FDOAqKmuhdS17K8XQFmrdOJWrAm0aXuzgRgcZxJe9K2F/4fRXGxm MIME-Version: 1.0 X-Received: by 10.202.219.87 with SMTP id s84mr12817774oig.114.1427494839973; Fri, 27 Mar 2015 15:20:39 -0700 (PDT) Received: by 10.202.57.195 with HTTP; Fri, 27 Mar 2015 15:20:39 -0700 (PDT) In-Reply-To: References: <20150321220246.GE14650@dft-labs.eu> <20150321232622.GF14650@dft-labs.eu> <20150327194920.GB18158@dft-labs.eu> Date: Fri, 27 Mar 2015 23:20:39 +0100 Message-ID: Subject: Re: How to traverse kernel threads? From: Oliver Pinter To: Yue Chen Content-Type: text/plain; charset=UTF-8 Cc: PaX Team , Benjamin Kaduk , Mateusz Guzik , HardenedBSD Core , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 22:20:41 -0000 On Fri, Mar 27, 2015 at 9:55 PM, Yue Chen wrote: >> Except it seems there routines are supposed to be only used when >> execution is 'frozen' (e.g. when escaped to the debugger). > > It means probably we can run the code in ``smp_rendezvous()'' function, > right? > >> Still nobody knows what you are trying to do. > > We are trying to enhance FreeBSD's security by randomizing kernel code basic > blocks periodically at runtime, to mitigate the attacks like return-oriented > programming (ROP). It is basically a stronger form of ASLR. > After each randomization procedure, the function return addresses saved in > the stack are the ``old'' addresses before randomization, so we need to > update them to the new addresses. > That's why we need to get all the stack ranges to find those addresses. > > Also, in kernel, we believe that not only the return addresses in stacks > need to be updated, there may exist other ``old'' saved instruction (PC) > addresses in memory. Like in exception handling (maybe, do not know), > debugging-purpose code and restartable atomic sequences (RAS) > implementation. That's why I asked how to traverse all the kernel pages and > get their virtual addresses here: > https://lists.freebsd.org/pipermail/freebsd-hackers/2015-March/047336.html > > Now we found that it seems needed to traverse the ``pv_entry'' structure for > x86_64 MMU. > > Another problem is that we do not know if FreeBSD has any form of special > encodings or offset form for 64-bit instruction addresses (e.g., saved %RIP) > on X86_64, instead of hard-coded addresses. For example, using a 32-bit > offset instead of the 64-bit full address; and doing what glibc does for the > setjmp/longjmp jmp_buf (special encodings (PTR_MANGLE) for the saved > register values). > > Any suggestion or correction are highly appreciated. > > Best, > Yue (Added HardenedBSD core and PaXTeam to CC.) Until you can not fixed all of the infoleaks from kernel (try sysctl -a | grep 0x or similar command) the KASLR and other kernel address space randomization techniques are easily bypass-able... > > > > On Fri, Mar 27, 2015 at 3:49 PM, Mateusz Guzik wrote: >> >> On Fri, Mar 27, 2015 at 02:35:55PM -0400, Yue Chen wrote: >> > When using the following code on kernel module loading: >> > >> > ------------------------------------------------------------------------------------------ >> > struct thread *td = kdb_thr_first(); >> > td = kdb_thr_next(td); >> > >> > ------------------------------------------------------------------------------------------ >> > The kernel panics. >> > >> >> Panics how? >> >> Also you can easily see these functions don't lock anything, so it would >> be assumed you took appropriate locks. >> >> Except it seems there routines are supposed to be only used when >> execution is 'frozen' (e.g. when escaped to the debugger). >> >> > >> > And when printing all threads in proc0 (all kernel threads?): >> > >> > ------------------------------------------------------------------------------------------ >> > struct proc *p = pfind(0); >> > FOREACH_THREAD_IN_PROC(p, td) { >> > uprintf("td: %x\n", td); >> > } >> > >> >> proc0 is an exported symbol, no need to pfind. >> >> > td = curthread; >> > uprintf("cur td: %x\n", td); >> > >> > ------------------------------------------------------------------------------------------ >> > The ``curthread'' (from this kernel module running the above code) is >> > not >> > in the 0 process group. >> > >> >> There is no 'curthread from kernel module'. >> >> My guess is you do this work from module initializator, and in that case >> curthread is the thread which loads the module, and such a thread is >> definitely not linked into proc0. >> >> Still nobody knows what you are trying to do. >> >> -- >> Mateusz Guzik > > From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 27 23:52:48 2015 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CFFAD5EA; Fri, 27 Mar 2015 23:52:48 +0000 (UTC) Received: from mail-qc0-x236.google.com (mail-qc0-x236.google.com [IPv6:2607:f8b0:400d:c01::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 81D47ABC; Fri, 27 Mar 2015 23:52:48 +0000 (UTC) Received: by qcbjx9 with SMTP id jx9so30280656qcb.0; Fri, 27 Mar 2015 16:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type; bh=pKDfa2eetgd6WU6iPMHFrt5aAk+2VWhhhxBBa/m2eis=; b=AW+ryVc6qX8Mt/+G1hRwV6weyoJX8hgWSelucx685+hnHywPEfDyVjdIeEX7veKUFX og3JzFfzGLoUyZ+JDwPZZvY1sLsqta2AK0V3rZ7FXXWh+dD/PGG6dAxKtolkrlRcFRG+ Be3/fxDUXpkWiv32L01hULa4WNKlpc6cgmUfTxssbaKsFJG0AN9c3CO14tkCTriaSc3h sn+7mNDPpe5x5y0Bv+NBh/ZioCNuhzZUytAaACjO7ND/tUyztt0dWUSVR6ga3IwgNofY DE96uw9dy9s6YTLlCAA98B7V11raL74st56FdybIaCNbCep0FZJDuEDu9d5TiUMYvrOm NyGA== X-Received: by 10.55.50.203 with SMTP id y194mr45486673qky.8.1427500367620; Fri, 27 Mar 2015 16:52:47 -0700 (PDT) Received: from kan ([2601:6:6780:7e0:226:18ff:fe00:232e]) by mx.google.com with ESMTPSA id n41sm2594786qkh.3.2015.03.27.16.52.46 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Mar 2015 16:52:46 -0700 (PDT) Date: Fri, 27 Mar 2015 19:52:37 -0400 From: Alexander Kabaev To: Chagin Dmitry Subject: Re: Linux core under FreeBSD. Message-ID: <20150327195237.3bddcb31@kan> In-Reply-To: <20150327154348.GA90856@dchagin.static.corbina.net> References: <20150327124058.GA3991@brick.home> <20150327154348.GA90856@dchagin.static.corbina.net> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.25; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/n_Sh5U6iDnAmoS4fEe2W7EJ"; protocol="application/pgp-signature" Cc: hackers@FreeBSD.org, Edward Tomasz Napierala X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2015 23:52:48 -0000 --Sig_/n_Sh5U6iDnAmoS4fEe2W7EJ Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Fri, 27 Mar 2015 18:43:48 +0300 Chagin Dmitry wrote: > On Fri, Mar 27, 2015 at 01:40:58PM +0100, Edward Tomasz Napierala > wrote: > > Is there a way to debug a core generated by Linux process running > > under Linuxulator on 11-CURRENT? Thanks! > >=20 > ptrace implemented only for i386, so ktrace/kdump and > machdep.uprintf_signal=3D1 >=20 IFF FreeBSD had implemented dumping the core in format compatible with Linux gdb, then normal Linux gdb binary can be used to examine cores, ptrace support is not required for that. Since sysentvec already provides sv_coredump entry, it should not even be too hard to implement this. --=20 Alexander Kabaev --Sig_/n_Sh5U6iDnAmoS4fEe2W7EJ Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlUV7UcACgkQQ6z1jMm+XZbaXwCdEYSaj6M08YmXd3q4Kj37qSpb JYEAoK+Cs0S6mNEzwa2TZqes7KSo37re =4fDx -----END PGP SIGNATURE----- --Sig_/n_Sh5U6iDnAmoS4fEe2W7EJ-- From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 01:32:34 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7510C635 for ; Sat, 28 Mar 2015 01:32:34 +0000 (UTC) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F337B6C2 for ; Sat, 28 Mar 2015 01:32:33 +0000 (UTC) Received: by wibgn9 with SMTP id gn9so52771402wib.1 for ; Fri, 27 Mar 2015 18:32:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=5c35P82ZdGrfFoX3vW/cAtgYS3X2JPrFgWGaw7f+eWE=; b=XFP8n6gdpsnkE14GW8P1hb+0PP/XDkMKwASOjT7YbZ6M0sKwB5Jrs6UCXTX8ILkVgq jCLnqhUAVhxmUBOkBYpBsAcuImls30z3juThBwjtl1YIjiTjyzBn4sOaW8gOGuNQOUCW SlvIDk/4VjVl8T5i+JUJ0b6vSyDtKgy93O5h+2cyqxUd8GvImhF7yh2gdDUhp1G7H9oD BzLvefb8Ch9IsIrjJPXwd4kI9AD5nW4FYZbIiDBq5jPOySN53jOd24bfWfB4tBXPOMqI EKwvkwdcpX91XAPfwHYlqI7tClq5tn1D+RSndW9Ah2OCZ+WejyQ62hnwfCHDY2a62KpC FzhA== MIME-Version: 1.0 X-Received: by 10.194.57.170 with SMTP id j10mr43329146wjq.102.1427506352467; Fri, 27 Mar 2015 18:32:32 -0700 (PDT) Received: by 10.194.18.37 with HTTP; Fri, 27 Mar 2015 18:32:32 -0700 (PDT) In-Reply-To: <55159CF0.9060700@delphij.net> References: <20150319013231.GR51048@funkthat.com> <55149E70.30608@delphij.net> <55159CF0.9060700@delphij.net> Date: Fri, 27 Mar 2015 22:32:32 -0300 Message-ID: Subject: Re: GELI support on /boot folder From: Pedro Arthur To: Xin LI Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "" , John-Mark Gurney X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 01:32:34 -0000 Disregard this boot2 comment, it has nothing to do with the gptboot 2015-03-27 15:09 GMT-03:00 Xin Li : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 03/26/15 19:56, Pedro Arthur wrote: > > I think that encrypting the boot folder will protect the boot > > configurations, kernel and kernel modules from being changed. > > I see... Have you considered other approaches for this goal, for > instance verifying signature? (But to make it useful, we still need > something in the BIOS/EFI to enforce the integrity of the boot code > itself). > > >> If we make changes to loader more often, it could be a bad idea > >> because merging both parties would make it harder for those who > >> develop loader changes. > >> > >> Additionally, it may be desirable to keep different copies of > >> loaders in different "boot environment" datasets, it's more > >> convenient for debugging: let's say one developer decided to make > >> some changes to ZFS support of loader, and that's installed to a > >> new boot environment, then they can try it out without making a > >> usable boot disk at hand before hand. Once the zfsloader is > >> proven to be working (we still have zfsloader.old or a different > >> boot environment available), we would have much more confident > >> that the system will boot after a gptzfsboot update because they > >> share the same code. > >> > >> I agree with you, but the boot2 has already reached its size > >> limit.For > > example if you try to compile the boot2 with clang < 3.5 (>=3.5 > > uses the enable-gvn flag) you will get an error saying boot2 > > exceeded its max size by ~20 bytes. I can't see other way to do it > > without merging. > > Hmm I don't quite follow -- we were discussing about whether to merge > gpt[zfs]boot with [zfs]loader, right? > > (I don't have strong opinion on whether we merge or not merge the two, > just would like to point out the pro/cons). > > Cheers, > - -- > Xin LI https://www.delphij.net/ > FreeBSD - The Power to Serve! Live free or die > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.1.2 (FreeBSD) > > iQIcBAEBCgAGBQJVFZztAAoJEJW2GBstM+nsBLUP/RuzlcrJ6+WW3h5vUF0gNwb+ > zEv/WAPtiH6pZIgcUmUkL2F4icKEiEknoTgPhObpgARGPx4xrm7pYHZ4Zsule/MS > KYE3Sys8eLwIONHSBl1sHJ3WV8K/Jv+buwRDWXsmwtjH8e7C5yxrmuytp4XJ4Lxp > pRIqNJmfdPJOI1bNMJCI4sPNHo/1pnxQGNTN2vxJAdjSzgh9FvIiH00CyHm+r23z > ZCQn1aAGded2Rnv4boG0EPklKQA38GG8kHdtQVaLySDZL13BvHFbF0P09b/1m0I7 > TXypho3xgHEI2vVDiLPPIgFdnFm95AJ2ibVu5UP3g+4iqiGMSwtq7XYZRnDIGVJ5 > MxZdgTgf1c7tvmf8SoQLFnfDVi8RfVzh+CpmbWr7+KotuW3BMfOgd20V2z/ItDhF > 9ptZDPUILrqEUL127HwSMENX8mwLmMDo14lPzRtan7YfoIgNMgAh0B0ZwP5Ow0yO > txsJ8/YQYgcCOP3sQinyu+OV3OD91qlK0OBIePrqX8eP5jI1paflXElikWhEYjvi > pNO2c+HenFm09OGGaW5PrHvIk4fjknkpq0ndwS2a8dSQS2zFaEvfzvKvoCr2x7Lh > KtTzdGrORXdYelHndeMg8dh9LXDWEQgNdWBNP+xKnL23xaXcVWo8qgWpLM4RIc72 > uGDJqiUysU9rDEf3oq7z > =H1bs > -----END PGP SIGNATURE----- > From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 02:45:07 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AAAB0F83; Sat, 28 Mar 2015 02:45:07 +0000 (UTC) Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id 6CFB5D51; Sat, 28 Mar 2015 02:45:07 +0000 (UTC) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id 8E0C5428F24; Sat, 28 Mar 2015 13:44:58 +1100 (AEDT) Date: Sat, 28 Mar 2015 13:44:57 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Alexander Motin Subject: Re: MAXBSIZE increase In-Reply-To: <5515C421.4040703@FreeBSD.org> Message-ID: <20150328111733.L963@besplex.bde.org> References: <5515C421.4040703@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=ZuzUdbLG c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=K_dCTBWPffNRIZAgkwQA:9 a=v5_RuTisAHe7God6:21 a=T73ABXJZ4VF9rtWC:21 a=CjuIK1q_8ugA:10 X-Mailman-Approved-At: Sat, 28 Mar 2015 02:52:18 +0000 Cc: freebsd-fs@freebsd.org, "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 02:45:07 -0000 > Experimenting with NFS and ZFS I found an inter-operation issue: ZFS by > default uses block of 128KB, while FreeBSD NFS (both client and server) > is limited to 64KB requests by the value of MAXBSIZE. On file rewrite > that limitation makes ZFS to do slow read-modify-write cycles for every > write operation, instead of just writing the new data. Trivial iozone > test show major difference between initial write and rewrite speeds > because of this issue. > > Looking through the sources I've found and in r280347 fixed number of > improper MAXBSIZE use cases in device drivers. After that I see no any > reason why MAXBSIZE can not be increased to at least 128KB to match ZFS > default (ZFS now supports block up to 1MB, but that is not default and > so far rare). I've made a test build and also successfully created UFS > file system with 128KB block -- not sure it is needed, but seems it > survives this change well too. > > Is there anything I am missing, or it is safe to rise this limit now? I see the following minor problems: - static and dynamic allocation of MAXBSIZE bytes would be more wasteful than before. - boot blocks used to do static allocation of MAXBSIZE bytes. Now they just do ffs's sanity check that the block size is less that that. A block size larger than this is not necessarily invalid, but just unsupported by the running instance of the buffer cache layer (so unsupported by the running instance of ffs too). Another or the same OS may have a larger MAXBSIZE, and the user may have broken portability by actually using this to create a file system that cannot be read by OS's with the historical MAXBSIZE. This check is bogus for boot blocks, since they don't use the buffer cache layer. ufsread.c uses a sort of anti-buffer-cache to avoid problems but give extreme slowness. It uses a virtual block size of 4K and does i/o 4K at a time with no caching. The buffer must not cross a 64K boundary on x86, and the MI code states this requirement for all arches. In i386/boot2, dmadat is 64K-aligned so the virtual buffer size could be up to 64K, except dmadat is used for 3 other buffers and only 4K is used for the data buffer. The data structure for this is: X /* Buffers that must not span a 64k boundary. */ X struct dmadat { X char blkbuf[VBLKSIZE]; /* filesystem blocks */ X char indbuf[VBLKSIZE]; /* indir blocks */ X char sbbuf[SBLOCKSIZE]; /* superblock */ X char secbuf[DEV_BSIZE]; /* for MBR/disklabel */ X }; X static struct dmadat *dmadat; I don't like the FreeBSD boot code, and use my version of biosboot if possible. I expanded its buffers and improved its caching a year or 2 ago. Old versions have 2 buffers of size MAXBSIZE in commented- out code since this doesn't work, especially when written in C. The commented-out code also sets a size of 4K for one of these buffers. This last worked, for the default ffs block size only, in about 1990 (this code is from Mach). The old code actually uses 3 buffers of size 8K, corresponding to 3 of the 4 buffers in dmadat. This broke about 15 years ago when the default and normal ffs block size was increased to 16K. I fixed this by allocating all of the buffers in asm. From start.S: X ENTRY(disklabel) X . = EXT(boot1) + 0x200 + 0*276 + 1*0x200 X X .globl EXT(buf) X .set EXT(buf), EXT(boot1) + 0x20000 X .globl EXT(iobuf) X .set EXT(iobuf), EXT(boot1) + 0x30000 X .globl EXT(mapbuf) X .set EXT(mapbuf), EXT(boot1) + 0x40000 boot1 is loaded at a 64K boundary and overlaid with boot2, the same as in the -current boot2. The above bits off 64K pieces of the heap for all large data structures. boot2 only (64K?) for dmadat instead, using hackish C code. Then I improved the caching. biosboot was using my old code which did caching mainly for floppies, since old systems were too slow to keep up with reading even floppies 1 512-block at a time. It used a read-ahead buffer of size 18*512 = 9K to optimize for floppies up to size 1440K. This worked OK for old hard disks with the old default ffs block size of 8K too. But it gives much the same anti-caching as -current's virtual 4K buffers when the ffs block size is 16K or larger. I didn't expand the cache to a large one on the heap, but just changed it to 32*512 = 16K to work well with my default ffs block size of 16K (32K is pessimal for my old disk), and fixed some alignment problems (the old code attempts to align on track boundaries but tracks don't exist for modern hard disks, and the alignment needs to be to ffs block boundaries else 16K-blocks would be split every time in the 16K "cache". Summary for the boot blocks: they seem to be unaffected by increasing MAXBSIZE, but their anti-cache works even better for fragmenting larger blocks. - the buffer cache is still optimized for i386 with BKVASIZE = 8K. 64-bit systems don't need the complications and pessimizations to fit in i386's limited kva, but have them anyway. When the default ffs block size was doubled to 16K, BKVASIZE was doubled to match, but the tuning wasn't doubled to match. This reduced the effective number of buffers by a factor of 2. This pessimization was mostly unnoticable, since memory sizes grew by more than a factor of 2 and and nbuf grew by about a factor of 2. But increasing (nbuf*BKVASIZE) much more isn't possible on i386 since it reaches a kva limit. Then when ffs's default block size was doubled to 32K, BKVASIZE wasn't even doubled to match. If anyone actually uses the doubled MAXBSIZE, then BKVASIZE will be mistuned by another factor of 2. They probably shouldn't do that. A block size of 64K already works poorly in ffs. Relative to a block size of 32K, It mainly doubles the size for metadata i/o without making much difference for data i/o, since data i/o is clustered. An fs block size equal to MAXPHYS also makes clustering useless, by limiting the maximum number of blocks per cluster to 1. That is better than the ratio of 4/32 in ufsread and 9/{8,16,32} in old biosboot, but still silly. Large fs block sizes (where "large" is about 2K) are only good when clustering doesn't work and the disk doesn't like small blocks. This may be the case for ffs on newer hard disks. Metdata is not clustered for ffs. My old hard disks like any size larger than 16K, but my not so old hard disks prefer 32K or above. nfs for zfs will actually use the new MAXBSIZE. I don't like it using a hard-coded size. It gives buffer cache fragmentation. The new MAXBSIZE will non-accidentally match the fs block size for zfs, but even the old MAXBSIZE doesn't match the usual fs block size for any file system. - cd9660 mount uses MAXBSIZE for a sanity check. It can only support block sizes up to that, but there must be an fs limit. It should probably use min(CD9660_MAXBSIZE, MAXBSIZE). - similarly in msdosfs, except I'm sure that there is an fs limit of 64K. Microsoft specifies that the limit is 32K, but 64K works in FreeBSD and perhaps even in Microsoft OS's. - similarly in ffs, except the ffs limit is historically identical to MAXBSIZE. I think it goes the other way -- MAXBSIZE = 64K is the historical ffs limit, and the buffer cache has to support that. Perhaps ffs should remain at its historical limit. The lower limit is still local in ffs. It is named MINBSIZE. Its value is 4K in -current but 512 in my version. ffs has no fundamental limit at either 4K or 64K, and can support any size supported by the hardware after fixing some bugs involving assumptions that the superblock fits in an ffs block. - many file systems use MAXBSIZE to limit the read-ahead for cluster_read(). This seems wrong. cluster_read() has a natural limit of geom's virtual "best" i/o size (normally MAXPHYS). The decision about the amount of read-ahead should be left to the clustering code if possible. But it is unclear what this should be. The clustering code gets this wrong anyway. It has sysctls vfs.read_max (default 64) and vfs.write_behind (default 1). The units for these are broken. They are fs-blocks. A read-ahead of 64 fs-blocks of size 512 is too different from a read-ahead of 64 fs-blocks of size MAXBSIZE whatever the latter is. My version uses a read-ahead scaled in 512-blocks (default 256 blocks = MAXPHYS bytes). The default read-ahead shouldn't vary much with either MAXPHYS, MAXBSIZE or the fs block size, but should vary with the device (don't read-ahead 64 large fs blocks on a floppy disk device, as asked for by -current's read_max ...). - ffs utilities like fsck are broken by limiting themselves to the buffer cache limit of MAXBSIZE, like the boot blocks but with less reason since they don't have space constraints and not being limited by the current OS is more useful. Unless MAXBSIZE = 64K is considered to be a private ffs that escaped. Then the ffs code should spell it FFS_MAXBSIZE or 64K. Bruce From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 08:47:56 2015 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 64D14FC3; Sat, 28 Mar 2015 08:47:56 +0000 (UTC) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA708687; Sat, 28 Mar 2015 08:47:55 +0000 (UTC) Received: by wibg7 with SMTP id g7so54490253wib.1; Sat, 28 Mar 2015 01:47:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=Eh0vVoRqbXnx6f3jBALdLVOvxpiBoVt/FJp6IgA/PU0=; b=gwnuSsXHZeLyK+IMNhgR+kDtpbXiTg0pcgFT8qt/YTWAoKHY1HO8Qr+o7cnrY+nXP7 w8HtTPGWEYuyqx9EkbqwRR5ryKeJTlL7Qb7bzL5UjI6YEdZsljNcFqiw3RmVPfiLA5Zi AkIwT5h/Y+F5a0h2jRhJZgJmDat4gC3iQ87yNeTpRVL3fWey8f9/If58eLw17yXsKf0/ 1BXu5KEDAHVn5cT8yEZm9wmIOb4dNsohv5p/6cRJReIqmi7py0Yd3Bog/Ry13e3SzdtH pKuwHzHuOUKsmNENWwrRS1T4qmoVuKZltX65GRoP/fdxniYBuUVP+AmY0ZBB3mYSWgPx Yb/Q== X-Received: by 10.180.87.66 with SMTP id v2mr4394359wiz.51.1427532474137; Sat, 28 Mar 2015 01:47:54 -0700 (PDT) Received: from brick.home (abug111.neoplus.adsl.tpnet.pl. [83.8.178.111]) by mx.google.com with ESMTPSA id fo8sm6263352wib.14.2015.03.28.01.47.52 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 Mar 2015 01:47:52 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Date: Sat, 28 Mar 2015 09:47:50 +0100 From: Edward Tomasz Napierala To: Chagin Dmitry Subject: Re: Linux core under FreeBSD. Message-ID: <20150328084750.GA6548@brick.home> References: <20150327124058.GA3991@brick.home> <20150327154348.GA90856@dchagin.static.corbina.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150327154348.GA90856@dchagin.static.corbina.net> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 08:47:56 -0000 On 0327T1843, Chagin Dmitry wrote: > On Fri, Mar 27, 2015 at 01:40:58PM +0100, Edward Tomasz Napierala wrote: > > Is there a way to debug a core generated by Linux process running > > under Linuxulator on 11-CURRENT? Thanks! > > > ptrace implemented only for i386, so ktrace/kdump and > machdep.uprintf_signal=1 Ok, but what about core files from Linux processes? Are they useless at this moment? If so, perhaps they should just be disabled? From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 11:10:06 2015 Return-Path: Delivered-To: hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 18689D8A; Sat, 28 Mar 2015 11:10:06 +0000 (UTC) Received: from dchagin.static.corbina.net (dchagin.static.corbina.ru [78.107.232.239]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "dchagin.static.corbina.net", Issuer "dchagin.static.corbina.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D3740333; Sat, 28 Mar 2015 11:10:04 +0000 (UTC) Received: from dchagin.static.corbina.net (localhost [127.0.0.1]) by dchagin.static.corbina.net (8.14.9/8.14.9) with ESMTP id t2SBA09x097479 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 28 Mar 2015 14:10:00 +0300 (MSK) (envelope-from dchagin@dchagin.static.corbina.net) Received: (from dchagin@localhost) by dchagin.static.corbina.net (8.14.9/8.14.9/Submit) id t2SBA0PG097478; Sat, 28 Mar 2015 14:10:00 +0300 (MSK) (envelope-from dchagin) Date: Sat, 28 Mar 2015 14:10:00 +0300 From: Chagin Dmitry To: Edward Tomasz Napierala Subject: Re: Linux core under FreeBSD. Message-ID: <20150328111000.GA97431@dchagin.static.corbina.net> References: <20150327124058.GA3991@brick.home> <20150327154348.GA90856@dchagin.static.corbina.net> <20150328084750.GA6548@brick.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150328084750.GA6548@brick.home> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 11:10:06 -0000 On Sat, Mar 28, 2015 at 09:47:50AM +0100, Edward Tomasz Napierala wrote: > On 0327T1843, Chagin Dmitry wrote: > > On Fri, Mar 27, 2015 at 01:40:58PM +0100, Edward Tomasz Napierala wrote: > > > Is there a way to debug a core generated by Linux process running > > > under Linuxulator on 11-CURRENT? Thanks! > > > > > ptrace implemented only for i386, so ktrace/kdump and > > machdep.uprintf_signal=1 > > Ok, but what about core files from Linux processes? Are they useless > at this moment? If so, perhaps they should just be disabled? I never debugged core from Linux procs. I can write sysctl to enable core of Linux procs. -- Have fun! chd From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 11:20:44 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C3E01148 for ; Sat, 28 Mar 2015 11:20:44 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F4C8600 for ; Sat, 28 Mar 2015 11:20:44 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Ybon5-000GRO-I0 for freebsd-hackers@freebsd.org; Sat, 28 Mar 2015 14:20:35 +0300 Date: Sat, 28 Mar 2015 14:20:35 +0300 From: Slawa Olhovchenkov To: freebsd-hackers@freebsd.org Subject: irq cpu binding Message-ID: <20150328112035.GZ23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 11:20:44 -0000 Can someone describe how on FreeBSD/amd64 do interrupt handling? Can be interrupt handler (hardware interrupt) direct dispatch to specific CPU core (and only to this core)? Can be all work be only on this core (ithread, device driver interrupt handler, finalise)? From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 10:51:22 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3934FBD5; Sat, 28 Mar 2015 10:51:22 +0000 (UTC) Received: from mx1.riseup.net (mx1.riseup.net [198.252.153.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.riseup.net", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 189151F7; Sat, 28 Mar 2015 10:51:21 +0000 (UTC) Received: from plantcutter.riseup.net (plantcutter-pn.riseup.net [10.0.1.121]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.riseup.net", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.riseup.net (Postfix) with ESMTPS id 1DB864083A; Sat, 28 Mar 2015 10:45:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1427539501; bh=LxMWdmc/Q0HfkDD7Ggcgu4EpOz102cKOp6deGvVozAU=; h=Date:From:To:Subject:From; b=H2yCVOS09+BSvw8hzw1Zr0GD6Z50U0NVDkkfQ37KKj4YlgQH2BWr2hcv97kVsxm4p r0Y3quCs0eOIUlqrMNBmuSIEcoLJeZAZdirkvI+2s9vuMhpFfoOwylD10vYMiL1Kcu gB4mLhVqd3G3I/tct0faP+SN7oqxnLh2QLI4qs88= Received: from [127.0.0.1] (localhost [127.0.0.1]) (Authenticated sender: pkubaj) with ESMTPSA id D441420968 Message-ID: <55168627.5060509@riseup.net> Date: Sat, 28 Mar 2015 11:44:55 +0100 From: Piotr Kubaj User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-current@freebsd.org, freebsd-hackers@freebsd.org Subject: Libreboot X200 and FreeBSD Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.98.6 at mx1 X-Virus-Status: Clean X-Mailman-Approved-At: Sat, 28 Mar 2015 11:27:46 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 10:51:22 -0000 I want to buy a Libreboot X200 notebook, however, I also want to use it with FreeBSD. I'm not sure if that works, so I asked Gluglug directly and got the following response: >I'm given the impression that text-mode graphics initialization used >to work on the X200, but was broken by a later commit. A bisect might >help. > > fchmmr: if it uses vbt or bios calls, it won't > fchmmr: first one should work, but FreeBSD works >only in text mode > >Text-mode is currently broken on the X200. VBT or "fake VBT" is >currently lacking in the coreboot port for X200 ("VBT calls"), and >lacks INT 10H video services ("BIOS calls"). > >I'm not sure if FreeBSD will work correctly or not. It should work >if you use the Video BIOS extracted from the original firmware >(libreboot won't use or recommend this, because it's proprietary; >instead, it uses a free but incomplete implementation called >"native graphics initialization"). Can some FreeBSD developers make a statement on this topic? If it doesn't work, then I can test some patches, if someone writes them, but I'm a sysadmin, not a programmer, so I'm not really able to write some code. From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 12:58:30 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6C1A11F5; Sat, 28 Mar 2015 12:58:30 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E5812FA5; Sat, 28 Mar 2015 12:58:29 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id t2SCwIfu082572 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 28 Mar 2015 13:58:19 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from laptop.wojtek.intra (localhost [127.0.0.1]) by laptop.wojtek.intra (8.14.9/8.14.9) with ESMTP id t2SCwEfB000824; Sat, 28 Mar 2015 13:58:14 +0100 (CET) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by laptop.wojtek.intra (8.14.9/8.14.9/Submit) with ESMTP id t2SCw9VZ000821; Sat, 28 Mar 2015 13:58:09 +0100 (CET) (envelope-from wojtek@puchar.net) X-Authentication-Warning: laptop.wojtek.intra: wojtek owned process doing -bs Date: Sat, 28 Mar 2015 13:58:09 +0100 (CET) From: Wojciech Puchar X-X-Sender: wojtek@laptop.wojtek.intra To: Alexander Motin Subject: Re: MAXBSIZE increase In-Reply-To: <5515C421.4040703@FreeBSD.org> Message-ID: References: <5515C421.4040703@FreeBSD.org> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Sat, 28 Mar 2015 13:58:20 +0100 (CET) Cc: freebsd-fs@FreeBSD.org, "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 12:58:30 -0000 i routinely put options MAXPHYS=2097152 on my servers. no problems, except MUCH enhanced performance on large files. with UFS. SOME SSDs (few) doesn't work properly with >1MB requests. no idea about ZFS. with NFS do not ever expect good performance with it's sync mode. if you like to take a risk add vfs.nfsd.async=1 to sysctl.conf unless your NFS clients depend on forced syncs, the risk is actually no higher that running normal UFS filesystem locally. And performance is excellent. From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 15:18:43 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B4E7E959; Sat, 28 Mar 2015 15:18:43 +0000 (UTC) Received: from mail-ig0-x22f.google.com (mail-ig0-x22f.google.com [IPv6:2607:f8b0:4001:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B58BE94; Sat, 28 Mar 2015 15:18:43 +0000 (UTC) Received: by igbqf9 with SMTP id qf9so41800027igb.1; Sat, 28 Mar 2015 08:18:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=XRDrHYvGudyO9Nr7THdZMCHKazwQ4kK1GyFbOGWpk+Y=; b=VvrUEmB5YMEZslcWL6fIppFMB0sqnNpb4Y5QqXoiD+NyMEO809Bf195qHbD3w0BprB F6tLu2zYCWD+lMmcYHo4b56xhQo4AZONJnH1j/FewNSHO2pLMTNm/vcIjmlcyn9BMCFq As+EB8hZf6BaRUuAv92cPwnb4Bq5+vRvXUw7OvHiyL0+AgIFejacv9KymzdGqouT7L6K CB7/DrkJkth6JGRwvQLvaZ/U3ntgFQ3gPSPH34flRQAFVx68q8fJn2G2tvowTKiB1n8G jv8g83m8uCsgJYZnHaXm2XOimemdHzOVhS8zpPBPuZn7CabJ6dSBFrMaQ00iCWkNOX7j 6pwA== MIME-Version: 1.0 X-Received: by 10.50.67.100 with SMTP id m4mr5900602igt.32.1427555922957; Sat, 28 Mar 2015 08:18:42 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 08:18:42 -0700 (PDT) In-Reply-To: <55168627.5060509@riseup.net> References: <55168627.5060509@riseup.net> Date: Sat, 28 Mar 2015 08:18:42 -0700 X-Google-Sender-Auth: I7BMolU9JSVxa9EH4ftB0LaQEdM Message-ID: Subject: Re: Libreboot X200 and FreeBSD From: Adrian Chadd To: Piotr Kubaj Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , freebsd-current X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 15:18:43 -0000 Which intel chipset is in that thing? -a From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 15:20:09 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7CF90BBE for ; Sat, 28 Mar 2015 15:20:09 +0000 (UTC) Received: from mail-ig0-x231.google.com (mail-ig0-x231.google.com [IPv6:2607:f8b0:4001:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43BF4F41 for ; Sat, 28 Mar 2015 15:20:09 +0000 (UTC) Received: by ignm3 with SMTP id m3so47918453ign.0 for ; Sat, 28 Mar 2015 08:20:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=PElbFA60ZLX/ES8i/nERjPWv2+VzkyBiTXD/6omu5pM=; b=XQd8Uu0yg7lFuIjv+MS4Z5gC8duxx4SZyX+fhiQaPMZdLoa3K6BZcvqe2QNNDaw1sm Ifka2NBQ00Ier9xRrm5Ma+40g3ZlYLtsv69adytJJorWddkAwJ9SNOH3MqRRG6fGeZWl 0BHREtFrXYdHuBT9IMdcLPpISJqVVCUAFlD7mLfRqwRFNpolTNp03NXcaWk7tDy1b4wL WZkXHadtrdO+13UpAweRO2bIt9VDUs6Epvu0CRipOUUbrXNQtf07aG1qtatAkOTjcHdb YIuvX6ifQbEM3KhgornotnxzTlPHDNI+ca1w5w5yRPwdz6RnmRUwohC27JKPTPEVhRYC WAmQ== MIME-Version: 1.0 X-Received: by 10.42.41.200 with SMTP id q8mr46925469ice.61.1427556008684; Sat, 28 Mar 2015 08:20:08 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 08:20:08 -0700 (PDT) In-Reply-To: <20150328112035.GZ23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 08:20:08 -0700 X-Google-Sender-Auth: MTITpZjAZ0CkQ2FxHSqMM84g5yc Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 15:20:09 -0000 On 28 March 2015 at 04:20, Slawa Olhovchenkov wrote: > Can someone describe how on FreeBSD/amd64 do interrupt handling? > Can be interrupt handler (hardware interrupt) direct dispatch to > specific CPU core (and only to this core)? > Can be all work be only on this core (ithread, device driver interrupt > handler, finalise)? Yes - you can use cpuset on the interrupt to get them bound that way. John and I are trying to make that whole process more automated and NUMA friendly. I'm debugging some of his work at the moment. -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 15:40:36 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 007CA5D5; Sat, 28 Mar 2015 15:40:35 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AD867188; Sat, 28 Mar 2015 15:40:35 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Ybsqe-000Kmu-0Y; Sat, 28 Mar 2015 18:40:32 +0300 Date: Sat, 28 Mar 2015 18:40:31 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328154031.GA23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 15:40:36 -0000 On Sat, Mar 28, 2015 at 08:20:08AM -0700, Adrian Chadd wrote: > On 28 March 2015 at 04:20, Slawa Olhovchenkov wrote: > > Can someone describe how on FreeBSD/amd64 do interrupt handling? > > Can be interrupt handler (hardware interrupt) direct dispatch to > > specific CPU core (and only to this core)? > > Can be all work be only on this core (ithread, device driver interrupt > > handler, finalise)? > > Yes - you can use cpuset on the interrupt to get them bound that way. > > John and I are trying to make that whole process more automated and > NUMA friendly. I'm debugging some of his work at the moment. cpuset don't work as expected -- I see irq handling on other cpu. From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 15:57:40 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 670A3E50; Sat, 28 Mar 2015 15:57:40 +0000 (UTC) Received: from mx1.riseup.net (mx1.riseup.net [198.252.153.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.riseup.net", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4379D2ED; Sat, 28 Mar 2015 15:57:39 +0000 (UTC) Received: from berryeater.riseup.net (berryeater-pn.riseup.net [10.0.1.120]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.riseup.net", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.riseup.net (Postfix) with ESMTPS id D684940A70; Sat, 28 Mar 2015 15:57:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1427558258; bh=phV38gyFBw4SvDzEjmBVeXX4FYYdM27bvOlbZH7IX4Y=; h=Date:From:To:CC:Subject:References:In-Reply-To:From; b=nXnocxujQewp62WGfRaPzk45iRmrk07Mn1CPTeWrofzeykxoRomCJTLBtOBR4C4BY yCFEc/ZRkvD0HcSTI6OZXQ2Ni4C491GgnVucJPD9e/mTSyX7d4Yi3CR6gPt5ORabWm vVrislp3KzsuuJtwmknhuCem3WxJzcFGX2sKjaqI= Received: from [127.0.0.1] (localhost [127.0.0.1]) (Authenticated sender: pkubaj) with ESMTPSA id B15893F772 Message-ID: <5516CF6C.6080808@riseup.net> Date: Sat, 28 Mar 2015 16:57:32 +0100 From: Piotr Kubaj User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: Libreboot X200 and FreeBSD References: <55168627.5060509@riseup.net> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.98.6 at mx1 X-Virus-Status: Clean X-Mailman-Approved-At: Sat, 28 Mar 2015 16:43:52 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 15:57:40 -0000 On 03/28/15 16:18, Adrian Chadd wrote: > Which intel chipset is in that thing? > > > > -a > It's a modified Thinkpad X200, so it uses the same chipset, i.e. GM45. From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 17:13:21 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BCF82247; Sat, 28 Mar 2015 17:13:21 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2DF39C43; Sat, 28 Mar 2015 17:13:21 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t2SHDFZ8056343 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 28 Mar 2015 19:13:15 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t2SHDFZ8056343 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t2SHDFlN056342; Sat, 28 Mar 2015 19:13:15 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 28 Mar 2015 19:13:15 +0200 From: Konstantin Belousov To: Alexander Motin Subject: Re: MAXBSIZE increase Message-ID: <20150328171315.GU2379@kib.kiev.ua> References: <5515C421.4040703@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5515C421.4040703@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-fs@FreeBSD.org, "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 17:13:21 -0000 On Fri, Mar 27, 2015 at 10:57:05PM +0200, Alexander Motin wrote: > Hi. > > Experimenting with NFS and ZFS I found an inter-operation issue: ZFS by > default uses block of 128KB, while FreeBSD NFS (both client and server) > is limited to 64KB requests by the value of MAXBSIZE. On file rewrite > that limitation makes ZFS to do slow read-modify-write cycles for every > write operation, instead of just writing the new data. Trivial iozone > test show major difference between initial write and rewrite speeds > because of this issue. > > Looking through the sources I've found and in r280347 fixed number of > improper MAXBSIZE use cases in device drivers. After that I see no any > reason why MAXBSIZE can not be increased to at least 128KB to match ZFS > default (ZFS now supports block up to 1MB, but that is not default and > so far rare). I've made a test build and also successfully created UFS > file system with 128KB block -- not sure it is needed, but seems it > survives this change well too. > > Is there anything I am missing, or it is safe to rise this limit now? This post is useless after the Bruce explanation, but I still want to highlidht the most important point from that long story: increasing MAXBSIZE without tuning other buffer cache parameters would dis-balance the buffer cache. Allowing bigger buffers increases fragmentation, while limiting the total number of buffers. Also, it changes the tuning for runtime limits for amount of io in flight, see hi/lo runningspace initialization. From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 17:21:01 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E78CB6B5; Sat, 28 Mar 2015 17:21:01 +0000 (UTC) Received: from mail-wg0-x22b.google.com (mail-wg0-x22b.google.com [IPv6:2a00:1450:400c:c00::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 88900D23; Sat, 28 Mar 2015 17:21:01 +0000 (UTC) Received: by wgdm6 with SMTP id m6so129733833wgd.2; Sat, 28 Mar 2015 10:21:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=1LxlOfOLUxU2KEooV0UB8p7GkDjYuebqp2Yl3dKx37E=; b=czemo3O70ZFPTpIKvn+HKdDo5xE7le6R/OtWrqe+kss7sZuLBmnI+RjH9nDbu5DoQz TSR9cs3Z9+MdJ/jtoYjjWfgAoueN5oUCiLJhyekhiVjf60g9Yw04eLOGiis7+1TbS/FN AB5ZQLJ2GgOhbiB2iIQcUdvU4EijvmRzQkgKFuVciE2nxQGH5Anpc0mdguqwSfdHEEIu ky6tXk67LEBNVHjIKqYeeGX4YGyxWicIj4eZwfQYXVGiPwr05T1QSnaDOxZR0uZmljfV DwDVWTkMMaQ4nj8AExPYw83ILWSKvqVaIqp3UEwCqniKjBjUsez4ydRVFpzmW2Ny3R4e es6Q== X-Received: by 10.194.185.68 with SMTP id fa4mr46456489wjc.111.1427563260021; Sat, 28 Mar 2015 10:21:00 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id ga8sm7884714wib.11.2015.03.28.10.20.58 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 Mar 2015 10:20:59 -0700 (PDT) Sender: Alexander Motin Message-ID: <5516E2F9.20205@FreeBSD.org> Date: Sat, 28 Mar 2015 19:20:57 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: MAXBSIZE increase References: <5515C421.4040703@FreeBSD.org> <20150328171315.GU2379@kib.kiev.ua> In-Reply-To: <20150328171315.GU2379@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 17:21:02 -0000 On 28.03.2015 19:13, Konstantin Belousov wrote: > On Fri, Mar 27, 2015 at 10:57:05PM +0200, Alexander Motin wrote: >> Experimenting with NFS and ZFS I found an inter-operation issue: ZFS by >> default uses block of 128KB, while FreeBSD NFS (both client and server) >> is limited to 64KB requests by the value of MAXBSIZE. On file rewrite >> that limitation makes ZFS to do slow read-modify-write cycles for every >> write operation, instead of just writing the new data. Trivial iozone >> test show major difference between initial write and rewrite speeds >> because of this issue. >> >> Looking through the sources I've found and in r280347 fixed number of >> improper MAXBSIZE use cases in device drivers. After that I see no any >> reason why MAXBSIZE can not be increased to at least 128KB to match ZFS >> default (ZFS now supports block up to 1MB, but that is not default and >> so far rare). I've made a test build and also successfully created UFS >> file system with 128KB block -- not sure it is needed, but seems it >> survives this change well too. >> >> Is there anything I am missing, or it is safe to rise this limit now? > > This post is useless after the Bruce explanation, but I still want to > highlidht the most important point from that long story: > > increasing MAXBSIZE without tuning other buffer cache parameters > would dis-balance the buffer cache. Allowing bigger buffers increases > fragmentation, while limiting the total number of buffers. Also, it > changes the tuning for runtime limits for amount of io in flight, see > hi/lo runningspace initialization. I would be happy if somebody with more skills in buffer cache brought some order into that area, hopefully once and forever. -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 17:38:42 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 20B3DEE7 for ; Sat, 28 Mar 2015 17:38:42 +0000 (UTC) Received: from mail-pa0-x233.google.com (mail-pa0-x233.google.com [IPv6:2607:f8b0:400e:c03::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E00F2E57 for ; Sat, 28 Mar 2015 17:38:41 +0000 (UTC) Received: by pabxg6 with SMTP id xg6so125880325pab.0 for ; Sat, 28 Mar 2015 10:38:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=hKlljpJF0wDB0RjAzmGFXmAHEKVZrECR3AAeDp0xg7A=; b=XBTryFEMhRhDVmaCsvkRWZw+z2GDaxcKqZXYEeX2K4cZcg2O9VQdAvGPR752EA4+cd pHWrR4xhytlYRCsShK9erOzTr1cGwOnZp5bb3RwF4omgApFObxZKzltYKRA9D4DYvgYZ ugzd1ZrwGLt+ajVMhySoyuePiGH2eGPjHA7g+wWsxPp41fblEJKb04PvZcN+fNAqHTiv u30TlTAUgTNHzDTx3XSijm+/7rswfPthcvB11tC7jB9qnhjy/uy7j12ktGB1NLhjmZKu Iv+DgAGLhnLkyM+QSxCfjszueThr4MoDqmTewZ0Igw7kzatCye7vJCba81g6Eat04LcT cQGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=hKlljpJF0wDB0RjAzmGFXmAHEKVZrECR3AAeDp0xg7A=; b=APbHpYsQUzVs3hQ+zl4iAGmnZZ/7N/xHO6+Qht4Btu7PuQ08O2bZ2fmeRhecNeI26B 8ReLH4tvJG0SHqN+iTOvd3SRtJn7HnIus/qj4GaqUmbGszdAeQrURuKSfoiYaYUBa026 xwiYDaEQQAGuGSWfecrbctFwHV8Og0RvbS5Ihcb9KNMTVNrUbCzWZC0eDNZ3rKoK7F5X 8tpZEhojG7+p7qFSwN08WuqngLQjoE7ksCsa6afNX2fNRRg0obgi/z7YYjDhqtZORLWA m+t7ltt+bhcWVfn0LNkjo6hSWclhSk87Hszdlq1/0NXLnWn+73ZtmWIbrwIXb6g4qUSl OCaA== X-Gm-Message-State: ALoCoQn8rxdP6GVcLC6D+hRrcDYpb4tWHgtmdLioPAhSjFomOBFe96RGkbjLdzlQuRd4xClsWTUQ MIME-Version: 1.0 X-Received: by 10.70.88.234 with SMTP id bj10mr44522553pdb.165.1427564321352; Sat, 28 Mar 2015 10:38:41 -0700 (PDT) Received: by 10.70.67.129 with HTTP; Sat, 28 Mar 2015 10:38:41 -0700 (PDT) Date: Sat, 28 Mar 2015 10:38:41 -0700 Message-ID: Subject: Export docs to EPUB project idea - any takers yet? From: In-Ho Yi To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 17:38:42 -0000 Hi, I've posted this earlier to freebsd-doc@ but I am re-posting it here. I've been looking through project ideas as I want to start participating in FreeBSD development. I found the export to EPUB idea, and I thought this would be a good project to take. I wonder if anyone has started working on this project idea. If not, could anyone enlighten me as to how I get started working on the idea please. Thanks! From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 17:42:31 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3350A251; Sat, 28 Mar 2015 17:42:31 +0000 (UTC) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC203F17; Sat, 28 Mar 2015 17:42:30 +0000 (UTC) Received: by ierf6 with SMTP id f6so24314510ier.2; Sat, 28 Mar 2015 10:42:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=NiBuzobvER+bULiTc6Tic+E5bx1jI9zCK0ts2zz6qpw=; b=U8tPAfWJpHOdCDeugxT0HiiWcPETnU8TBHbUKK5sI3JdSWmhyAnl1srCAUaclLasep zK4DeqVnpjY9Qvd4FuKl4+j7z+lK8gL74ST/fImhVhaatCy/oio69ahAXuh8UcBFx5Qq Wzvl6PWkK/lIrWFW3ogc+jGjzr5ZJfBMOtB7lc7SIbED/qTbAKwuEPuWPLVS6xXB4GNj z9SMJo5CI9Z0dXbDQ4vAWTomXAMVJfI/8OlXJkRev/9tZOF1MALDbTyJFuviYoNDK+Vg O6jtNCdGFLJAaFUmXLkrGsKuQsN+KCfhC4BZv7wdnUWMiRWmEhe/w9xVCymVXrDgA9QA 2OfA== MIME-Version: 1.0 X-Received: by 10.107.39.72 with SMTP id n69mr22311356ion.8.1427564550309; Sat, 28 Mar 2015 10:42:30 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 10:42:30 -0700 (PDT) In-Reply-To: <5516CF6C.6080808@riseup.net> References: <55168627.5060509@riseup.net> <5516CF6C.6080808@riseup.net> Date: Sat, 28 Mar 2015 10:42:30 -0700 X-Google-Sender-Auth: 74pcj3PHQakpbC-W9oBnukEN0Y0 Message-ID: Subject: Re: Libreboot X200 and FreeBSD From: Adrian Chadd To: Piotr Kubaj Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , freebsd-current X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 17:42:31 -0000 Oh, in that case, someone should send me one so I can use it to verify that my frame-buffer bootloader hack will work correctly on it. Then yeah, we won't have to worry about such evil BIOSes. -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 17:43:08 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9788542B for ; Sat, 28 Mar 2015 17:43:08 +0000 (UTC) Received: from mail-ig0-x235.google.com (mail-ig0-x235.google.com [IPv6:2607:f8b0:4001:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B485F27 for ; Sat, 28 Mar 2015 17:43:08 +0000 (UTC) Received: by igbud6 with SMTP id ud6so44720286igb.1 for ; Sat, 28 Mar 2015 10:43:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=G3PSJH1uhyR8qWe5i8Q1j6DsrgDzG68/04WOQvx7S2Q=; b=0khJ1TPfJoVlDwtikfqArEoUMGrm6mx0ausdvUZ1HDDSYujSGEqr71WHyUBwjkMpSl j3EKooIzt+Migd2Rsn7VVDCYTV3TkEzqQhqU3Y4/tp2K22ZNdGTkoAC1opB7UC4G+yTC VG0+MO85PgVq4uT9CKGlusqRy13v8zXWrQmKEsbDT1f0WehekJcLRbDBMm1navVi3otD Tk1/T9ll11xUG17B6gBBCZhdHAa5daRsqsYNChKUgSExzL/amOHccvcKlsdFG+EsCFzc 9OqEt3UGW5Q1NvQSvukX14AkCQjOOsGzJ7DnqnwQQo2VRxnjWMhxc9Ja5jL/BcGfQ9K8 YXTA== MIME-Version: 1.0 X-Received: by 10.107.155.13 with SMTP id d13mr37069376ioe.29.1427564587791; Sat, 28 Mar 2015 10:43:07 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 10:43:07 -0700 (PDT) In-Reply-To: <20150328154031.GA23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> <20150328154031.GA23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 10:43:07 -0700 X-Google-Sender-Auth: -iHmPfe5GFn6vKZ0in9l6PJMYpA Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 17:43:08 -0000 On 28 March 2015 at 08:40, Slawa Olhovchenkov wrote: > On Sat, Mar 28, 2015 at 08:20:08AM -0700, Adrian Chadd wrote: > >> On 28 March 2015 at 04:20, Slawa Olhovchenkov wrote: >> > Can someone describe how on FreeBSD/amd64 do interrupt handling? >> > Can be interrupt handler (hardware interrupt) direct dispatch to >> > specific CPU core (and only to this core)? >> > Can be all work be only on this core (ithread, device driver interrupt >> > handler, finalise)? >> >> Yes - you can use cpuset on the interrupt to get them bound that way. >> >> John and I are trying to make that whole process more automated and >> NUMA friendly. I'm debugging some of his work at the moment. > > cpuset don't work as expected -- I see irq handling on other cpu. Well, when you see "irq handling on other cpu", what do you mean? How are you using cpuset to move things around? -a From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 18:10:30 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5D36FFB1; Sat, 28 Mar 2015 18:10:30 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1331324C; Sat, 28 Mar 2015 18:10:30 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YbvBi-000Okc-NQ; Sat, 28 Mar 2015 21:10:26 +0300 Date: Sat, 28 Mar 2015 21:10:26 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328181026.GB23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> <20150328154031.GA23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 18:10:30 -0000 On Sat, Mar 28, 2015 at 10:43:07AM -0700, Adrian Chadd wrote: > On 28 March 2015 at 08:40, Slawa Olhovchenkov wrote: > > On Sat, Mar 28, 2015 at 08:20:08AM -0700, Adrian Chadd wrote: > > > >> On 28 March 2015 at 04:20, Slawa Olhovchenkov wrote: > >> > Can someone describe how on FreeBSD/amd64 do interrupt handling? > >> > Can be interrupt handler (hardware interrupt) direct dispatch to > >> > specific CPU core (and only to this core)? > >> > Can be all work be only on this core (ithread, device driver interrupt > >> > handler, finalise)? > >> > >> Yes - you can use cpuset on the interrupt to get them bound that way. > >> > >> John and I are trying to make that whole process more automated and > >> NUMA friendly. I'm debugging some of his work at the moment. > > > > cpuset don't work as expected -- I see irq handling on other cpu. > > Well, when you see "irq handling on other cpu", what do you mean? > How are you using cpuset to move things around? I have dual Xenon and dual 82599EB (handled bu second socket). irq270: ix0:que 0 293090597 2483 irq271: ix0:que 1 290379344 2460 irq272: ix0:que 2 292648245 2479 irq273: ix0:link 1 0 irq274: ix1:que 0 294816977 2498 irq275: ix1:que 1 292665696 2479 irq276: ix1:que 2 294411404 2494 irq277: ix1:link 2 0 First, I do from 'cpuset -l 6 -x 270' to 'cpuset -l 11 -x 276'. I try 'cpuset -l 0 pmcstat -S CPU_CLK_UNHALTED_CORE -O sample.out -c 2 -l 10' and next pmcstat -R sample.out -G out.txt. And I see many ixgbe_msix_que in out.txt (I save out.txt, you can see this, plot flame graph, etc). After this I see in `ps -axdHO lwp | grep ix` more then one irq handler: # ps -axdHO lwp | grep ix 94661 100976 0 S+ 0:00.00 | | `-- grep ix 12 100064 - WL 840:32.50 - [intr/irq270: ix0:qu] 12 100066 - WL 843:33.09 - [intr/irq271: ix0:qu] 12 100068 - RL 828:25.61 - [intr/irq272: ix0:qu] 12 100070 - WL 0:00.00 - [intr/irq273: ix0:li] 12 100072 - RL 858:21.86 - [intr/irq274: ix1:qu] 12 100074 - RL 858:43.72 - [intr/irq275: ix1:qu] 12 100076 - WL 843:01.04 - [intr/irq276: ix1:qu] 12 100078 - WL 0:00.00 - [intr/irq277: ix1:li] 0 100065 - RLs 70:03.18 [kernel/ix0 que] 0 100067 - RLs 71:06.67 [kernel/ix0 que] 0 100069 - DLs 68:23.48 [kernel/ix0 que] 0 100071 - DLs 0:01.48 [kernel/ix0 linkq] 0 100073 - DLs 50:25.37 [kernel/ix1 que] 0 100075 - DLs 49:08.89 [kernel/ix1 que] 0 100077 - DLs 48:18.12 [kernel/ix1 que] 0 100079 - DLs 0:00.00 [kernel/ix1 linkq] I am don't know what thread binded by 'cpuset -x'. I think intr/irqNNN (as more CPU spending). I do 'cpuset -l 6 -t 100065' .. 'cpuset -l 11 -t 100077'. After this I do `cpuset -l 0 pmcstat -S CPU_CLK_UNHALTED_CORE -O sample2.out -c 2 -l 10` and `pmcstat -R sample2.out -G out2.txt` (I save out2.txt). I still see ixgbe_msix_que: 06.94% [17981] tcp_output @ /boot/kernel/kernel 100.0% [17981] tcp_do_segment 100.0% [17981] tcp_input 100.0% [17981] ip_input 100.0% [17981] netisr_dispatch_src 100.0% [17981] ether_demux 100.0% [17981] ether_nh_input 100.0% [17981] netisr_dispatch_src 97.56% [17542] ixgbe_rxeof @ /boot/kernel/if_ixgbe.ko 75.54% [13252] ixgbe_msix_que 100.0% [13252] intr_event_execute_handlers @ /boot/kernel/kernel 100.0% [13252] ithread_loop 100.0% [13252] fork_exit 24.46% [4290] ixgbe_handle_que @ /boot/kernel/if_ixgbe.ko 100.0% [4290] taskqueue_run_locked @ /boot/kernel/kernel 100.0% [4290] taskqueue_thread_loop 100.0% [4290] fork_exit 02.44% [439] tcp_lro_flush 96.58% [424] ixgbe_rxeof @ /boot/kernel/if_ixgbe.ko 100.0% [424] ixgbe_msix_que 100.0% [424] intr_event_execute_handlers @ /boot/kernel/kernel 100.0% [424] ithread_loop 100.0% [424] fork_exit 03.42% [15] tcp_lro_rx 100.0% [15] ixgbe_rxeof @ /boot/kernel/if_ixgbe.ko 100.0% [15] ixgbe_msix_que 100.0% [15] intr_event_execute_handlers @ /boot/kernel/kernel 100.0% [15] ithread_loop 100.0% [15] fork_exit From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 18:23:57 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 15653533 for ; Sat, 28 Mar 2015 18:23:57 +0000 (UTC) Received: from mail-ie0-x22b.google.com (mail-ie0-x22b.google.com [IPv6:2607:f8b0:4001:c03::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CCD58391 for ; Sat, 28 Mar 2015 18:23:56 +0000 (UTC) Received: by iecvj10 with SMTP id vj10so91778074iec.0 for ; Sat, 28 Mar 2015 11:23:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=n6yIG4ueYCfBXMlJy3UKQra5AxvheW/OA1hicM4fcg8=; b=dFHpk6dhBM/xojIxpxM4T6wTZ6QgH1tijauXzcJ3TbPt+c0SxXjLUx6r5dFAyFobqM TVIitqKgqyeQkkW0fQu41EKBgaQ9Wa6OjVGeK4gSw1K1I6z15QgK+ITrtmeJWtxx2441 npz5+FlfExc/AEpQMrvXOSt6PD1KaoHsc+kL7gyi9VM2muy490DJOsDCTqSDoXoEsqVQ 9xQubXTth3SQEbVwpDq2y97bx4OHQO4l/pbp9T9ARNNbzWDhwBKzlZjSmwPAFDQKCsTz oVJVGBUD/R0hmbEHTPqOvsjp/x2CmQNbtxNPgOG4QQblPuhsFgB4QisMWUcGmTXi1Mld ZR1Q== MIME-Version: 1.0 X-Received: by 10.107.39.72 with SMTP id n69mr22501581ion.8.1427567036073; Sat, 28 Mar 2015 11:23:56 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 11:23:56 -0700 (PDT) In-Reply-To: <20150328181026.GB23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 11:23:56 -0700 X-Google-Sender-Auth: xMd557BZ8tuxEoHLMVSoU0yg_Eo Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 18:23:57 -0000 Ah, I think that's because the taskqueues in the driver for deferred handling aren't also being pinned. I've talked to John about this - the problem is that all the taskqueues for all the drivers run under one kernel process. Find out their threadids and pin them too. Eg: # procstat -ta | grep em0 0 100024 kernel em0 que -1 8 sleep - 0 100025 kernel em0 txq -1 8 sleep - # vmstat -ia | grep em irq256: em0 68465 1 # cpuset -g -x 256 irq 256 mask: 0, 1 # cpuset -g -t 100024 tid 100024 mask: 0, 1 # cpuset -g -t 100025 tid 100025 mask: 0, 1 So you'd have to manually do that - there's no generic interface at the moment to be able to ask a device driver to re-mask its taskqueue thread(s) for a given queue and rewire its interrupt(s) for that queue. (That would be a nice smallish project to prototype, btw.) -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 18:31:50 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 05E31A8C; Sat, 28 Mar 2015 18:31:50 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AEFA8692; Sat, 28 Mar 2015 18:31:49 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YbvWN-000P9t-Fn; Sat, 28 Mar 2015 21:31:47 +0300 Date: Sat, 28 Mar 2015 21:31:47 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328183147.GC23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 18:31:50 -0000 On Sat, Mar 28, 2015 at 11:23:56AM -0700, Adrian Chadd wrote: > Ah, I think that's because the taskqueues in the driver for deferred > handling aren't also being pinned. > > I've talked to John about this - the problem is that all the > taskqueues for all the drivers run under one kernel process. Find out > their threadids and pin them too. > > Eg: > > # procstat -ta | grep em0 > 0 100024 kernel em0 que -1 8 sleep - > 0 100025 kernel em0 txq -1 8 sleep - # procstat -ta | grep ix0 0 100065 kernel ix0 que 6 8 run - 0 100067 kernel ix0 que 7 8 run - 0 100069 kernel ix0 que 8 8 run - 0 100071 kernel ix0 linkq 7 8 sleep - 12 100064 intr irq270: ix0:que 6 8 run - 12 100066 intr irq271: ix0:que 7 8 run - 12 100068 intr irq272: ix0:que 8 8 run - 12 100070 intr irq273: ix0:link 0 8 wait - As you see -- I am already pined all. # cpuset -g -x 270 irq 270 mask: 6 # cpuset -g -t 100065 tid 100065 mask: 6 # cpuset -g -t 100064 tid 100064 mask: 6 > # vmstat -ia | grep em > irq256: em0 68465 1 > # cpuset -g -x 256 > irq 256 mask: 0, 1 > # cpuset -g -t 100024 > tid 100024 mask: 0, 1 > # cpuset -g -t 100025 > tid 100025 mask: 0, 1 > > So you'd have to manually do that - there's no generic interface at > the moment to be able to ask a device driver to re-mask its taskqueue > thread(s) for a given queue and rewire its interrupt(s) for that > queue. > > (That would be a nice smallish project to prototype, btw.) From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 19:21:36 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 164C99B for ; Sat, 28 Mar 2015 19:21:36 +0000 (UTC) Received: from mail-ie0-x22d.google.com (mail-ie0-x22d.google.com [IPv6:2607:f8b0:4001:c03::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC8D4C28 for ; Sat, 28 Mar 2015 19:21:35 +0000 (UTC) Received: by iecvj10 with SMTP id vj10so92261227iec.0 for ; Sat, 28 Mar 2015 12:21:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=pE5d2whU6rlao1ipaBZ4YoaEXue9K6YhmDS9gHuhn8w=; b=lB77i7bOMPZBLrIoHGkEAv1SzcmXEVQhFZN0PsS09rd2iEf+OIbV7QlKLRAFKRI+2B WkDRNM6S5wSULdhrcSiCXScchpOpenTlL14B+TeHVpbyPaT5rnsSgiZFi84cA03eURxD AAZAYaU3ai7trwFeuNgBo8VeyOVvvOM/VSPbm2kGQG/HvBCohdRlcREmIVbR4VYH9Gup W7gbuCaEGHqxbJKY/9JLTCYoiEtifel1FMZY8aw92Ez5nD0wx2EZEt+P3PxQLZHv6Ap6 gL2U3WjLLwtvTSmK5X+liMEVhMIIyRqVs14tfGiJJLY3/sH+aqWVRMKwU0hLSa3iYOTA yK5Q== MIME-Version: 1.0 X-Received: by 10.50.143.42 with SMTP id sb10mr6986800igb.49.1427570495232; Sat, 28 Mar 2015 12:21:35 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 12:21:35 -0700 (PDT) In-Reply-To: <20150328183147.GC23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 12:21:35 -0700 X-Google-Sender-Auth: Fc6TZnlz0oL4rWvJVmLX0McdimI Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 19:21:36 -0000 Hm, there's some other taskqueue thread oddness going on. I know that when I call it using taskqueue_start_threads_cpuset() (with RSS enabled), it's doing the right thing. Hm! -a From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 19:25:09 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 49AC0219; Sat, 28 Mar 2015 19:25:09 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F2C38C4F; Sat, 28 Mar 2015 19:25:08 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YbwLx-00009k-NN; Sat, 28 Mar 2015 22:25:05 +0300 Date: Sat, 28 Mar 2015 22:25:05 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328192505.GD23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 19:25:09 -0000 On Sat, Mar 28, 2015 at 12:21:35PM -0700, Adrian Chadd wrote: > Hm, there's some other taskqueue thread oddness going on. I know that > when I call it using taskqueue_start_threads_cpuset() (with RSS > enabled), it's doing the right thing. > > Hm! What about APIC? Where IRQ handled initiated? What CPU handled APIC request? From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 19:33:53 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CD1C33E6 for ; Sat, 28 Mar 2015 19:33:53 +0000 (UTC) Received: from mail-ie0-x22e.google.com (mail-ie0-x22e.google.com [IPv6:2607:f8b0:4001:c03::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8DDA2D4A for ; Sat, 28 Mar 2015 19:33:53 +0000 (UTC) Received: by iedfl3 with SMTP id fl3so104017868ied.1 for ; Sat, 28 Mar 2015 12:33:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=dNfRxC3bBN3r10LMzdCSCprIunjcrADXZKtCpjQJF3A=; b=x+2MqsnyWYeCucJEgb4MzxxresnT1UHDXi+qTVz1fHUFqIMLHgXcIUrpT0F4Z3qFOl BOeQhhX3cpzEUwy7IxU8GRMt4fjRXq8+NptJ6Wkkj21ZdJl+ybPAnnzeIFGR3aLufnPm c1QqGM0NrVthROOriR3jlO0uKIMHBEhv26YEbLhcIBlleoHr85stKMIQT6rKLVy5Csv7 ivyKRn2m/cAS7p8mhow9Nc3vv7DVKBISQC/UiYOemNcoLW9mREJR/JAqq+CMcmFtS96y oab4lhtROeKayd8UV4F3jrHoQEfByxBZaMYzgYLgZLDDFqFarHEn/As64TrPWh3GIlTo iwGQ== MIME-Version: 1.0 X-Received: by 10.50.67.100 with SMTP id m4mr7118263igt.32.1427571232963; Sat, 28 Mar 2015 12:33:52 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 12:33:52 -0700 (PDT) In-Reply-To: <20150328192505.GD23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 12:33:52 -0700 X-Google-Sender-Auth: TOSVHU-RQykHyBh_V4b3-rIuYok Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 19:33:53 -0000 That's done deferred by the bus interrupt wiring. That's something John's been looking into as part of the general NUMA work (and I'm trying to debug right now, on dual-socket boxes with ixgbe. :-) Look at bus_bind_intr() and the twisty path to intr_event_bind(), then x86/x86/intr_machdep.c:intr_assign_cpu(), then intr_shuffle_cpus() at boot, versus what happens via calls to pic_assign_cpu to setup the wiring. -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 19:50:01 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BAFEEAB4; Sat, 28 Mar 2015 19:50:01 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6DD25E93; Sat, 28 Mar 2015 19:50:01 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Ybwk3-0000Zy-DI; Sat, 28 Mar 2015 22:49:59 +0300 Date: Sat, 28 Mar 2015 22:49:59 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328194959.GE23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 19:50:01 -0000 On Sat, Mar 28, 2015 at 12:33:52PM -0700, Adrian Chadd wrote: > That's done deferred by the bus interrupt wiring. That's something > John's been looking into as part of the general NUMA work (and I'm > trying to debug right now, on dual-socket boxes with ixgbe. :-) > > Look at bus_bind_intr() and the twisty path to intr_event_bind(), then > x86/x86/intr_machdep.c:intr_assign_cpu(), then intr_shuffle_cpus() at > boot, versus what happens via calls to pic_assign_cpu to setup the > wiring. This is very complex to me -- I am not familar with current x86 hardware. Hmm. I see intr_setaffinity and intr_bind. And I don't see using this nor ixgbe nor cxgbe. Anyway, I am re-set CPUs compared to initial assigned in ixgbe/cxgbe drivers. From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 19:54:06 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B8BC1BE3 for ; Sat, 28 Mar 2015 19:54:06 +0000 (UTC) Received: from mail-ig0-x232.google.com (mail-ig0-x232.google.com [IPv6:2607:f8b0:4001:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8146BF46 for ; Sat, 28 Mar 2015 19:54:06 +0000 (UTC) Received: by ignm3 with SMTP id m3so50228089ign.0 for ; Sat, 28 Mar 2015 12:54:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=NejX1r5vcXjdBLY97RL3Of7P6ByARPPDl4eVtswvX/c=; b=sJVNlmKUbrFEuvxvOy7/XQwRvznDYRJ5ABbG4uYy7lCWEpLJeeuSs5Y2R8sKjfineX YJydOA5Qux82yXFUoGf1uTKJNEq+S2OhugGy0A5JY9QabF7PSrhp3NxzqLbKIDd3Br8W OLwLvdgdCtCSH1DJBxpFUPH/2EKZ0Ci/4Pma92ORVgAA4X+PHBikr8aktZ396bYS5aZC 7k747Tm4TKW/L3no4OVRu8MYmvLrHyYYJKfp6cAFSkgjlpX20l2qif13ybIeoS7y0dSw cvw/7+QNrn+z+GvhgNmACDsA9jKTS6eNm/Hx4vsAoHJUlZB1qA6b7EhUcvcDLeQz837B +KiA== MIME-Version: 1.0 X-Received: by 10.50.67.100 with SMTP id m4mr7201774igt.32.1427572445969; Sat, 28 Mar 2015 12:54:05 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 12:54:05 -0700 (PDT) In-Reply-To: <20150328194959.GE23643@zxy.spb.ru> References: <20150328112035.GZ23643@zxy.spb.ru> <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> <20150328194959.GE23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 12:54:05 -0700 X-Google-Sender-Auth: zz6AFgyCosTAZoqGp857-pzCz7k Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 19:54:06 -0000 This is why I started fleshing out more of the RSS stuff - the defaults aren't .. all that useful. ixgbe should be calling bus_bind_intr(). -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 20:12:22 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0D6D344C; Sat, 28 Mar 2015 20:12:22 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B53441E6; Sat, 28 Mar 2015 20:12:21 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Ybx5f-0000wc-G1; Sat, 28 Mar 2015 23:12:19 +0300 Date: Sat, 28 Mar 2015 23:12:19 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328201219.GF23643@zxy.spb.ru> References: <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> <20150328194959.GE23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 20:12:22 -0000 On Sat, Mar 28, 2015 at 12:54:05PM -0700, Adrian Chadd wrote: > This is why I started fleshing out more of the RSS stuff - the > defaults aren't .. all that useful. Do you review implementing RSS as accpet_filter? > ixgbe should be calling bus_bind_intr(). And how to re-bind this later?.. From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 21:52:50 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B568D308 for ; Sat, 28 Mar 2015 21:52:50 +0000 (UTC) Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7539BEC0 for ; Sat, 28 Mar 2015 21:52:50 +0000 (UTC) Received: by igbud6 with SMTP id ud6so47013574igb.1 for ; Sat, 28 Mar 2015 14:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Pagvpr/sQGJ/AATqrCRAkgdq6PD/a6KWwzd5dCDGXbM=; b=QnxjxbKrjcXPJvrDkbM971Rf5R/qWNiDpQhBBHQ3OU5tYxr2z5nGaKkA297ZobrXZ+ D1bzp6lLltzLurAdXd6Ykj0I38uoWyns9gklm1SaR9NVxwLoypmHZQtjT82EUlzsUMjl PFaf7SwthNOVwAEh4027FdZgEQEtYff+ClnUwdAQvWE6ogLqBc903z5/L8INe1QdyoZ4 W2tze4V/VQMPOiwXkyU+TBgMk2a4NVMFdsDrOB/bw1CfTq3qpApU+VbymW3FSRG8g3i2 fu39kDAHgJDyyKN+tTfRKOmcLrqajbu/0Oicns0hfIZk48fECTN54uzXhkyWZnxFTVU4 j6Kw== MIME-Version: 1.0 X-Received: by 10.50.36.65 with SMTP id o1mr7640655igj.32.1427579569883; Sat, 28 Mar 2015 14:52:49 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 14:52:49 -0700 (PDT) In-Reply-To: <20150328201219.GF23643@zxy.spb.ru> References: <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 14:52:49 -0700 X-Google-Sender-Auth: iF7Ysaipi4kEAvrZfD45dWHqS7o Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 21:52:50 -0000 On 28 March 2015 at 13:12, Slawa Olhovchenkov wrote: > On Sat, Mar 28, 2015 at 12:54:05PM -0700, Adrian Chadd wrote: > >> This is why I started fleshing out more of the RSS stuff - the >> defaults aren't .. all that useful. > > Do you review implementing RSS as accpet_filter? If someone has a hack to do that, sure. Thing is, I'm not all that interested in trying to support a hack to make things transparent as there are other things to get "right" for multicore / NUMA performance, and those already require much cpu pinning and such. I'd rather finish off what I've started - expose this stuff via some freebsd libraries to make application developers lives easier, and not try to do more hacks. :) >> ixgbe should be calling bus_bind_intr(). > > And how to re-bind this later?.. That's a question for john. I thought it was being rebound when you reset the irq interrupt via cpuset, but maybe I'm wrong. -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 22:16:23 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DD901962; Sat, 28 Mar 2015 22:16:23 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8FFC610F; Sat, 28 Mar 2015 22:16:23 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Ybz1h-00033V-H2; Sun, 29 Mar 2015 01:16:21 +0300 Date: Sun, 29 Mar 2015 01:16:21 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328221621.GG23643@zxy.spb.ru> References: <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 22:16:24 -0000 On Sat, Mar 28, 2015 at 02:52:49PM -0700, Adrian Chadd wrote: > On 28 March 2015 at 13:12, Slawa Olhovchenkov wrote: > > On Sat, Mar 28, 2015 at 12:54:05PM -0700, Adrian Chadd wrote: > > > >> This is why I started fleshing out more of the RSS stuff - the > >> defaults aren't .. all that useful. > > > > Do you review implementing RSS as accpet_filter? > > If someone has a hack to do that, sure. Thing is, I'm not all that > interested in trying to support a hack to make things transparent as > there are other things to get "right" for multicore / NUMA > performance, and those already require much cpu pinning and such. > > I'd rather finish off what I've started - expose this stuff via some > freebsd libraries to make application developers lives easier, and not > try to do more hacks. :) accpet_filter allow use RSS in nginx w/o any modification, I think. > >> ixgbe should be calling bus_bind_intr(). > > > > And how to re-bind this later?.. > > That's a question for john. I thought it was being rebound when you > reset the irq interrupt via cpuset, but maybe I'm wrong. Do you forward this to john? From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 22:27:37 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 237C3B72 for ; Sat, 28 Mar 2015 22:27:37 +0000 (UTC) Received: from mail-ie0-x22e.google.com (mail-ie0-x22e.google.com [IPv6:2607:f8b0:4001:c03::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D68BE1F0 for ; Sat, 28 Mar 2015 22:27:36 +0000 (UTC) Received: by iedm5 with SMTP id m5so93651109ied.3 for ; Sat, 28 Mar 2015 15:27:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=nYuwD6W9YZlX6bPLSXCKDs0rcY+AZo8UW9Z9b4a6f38=; b=P2/xnw4pQMziK3sPhUThupSlcUcUm3nn+zug5/D/zMWFwDizxoChQCsOSo3MLPWKY2 0G8VEBmjvXmzJKLk17GGfx9vmyrHcuYxvWNjdsCHYm7mMyUka307P1PHjMZtOJEJnADk V6638lId40McSL+PYjrf5ZfWFgLJv90DTD6/SUfZgfXYoMRjQpuTROQJETpyjPR9JAAe 9vT6SM14STHrTkem+Iq0Zf+BXw490K3yNhC25Wf11U7ayRQ+mfjqLnOu8P6FzRUZ4RyR tKRy9e81fVW6ODeCNX9odyTrJBaVShp+jgpZpodq2PYhsQhuu+ZL1f74XM38TJUhKO8t xy/Q== MIME-Version: 1.0 X-Received: by 10.107.5.131 with SMTP id 125mr37836415iof.88.1427581656320; Sat, 28 Mar 2015 15:27:36 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 15:27:36 -0700 (PDT) In-Reply-To: <20150328221621.GG23643@zxy.spb.ru> References: <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> <20150328221621.GG23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 15:27:36 -0700 X-Google-Sender-Auth: 7p7ggFE3xaQ2XAAIq3jOkah6o0o Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 22:27:37 -0000 You still have to pin the nginx threads to whatever your limited set of RSS CPUs are, or the benefits aren't really there. I'd rather we just get librss defined in FreeBSD-HEAD and then have nginx "know" about that for FreeBSD. It already has to know about the linux way of doing it and it has to be conditional on it running Linux. So it's not a big stretch to need to know about FreeBSD. I haven't forwarded anything on to john yet, I'm busy trying to debug the NUMA enumeration/configuration stuff i'm working on and why jhb's patches break that for me. :) -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 22:46:38 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 537E6FDF; Sat, 28 Mar 2015 22:46:38 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 052A23DC; Sat, 28 Mar 2015 22:46:38 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YbzUw-0003VX-Rp; Sun, 29 Mar 2015 01:46:35 +0300 Date: Sun, 29 Mar 2015 01:46:34 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328224634.GH23643@zxy.spb.ru> References: <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> <20150328221621.GG23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 22:46:38 -0000 On Sat, Mar 28, 2015 at 03:27:36PM -0700, Adrian Chadd wrote: > You still have to pin the nginx threads to whatever your limited set > of RSS CPUs are, or the benefits aren't really there. nginx already have this ability: worker_processes 6; worker_cpu_affinity 000001 000010 000100 001000 010000 100000; > I'd rather we just get librss defined in FreeBSD-HEAD and then have > nginx "know" about that for FreeBSD. It already has to know about the > linux way of doing it and it has to be conditional on it running I am don't know about nginx support for RSS in Linux. > Linux. So it's not a big stretch to need to know about FreeBSD. I am try to read RSS implementation. I am don't completely understund it, may be. I found it overingenering and useless in complex setup. For example: dual NIC, LACP. Uniqueue map NIC queue to CPU core: NIC 1 que 0 <=> Core 0 NIC 1 que 1 <=> Core 1 NIC 1 que 2 <=> Core 2 NIC 1 que 3 <=> Core 3 NIC 2 que 0 <=> Core 4 NIC 2 que 1 <=> Core 5 NIC 2 que 2 <=> Core 6 NIC 2 que 3 <=> Core 7 ISP network equipment use proper hash for map flow to network link, you can't predict what NIC got incoming packet (you may assume that all not-fragment packets dispatch to same NIC, yes). This is about case of outgoing connections. From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 22:49:49 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 22534210 for ; Sat, 28 Mar 2015 22:49:49 +0000 (UTC) Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com [IPv6:2607:f8b0:4001:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D5E6C5FB for ; Sat, 28 Mar 2015 22:49:48 +0000 (UTC) Received: by igcau2 with SMTP id au2so51372432igc.1 for ; Sat, 28 Mar 2015 15:49:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=1G1X/cONbsyjoRChSixwwF6eF1QraQu048urPFKrzRc=; b=XGe25eJkV24YwDRenw0jiKK0XtUJdhGktBUwmco+J+yiLCLp7kjebYsR6NNdPk+RaY lXyz/3CKf6a7uJ66YOHoGOOcfCjkW1yHApDd9OZKGxTVOyfsvvPIyoGkjMbhfRL8VtBU y0+YVlIbYkSNTRC++l/EMU9aEeqECLh7i4hJV0WIUlLnMBt7rq0/LSerV34vC54x7xzr VuN0OVXDlIVvA8MWU6oC5IXvQ51M0z083GeFAS+Kulbtk4ihZsaNN7U0udGnLSoo/pBs CBhNyIWYCjd+a9kmqWayEDfX3NwQgCIueggJ5kFBaSRBAwNmWjKwRXCC0Z1hnvq8vcFT wntQ== MIME-Version: 1.0 X-Received: by 10.42.109.12 with SMTP id j12mr11705456icp.22.1427582988253; Sat, 28 Mar 2015 15:49:48 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 15:49:48 -0700 (PDT) In-Reply-To: <20150328224634.GH23643@zxy.spb.ru> References: <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> <20150328221621.GG23643@zxy.spb.ru> <20150328224634.GH23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 15:49:48 -0700 X-Google-Sender-Auth: Ah6R5tSoY-lngA0C4pE8XKvS2Ac Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 22:49:49 -0000 You should totally join #bsdcode on efnet and ask me about it. :) on RSS, this is what would happen: * ALL NICs RSS BUCKET 0 -> core 0 * ... * ALL NICs RSS BUCKET 7 -> core 7 Now, that's not really 100% optimal for NUMA and multiple PCIe controllers, but we're not there yet. Hopefully I can twist/cajole navdeep @ chelsio to continue doing a little more RSS work so I can teach cxgbe/cxl about RSS configuration, but ixgbe, igb and ixl all do the above when RSS is enabled. -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 23:05:36 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5382D77D; Sat, 28 Mar 2015 23:05:36 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 05419843; Sat, 28 Mar 2015 23:05:36 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YbznJ-0003np-T0; Sun, 29 Mar 2015 02:05:33 +0300 Date: Sun, 29 Mar 2015 02:05:33 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328230533.GI23643@zxy.spb.ru> References: <20150328192505.GD23643@zxy.spb.ru> <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> <20150328221621.GG23643@zxy.spb.ru> <20150328224634.GH23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 23:05:36 -0000 On Sat, Mar 28, 2015 at 03:49:48PM -0700, Adrian Chadd wrote: > You should totally join #bsdcode on efnet and ask me about it. :) I am totaly don't use IRC (last 20 years). May be skype? > on RSS, this is what would happen: > > * ALL NICs RSS BUCKET 0 -> core 0 > * ... > * ALL NICs RSS BUCKET 7 -> core 7 My expirens: this is worse vs dedicated core (one core handeled only one bucket of one NIC). > Now, that's not really 100% optimal for NUMA and multiple PCIe > controllers, but we're not there yet. > > Hopefully I can twist/cajole navdeep @ chelsio to continue doing a > little more RSS work so I can teach cxgbe/cxl about RSS configuration, > but ixgbe, igb and ixl all do the above when RSS is enabled. Most part of my setup use cxgbe. From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 23:23:55 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9575EBE6 for ; Sat, 28 Mar 2015 23:23:55 +0000 (UTC) Received: from mail-ig0-x236.google.com (mail-ig0-x236.google.com [IPv6:2607:f8b0:4001:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 53EEAA25 for ; Sat, 28 Mar 2015 23:23:55 +0000 (UTC) Received: by igbud6 with SMTP id ud6so47731976igb.1 for ; Sat, 28 Mar 2015 16:23:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=+ecSGIKrf2+x8teauMOrKE5qA6r6O2NYzeLt/m0YyKQ=; b=tSWEaNkHYIep5lE7fA+YcmQr0aXpONIamwwDdKT6/U+hSGNbzJD0uL4DoPv6HoB9zY pQYMDKunyit6efk8J1D6EPg0x2Sc7Z1RK0MPVZSEqDj2lAcQ+Lvfkv6xQ09S3jR+dqpn dxfPWVlnz6nOEpynRwgADpdru0pR4VYMnaqP6/sUome4zzOvdUo1DmeazUBbsnKysjpI S4pou+TPMeClxqAio9ULHlEGrCxhnX47qTMNn+dkP2Ylpi4eBeIAJxvU4FI2MQFkxQGl 2q4ADsy3QE4Z3ieXA9VgOJnEG/NL6NcHa0jCQyR2iKWbA5gwTucpnOrwdRWs/QYR2ux+ xQJA== MIME-Version: 1.0 X-Received: by 10.42.93.83 with SMTP id w19mr52896138icm.37.1427585034627; Sat, 28 Mar 2015 16:23:54 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 16:23:54 -0700 (PDT) In-Reply-To: <20150328230533.GI23643@zxy.spb.ru> References: <20150328192505.GD23643@zxy.spb.ru> <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> <20150328221621.GG23643@zxy.spb.ru> <20150328224634.GH23643@zxy.spb.ru> <20150328230533.GI23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 16:23:54 -0700 X-Google-Sender-Auth: 7pRGK4dG78tToBhyNTgYSlu5t90 Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 23:23:55 -0000 On 28 March 2015 at 16:05, Slawa Olhovchenkov wrote: > On Sat, Mar 28, 2015 at 03:49:48PM -0700, Adrian Chadd wrote: > >> You should totally join #bsdcode on efnet and ask me about it. :) > > I am totaly don't use IRC (last 20 years). > May be skype? Heh, IRC is better. There are more FreeBSD people in the channel. :) >> on RSS, this is what would happen: >> >> * ALL NICs RSS BUCKET 0 -> core 0 >> * ... >> * ALL NICs RSS BUCKET 7 -> core 7 > > My expirens: this is worse vs dedicated core (one core handeled only > one bucket of one NIC). The only reason(s) this becomes problematic is if things preempt other things on that CPU. Hopefully enough work gets done in each interrupt run - but, maybe the scheduler is doing something odd and interleaving all the supposedly-equivalent-ithreads based on what's blocking in locks and what isn't. It's worth digging into. Not only that, but I also do handle the case of fragments going to the "wrong" queue - then getting reassembled and reinjected back into the right RSS CPU. That way things are correctly in-order. > >> Now, that's not really 100% optimal for NUMA and multiple PCIe >> controllers, but we're not there yet. >> >> Hopefully I can twist/cajole navdeep @ chelsio to continue doing a >> little more RSS work so I can teach cxgbe/cxl about RSS configuration, >> but ixgbe, igb and ixl all do the above when RSS is enabled. > > Most part of my setup use cxgbe. Ok. Well, that (and other stuff) will happen at the speed of "adrian's doing this for fun as his home project", so if you/others would like to help out then please do. I'd like to get this stuff very much done and in -11 before it's released next year. -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 23:41:18 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B25A2196; Sat, 28 Mar 2015 23:41:18 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 63641C4A; Sat, 28 Mar 2015 23:41:18 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Yc0Ls-0004LU-Ag; Sun, 29 Mar 2015 02:41:16 +0300 Date: Sun, 29 Mar 2015 02:41:16 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328234116.GJ23643@zxy.spb.ru> References: <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> <20150328221621.GG23643@zxy.spb.ru> <20150328224634.GH23643@zxy.spb.ru> <20150328230533.GI23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 23:41:18 -0000 On Sat, Mar 28, 2015 at 04:23:54PM -0700, Adrian Chadd wrote: > On 28 March 2015 at 16:05, Slawa Olhovchenkov wrote: > > On Sat, Mar 28, 2015 at 03:49:48PM -0700, Adrian Chadd wrote: > > > >> You should totally join #bsdcode on efnet and ask me about it. :) > > > > I am totaly don't use IRC (last 20 years). > > May be skype? > > Heh, IRC is better. There are more FreeBSD people in the channel. :) I don't understund IRC. I don't know how I can got history of chat in case of long disconnect. > >> on RSS, this is what would happen: > >> > >> * ALL NICs RSS BUCKET 0 -> core 0 > >> * ... > >> * ALL NICs RSS BUCKET 7 -> core 7 > > > > My expirens: this is worse vs dedicated core (one core handeled only > > one bucket of one NIC). > > The only reason(s) this becomes problematic is if things preempt other > things on that CPU. > Hopefully enough work gets done in each interrupt run - but, maybe the > scheduler is doing something odd and interleaving all the > supposedly-equivalent-ithreads based on what's blocking in locks and > what isn't. It's worth digging into. Sorry, I am missunderstund you there. Or, may be, I use unintelligible explanation. dedicated core != core only NIC and don't do other work. dedicated core :== for this NIC bucket only this core. on this core no other NIC buckets. Other task may be used this core. What you mean? Can you explain? > Not only that, but I also do handle the case of fragments going to the > "wrong" queue - then getting reassembled and reinjected back into the > right RSS CPU. That way things are correctly in-order. fragment reassembled done in-kernel and (for TCP) go to right queue automatic. > > > >> Now, that's not really 100% optimal for NUMA and multiple PCIe > >> controllers, but we're not there yet. > >> > >> Hopefully I can twist/cajole navdeep @ chelsio to continue doing a > >> little more RSS work so I can teach cxgbe/cxl about RSS configuration, > >> but ixgbe, igb and ixl all do the above when RSS is enabled. > > > > Most part of my setup use cxgbe. > > Ok. > > Well, that (and other stuff) will happen at the speed of "adrian's > doing this for fun as his home project", so if you/others would like > to help out then please do. I'd like to get this stuff very much done > and in -11 before it's released next year. I am understand that you home project. I can't request you. I only think that propose more light and simple solution (don't need drivers modification, don't break API/ABI/KBI). From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 23:57:12 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F1B91816; Sat, 28 Mar 2015 23:57:11 +0000 (UTC) Received: from mail.highsecure.ru (mail6.highsecure.ru [IPv6:2a01:4f8:190:43b5::99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9FDC7D7C; Sat, 28 Mar 2015 23:57:11 +0000 (UTC) Received: from secret-bunker.localdomain (global-2-12.nat.csx.cam.ac.uk [131.111.185.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: vsevolod@highsecure.ru) by mail.highsecure.ru (Postfix) with ESMTPSA id EF1DA3004F0; Sun, 29 Mar 2015 00:57:07 +0100 (CET) Received: from [127.0.0.1] (localhost [127.0.0.1]) by secret-bunker.localdomain (Postfix) with ESMTP id 6E2F3243F59; Sat, 28 Mar 2015 23:57:06 +0000 (GMT) Message-ID: <55173FD2.2000708@highsecure.ru> Date: Sat, 28 Mar 2015 23:57:06 +0000 From: Vsevolod Stakhov User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Slawa Olhovchenkov , Adrian Chadd Subject: Re: irq cpu binding References: <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> <20150328221621.GG23643@zxy.spb.ru> <20150328224634.GH23643@zxy.spb.ru> <20150328230533.GI23643@zxy.spb.ru> <20150328234116.GJ23643@zxy.spb.ru> In-Reply-To: <20150328234116.GJ23643@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=highsecure.ru; s=dkim; t=1427587028; bh=sHERedw8B6rJ6ZH5T0/PLH4wcoUyh3jbeIv6H+VQ8Hw=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=F/t1iO53XzoqOJKEZMXOoZhvlAhqCc1cNByWDAcI/UuLuqAMb2H+j8Sm+AuByx18280fS5k7IcL83+mmhNYJ85b3Y7DvnQUeYOlAGnLJj+m/gESrv0vID1rHewCF6xZppNeUxndwQGPKCY1XFpcNMoDcatZ9aTgbzmWYZor4JSk= Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 23:57:12 -0000 On 28/03/2015 23:41, Slawa Olhovchenkov wrote: > On Sat, Mar 28, 2015 at 04:23:54PM -0700, Adrian Chadd wrote: > >> On 28 March 2015 at 16:05, Slawa Olhovchenkov wrote: >>> On Sat, Mar 28, 2015 at 03:49:48PM -0700, Adrian Chadd wrote: >>> >>>> You should totally join #bsdcode on efnet and ask me about it. :) >>> >>> I am totaly don't use IRC (last 20 years). >>> May be skype? >> >> Heh, IRC is better. There are more FreeBSD people in the channel. :) > > I don't understund IRC. I don't know how I can got history of chat in > case of long disconnect. Some years ago, that was a real problem since all you could do was to use some sort of IRC bouncer. However, at present, you could use some sort of web 2.0 providers to manage history and persistent online status automatically, for example, you might check irccloud: https://irccloud.com -- Vsevolod Stakhov From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 23:58:54 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 607ED91F for ; Sat, 28 Mar 2015 23:58:54 +0000 (UTC) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1F5F4D96 for ; Sat, 28 Mar 2015 23:58:54 +0000 (UTC) Received: by igcau2 with SMTP id au2so47946412igc.0 for ; Sat, 28 Mar 2015 16:58:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=IKhdY8s1RmyHO02aZDfv0SkJmYldKN1HDhGjpGGF3jo=; b=CvusWVXztV4NEf4VVqrCdXZ78cB7rs5wvjrasZ0fWX/8+2qiMqmLFPc8/K3S8D0oZx TGqQYJzizn4RyPmdWv+iUQuYu6pEfghTrAOzuiV/iITBG1YgfPrP0EJBQHd1rrPI8ucS ZwcF/qhtl5wx9b8K7B47cQYwlElejR4hmBNFT/pT9b+t5L0a0J3BROuOnzKL1Ij3Y+rK cr1COH7eIEBbnCrXE1mjkRO73TTcZq4QIlUglUP9MsJC4t8cfX/hUBQKiX2yqVCkgCQ7 sonN3rAgd+Noj0fs5p85/9fpwCk3rJN60Kn9H2CjKRmYsEAEos5tHtawR6S4VF4JZQTN lJQg== MIME-Version: 1.0 X-Received: by 10.42.41.200 with SMTP id q8mr48738692ice.61.1427587133585; Sat, 28 Mar 2015 16:58:53 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Sat, 28 Mar 2015 16:58:53 -0700 (PDT) In-Reply-To: <20150328234116.GJ23643@zxy.spb.ru> References: <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> <20150328221621.GG23643@zxy.spb.ru> <20150328224634.GH23643@zxy.spb.ru> <20150328230533.GI23643@zxy.spb.ru> <20150328234116.GJ23643@zxy.spb.ru> Date: Sat, 28 Mar 2015 16:58:53 -0700 X-Google-Sender-Auth: M1ggTk0dZnlsjfcQJRuA2gZRymE Message-ID: Subject: Re: irq cpu binding From: Adrian Chadd To: Slawa Olhovchenkov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 23:58:54 -0000 Hi, * It turns out that fragments were being 100% handled out of order (compared to non-fragments in the same stream) when doing fragment reassembly, because the current system was assuming direct dispatch netisr and not checking any packet contents for whether they're on the wrong CPU. I checked. It's not noticable unless you go digging, but it's absolutely happening. That's why I spun a lot of cycles looking at the IP fragment reassembly path and which methods get called on the frames as they're reinjected. * We're going to have modify drivers, because the way drivers currently assign interrupts, pick CPUs for queues, auto-select how many queues to use, etc is all completely adhoc and not consistent. So yeah, we're going to change the drivers and they're going to be consistent and configurable. That way you can choose how you want to distribute work and pin or not pin things - and it's not done adhoc differently in each driver. Even igb, ixgbe and cxgbe differ in how they implement these three things. * For RSS, there'll be a consistent configuration for what the hardware is doing with hashing, rather than it being driver dependent. Again, otherwise you may end up with some NICs doing 2-tuple hashing where others are doing 4-tuple hashing, and behaviour changes dramatically based on what NIC you're using. * For applications - I'm not sure yet, but at the minimum the librss API I have vaguely sketched out and coded up in a git branch lets you pull out the list of buckets and which CPU it's on. I'm going to extend that a bit more, but it should be enough for things like nginx to say "ok, start up one nginx process per RSS bucket, and here's the CPU set for it to bind to." You said it has worker groups - that's great; I want that to be auto configured. -adrian