From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 06:32:21 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1A277909; Mon, 22 Jul 2013 06:32:21 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-la0-x22a.google.com (mail-la0-x22a.google.com [IPv6:2a00:1450:4010:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1BC152408; Mon, 22 Jul 2013 06:32:19 +0000 (UTC) Received: by mail-la0-f42.google.com with SMTP id eh20so2600561lab.29 for ; Sun, 21 Jul 2013 23:32:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; bh=BRjAejkO3k3Qi5QfaPOi8/JkAM5ql9nuw1zlJZN8X+Q=; b=iLVMOq9tfrDs6Dddka0faB8XZmQep87xI4PUNgmry5jO5aM75nDHSMpldsehBpFaEf sPaqpogBUM3a1r2UdGd27D/FVLUGGS79IrAAyZnpCIO04hPMfZsLfTxedttLHuTHOaVC YNGWBpjLMZOc1B5EN+d9D8L5D5xd3Cu0J4X9uBI7qeAc2YpUSy+bOw3SCRje8byxiGDV RLLNPEXNhbV29tuCq57inqd6mF6HfR3hZJtXRE2xTvUfZEUxfmKO59FkfRAeqxGd3iKC eohyPi7DzmGOsYhVIDg8vTnRiSQSy/oICLLKXV72VeeRkjJ0iNJdjPFR9T6jZxSBjbXT FPpQ== MIME-Version: 1.0 X-Received: by 10.112.3.234 with SMTP id f10mr11033009lbf.81.1374474737858; Sun, 21 Jul 2013 23:32:17 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.149.38 with HTTP; Sun, 21 Jul 2013 23:32:17 -0700 (PDT) Date: Sun, 21 Jul 2013 23:32:17 -0700 X-Google-Sender-Auth: 43Ak-TDYf3ata2p5DxACjAN2548 Message-ID: Subject: VIMAGE + PF crash in mbuf destructor From: Craig Rodrigues To: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Adrian Chadd , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 06:32:21 -0000 Hi, I used a kernel config with the following lines: include GENERIC options VIMAGE and compiled a CURRENT kernel from svn://svn.freebsd.org/base/head@253346 . I also have PF enabled on my system. Once in a while I have been getting kernel panics like these: ==================================================================== (kgdb) #0 doadump (textdump=1) at pcpu.h:236 #1 0xffffffff808bc617 in kern_reboot (howto=260) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:447 #2 0xffffffff808bcb25 in vpanic (fmt=, ap=) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:754 #3 0xffffffff808bcb73 in panic (fmt=) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:683 #4 0xffffffff8033dff7 in db_panic (addr=, have_addr=, count=, modif=) at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:482 #5 0xffffffff8033dbcd in db_command (cmd_table=) at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:449 #6 0xffffffff8033d944 in db_command_loop () at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:502 #7 0xffffffff803402f0 in db_trap (type=, code=0) at /usr/home/rodrigc/freebsd/head/sys/ddb/db_main.c:231 #8 0xffffffff808f3623 in kdb_trap (type=12, code=0, tf=) at /usr/home/rodrigc/freebsd/head/sys/kern/subr_kdb.c:654 #9 0xffffffff80cda43a in trap_fatal (frame=0xffffff811dbab6b0, eva=) at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:868 #10 0xffffffff80cda6f4 in trap_pfault (frame=0x0, usermode=0) at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:699 #11 0xffffffff80cd9ef0 in trap (frame=0xffffff811dbab6b0) at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:463 #12 0xffffffff80cc31a2 in calltrap () at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:232 #13 0xffffffff8208f7b7 in pf_mtag_free (t=0xfffffe00a8797870) at /usr/home/rodrigc/freebsd/head/sys/modules/pf/../../netpfil/pf/pf.c:830 #14 0xffffffff808a51c9 in mb_dtor_mbuf (mem=0xfffffe000d0bc500, size=256, arg=0x0) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_mbuf.c:499 #15 0xffffffff80b55d4d in uma_zfree_arg (zone=0xfffffe000b4ab900, item=0xfffffe000d0bc500, udata=0x0) at /usr/home/rodrigc/freebsd/head/sys/vm/uma_core.c:2560 #16 0xffffffff8092d1f5 in m_freem (mb=) at uma.h:364 #17 0xffffffff8058ba72 in iwn_tx_done (sc=0xffffff8000974000, desc=, ackfailcnt=16, status=131 '\203') at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:2817 #18 0xffffffff80583e60 in iwn_notif_intr (sc=0xffffff8000974000) at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3015 #19 0xffffffff80583684 in iwn_intr (arg=0xffffff8000974000) at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3306 #20 0xffffffff8088daf3 in intr_event_execute_handlers ( p=, ie=0xfffffe000b696600) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1263 #21 0xffffffff8088e4c6 in ithread_loop (arg=0xfffffe000b31b040) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1276 #22 0xffffffff8088b3f4 in fork_exit ( callout=0xffffffff8088e420 , arg=0xfffffe000b31b040, frame=0xffffff811dbabac0) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_fork.c:991 #23 0xffffffff80cc36de in fork_trampoline () at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:606 #24 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) ==================================================================== It turns out that in this file: src/sys/netpfil/pf/pf.c 826 static void 827 pf_mtag_free(struct m_tag *t) 828 { 829 830 uma_zfree(V_pf_mtag_z, t); 831 } when line 830 is hit, it turns out that curthread->td_vnet is NULL. Does anyone have an idea as to the best place to put CURVNET_SET() to avoid this problem? I am a little less famiiar with mbuf and pf. Thanks. -- Craig From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 06:38:59 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0EE4E9C1; Mon, 22 Jul 2013 06:38:59 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-x232.google.com (mail-we0-x232.google.com [IPv6:2a00:1450:400c:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 44407243A; Mon, 22 Jul 2013 06:38:58 +0000 (UTC) Received: by mail-we0-f178.google.com with SMTP id u57so1126143wes.37 for ; Sun, 21 Jul 2013 23:38:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=7E4zLFltRebQSfMPTjoml4w9s3pN4SpPu5xymdj34Qs=; b=oAAInPUeP3OPwFlO9jxwbR2xxvtkZtNOEQFHZk19/mImLN6CIOLNg2K/1hzvz/IhDI DPUyz1THZ1cLu8m8Lm6AMMb+uHA8CSzGDwfGZd7XXs6OnMHl/OcJRRQmSe9TVwLxY8rL a3cbZjPa+W3X3I4pg2eVfTqbjUBXTmE/De8V4AzpVTJBQ2JRCkFM17k7e0lH1feMnW4D E8L9Qx2EYkP5yubKC9SsMNhWrjrty/Qs6ndCob4zFaGuFIQNMUUwI8mjd1TbIlmwJl+7 6vRPxlRWB3akk6hTQzj2GScwf7ml1GJ/vP2Qq4y7a06N8IL5Q8XPzZj5Crlun5KccSyg sWHQ== MIME-Version: 1.0 X-Received: by 10.180.14.105 with SMTP id o9mr22493980wic.30.1374475136458; Sun, 21 Jul 2013 23:38:56 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Sun, 21 Jul 2013 23:38:56 -0700 (PDT) In-Reply-To: References: Date: Sun, 21 Jul 2013 23:38:56 -0700 X-Google-Sender-Auth: 7hLqRJwskX0ETfzYi7S_vbSqDIk Message-ID: Subject: Re: VIMAGE + PF crash in mbuf destructor From: Adrian Chadd To: Craig Rodrigues Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 06:38:59 -0000 hm. There's lots of mbuf free calls in the net80211 TX and RX path; do we have to have to set the vnet context during the whole tx/rx path? -adrian On 21 July 2013 23:32, Craig Rodrigues wrote: > Hi, > > I used a kernel config with the following lines: > > include GENERIC > options VIMAGE > > and compiled a CURRENT kernel from svn://svn.freebsd.org/base/head@253346 . > > I also have PF enabled on my system. > > Once in a while I have been getting kernel panics like these: > > > ==================================================================== > (kgdb) #0 doadump (textdump=1) at pcpu.h:236 > #1 0xffffffff808bc617 in kern_reboot (howto=260) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:447 > #2 0xffffffff808bcb25 in vpanic (fmt=, > ap=) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:754 > #3 0xffffffff808bcb73 in panic (fmt=) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:683 > #4 0xffffffff8033dff7 in db_panic (addr=, > have_addr=, count=, > modif=) > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:482 > #5 0xffffffff8033dbcd in db_command (cmd_table=) > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:449 > #6 0xffffffff8033d944 in db_command_loop () > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:502 > #7 0xffffffff803402f0 in db_trap (type=, code=0) > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_main.c:231 > #8 0xffffffff808f3623 in kdb_trap (type=12, code=0, tf= out>) > at /usr/home/rodrigc/freebsd/head/sys/kern/subr_kdb.c:654 > #9 0xffffffff80cda43a in trap_fatal (frame=0xffffff811dbab6b0, > eva=) > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:868 > #10 0xffffffff80cda6f4 in trap_pfault (frame=0x0, usermode=0) > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:699 > #11 0xffffffff80cd9ef0 in trap (frame=0xffffff811dbab6b0) > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:463 > #12 0xffffffff80cc31a2 in calltrap () > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:232 > #13 0xffffffff8208f7b7 in pf_mtag_free (t=0xfffffe00a8797870) > at > /usr/home/rodrigc/freebsd/head/sys/modules/pf/../../netpfil/pf/pf.c:830 > #14 0xffffffff808a51c9 in mb_dtor_mbuf (mem=0xfffffe000d0bc500, size=256, > arg=0x0) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_mbuf.c:499 > #15 0xffffffff80b55d4d in uma_zfree_arg (zone=0xfffffe000b4ab900, > item=0xfffffe000d0bc500, udata=0x0) > at /usr/home/rodrigc/freebsd/head/sys/vm/uma_core.c:2560 > #16 0xffffffff8092d1f5 in m_freem (mb=) at uma.h:364 > #17 0xffffffff8058ba72 in iwn_tx_done (sc=0xffffff8000974000, > desc=, ackfailcnt=16, status=131 '\203') > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:2817 > #18 0xffffffff80583e60 in iwn_notif_intr (sc=0xffffff8000974000) > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3015 > #19 0xffffffff80583684 in iwn_intr (arg=0xffffff8000974000) > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3306 > #20 0xffffffff8088daf3 in intr_event_execute_handlers ( > p=, ie=0xfffffe000b696600) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1263 > #21 0xffffffff8088e4c6 in ithread_loop (arg=0xfffffe000b31b040) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1276 > #22 0xffffffff8088b3f4 in fork_exit ( > callout=0xffffffff8088e420 , arg=0xfffffe000b31b040, > frame=0xffffff811dbabac0) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_fork.c:991 > #23 0xffffffff80cc36de in fork_trampoline () > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:606 > #24 0x0000000000000000 in ?? () > Current language: auto; currently minimal > (kgdb) > ==================================================================== > > > It turns out that in this file: src/sys/netpfil/pf/pf.c > > 826 static void > 827 pf_mtag_free(struct m_tag *t) > 828 { > 829 > 830 uma_zfree(V_pf_mtag_z, t); > 831 } > > when line 830 is hit, it turns out that curthread->td_vnet is NULL. > > Does anyone have an idea as to the best place > to put CURVNET_SET() to avoid this problem? > > I am a little less famiiar with mbuf and pf. > > Thanks. > -- > Craig > > From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 06:57:46 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C0139231; Mon, 22 Jul 2013 06:57:46 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C29C624CD; Mon, 22 Jul 2013 06:57:45 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id z5so4980433lbh.35 for ; Sun, 21 Jul 2013 23:57:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=ZV4bFKg5tUhFkjt544/ufAT/r2TJTHUz3qDLzRLH9f0=; b=WCkHeMbw3YZS+BtqlLKe6c+ATPU3BnGbuk1GsiZ7XWs5vOyUzXhVUqpaOtaCVSQnXG y77hkzriOheNNWtuhiR5gPFHiBjkeSLzzRg3SyNAfxKIRcCA2RLPX1M1iASncQSTt3bn eZrUgMDICCGmoouMfLakE3uZdOFelAzzgtmrGEpTOUzcvd0wrnM0W/O01/5JtCEQksM3 HYEO16T+oJkmpANFbokKxPdBC4/jYzXIpgUn9qKf7tdpi2xKIenK3B03Kg8Si6WBsbdx 4sph9GTrqj0KGU289qFdQg5m9Jp7iXuHK3Lq9KGxqhnv418GZZJjnG7zKKo3ygDZYfHq /YMw== MIME-Version: 1.0 X-Received: by 10.112.150.231 with SMTP id ul7mr11876865lbb.92.1374476263543; Sun, 21 Jul 2013 23:57:43 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.149.38 with HTTP; Sun, 21 Jul 2013 23:57:43 -0700 (PDT) In-Reply-To: References: Date: Sun, 21 Jul 2013 23:57:43 -0700 X-Google-Sender-Auth: 1X-fsdhdb5dO17S5pVBIBHky6Po Message-ID: Subject: Re: VIMAGE + PF crash in mbuf destructor From: Craig Rodrigues To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 06:57:46 -0000 On Sun, Jul 21, 2013 at 11:38 PM, Adrian Chadd wrote: > hm. There's lots of mbuf free calls in the net80211 TX and RX path; do > we have to have to set the vnet context during the whole tx/rx path? > I'm not sure about that. In src/sys/netpfil/pf/pf.c, we have this in pf_initialize(): 751 /* Mbuf tags */ 752 V_pf_mtag_z = uma_zcreate("pf mtags", sizeof(struct m_tag) + 753 sizeof(struct pf_mtag), NULL, NULL, pf_mtag_init, NULL, 754 UMA_ALIGN_PTR, 0); and further down this: 812 static int 813 pf_mtag_init(void *mem, int size, int how) 814 { 815 struct m_tag *t; 816 817 t = (struct m_tag *)mem; 818 t->m_tag_cookie = MTAG_ABI_COMPAT; 819 t->m_tag_id = PACKET_TAG_PF; 820 t->m_tag_len = sizeof(struct pf_mtag); 821 t->m_tag_free = pf_mtag_free; 822 823 return (0); 824 } 825 826 static void 827 pf_mtag_free(struct m_tag *t) 828 { 829 830 uma_zfree(V_pf_mtag_z, t); 831 } Can we somehow modify pf_mtag_init() so that it passes the vnet into the pf_mtag? Then we can call CURVNET_SET/CURVNET_RESTORE in pf_mtag_free(). -- Craig > > > > -adrian > > On 21 July 2013 23:32, Craig Rodrigues wrote: > > Hi, > > > > I used a kernel config with the following lines: > > > > include GENERIC > > options VIMAGE > > > > and compiled a CURRENT kernel from svn:// > svn.freebsd.org/base/head@253346 . > > > > I also have PF enabled on my system. > > > > Once in a while I have been getting kernel panics like these: > > > > > > ==================================================================== > > (kgdb) #0 doadump (textdump=1) at pcpu.h:236 > > #1 0xffffffff808bc617 in kern_reboot (howto=260) > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:447 > > #2 0xffffffff808bcb25 in vpanic (fmt=, > > ap=) > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:754 > > #3 0xffffffff808bcb73 in panic (fmt=) > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:683 > > #4 0xffffffff8033dff7 in db_panic (addr=, > > have_addr=, count=, > > modif=) > > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:482 > > #5 0xffffffff8033dbcd in db_command (cmd_table=) > > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:449 > > #6 0xffffffff8033d944 in db_command_loop () > > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:502 > > #7 0xffffffff803402f0 in db_trap (type=, code=0) > > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_main.c:231 > > #8 0xffffffff808f3623 in kdb_trap (type=12, code=0, tf= > out>) > > at /usr/home/rodrigc/freebsd/head/sys/kern/subr_kdb.c:654 > > #9 0xffffffff80cda43a in trap_fatal (frame=0xffffff811dbab6b0, > > eva=) > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:868 > > #10 0xffffffff80cda6f4 in trap_pfault (frame=0x0, usermode=0) > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:699 > > #11 0xffffffff80cd9ef0 in trap (frame=0xffffff811dbab6b0) > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:463 > > #12 0xffffffff80cc31a2 in calltrap () > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:232 > > #13 0xffffffff8208f7b7 in pf_mtag_free (t=0xfffffe00a8797870) > > at > > /usr/home/rodrigc/freebsd/head/sys/modules/pf/../../netpfil/pf/pf.c:830 > > #14 0xffffffff808a51c9 in mb_dtor_mbuf (mem=0xfffffe000d0bc500, size=256, > > arg=0x0) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_mbuf.c:499 > > #15 0xffffffff80b55d4d in uma_zfree_arg (zone=0xfffffe000b4ab900, > > item=0xfffffe000d0bc500, udata=0x0) > > at /usr/home/rodrigc/freebsd/head/sys/vm/uma_core.c:2560 > > #16 0xffffffff8092d1f5 in m_freem (mb=) at uma.h:364 > > #17 0xffffffff8058ba72 in iwn_tx_done (sc=0xffffff8000974000, > > desc=, ackfailcnt=16, status=131 '\203') > > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:2817 > > #18 0xffffffff80583e60 in iwn_notif_intr (sc=0xffffff8000974000) > > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3015 > > #19 0xffffffff80583684 in iwn_intr (arg=0xffffff8000974000) > > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3306 > > #20 0xffffffff8088daf3 in intr_event_execute_handlers ( > > p=, ie=0xfffffe000b696600) > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1263 > > #21 0xffffffff8088e4c6 in ithread_loop (arg=0xfffffe000b31b040) > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1276 > > #22 0xffffffff8088b3f4 in fork_exit ( > > callout=0xffffffff8088e420 , arg=0xfffffe000b31b040, > > frame=0xffffff811dbabac0) > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_fork.c:991 > > #23 0xffffffff80cc36de in fork_trampoline () > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:606 > > #24 0x0000000000000000 in ?? () > > Current language: auto; currently minimal > > (kgdb) > > ==================================================================== > > > > > > It turns out that in this file: src/sys/netpfil/pf/pf.c > > > > 826 static void > > 827 pf_mtag_free(struct m_tag *t) > > 828 { > > 829 > > 830 uma_zfree(V_pf_mtag_z, t); > > 831 } > > > > when line 830 is hit, it turns out that curthread->td_vnet is NULL. > > > > Does anyone have an idea as to the best place > > to put CURVNET_SET() to avoid this problem? > > > > I am a little less famiiar with mbuf and pf. > > > > Thanks. > > -- > > Craig > > > > > From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 11:06:49 2013 Return-Path: Delivered-To: freebsd-pf@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C81E849A for ; Mon, 22 Jul 2013 11:06:49 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B966224A7 for ; Mon, 22 Jul 2013 11:06:49 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6MB6ngR053788 for ; Mon, 22 Jul 2013 11:06:49 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6MB6nbx053786 for freebsd-pf@FreeBSD.org; Mon, 22 Jul 2013 11:06:49 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 22 Jul 2013 11:06:49 GMT Message-Id: <201307221106.r6MB6nbx053786@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-pf@FreeBSD.org Subject: Current problem reports assigned to freebsd-pf@FreeBSD.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 11:06:49 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/179392 pf [pf] [ip6] Incorrect TCP checksums in rdr return packe o kern/177810 pf [pf] traffic dropped by accepting rules is not counted o kern/177808 pf [pf] [patch] route-to rule forwarding traffic inspite o kern/176763 pf [pf] [patch] Removing pf Source entries locks kernel. o kern/176268 pf [pf] [patch] synproxy not working with route-to o kern/173659 pf [pf] PF fatal trap on 9.1 (taskq fatal trap on pf_test o bin/172888 pf [patch] authpf(8) feature enhancement o kern/172648 pf [pf] [ip6]: 'scrub reassemble tcp' breaks IPv6 packet o kern/171733 pf [pf] PF problem with modulate state in [regression] o kern/169630 pf [pf] [patch] pf fragment reassembly of padded (undersi o kern/168952 pf [pf] direction scrub rules don't work o kern/168190 pf [pf] panic when using pf and route-to (maybe: bad frag o kern/166336 pf [pf] kern.securelevel 3 +pf reload o kern/165315 pf [pf] States never cleared in PF with DEVICE_POLLING o kern/164402 pf [pf] pf crashes with a particular set of rules when fi o kern/164271 pf [pf] not working pf nat on FreeBSD 9.0 [regression] o kern/163208 pf [pf] PF state key linking mismatch o kern/160370 pf [pf] Incorrect pfctl check of pf.conf o kern/155736 pf [pf] [altq] borrow from parent queue does not work wit o kern/153307 pf [pf] Bug with PF firewall o kern/148290 pf [pf] "sticky-address" option of Packet Filter (PF) blo o kern/148260 pf [pf] [patch] pf rdr incompatible with dummynet o kern/147789 pf [pf] Firewall PF no longer drops connections by sendin o kern/143543 pf [pf] [panic] PF route-to causes kernel panic o bin/143504 pf [patch] outgoing states are not killed by authpf(8) o conf/142961 pf [pf] No way to adjust pidfile in pflogd o conf/142817 pf [patch] etc/rc.d/pf: silence pfctl o kern/141905 pf [pf] [panic] pf kernel panic on 7.2-RELEASE with empty o kern/140697 pf [pf] pf behaviour changes - must be documented o kern/137982 pf [pf] when pf can hit state limits, random IP failures o kern/136781 pf [pf] Packets appear to drop with pf scrub and if_bridg o kern/135948 pf [pf] [gre] pf not natting gre protocol o kern/134996 pf [pf] Anchor tables not included when pfctl(8) is run w o kern/133732 pf [pf] max-src-conn issue o conf/130381 pf [rc.d] [pf] [ip6] ipv6 not fully configured when pf st o kern/127920 pf [pf] ipv6 and synproxy don't play well together o conf/127814 pf [pf] The flush in pf_reload in /etc/rc.d/pf does not w o kern/127121 pf [pf] [patch] pf incorrect log priority o kern/127042 pf [pf] [patch] pf recursion panic if interface group is o kern/125467 pf [pf] pf keep state bug while handling sessions between s kern/124933 pf [pf] [ip6] pf does not support (drops) IPv6 fragmented o kern/122773 pf [pf] pf doesn't log uid or pid when configured to o kern/122014 pf [pf] [panic] FreeBSD 6.2 panic in pf o kern/120281 pf [pf] [request] lost returning packets to PF for a rdr o kern/120057 pf [pf] [patch] Allow proper settings of ALTQ_HFSC. The c o bin/118355 pf [pf] [patch] pfctl(8) help message options order false o kern/114567 pf [pf] [lor] pf_ioctl.c + if.c o kern/103283 pf pfsync fails to sucessfully transfer some sessions o kern/93825 pf [pf] pf reply-to doesn't work o sparc/93530 pf [pf] Incorrect checksums when using pf's route-to on s o kern/92949 pf [pf] PF + ALTQ problems with latency o kern/87074 pf [pf] pf does not log dropped packets when max-* statef a kern/86752 pf [pf] pf does not use default timeouts when reloading c o bin/86635 pf [patch] pfctl(8): allow new page character (^L) in pf. o kern/82271 pf [pf] cbq scheduler cause bad latency 55 problems total. From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 13:06:41 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9B0915A9; Mon, 22 Jul 2013 13:06:41 +0000 (UTC) (envelope-from zec@fer.hr) Received: from mail.fer.hr (mail.fer.hr [161.53.72.233]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EE4E82D0D; Mon, 22 Jul 2013 13:06:40 +0000 (UTC) Received: from dyn10.nxlab.fer.hr (161.53.63.210) by MAIL.fer.hr (161.53.72.233) with Microsoft SMTP Server (TLS) id 14.2.342.3; Mon, 22 Jul 2013 15:05:29 +0200 From: Marko Zec To: Subject: Re: VIMAGE + PF crash in mbuf destructor Date: Mon, 22 Jul 2013 15:05:29 +0200 User-Agent: KMail/1.9.10 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-ID: <201307221505.29495.zec@fer.hr> X-Originating-IP: [161.53.63.210] Cc: Adrian Chadd , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 13:06:41 -0000 On Monday 22 July 2013 08:57:43 Craig Rodrigues wrote: > On Sun, Jul 21, 2013 at 11:38 PM, Adrian Chadd wrote: > > hm. There's lots of mbuf free calls in the net80211 TX and RX path; do > > we have to have to set the vnet context during the whole tx/rx path? > > I'm not sure about that. > In src/sys/netpfil/pf/pf.c, we have this in pf_initialize(): > > 751 /* Mbuf tags */ > 752 V_pf_mtag_z = uma_zcreate("pf mtags", sizeof(struct > m_tag) + 753 sizeof(struct pf_mtag), NULL, NULL, > pf_mtag_init, NULL, 754 UMA_ALIGN_PTR, 0); > > and further down this: > > 812 static int > 813 pf_mtag_init(void *mem, int size, int how) > 814 { > 815 struct m_tag *t; > 816 > 817 t = (struct m_tag *)mem; > 818 t->m_tag_cookie = MTAG_ABI_COMPAT; > 819 t->m_tag_id = PACKET_TAG_PF; > 820 t->m_tag_len = sizeof(struct pf_mtag); > 821 t->m_tag_free = pf_mtag_free; > 822 > 823 return (0); > 824 } > 825 > 826 static void > 827 pf_mtag_free(struct m_tag *t) > 828 { > 829 > 830 uma_zfree(V_pf_mtag_z, t); > 831 } > > > Can we somehow modify pf_mtag_init() so that it passes the > vnet into the pf_mtag? > Then we can call CURVNET_SET/CURVNET_RESTORE in pf_mtag_free(). I'd say just de-virtualize V_pf_mtag_z, and you're done. Marko > -- > Craig > > > -adrian > > > > On 21 July 2013 23:32, Craig Rodrigues wrote: > > > Hi, > > > > > > I used a kernel config with the following lines: > > > > > > include GENERIC > > > options VIMAGE > > > > > > and compiled a CURRENT kernel from svn:// > > > > svn.freebsd.org/base/head@253346 . > > > > > I also have PF enabled on my system. > > > > > > Once in a while I have been getting kernel panics like these: > > > > > > > > > ==================================================================== > > > (kgdb) #0 doadump (textdump=1) at pcpu.h:236 > > > #1 0xffffffff808bc617 in kern_reboot (howto=260) > > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:447 > > > #2 0xffffffff808bcb25 in vpanic (fmt=, > > > ap=) > > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:754 > > > #3 0xffffffff808bcb73 in panic (fmt=) > > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:683 > > > #4 0xffffffff8033dff7 in db_panic (addr=, > > > have_addr=, count=, > > > modif=) > > > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:482 > > > #5 0xffffffff8033dbcd in db_command (cmd_table= > > out>) at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:449 #6 > > > 0xffffffff8033d944 in db_command_loop () > > > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:502 > > > #7 0xffffffff803402f0 in db_trap (type=, > > > code=0) at /usr/home/rodrigc/freebsd/head/sys/ddb/db_main.c:231 > > > #8 0xffffffff808f3623 in kdb_trap (type=12, code=0, tf= > > optimized out>) > > > at /usr/home/rodrigc/freebsd/head/sys/kern/subr_kdb.c:654 > > > #9 0xffffffff80cda43a in trap_fatal (frame=0xffffff811dbab6b0, > > > eva=) > > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:868 > > > #10 0xffffffff80cda6f4 in trap_pfault (frame=0x0, usermode=0) > > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:699 > > > #11 0xffffffff80cd9ef0 in trap (frame=0xffffff811dbab6b0) > > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:463 > > > #12 0xffffffff80cc31a2 in calltrap () > > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:232 > > > #13 0xffffffff8208f7b7 in pf_mtag_free (t=0xfffffe00a8797870) > > > at > > > /usr/home/rodrigc/freebsd/head/sys/modules/pf/../../netpfil/pf/pf.c:8 > > >30 #14 0xffffffff808a51c9 in mb_dtor_mbuf (mem=0xfffffe000d0bc500, > > > size=256, arg=0x0) at > > > /usr/home/rodrigc/freebsd/head/sys/kern/kern_mbuf.c:499 #15 > > > 0xffffffff80b55d4d in uma_zfree_arg (zone=0xfffffe000b4ab900, > > > item=0xfffffe000d0bc500, udata=0x0) > > > at /usr/home/rodrigc/freebsd/head/sys/vm/uma_core.c:2560 > > > #16 0xffffffff8092d1f5 in m_freem (mb=) at > > > uma.h:364 #17 0xffffffff8058ba72 in iwn_tx_done > > > (sc=0xffffff8000974000, desc=, ackfailcnt=16, > > > status=131 '\203') at > > > /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:2817 #18 > > > 0xffffffff80583e60 in iwn_notif_intr (sc=0xffffff8000974000) at > > > /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3015 #19 > > > 0xffffffff80583684 in iwn_intr (arg=0xffffff8000974000) > > > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3306 > > > #20 0xffffffff8088daf3 in intr_event_execute_handlers ( > > > p=, ie=0xfffffe000b696600) > > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1263 > > > #21 0xffffffff8088e4c6 in ithread_loop (arg=0xfffffe000b31b040) > > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1276 > > > #22 0xffffffff8088b3f4 in fork_exit ( > > > callout=0xffffffff8088e420 , > > > arg=0xfffffe000b31b040, frame=0xffffff811dbabac0) > > > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_fork.c:991 > > > #23 0xffffffff80cc36de in fork_trampoline () > > > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:606 > > > #24 0x0000000000000000 in ?? () > > > Current language: auto; currently minimal > > > (kgdb) > > > ==================================================================== > > > > > > > > > It turns out that in this file: src/sys/netpfil/pf/pf.c > > > > > > 826 static void > > > 827 pf_mtag_free(struct m_tag *t) > > > 828 { > > > 829 > > > 830 uma_zfree(V_pf_mtag_z, t); > > > 831 } > > > > > > when line 830 is hit, it turns out that curthread->td_vnet is NULL. > > > > > > Does anyone have an idea as to the best place > > > to put CURVNET_SET() to avoid this problem? > > > > > > I am a little less famiiar with mbuf and pf. > > > > > > Thanks. > > > -- > > > Craig > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 15:43:26 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AA148854 for ; Mon, 22 Jul 2013 15:43:26 +0000 (UTC) (envelope-from nvass@gmx.com) Received: from mout.gmx.net (mout.gmx.net [74.208.4.201]) by mx1.freebsd.org (Postfix) with ESMTP id 6D2E72543 for ; Mon, 22 Jul 2013 15:43:26 +0000 (UTC) Received: from moby.local ([178.128.212.170]) by mail.gmx.com (mrgmxus001) with ESMTPSA (Nemesis) id 0MH0Nw-1UxSWM2bqM-00DoIr for ; Mon, 22 Jul 2013 17:43:25 +0200 Message-ID: <51ED5308.3020008@gmx.com> Date: Mon, 22 Jul 2013 18:43:04 +0300 From: Nikos Vassiliadis User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:17.0) Gecko/20130714 Thunderbird/17.0.7 MIME-Version: 1.0 To: Craig Rodrigues Subject: Re: VIMAGE + PF crash in mbuf destructor References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:Z5D+X8087nvyIHCLdFveVPOuSV3wnRvUt348Y32ktWAhd2/ISa0 4ofWXPIlsl47OEKWfMBtvgMVRKA5Ofp9ARTb0kgZt/U6iZjS8nhE6Y52feAMz0h29DpAK/a tnVpLPc2ptUjRJddYU+o7i91QVgnPlwTIvsrsyvrWF5pX1cV4VbZcRR9ut58bCDpt09IXPj SShfu3V6IMg26AVxG/CAg== Cc: Adrian Chadd , "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 15:43:26 -0000 On 07/22/13 09:32, Craig Rodrigues wrote: > Hi, > > I used a kernel config with the following lines: > > include GENERIC > options VIMAGE > > and compiled a CURRENT kernel from svn://svn.freebsd.org/base/head@253346 . > > I also have PF enabled on my system. > > Once in a while I have been getting kernel panics like these: > > > ==================================================================== > (kgdb) #0 doadump (textdump=1) at pcpu.h:236 > #1 0xffffffff808bc617 in kern_reboot (howto=260) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:447 > #2 0xffffffff808bcb25 in vpanic (fmt=, > ap=) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:754 > #3 0xffffffff808bcb73 in panic (fmt=) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_shutdown.c:683 > #4 0xffffffff8033dff7 in db_panic (addr=, > have_addr=, count=, > modif=) > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:482 > #5 0xffffffff8033dbcd in db_command (cmd_table=) > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:449 > #6 0xffffffff8033d944 in db_command_loop () > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_command.c:502 > #7 0xffffffff803402f0 in db_trap (type=, code=0) > at /usr/home/rodrigc/freebsd/head/sys/ddb/db_main.c:231 > #8 0xffffffff808f3623 in kdb_trap (type=12, code=0, tf= out>) > at /usr/home/rodrigc/freebsd/head/sys/kern/subr_kdb.c:654 > #9 0xffffffff80cda43a in trap_fatal (frame=0xffffff811dbab6b0, > eva=) > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:868 > #10 0xffffffff80cda6f4 in trap_pfault (frame=0x0, usermode=0) > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:699 > #11 0xffffffff80cd9ef0 in trap (frame=0xffffff811dbab6b0) > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/trap.c:463 > #12 0xffffffff80cc31a2 in calltrap () > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:232 > #13 0xffffffff8208f7b7 in pf_mtag_free (t=0xfffffe00a8797870) > at > /usr/home/rodrigc/freebsd/head/sys/modules/pf/../../netpfil/pf/pf.c:830 > #14 0xffffffff808a51c9 in mb_dtor_mbuf (mem=0xfffffe000d0bc500, size=256, > arg=0x0) at /usr/home/rodrigc/freebsd/head/sys/kern/kern_mbuf.c:499 > #15 0xffffffff80b55d4d in uma_zfree_arg (zone=0xfffffe000b4ab900, > item=0xfffffe000d0bc500, udata=0x0) > at /usr/home/rodrigc/freebsd/head/sys/vm/uma_core.c:2560 > #16 0xffffffff8092d1f5 in m_freem (mb=) at uma.h:364 > #17 0xffffffff8058ba72 in iwn_tx_done (sc=0xffffff8000974000, > desc=, ackfailcnt=16, status=131 '\203') > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:2817 > #18 0xffffffff80583e60 in iwn_notif_intr (sc=0xffffff8000974000) > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3015 > #19 0xffffffff80583684 in iwn_intr (arg=0xffffff8000974000) > at /usr/home/rodrigc/freebsd/head/sys/dev/iwn/if_iwn.c:3306 > #20 0xffffffff8088daf3 in intr_event_execute_handlers ( > p=, ie=0xfffffe000b696600) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1263 > #21 0xffffffff8088e4c6 in ithread_loop (arg=0xfffffe000b31b040) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_intr.c:1276 > #22 0xffffffff8088b3f4 in fork_exit ( > callout=0xffffffff8088e420 , arg=0xfffffe000b31b040, > frame=0xffffff811dbabac0) > at /usr/home/rodrigc/freebsd/head/sys/kern/kern_fork.c:991 > #23 0xffffffff80cc36de in fork_trampoline () > at /usr/home/rodrigc/freebsd/head/sys/amd64/amd64/exception.S:606 > #24 0x0000000000000000 in ?? () > Current language: auto; currently minimal > (kgdb) > ==================================================================== > > > It turns out that in this file: src/sys/netpfil/pf/pf.c > > 826 static void > 827 pf_mtag_free(struct m_tag *t) > 828 { > 829 > 830 uma_zfree(V_pf_mtag_z, t); > 831 } > > when line 830 is hit, it turns out that curthread->td_vnet is NULL. > > Does anyone have an idea as to the best place > to put CURVNET_SET() to avoid this problem? > > I am a little less famiiar with mbuf and pf. Hi, I think this comes from the eventhandlers pf installs to handle ifnet events. It seems like a wifi event causes this code to run and the context is not set. Does the panic happen only when you use vnet jails? Could you try putting all evenhandlers in an 'if (IS_DEFAULT_VNET(curvnet))' block? It's here: http://fxr.watson.org/fxr/source/netpfil/pf/pf_if.c#L127 the pfi_*_cookie = ... lines. I am not sure if this would be enough though since it might panic in other places. HTH, Nikos From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 17:11:59 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 91650170; Mon, 22 Jul 2013 17:11:59 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D1FA92960; Mon, 22 Jul 2013 17:11:58 +0000 (UTC) Received: by mail-wi0-f175.google.com with SMTP id m6so2159850wiv.2 for ; Mon, 22 Jul 2013 10:11:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=YbN64UG9gjLS2ugbrB9SAtdRrLPVCm3M/xVAMBbefg0=; b=OjGCVlM3dX6+yHeVE2MqD3NCozRr3JJH+OdbPmhVQPlgasGiKUIaWa8aUuDgQPJG7v dn1ZkHX0b99lMapVfU2iviAOoFE3oBoSfXe18eH/zamzZ2Q3KYYMd7Ue6t3HuojSzs4f 4S2toLF7IhxIAnzxyc8+hbkbNSg+Q9Y2J6By5uZgjm5agNGE5fPJ+JiMLktCil1Yn0He +YWN7YvoyVldJEQxiMEATtvGkjBlRwGTbEnq9Yv04QXe4ylMaTousuZ14/Yurm8Rae+L U0dn0yqATdZVbPbVDFvmtqHLjcP7XKUpHlvPMdLWpNQob+/Xhr9ilPJ1mbyLQvUQn0D+ DdPg== MIME-Version: 1.0 X-Received: by 10.194.11.72 with SMTP id o8mr20693027wjb.0.1374513117128; Mon, 22 Jul 2013 10:11:57 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Mon, 22 Jul 2013 10:11:57 -0700 (PDT) In-Reply-To: <51ED5308.3020008@gmx.com> References: <51ED5308.3020008@gmx.com> Date: Mon, 22 Jul 2013 10:11:57 -0700 X-Google-Sender-Auth: izKPMuXWKHEBcYxYnS4wEWHbiwA Message-ID: Subject: Re: VIMAGE + PF crash in mbuf destructor From: Adrian Chadd To: Nikos Vassiliadis Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 17:11:59 -0000 On 22 July 2013 08:43, Nikos Vassiliadis wrote: > Hi, > > I think this comes from the eventhandlers pf installs to handle > ifnet events. It seems like a wifi event causes this code to run > and the context is not set. Does the panic happen only when you > use vnet jails? > > Could you try putting all evenhandlers in an > 'if (IS_DEFAULT_VNET(curvnet))' block? > > It's here: > http://fxr.watson.org/fxr/source/netpfil/pf/pf_if.c#L127 > the pfi_*_cookie = ... lines. > > I am not sure if this would be enough though since it might > panic in other places. I don't think the default vnet context is the correct behaviour there. We'd need to figure out what the vnet context of the mbuf is and set that. -adrian From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 19:00:03 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 257C5FAE; Mon, 22 Jul 2013 19:00:03 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-la0-x22f.google.com (mail-la0-x22f.google.com [IPv6:2a00:1450:4010:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4419E2EE0; Mon, 22 Jul 2013 19:00:02 +0000 (UTC) Received: by mail-la0-f47.google.com with SMTP id fe20so5524241lab.20 for ; Mon, 22 Jul 2013 12:00:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=+S2YSubdM7jkzMXwiXCVOXe7QHiiMYCq7OQYlinWkpc=; b=ACTz9sTWiOWxzPhA6wOKRVsiEW5donWHbnXNw2ntm4U7pUieXcypx0rTT4+F3w6sGd pzfQaEFRGa9ro3IFbyuqzMM1wgZmrdyn8kgIcg2JXZqNoc7fRJseYTxKqD4vbgYthzFu eIZe02UBBn6GGeFBcfglKYNcummVtatqkzh61dRCLghAA6n6oBgot5QK5n0bCfCY981A 4pLw/yVY18GK1a2hQIIbYnCXV7FtgU36K2v1X2SDbeDSyVnQKGMyPvgaA4QjT7shC/Xi waCC3MH+GM7KkRiTsLGvvnR7JbWbx42yajtKtEBaxuldKerwJNz8GEYPDprxdPrs4Szy 5pGQ== MIME-Version: 1.0 X-Received: by 10.112.12.225 with SMTP id b1mr13000681lbc.3.1374519600114; Mon, 22 Jul 2013 12:00:00 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.149.38 with HTTP; Mon, 22 Jul 2013 11:59:59 -0700 (PDT) In-Reply-To: References: <51ED5308.3020008@gmx.com> Date: Mon, 22 Jul 2013 11:59:59 -0700 X-Google-Sender-Auth: tp7sirbKL9vvGWx44PYvYdEp_CI Message-ID: Subject: Re: VIMAGE + PF crash in mbuf destructor From: Craig Rodrigues To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 19:00:03 -0000 On Mon, Jul 22, 2013 at 10:11 AM, Adrian Chadd wrote: > > I don't think the default vnet context is the correct behaviour there. > We'd need to figure out what the vnet context of the mbuf is and set > that. > > What do you think about Marko's suggestion to de-virtualize V_pf_mtag_z? What would be the down side of that? I don't understand enough of the PF code to understand which variables need to be virtual and which don't. -- Craig From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 19:01:34 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 079FFD0; Mon, 22 Jul 2013 19:01:34 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x232.google.com (mail-wg0-x232.google.com [IPv6:2a00:1450:400c:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4660D2EF0; Mon, 22 Jul 2013 19:01:33 +0000 (UTC) Received: by mail-wg0-f50.google.com with SMTP id m15so934755wgh.5 for ; Mon, 22 Jul 2013 12:01:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=lW+wnwyCYHpMvmJkx+B1JdzE65vWVla1R7YXs5jGOJs=; b=BHquAVDNXDDsAyU0LaBgg8AkA7wD+QoJnQ2ddY7irTxClFZNtOjFFUMxx9BRELu5GV Lmbiik3YDdXTiiC1tL82J87Hyq06gbDIk00oeNh9J0vWXjv4q9LWbHTdxYvU4KETmSY3 c0QzEXEqF/Ra8ltVQR65HTJtPklTDNqJkH6WhUSzdHuUgnBcU796YuA9nSikf+MxBC/N GNsFdlU3AH7PqaPxcKskJ4lXcylSA9IVdaIXDxZLwKmImTjOSiz86+abXvoP3bmhVZaS BYKnxPaa2N86huRoiaaY8hSdBpvCBZHpodtZGpqgp/rXsnxrYDxyAy3KaufthvgCSVc7 5eNQ== MIME-Version: 1.0 X-Received: by 10.180.82.196 with SMTP id k4mr31144532wiy.0.1374519691607; Mon, 22 Jul 2013 12:01:31 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Mon, 22 Jul 2013 12:01:31 -0700 (PDT) In-Reply-To: References: <51ED5308.3020008@gmx.com> Date: Mon, 22 Jul 2013 12:01:31 -0700 X-Google-Sender-Auth: Z37cUi3HUr_PvFTKaLqL8JHaA-o Message-ID: Subject: Re: VIMAGE + PF crash in mbuf destructor From: Adrian Chadd To: Craig Rodrigues Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 19:01:34 -0000 Well I'm worried about _other_ stuff causing issues here. So - what's the "right" behaviour? Does vnet/vimage make the assumption that for all the mbuf processing/free operations, the vnet tag/state is set? -adrian On 22 July 2013 11:59, Craig Rodrigues wrote: > On Mon, Jul 22, 2013 at 10:11 AM, Adrian Chadd wrote: >> >> >> I don't think the default vnet context is the correct behaviour there. >> We'd need to figure out what the vnet context of the mbuf is and set >> that. >> > > What do you think about Marko's suggestion to de-virtualize > V_pf_mtag_z? What would be the down side of that? > > I don't understand enough of the PF code to understand which variables need > to > be virtual and which don't. > > -- > Craig From owner-freebsd-pf@FreeBSD.ORG Mon Jul 22 21:38:12 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 66F93FF3; Mon, 22 Jul 2013 21:38:12 +0000 (UTC) (envelope-from zec@fer.hr) Received: from mail.fer.hr (mail.fer.hr [161.53.72.233]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F09852661; Mon, 22 Jul 2013 21:38:11 +0000 (UTC) Received: from x23.lan (141.136.246.215) by MAIL.fer.hr (161.53.72.233) with Microsoft SMTP Server (TLS) id 14.2.342.3; Mon, 22 Jul 2013 23:38:09 +0200 From: Marko Zec To: Subject: Re: VIMAGE + PF crash in mbuf destructor Date: Mon, 22 Jul 2013 23:38:09 +0200 User-Agent: KMail/1.9.10 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-ID: <201307222338.09833.zec@fer.hr> X-Originating-IP: [141.136.246.215] Cc: Adrian Chadd , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 21:38:12 -0000 On Monday 22 July 2013 21:01:31 Adrian Chadd wrote: > Well I'm worried about _other_ stuff causing issues here. > > So - what's the "right" behaviour? Does vnet/vimage make the > assumption that for all the mbuf processing/free operations, the vnet > tag/state is set? To the best of my knowledge, there's nothing vnet-specific in any of the mbuf-handling routines, i.e. they should all work fine on a VIMAGE kernel even if curvnet isn't set. My original motivation behind keeping separate UMA zones for each vnet was solely to capture resource leaks between vnets in the early days of VIMAGE prototyping. Nothing prevents a single UMA zone to be shared by multiple vnets, unless we want to enforce per-vnet limitations on the number of items in a zone. Marko > > -adrian > > On 22 July 2013 11:59, Craig Rodrigues wrote: > > On Mon, Jul 22, 2013 at 10:11 AM, Adrian Chadd wrote: > >> I don't think the default vnet context is the correct behaviour there. > >> We'd need to figure out what the vnet context of the mbuf is and set > >> that. > > > > What do you think about Marko's suggestion to de-virtualize > > V_pf_mtag_z? What would be the down side of that? > > > > I don't understand enough of the PF code to understand which variables > > need to > > be virtual and which don't. > > > > -- > > Craig > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-pf@FreeBSD.ORG Fri Jul 26 07:45:15 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 27830E9C for ; Fri, 26 Jul 2013 07:45:15 +0000 (UTC) (envelope-from h.tugrul.erdogan@gmail.com) Received: from mail-vb0-x22c.google.com (mail-vb0-x22c.google.com [IPv6:2607:f8b0:400c:c02::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DBC342A32 for ; Fri, 26 Jul 2013 07:45:14 +0000 (UTC) Received: by mail-vb0-f44.google.com with SMTP id e15so1005891vbg.31 for ; Fri, 26 Jul 2013 00:45:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=WMKE2kBUR9kq44nqylkQjzMISGm/ntCyjf6jbrAUZ2s=; b=rovhhORAQyuSinshr0TpGbNFp8lwXUhxctJ5TqPojfgaPBu0NHpAbSCtM2hE2RyOYA 3GXWb6ygdVrNdU45585zNjygPNJJ59IR/a936+7y26NwzzI/bHZ9a2S+o6KEBE/A8NkR iMJaqOEqlwwXdTVwqrIzgGArJGQ/evdhOdUyWoAL/cA5kn6MLRYyOc+Il6H3ec08Dhij NXnSQYB63WZ/wUVZkHe4midV+ghWDFpxI9dGICNFJwGQCzW3lfOG6sKM0vTKgiWUeeDt l3f+KekXQ2SVUeKmw2y3YYgmTzKQppXB4SEnwTD2vx+imgD2gJcfStaIR73sRRCtFbkK AcsQ== MIME-Version: 1.0 X-Received: by 10.52.164.227 with SMTP id yt3mr16617650vdb.107.1374824714062; Fri, 26 Jul 2013 00:45:14 -0700 (PDT) Received: by 10.58.54.103 with HTTP; Fri, 26 Jul 2013 00:45:13 -0700 (PDT) Date: Fri, 26 Jul 2013 10:45:13 +0300 Message-ID: Subject: panic: kmem_map too small at heavy packet traffic From: Tugrul Erdogan To: freebsd-pf@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 07:45:15 -0000 howdy all, At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB ram. I am taking "panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated" message with configuration below: [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size: 16686845952 vm.kmem_size_scale: 1 [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem hw.physmem: 17151787008 hw.usermem: 8282652672 hw.realmem: 18253611008 [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages hw.pagesize: 4096 hw.pagesizes: 4096 2097152 0 hw.availpages: 4187448 When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at: pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128 and after the panic at the core dump file the major vmstat difference is: temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2 Specifically, I am taking this panic when doing ip spoof attack while syn-proxy activated. The output of system arguments below: kern.malloc_count: 315 vm.md_malloc_wait: 0 vfs.bufmallocspace: 0 vfs.maxmallocbufspace: 86269952 vm.kmem_size: 16686845952 vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size_scale: 1 vm.kmem_map_size: 543973376 vm.kmem_map_free: 15974895616 kern.maxvnodes: 350097 kern.minvnodes: 87524 vfs.numvnodes: 112329 vfs.wantfreevnodes: 87524 vfs.freevnodes: 87502 [root@ ~]# pfctl -si No ALTQ support in kernel ALTQ related functions disabled Status: Enabled for 0 days 00:17:39 Debug: Urgent State Table Total Rate current entries 5142886 searches 26982141 25478.9/s inserts 29055053 27436.3/s removals 24218654 22869.4/s Counters match 24901305 23514.0/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 18 0.0/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 29378439 27741.7/s [root@~]# panic: kmem_malloc(-1 814 425 600): kmem_map too small: 543 956 992 to tal allocated cpuid = 8 Uptime: 1d18h2m14s (ada0:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada0:ahcich1:0:0:0): CAM status: CCB request is in progress (ada0:ahcich1:0:0:0): Error 5, Retries exhausted (ada0:ahcich1:0:0:0): Synchronize cache failed (ada1:ahcich2:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada1:ahcich2:0:0:0): CAM status: CCB request is in progress (ada1:ahcich2:0:0:0): Error 5, Retries exhausted (ada1:ahcich2:0:0:0): Synchronize cache failed Dumping 8243 out of 16357 MB:..1%..11% When I explore the source code of kernel (at vm_kern.c and vm_map.c), I see that the panic can occur with the cases at below: * negative malloc size parameter * longer than free buffer respect to kmem_map min_offset and max_offset values * try to allocate when the root entry of map is the rightmost entry of map * try to allocate bigger than map's max_free value I think the panic occurs at mbuf creation process when calling malloc() as a result of couldn't be able to allocate memory; but I don't understand why one of this panic case activating? The memory is almost empty but the device is saying kmem_map small when using about 0.5GB memory purely. How can i solve this panic problem? Thank you all for your time. -- Best Wishes, Tugrul Erdogan From owner-freebsd-pf@FreeBSD.ORG Fri Jul 26 12:25:00 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 37AE4F24 for ; Fri, 26 Jul 2013 12:25:00 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [128.127.144.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A4F662677 for ; Fri, 26 Jul 2013 12:24:58 +0000 (UTC) Received: from bsdrookie.norma.com. (bsdrookie.norma.com [192.168.7.159] (may be forged)) by elf.hq.norma.perm.ru (8.14.5/8.14.5) with ESMTP id r6QCOnAg018918 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Fri, 26 Jul 2013 18:24:49 +0600 (YEKT) (envelope-from emz@norma.perm.ru) Message-ID: <51F26A91.202@norma.perm.ru> Date: Fri, 26 Jul 2013 18:24:49 +0600 From: "Eugene M. Zheganin" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130709 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-pf@freebsd.org Subject: carp-ng and vhid>9 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (elf.hq.norma.perm.ru [192.168.3.10]); Fri, 26 Jul 2013 18:24:49 +0600 (YEKT) X-Spam-Status: No hits=-101.0 bayes=0.5 testhits ALL_TRUSTED=-1, USER_IN_WHITELIST=-100 autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on elf.hq.norma.perm.ru X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 12:25:00 -0000 Hi. I use FreeBSD 10.0-CURRENT. Call me a dumbhead, but I have a strong impression that using vhid>9 causes problems: Host A (r251857): # ifconfig vlan3 vlan3: flags=8943 metric 0 mtu 1500 options=3 ether 00:1a:64:21:94:89 inet 128.127.145.2 netmask 0xfffffff8 broadcast 128.127.145.7 inet6 fe80::21a:64ff:fe21:9489%vlan3 prefixlen 64 scopeid 0x8 inet6 2a00:7540::2 prefixlen 120 inet 128.127.145.1 netmask 0xfffffff8 broadcast 128.127.145.7 vhid 10 inet6 2a00:7540::1 prefixlen 120 vhid 10 nd6 options=21 media: Ethernet autoselect (100baseTX ) status: active vlan: 3 parent interface: bge0 carp: MASTER vhid 10 advbase 1 advskew 100 Host B (r253079): # ifconfig vlan3 vlan3: flags=8943 metric 0 mtu 1500 options=3 ether 00:1a:64:21:95:28 inet 128.127.145.3 netmask 0xfffffff8 broadcast 128.127.145.7 inet6 fe80::21a:64ff:fe21:9528%vlan3 prefixlen 64 scopeid 0x8 inet6 2a00:7540::3 prefixlen 120 inet 128.127.145.1 netmask 0xfffffff8 broadcast 128.127.145.7 vhid 10 inet6 2a00:7540::1 prefixlen 120 vhid 10 nd6 options=21 media: Ethernet autoselect (100baseTX ) status: active vlan: 3 parent interface: bge0 carp: BACKUP vhid 10 advbase 1 advskew 200 but in dmesg on host B I see (net.inet.carp.log: 2): [...] carp: VHID 10@vlan3: MASTER -> BACKUP (more frequent advertisement received) carp: VHID 11@vlan600: BACKUP -> MASTER (master down) carp: VHID 11@vlan600: MASTER -> BACKUP (more frequent advertisement received) carp: VHID 10@vlan3: BACKUP -> MASTER (master down) carp: VHID 10@vlan3: MASTER -> BACKUP (more frequent advertisement received) carp: VHID 11@vlan600: BACKUP -> MASTER (master down) carp: VHID 11@vlan600: MASTER -> BACKUP (more frequent advertisement received) carp: VHID 10@vlan3: BACKUP -> MASTER (master down) carp: VHID 10@vlan3: MASTER -> BACKUP (more frequent advertisement received) carp: VHID 11@vlan600: BACKUP -> MASTER (master down) carp: VHID 11@vlan600: MASTER -> BACKUP (more frequent advertisement received) carp: VHID 10@vlan3: BACKUP -> MASTER (master down) carp: VHID 10@vlan3: MASTER -> BACKUP (more frequent advertisement received) carp: VHID 11@vlan600: BACKUP -> MASTER (master down) [...] I checked and rechecked the passwords on both machines - identical. As you can see - address pairs are identical too. The fact that host B receives some vrrp announces tells that the packet filter/passwords are configured properly (in fact, I turned it off on B, but this didn't resolve the situation). netstat: Host A: # netstat -s -p carp carp: 524 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for bad vhid 0 discarded because of a bad address list 14064 packets sent (IPv4) 3516 packets sent (IPv6) 0 send failed due to mbuf memory error Host B: # netstat -s -p carp carp: 12881 packets received (IPv4) 1171 packets received (IPv6) 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for bad vhid 0 discarded because of a bad address list 648 packets sent (IPv4) 570 packets sent (IPv6) 0 send failed due to mbuf memory error I would be really glad if someone could help me with this. I have seen this situation when host B was running FreeBSD 8.2-STABLE, but first I thought that there's some incompatibility between old carp and carp-ng. I also have 9 other vhids that work just fine. The only explanation I have so far - using vhid>9 causes some errors on the backup side. Thanks. Eugene. From owner-freebsd-pf@FreeBSD.ORG Fri Jul 26 13:27:50 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D4B9599A for ; Fri, 26 Jul 2013 13:27:50 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [128.127.144.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4D8C02B90 for ; Fri, 26 Jul 2013 13:27:49 +0000 (UTC) Received: from bsdrookie.norma.com. (bsdrookie.norma.com [192.168.7.159] (may be forged)) by elf.hq.norma.perm.ru (8.14.5/8.14.5) with ESMTP id r6QDRjv0020586 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Fri, 26 Jul 2013 19:27:45 +0600 (YEKT) (envelope-from emz@norma.perm.ru) Message-ID: <51F27951.5080809@norma.perm.ru> Date: Fri, 26 Jul 2013 19:27:45 +0600 From: "Eugene M. Zheganin" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130709 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-pf@freebsd.org Subject: Re: carp-ng and vhid>9 References: <51F26A91.202@norma.perm.ru> In-Reply-To: <51F26A91.202@norma.perm.ru> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (elf.hq.norma.perm.ru [192.168.3.10]); Fri, 26 Jul 2013 19:27:45 +0600 (YEKT) X-Spam-Status: No hits=-101.0 bayes=0.5 testhits ALL_TRUSTED=-1, USER_IN_WHITELIST=-100 autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on elf.hq.norma.perm.ru X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 13:27:50 -0000 Hi. On 26.07.2013 18:24, Eugene M. Zheganin wrote: > Hi. > > I use FreeBSD 10.0-CURRENT. > Call me a dumbhead, but I have a strong impression that using vhid>9 > causes problems: > Discard my last message, I found that ipv6 carp was filtered out on these interfaces. Sorry for disturbing. Eugene. From owner-freebsd-pf@FreeBSD.ORG Sat Jul 27 06:31:49 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 282E9731; Sat, 27 Jul 2013 06:31:49 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-oa0-x236.google.com (mail-oa0-x236.google.com [IPv6:2607:f8b0:4003:c02::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BCA312987; Sat, 27 Jul 2013 06:31:48 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id o17so117514oag.27 for ; Fri, 26 Jul 2013 23:31:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; bh=ygTXVXI0hjEfzZRs3AT9CaENC0D/NWlXPfgfvbXM8Xg=; b=Q8r2gHvBpWT9AdlvhDwJG8yOZgNC0QzKjKYRmw18Yh3ThKhWhA12HnfA4QcbKM9TRi aZrQbAbjd9DzEW52OfygKr+O03ZOnsWDXR5VlqU0KqpWVlQ1OqOliaonhbYuP73KQjQz EImkP/KIgq7wlfVhMW2F4RvHZHgp9DkoxEcxNDnIfyhJIqDNdNp8J/r+Ey4M1tjqPziA bQQROyqOezljQCBua3PyirfFL297ZW+a+c/N4whn/9QGju58cLIjlTbMlIAxr9pniYnX U7sKEKi/zfTq5A5LpKFQLAfWnyIewTopWb1Q2+8Qewgch0G6cw9Hi2V56mLuuPZsiGVf 0k/w== MIME-Version: 1.0 X-Received: by 10.60.133.14 with SMTP id oy14mr51312606oeb.84.1374906708080; Fri, 26 Jul 2013 23:31:48 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.60.132.243 with HTTP; Fri, 26 Jul 2013 23:31:48 -0700 (PDT) Date: Fri, 26 Jul 2013 23:31:48 -0700 X-Google-Sender-Auth: Nrh4OGGgAcLC7znvcg2Jor9gp_U Message-ID: Subject: De-virtualize V_pf_mtag_z to eliminate kernel panics. From: Craig Rodrigues To: Gleb Smirnoff Content-Type: multipart/mixed; boundary=047d7b472746ef40e904e278696f X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Adrian Chadd , Marko Zec , "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 06:31:49 -0000 --047d7b472746ef40e904e278696f Content-Type: text/plain; charset=ISO-8859-1 Gleb, Since you did a lot of work in GRN 240233 to fix PF issues, especially for VIMAGE, I thought I would ask your opinion on the attached patch. In this post: http://lists.freebsd.org/pipermail/freebsd-virtualization/2013-July/001405.html I reported multiple PF-related panics when VIMAGE was enabled in my kernel config. In these posts: http://lists.freebsd.org/pipermail/freebsd-virtualization/2013-July/001413.html http://lists.freebsd.org/pipermail/freebsd-virtualization/2013-July/001420.html Marko Zec seemed to think that de-virtualizing V_pf_mtag_z would be a valid solution to this problem, and that keeping V_pf_mtag_z as a per-vnet variable is not necessary. What do you think of Marko's comments, and this patch? Thanks. -- Craig --047d7b472746ef40e904e278696f Content-Type: text/plain; charset=US-ASCII; name="diff.txt" Content-Disposition: attachment; filename="diff.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hjmfwbmu0 SW5kZXg6IHN5cy9uZXRwZmlsL3BmL3BmLmMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lzL25ldHBmaWwvcGYv cGYuYwkocmV2aXNpb24gMjUzMzQ2KQorKysgc3lzL25ldHBmaWwvcGYvcGYuYwkod29ya2luZyBj b3B5KQpAQCAtMTg3LDggKzE4Nyw3IEBACiAKIHN0YXRpYyBWTkVUX0RFRklORSh1bWFfem9uZV90 LAlwZl9zb3VyY2VzX3opOwogI2RlZmluZQlWX3BmX3NvdXJjZXNfeglWTkVUKHBmX3NvdXJjZXNf eikKLXN0YXRpYyBWTkVUX0RFRklORSh1bWFfem9uZV90LAlwZl9tdGFnX3opOwotI2RlZmluZQlW X3BmX210YWdfeglWTkVUKHBmX210YWdfeikKK3VtYV96b25lX3QgcGZfbXRhZ196OwogVk5FVF9E RUZJTkUodW1hX3pvbmVfdCwJIHBmX3N0YXRlX3opOwogVk5FVF9ERUZJTkUodW1hX3pvbmVfdCwJ IHBmX3N0YXRlX2tleV96KTsKIApAQCAtNzQ5LDcgKzc0OCw3IEBACiAJVl9wZl9hbHRxc19pbmFj dGl2ZSA9ICZWX3BmX2FsdHFzWzFdOwogCiAJLyogTWJ1ZiB0YWdzICovCi0JVl9wZl9tdGFnX3og PSB1bWFfemNyZWF0ZSgicGYgbXRhZ3MiLCBzaXplb2Yoc3RydWN0IG1fdGFnKSArCisJcGZfbXRh Z196ID0gdW1hX3pjcmVhdGUoInBmIG10YWdzIiwgc2l6ZW9mKHN0cnVjdCBtX3RhZykgKwogCSAg ICBzaXplb2Yoc3RydWN0IHBmX210YWcpLCBOVUxMLCBOVUxMLCBwZl9tdGFnX2luaXQsIE5VTEws CiAJICAgIFVNQV9BTElHTl9QVFIsIDApOwogCkBAIC04MDMsNyArODAyLDcgQEAKIAltdHhfZGVz dHJveSgmcGZfb3ZlcmxvYWRxdWV1ZV9tdHgpOwogCW10eF9kZXN0cm95KCZwZl91bmxua2RydWxl c19tdHgpOwogCi0JdW1hX3pkZXN0cm95KFZfcGZfbXRhZ196KTsKKwl1bWFfemRlc3Ryb3kocGZf bXRhZ196KTsKIAl1bWFfemRlc3Ryb3koVl9wZl9zb3VyY2VzX3opOwogCXVtYV96ZGVzdHJveShW X3BmX3N0YXRlX3opOwogCXVtYV96ZGVzdHJveShWX3BmX3N0YXRlX2tleV96KTsKQEAgLTgyNyw3 ICs4MjYsNyBAQAogcGZfbXRhZ19mcmVlKHN0cnVjdCBtX3RhZyAqdCkKIHsKIAotCXVtYV96ZnJl ZShWX3BmX210YWdfeiwgdCk7CisJdW1hX3pmcmVlKHBmX210YWdfeiwgdCk7CiB9CiAKIHN0cnVj dCBwZl9tdGFnICoKQEAgLTgzOCw3ICs4MzcsNyBAQAogCWlmICgobXRhZyA9IG1fdGFnX2ZpbmQo bSwgUEFDS0VUX1RBR19QRiwgTlVMTCkpICE9IE5VTEwpCiAJCXJldHVybiAoKHN0cnVjdCBwZl9t dGFnICopKG10YWcgKyAxKSk7CiAKLQltdGFnID0gdW1hX3phbGxvYyhWX3BmX210YWdfeiwgTV9O T1dBSVQpOworCW10YWcgPSB1bWFfemFsbG9jKHBmX210YWdfeiwgTV9OT1dBSVQpOwogCWlmICht dGFnID09IE5VTEwpCiAJCXJldHVybiAoTlVMTCk7CiAJYnplcm8obXRhZyArIDEsIHNpemVvZihz dHJ1Y3QgcGZfbXRhZykpOwo= --047d7b472746ef40e904e278696f-- From owner-freebsd-pf@FreeBSD.ORG Sat Jul 27 14:20:01 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1FBBB8E5; Sat, 27 Jul 2013 14:20:01 +0000 (UTC) (envelope-from zec@fer.hr) Received: from mail.fer.hr (mail.fer.hr [161.53.72.233]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A6AFC27BE; Sat, 27 Jul 2013 14:20:00 +0000 (UTC) Received: from x23.lan (89.164.3.121) by MAIL.fer.hr (161.53.72.233) with Microsoft SMTP Server (TLS) id 14.2.342.3; Sat, 27 Jul 2013 16:19:51 +0200 From: Marko Zec To: Craig Rodrigues Subject: Re: De-virtualize V_pf_mtag_z to eliminate kernel panics. Date: Sat, 27 Jul 2013 16:19:51 +0200 User-Agent: KMail/1.9.10 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-ID: <201307271619.51409.zec@fer.hr> X-Originating-IP: [89.164.3.121] Cc: Adrian Chadd , "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 14:20:01 -0000 On Saturday 27 July 2013 08:31:48 Craig Rodrigues wrote: > Gleb, > > Since you did a lot of work in GRN 240233 > to fix PF issues, especially for VIMAGE, I thought I would > ask your opinion on the attached patch. > > In this post: > > http://lists.freebsd.org/pipermail/freebsd-virtualization/2013-July/00140 >5.html > > I reported multiple PF-related panics when VIMAGE was enabled > in my kernel config. > > In these posts: > http://lists.freebsd.org/pipermail/freebsd-virtualization/2013-July/00141 >3.html > http://lists.freebsd.org/pipermail/freebsd-virtualization/2013-July/00142 >0.html > > Marko Zec seemed to think that de-virtualizing V_pf_mtag_z > would be a valid solution to this problem, and that keeping > V_pf_mtag_z as a per-vnet variable is not necessary. > > What do you think of Marko's comments, and this patch? Hi Craig, while in principle I agree with the intent to de-virtualize V_pf_mtag_z (after all, this was my suggestion in the first place), your proposed patch isn't the valid solution, since on each vnet creation you'll be clobbering (now global) variable pf_mtag_z, which would imminently cause double-free or similar issues later during runtime. You should move pf_mtag_z = uma_zcreate(...) to some other function which isn't called for each vnet, or at least as a crude kludge, do this conditionally if (curvnet == vnet0). The same goes for uma_zdestroy(pf_mtag_z)... Cheers, Marko From owner-freebsd-pf@FreeBSD.ORG Sat Jul 27 14:48:03 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7AC1FCE6; Sat, 27 Jul 2013 14:48:03 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 901DF2875; Sat, 27 Jul 2013 14:48:02 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id j17so1170311wiw.1 for ; Sat, 27 Jul 2013 07:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=UjO7gUdWXYYWpPRALND96WR/HtJEoieKqnBhh/YCqMA=; b=masB/uE3iuk6E9IPcv7eCzEQHSALuQZve6Sxg80kcnJWzYVpVY73Ekdt1bVbCzHhXt 9+D8ce47tkpg8/ANP58IP96JXxAfs37J8Dh69KAYd8h2od/DMhEh6LLBnAxwYVTB7DV5 ZsDo4j38YbpZxYANteXBKB3qPOgoVbO8kEo5PPVNOAaNjFyaQJKU+B9UpuUnSvp/ceyr Si1TyamjVis4+Z9HNCNlFSn3+THB7u4sUCWv5J0XKBDcmFI8xejRcrao0KI1Lbhp0j9C JzFPBlAEgZ8UUI1j5y6gyJMMbjZWOTOxmWg/kHJsCR1fX8uqIxinrBtujcXZqvVVY/VW EsDg== MIME-Version: 1.0 X-Received: by 10.194.92.6 with SMTP id ci6mr13605838wjb.79.1374936480746; Sat, 27 Jul 2013 07:48:00 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Sat, 27 Jul 2013 07:48:00 -0700 (PDT) In-Reply-To: <201307271619.51409.zec@fer.hr> References: <201307271619.51409.zec@fer.hr> Date: Sat, 27 Jul 2013 07:48:00 -0700 X-Google-Sender-Auth: iUyblCNVzXLHjAOT63qxpFTT2-M Message-ID: Subject: Re: De-virtualize V_pf_mtag_z to eliminate kernel panics. From: Adrian Chadd To: Marko Zec Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 14:48:03 -0000 I'm happy keeping it virtual (it lets us do things like set per-vimage mbuf tag limits, and having per-vimage mbufs may be a useful long term stretch goal to have).. we just have to think about this stuff in more detail. -adrian From owner-freebsd-pf@FreeBSD.ORG Sat Jul 27 15:29:00 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E8A87747; Sat, 27 Jul 2013 15:29:00 +0000 (UTC) (envelope-from zec@fer.hr) Received: from mail.fer.hr (mail.fer.hr [161.53.72.233]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7869A29D2; Sat, 27 Jul 2013 15:29:00 +0000 (UTC) Received: from x23.lan (89.164.3.121) by MAIL.fer.hr (161.53.72.233) with Microsoft SMTP Server (TLS) id 14.2.342.3; Sat, 27 Jul 2013 17:28:57 +0200 From: Marko Zec To: Adrian Chadd Subject: Re: De-virtualize V_pf_mtag_z to eliminate kernel panics. Date: Sat, 27 Jul 2013 17:28:56 +0200 User-Agent: KMail/1.9.10 References: <201307271619.51409.zec@fer.hr> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-ID: <201307271728.56777.zec@fer.hr> X-Originating-IP: [89.164.3.121] Cc: "freebsd-virtualization@freebsd.org" , freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 15:29:01 -0000 On Saturday 27 July 2013 16:48:00 Adrian Chadd wrote: > I'm happy keeping it virtual (it lets us do things like set per-vimage > mbuf tag limits, and having per-vimage mbufs may be a useful long term > stretch goal to have).. we just have to think about this stuff in more > detail. The V_pf_mtag_z zone apparently doesn't have any size limits, so it doesn't enforce mbuf tag limits in pf, and certainly it doesn't enforce any limits outside of PF? And while otherwise I wouldn't terribly object to keep V_pf_mtag_z virtual, if we don't de-virtualize it, would you have an alternative proposal to resolve the panics in PF which Craig wants to have fixed? Finally, while the idea about enforcing per-vnet mbuf (and tag) limits may look neat on the surface, the fact that mbufs may flow freely between vnets makes this idea less trivial to implement than it may appear at the first glance. In any case, IMO that issue is entirely unrelated to the patch Craig proposed, and should be discussed separately. Marko From owner-freebsd-pf@FreeBSD.ORG Sat Jul 27 16:12:24 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4F6B9FCD for ; Sat, 27 Jul 2013 16:12:24 +0000 (UTC) (envelope-from yeris@netcrc.net) Received: from mail.netcrc.net (mail.netcrc.net [190.124.246.246]) by mx1.freebsd.org (Postfix) with ESMTP id 195242B2E for ; Sat, 27 Jul 2013 16:12:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.netcrc.net (Postfix) with ESMTP id 5D4525343C6 for ; Sat, 27 Jul 2013 10:06:57 -0600 (CST) X-Virus-Scanned: Debian amavisd-new at mail.netcrc.net Received: from mail.netcrc.net ([127.0.0.1]) by localhost (mail.netcrc.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aNZpHeogky0K for ; Sat, 27 Jul 2013 10:06:56 -0600 (CST) Received: from [192.168.50.110] (unknown [190.124.246.29]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: yeris@netcrc.net) by mail.netcrc.net (Postfix) with ESMTPSA id A9E825343C5 for ; Sat, 27 Jul 2013 10:06:56 -0600 (CST) Message-ID: <51F3F06F.4000101@netcrc.net> Date: Sat, 27 Jul 2013 10:08:15 -0600 From: Yeris Antonio Madrigal Castro User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130623 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-pf@freebsd.org Subject: pf log script issue References: <51F30C00.2000007@netcrc.cr> In-Reply-To: <51F30C00.2000007@netcrc.cr> X-Enigmail-Version: 1.5.2 X-Forwarded-Message-Id: <51F30C00.2000007@netcrc.cr> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 16:12:24 -0000 Hi I am using freebsd 9.1 R, and i am trying to make the pf firewall to log to the syslog. I am using the script at the openbsd fag: http://www.openbsd.org/faq/pf/logging.html The script works 100% if I run it manually, but when crontab runs it as root, the tcpdump wont work. I would appreciate any help. Regards ** ** From owner-freebsd-pf@FreeBSD.ORG Sat Jul 27 16:18:03 2013 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D9351CC for ; Sat, 27 Jul 2013 16:18:03 +0000 (UTC) (envelope-from jhellenthal@dataix.net) Received: from mail-gh0-x234.google.com (mail-gh0-x234.google.com [IPv6:2607:f8b0:4002:c05::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 909EE2B4C for ; Sat, 27 Jul 2013 16:18:03 +0000 (UTC) Received: by mail-gh0-f180.google.com with SMTP id f18so1227894ghb.11 for ; Sat, 27 Jul 2013 09:18:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dataix.net; s=rsa; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to; bh=gfeCxZTu8sOd5UonYWAYbzHT+xoGmdnAzwpxE967c3M=; b=R8VaI0LG8sg9oICX2EV75gkIyqSRdEetRBxcl7LijO8IvMelC71PQ+0Gf99FpQgH5+ 68tB5pMBK3eWAQY3Bnh5cDFH3T2AmdStPdf/eYd4Ykutx1ke4TmJsu9hVn2plvw5+41Z sz7SkT/VN1KPS18Q9oaqljLHoDGT9BBQB2uAw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to:x-gm-message-state; bh=gfeCxZTu8sOd5UonYWAYbzHT+xoGmdnAzwpxE967c3M=; b=OqyiPyHoOxatmPEz2dnu439parHzZfUpLao8NpXWdyxOaT/8VjU9rPeF3kPM/2HUl5 u/YulJuzpJvpaN9R7HDkHWlNsvnnyQUfkQWFVTqAQKIh9s0amtU8HqF5OqIfbWTf11GF UcXsvAc4kXGCfi6c02M60iC4ihW32AMtTzNJF8xGBQUOFMlgHNJsQXoYyLtURXOBSpmG lLdUQ0mLaYpa3U+TsLGy7AdNTcp7UTwgMt9MbTxvIF/fu0wX7+TrBGiyxyHaCRHgtKTz +eHa93uCUVLHg2GwffrhgPqZHzef7NcO9Znjvye+hcQdL4mb5Z6N43STLZaYyC6Xmx+b gv3w== X-Received: by 10.236.216.77 with SMTP id f73mr23449665yhp.101.1374941882577; Sat, 27 Jul 2013 09:18:02 -0700 (PDT) Received: from [192.168.31.77] (68-188-148-148.dhcp.aldl.mi.charter.com. [68.188.148.148]) by mx.google.com with ESMTPSA id n4sm6500746yhk.5.2013.07.27.09.18.00 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 27 Jul 2013 09:18:01 -0700 (PDT) References: <51F30C00.2000007@netcrc.cr> <51F3F06F.4000101@netcrc.net> Mime-Version: 1.0 (1.0) In-Reply-To: <51F3F06F.4000101@netcrc.net> Content-Type: multipart/signed; micalg=sha1; boundary=Apple-Mail-D4F05C8E-9464-47CC-BDB1-6CCCA3F5B909; protocol="application/pkcs7-signature" Content-Transfer-Encoding: 7bit Message-Id: <166BA3A5-3876-4383-B78B-7BCB32BCDE80@dataix.net> X-Mailer: iPhone Mail (10B350) From: Jason Hellenthal Subject: Re: pf log script issue Date: Sat, 27 Jul 2013 12:17:57 -0400 To: Yeris Antonio Madrigal Castro X-Gm-Message-State: ALoCoQkNW8Egu/9o5iWp/FF4I6LI4BQHwIDZ6sNyxg2kqSxEe/1YW+qZsQ2x/XkLFRX+m+gvRek2 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-pf@freebsd.org" X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 16:18:04 -0000 --Apple-Mail-D4F05C8E-9464-47CC-BDB1-6CCCA3F5B909 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Are you using /etc/crontab ? --=20 Jason Hellenthal Inbox: jhellenthal@DataIX.net Voice: +1 (616) 953-0176 JJH48-ARIN On Jul 27, 2013, at 12:08, Yeris Antonio Madrigal Castro w= rote: >=20 > Hi >=20 > I am using freebsd 9.1 R, and i am trying to make the pf firewall to log > to the syslog. >=20 > I am using the script at the openbsd fag: >=20 > http://www.openbsd.org/faq/pf/logging.html >=20 > The script works 100% if I run it manually, but when crontab runs it as > root, the tcpdump wont work. >=20 > I would appreciate any help. >=20 > Regards >=20 > ** > ** >=20 >=20 >=20 > _______________________________________________ > freebsd-pf@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" --Apple-Mail-D4F05C8E-9464-47CC-BDB1-6CCCA3F5B909 Content-Type: application/pkcs7-signature; name=smime.p7s Content-Disposition: attachment; filename=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIUOTCCBjAw ggUYoAMCAQICAwaijjANBgkqhkiG9w0BAQsFADCBjDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0 YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcx ODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRlcm1lZGlhdGUgQ2xpZW50IENB MB4XDTEzMDUxODA4NTA0OFoXDTE0MDUxOTIyMDk0N1owSDEfMB0GA1UEAwwWamhlbGxlbnRoYWxA ZGF0YWl4Lm5ldDElMCMGCSqGSIb3DQEJARYWamhlbGxlbnRoYWxAZGF0YWl4Lm5ldDCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBALgnYFS1bWZr3KhKBzWAdRwrY+En+RRV8nCaYubqrMG+ YJbuenaIKSbIuFiDWipW4RHYTpE28pKaSnaVTG9WtAZvsWj0gYN9g2fYCnCOUceES2Yvi3RavxpB hsuzKIfsHb8iNNSEuczLu6gn4mQyaHwE4x6xSUKmbK8njR+YoF522F60wjsnq5dlOJdTrhDfObE5 5P23279WbRp8azgZX1VRB66wdKRDuSI1vBts4Nsha2paXd6HUUduHrPACBQREJTGXN8XtEKVwo63 aKUhRgtUwHNEuSWck/xwVl7PBUWH2dORAWTCqHjNuCKNOQ1/0LMiyMj7FdsBjN4dgL4YZpsCAwEA AaOCAtwwggLYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggr BgEFBQcDBDAdBgNVHQ4EFgQU29qUrmZtgQ7ZVoDKogfpJOSfk+YwHwYDVR0jBBgwFoAUU3Ltkpzg 2ssBXHx+ljVO8tS4UYIwIQYDVR0RBBowGIEWamhlbGxlbnRoYWxAZGF0YWl4Lm5ldDCCAUwGA1Ud IASCAUMwggE/MIIBOwYLKwYBBAGBtTcBAgMwggEqMC4GCCsGAQUFBwIBFiJodHRwOi8vd3d3LnN0 YXJ0c3NsLmNvbS9wb2xpY3kucGRmMIH3BggrBgEFBQcCAjCB6jAnFiBTdGFydENvbSBDZXJ0aWZp Y2F0aW9uIEF1dGhvcml0eTADAgEBGoG+VGhpcyBjZXJ0aWZpY2F0ZSB3YXMgaXNzdWVkIGFjY29y ZGluZyB0byB0aGUgQ2xhc3MgMSBWYWxpZGF0aW9uIHJlcXVpcmVtZW50cyBvZiB0aGUgU3RhcnRD b20gQ0EgcG9saWN5LCByZWxpYW5jZSBvbmx5IGZvciB0aGUgaW50ZW5kZWQgcHVycG9zZSBpbiBj b21wbGlhbmNlIG9mIHRoZSByZWx5aW5nIHBhcnR5IG9ibGlnYXRpb25zLjA2BgNVHR8ELzAtMCug KaAnhiVodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9jcnR1MS1jcmwuY3JsMIGOBggrBgEFBQcBAQSB gTB/MDkGCCsGAQUFBzABhi1odHRwOi8vb2NzcC5zdGFydHNzbC5jb20vc3ViL2NsYXNzMS9jbGll bnQvY2EwQgYIKwYBBQUHMAKGNmh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3N1Yi5jbGFz czEuY2xpZW50LmNhLmNydDAjBgNVHRIEHDAahhhodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS8wDQYJ KoZIhvcNAQELBQADggEBAHsw8/Hw07gsNTKYnld74NBFtHnQOPkXYuccWx3j0PGQe9nqNxeingBf 2yvx+xBQzBoi4J1u84Jbrbe8Ii3+LLD/QMW9cN0SBIgRStPQLVee4STdjeabGmpXQa7omC02wYYO 83qh6CgJEIbmrsBSZH8ZSVrjkC4UmZS8wAQMS3qTWAPF0ZQGWx2+Gks2fXuacyt2LpNR+p9ogjAZ 1/rmUKjNhQZLswytaLRUdwAwSfQ3+TNs68h6Kv1LC3bNGBT3NEtr2q/nzzb5MzuFcDE6f9exroAC 4BHmokAprhna/vZdb6BrPjpXgRAlWAh3wEMxw75M9S/Nbzj/jNp+I+lvUJYwggY0MIIEHKADAgEC AgEeMA0GCSqGSIb3DQEBBQUAMH0xCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQu MSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMSkwJwYDVQQDEyBT dGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wNzEwMjQyMTAxNTVaFw0xNzEwMjQy MTAxNTVaMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMi U2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xh c3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IB DwAwggEKAoIBAQDHCYPMzi3YGrEppC4Tq5a+ijKDjKaIQZZVR63UbxIP6uq/I0fhCu+cQhoUfE6E RKKnu8zPf1Jwuk0tsvVCk6U9b+0UjM0dLep3ZdE1gblK/1FwYT5Pipsu2yOMluLqwvsuz9/9f1+1 PKHG/FaR/wpbfuIqu54qzHDYeqiUfsYzoVflR80DAC7hmJ+SmZnNTWyUGHJbBpA8Q89lGxahNvur yGaC/o2/ceD2uYDX9U8Eg5DpIpGQdcbQeGarV04WgAUjjXX5r/2dabmtxWMZwhZna//jdiSyrrSM TGKkDiXm6/3/4ebfeZuCYKzN2P8O2F/Xe2AC/Y7zeEsnR7FOp+uXAgMBAAGjggGtMIIBqTAPBgNV HRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUU3Ltkpzg2ssBXHx+ljVO8tS4 UYIwHwYDVR0jBBgwFoAUTgvvGqRAW6UXaYcwyjRoQ9BBrvIwZgYIKwYBBQUHAQEEWjBYMCcGCCsG AQUFBzABhhtodHRwOi8vb2NzcC5zdGFydHNzbC5jb20vY2EwLQYIKwYBBQUHMAKGIWh0dHA6Ly93 d3cuc3RhcnRzc2wuY29tL3Nmc2NhLmNydDBbBgNVHR8EVDBSMCegJaAjhiFodHRwOi8vd3d3LnN0 YXJ0c3NsLmNvbS9zZnNjYS5jcmwwJ6AloCOGIWh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL3Nmc2Nh LmNybDCBgAYDVR0gBHkwdzB1BgsrBgEEAYG1NwECATBmMC4GCCsGAQUFBwIBFiJodHRwOi8vd3d3 LnN0YXJ0c3NsLmNvbS9wb2xpY3kucGRmMDQGCCsGAQUFBwIBFihodHRwOi8vd3d3LnN0YXJ0c3Ns LmNvbS9pbnRlcm1lZGlhdGUucGRmMA0GCSqGSIb3DQEBBQUAA4ICAQAKgwh9eKssBly4Y4xerhy5 I3dNoXHYfYa8PlVLL/qtXnkFgdtY1o95CfegFJTwqBBmf8pyTUnFsukDFUI22zF5bVHzuJ+GxhnS qN2sD1qetbYwBYK2iyYA5Pg7Er1A+hKMIzEzcduRkIMmCeUTyMyikfbUFvIBivtvkR8ZFAk22BZy +pJfAoedO61HTz4qSfQoCRcLN5A0t4DkuVhTMXIzuQ8CnykhExD6x4e6ebIbrjZLb7L+ocR0y4Yj Cl/Pd4MXU91y0vTipgr/O75CDUHDRHCCKBVmz/Rzkc/b970MEeHt5LC3NiWTgBSvrLEuVzBKM586 YoRD9Dy3OHQgWI270g+5MYA8GfgI/EPT5G7xPbCDz+zjdH89PeR3U4So4lSXur6H6vp+m9TQXPF3 a0LwZrp8MQ+Z77U1uL7TelWO5lApsbAonrqASfTpaprFVkL4nyGH+NHST2ZJPWIBk81i6Vw0ny0q ZW2Niy/QvVNKbb43A43ny076khXO7cNbBIRdJ/6qQNq9Bqb5C0Q5nEsFcj75oxQRqlKf6TcvGbjx kJh8BYtv9ePsXklAxtm8J7GCUBthHSQgepbkOexhJ0wP8imUkyiPHQ0GvEnd83129fZjoEhdGwXV 27ioRKbj/cIq7JRXun0NbeY+UdMYu9jGfIpDLtUUGSgsg2zMGs5R4jCCB8kwggWxoAMCAQICAQEw DQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzAp BgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAnBgNVBAMTIFN0YXJ0 Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA2MDkxNzE5NDYzNloXDTM2MDkxNzE5NDYz NlowfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3Vy ZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAnBgNVBAMTIFN0YXJ0Q29tIENlcnRpZmlj YXRpb24gQXV0aG9yaXR5MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwYjbCbxsRnx4 n5V7tTOQ8nJi1sE2ICIkXs7pd/JDCqIGZKTMjjb4OOYj8G5tsTzdcqOFHKHTPbQzK9Mvr/7qsEFZ Z7bEBn0KnnSF1nlMgDd63zkFUln39BtGQ6TShYXSw3HzdWI0uiyKfx6P7u000BHHls1SPboz1t1N 3gs7SkufwiYv+rUWHHI1d8o8XebK4SaLGjZ2XAHbdBQl/u21oIgP3XjKLR8HlzABLXJ5+kbWEyqo uaarg0kd5fLv3eQBjhgKj2NTFoViqQ4ZOsy1ZqbCa3QH5Cvhdj60bdj2ROFzYh87xL6gU1YlbFEJ 96qryr92/W2b853bvz1mvAxWqq+YSJU6S9+nWFDZOHWpW+pDDAL/mevobE1wWyllnN2qXcyvATHs DOvSjejqnHvmbvcnZgwaSNduQuM/3iE+e+ENcPtjqqhsGlS0XCV6yaLJixamuyx+F14FTVhuEh0B 7hIQDcYyfxj//PT6zW6R6DZJvhpIaYvClk0aErJpF8EKkNb6eSJIv7p7afhwx/p6N9jYDdJ2T1f/ kLfjkdLd78Jgt2c63f6qnPDUi39yIs7Gn5e2+K+KoBCo2fsYxra1XFI8ibYZKnMBCg8DsxJg8nov gdujbv8mMJf1i92JV7atPbOvK8W3dgLwpdYrmoYUKnL24zOMXQlLE9+7jHQTUksCAwEAAaOCAlIw ggJOMAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgGuMB0GA1UdDgQWBBROC+8apEBbpRdphzDKNGhD 0EGu8jBkBgNVHR8EXTBbMCygKqAohiZodHRwOi8vY2VydC5zdGFydGNvbS5vcmcvc2ZzY2EtY3Js LmNybDAroCmgJ4YlaHR0cDovL2NybC5zdGFydGNvbS5vcmcvc2ZzY2EtY3JsLmNybDCCAV0GA1Ud IASCAVQwggFQMIIBTAYLKwYBBAGBtTcBAQEwggE7MC8GCCsGAQUFBwIBFiNodHRwOi8vY2VydC5z dGFydGNvbS5vcmcvcG9saWN5LnBkZjA1BggrBgEFBQcCARYpaHR0cDovL2NlcnQuc3RhcnRjb20u b3JnL2ludGVybWVkaWF0ZS5wZGYwgdAGCCsGAQUFBwICMIHDMCcWIFN0YXJ0IENvbW1lcmNpYWwg KFN0YXJ0Q29tKSBMdGQuMAMCAQEagZdMaW1pdGVkIExpYWJpbGl0eSwgcmVhZCB0aGUgc2VjdGlv biAqTGVnYWwgTGltaXRhdGlvbnMqIG9mIHRoZSBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhv cml0eSBQb2xpY3kgYXZhaWxhYmxlIGF0IGh0dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9wb2xpY3ku cGRmMBEGCWCGSAGG+EIBAQQEAwIABzA4BglghkgBhvhCAQ0EKxYpU3RhcnRDb20gRnJlZSBTU0wg Q2VydGlmaWNhdGlvbiBBdXRob3JpdHkwDQYJKoZIhvcNAQEFBQADggIBABZsmfRmDDT10IVefQrs 2hBOOBxe36YlBUuRMsHoO/E93UQJWwdJiinLZgK3sZr3JZgJPI4b4d02hytLu2jTOWY9oCbH8jmR HVGrgnt+1c5a5OIDV3Bplwj5XlimCt+MBppFFhY4Cl5X9mLHegIF5rwetfKe9Kkpg/iyFONuKIdE w5Aa3jipPKxDTWRFzt0oqVzyc3sE+Bfoq7HzLlxkbnMxOhK4vLMR5H2PgVGaO42J9E2TZns8A+3T mh2a82VQ9aDQdZ8vr/DqgkOY+GmciXnEQ45GcuNkNhKv9yUeOImQd37Da2q5w8tES6x4kIvnxywe SxFEyDRSJ80KXZ+FwYnVGnjylRBTMt2AhGZ12bVoKPthLr6EqDjAmRKGpR5nZK0GLi+pcIXHlg98 iWX1jkNUDqvdpYA5lGDANMmWcCyjEvUfSHu9HH5rt52Q9CI7rvj8Ksr6glKg769LVZPrwbXwIous NE4mIgShhyx1SrflfRPXuAxkwDbSyS+GEowjCcEbgjtzSaNqV4eU5dZ4xZlDY+NN4Hct4WWZcmkE GkcJ5g8BViT7H78OealYLrnECQF+lbptAAY+supKEDnY0Cv1v+x1v5cCxQkbCNxVN+KB+zeEQ2Ig yudWS2Xq/mzBJJMkoTTrBf+aIq6bfT/xZVEKpjBqs/SIHIAN/HKK6INeMYIDbzCCA2sCAQEwgZQw gYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUg RGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFBy aW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQIDBqKOMAkGBSsOAwIaBQCgggGvMBgGCSqGSIb3 DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEzMDcyNzE2MTc1OVowIwYJKoZIhvcN AQkEMRYEFCw+nzQPnYxdO3QUxJ7YNSIQ0JwpMIGlBgkrBgEEAYI3EAQxgZcwgZQwgYwxCzAJBgNV BAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBD ZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50 ZXJtZWRpYXRlIENsaWVudCBDQQIDBqKOMIGnBgsqhkiG9w0BCRACCzGBl6CBlDCBjDELMAkGA1UE BhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENl cnRpZmljYXRlIFNpZ25pbmcxODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRl cm1lZGlhdGUgQ2xpZW50IENBAgMGoo4wDQYJKoZIhvcNAQEBBQAEggEARMe1t6sgf09H//eqUdeg XSKpUy9CPsvmP2Rilguke6CBFmfHCgSvI9qWGQLu0OZ/HIDdXlov2m2lJFRBru0LMYCF3o2Vwvtl ijGg4xDUnIQwpQyRUW3pPuYCw+8OQj39dw0P5Zwn7MwJFwTlFLB0tbtZD98qWy7TwNHYjRCP8TLH 9PuvzGHMATwLn6eLiQ9XfNSynJcIv0a9RYY+eK+DmERk+ow1R2tn34tQbPsFSwAwG8hosDDdGnCg P0oWTpPMP46nL21EQz3Vp2uRV5wyXPx52EvgKr6RtGIPN7mQO+rcrGa6VNJq5ZqtayW0qd4kxTws eSKgYRUJ/Upv+c0/hwAAAAAAAA== --Apple-Mail-D4F05C8E-9464-47CC-BDB1-6CCCA3F5B909--