From owner-freebsd-stable@FreeBSD.ORG Fri Jan 22 08:08:51 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1530E106566B; Fri, 22 Jan 2010 08:08:51 +0000 (UTC) (envelope-from flo@smeets.im) Received: from mail.solomo.de (mail.solomo.de [85.214.124.163]) by mx1.freebsd.org (Postfix) with ESMTP id A43298FC0C; Fri, 22 Jan 2010 08:08:50 +0000 (UTC) Received: from mail.solomo.de (localhost [127.0.0.1]) by mail.solomo.de (Postfix) with ESMTP id 654C861ED3; Fri, 22 Jan 2010 09:08:49 +0100 (CET) X-Virus-Scanned: amavisd-new at vistream.de Received: from mail.solomo.de ([127.0.0.1]) by mail.solomo.de (mail.solomo.de [127.0.0.1]) (amavisd-new, port 10024) with LMTP id e6WkBHxMqCLA; Fri, 22 Jan 2010 09:08:47 +0100 (CET) Received: from nibbler.vistream.local (relay3.vistream.de [87.139.10.28]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.solomo.de (Postfix) with ESMTPSA id 07AF261C94; Fri, 22 Jan 2010 09:08:47 +0100 (CET) Message-ID: <4B595D0D.3060904@smeets.im> Date: Fri, 22 Jan 2010 09:08:45 +0100 From: Florian Smeets User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.3a1pre) Gecko/20100121 Shredder/3.2a1pre MIME-Version: 1.0 To: John Baldwin References: <4B58280C.50602@smeets.im> <201001211405.35615.jhb@freebsd.org> <4B58A66E.3040900@smeets.im> <201001211515.32562.jhb@freebsd.org> In-Reply-To: <201001211515.32562.jhb@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: Luigi Rizzo , freebsd-stable@freebsd.org Subject: Re: 7.2-STABLE page fault with kernel from 12.01.2010 / crashinfo available X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2010 08:08:51 -0000 On 1/21/10 9:15 PM, John Baldwin wrote: > On Thursday 21 January 2010 2:09:34 pm Florian Smeets wrote: >> On 1/21/10 8:05 PM, John Baldwin wrote: >>> On Thursday 21 January 2010 1:33:35 pm Florian Smeets wrote: >>>> On 1/21/10 6:58 PM, John Baldwin wrote: >>>>> On Thursday 21 January 2010 8:25:22 am Florian Smeets wrote: >>>>>> (kgdb) frame 8 >>>>>> #8 0xc05f8b28 in ip_forward (m=0xc23dc900, srcrt=0) at >>>>>> /usr/src/sys/netinet/ip_input.c:1307 >>>>>> 1307 m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t)); >>>>>> (kgdb) l >>>>>> 1302 mcopy = NULL; >>>>>> 1303 } >>>>>> 1304 if (mcopy != NULL) { >>>>>> 1305 mcopy->m_len = min(ip->ip_len, M_TRAILINGSPACE(mcopy)); >>>>>> 1306 mcopy->m_pkthdr.len = mcopy->m_len; >>>>>> 1307 m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t)); >>>>>> 1308 } >>>>>> 1309 >>>>>> 1310 #ifdef IPSTEALTH >>>>>> 1311 if (!ipstealth) { >>>>>> (kgdb) p *m >>>>>> $1 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc271e80e >>>>>> "E\020", mh_len = 164, mh_flags = 3, mh_type = 1, pad = "\000"}, M_dat > = >>>>>> {MH = {MH_pkthdr = {rcvif = 0xc20a4800, header = 0x0, len = 164, >>>>>> csum_flags = 3072, >>>>>> csum_data = 65535, tso_segsz = 0, ether_vtag = 0, tags = >>>>>> {slh_first = 0xc35bc380}}, MH_dat = {MH_ext = {ext_buf = 0xc271e800 "", >>>>>> ext_free = 0, ext_args = 0x0, ext_size = 2048, ref_cnt = 0xc2703ab4, >>>>>> ext_type = 6}, >>>>>> MH_databuf = >>>>>> "\000?q?\000\000\000\000\000\000\000\000\000\b\000\000?:p? >>>>> \006\000\000\000dL?\t<+?\202\200\020 >>>>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l? > \000\000\001%r??? >>>>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b߼&? >>>>> >>> > \237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Ljava/l\000\220\032Ae\207\000\002? >>>>> 36@\210d\021\000\001? >>> \001B\000!E\000\001@bV\000\000@2\032$W\213\n\034"...}}, >>>>>> >>>>>> M_databuf = >>>>>> "\000H\n?\000\000\000\000?\000\000\000\000\f\000\000?? >>>>> \000\000\000\000\000\000\200?[?\000?q? >>>>> \000\000\000\000\000\000\000\000\000\b\000\000?:p?\006\000\000\000dL? > \t<+? >>>>> \202\200\020 >>>>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l? > \000\000\001%r??? >>>>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b߼&? >>>>> >>> > \237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Ljava/l\000\220\032Ae\207\000\002? >>>>> 3"...}} >>>>> >>>>> Ok, can you do 'p *m_copy'? >>>>> >>>> >>>> What ever you want :-) >>>> >>>> (kgdb) p *m_copy >>>> No symbol "m_copy" in current context. >>>> (kgdb) p *m_copydata >>>> $2 = {void (const struct mbuf *, int, int, caddr_t)} > 0xc0572e10 >>>> (kgdb) p *mcopy >>>> $1 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc23cce34 >>>> "E\020", mh_len = 204, mh_flags = 2, mh_type = 1, pad = "\000"}, M_dat = >>>> {MH = {MH_pkthdr = {rcvif = 0xc20a4800, header = 0x0, >>>> len = 204, csum_flags = 3072, csum_data = 65535, tso_segsz = 0, >>>> ether_vtag = 0, tags = {slh_first = 0xc23c3e00}}, MH_dat = {MH_ext = >>>> {ext_buf = 0x84001045
, >>> >>> Hmm, ok. Can you do 'p *ip'? mcopy->m_len (204) is larger than m->m_len >>> (164). That shouldn't be the case unless ip->ip_len is somehow larger > than m- >>>> m_len. >>> >> >> (kgdb) p *ip >> $3 = {ip_hl = 5, ip_v = 4, ip_tos = 16 '\020', ip_len = 33792, ip_id = >> 61492, ip_off = 64, ip_ttl = 64 '@', ip_p = 6 '\006', ip_sum = 34849, >> ip_src = {s_addr = 355576000}, ip_dst = { >> s_addr = 2251401408}} > > Looks like ip_len is in network byte order instead of host byte order and that > is causing the problem. 33792 == 0x8400. Swapping that gives 0x84 == 132 > which would be a reasonable length. Are you using any firewall rules that > would rewrite packets? I wonder if you are having a packet rewritten and the > new IP header is written in network byte order, but we swap the IP header len > field to host byte order earlier in ip_input(). Luigi Rizzo may have some > insight into this. > Well, when looking at MH_databuf i see Jboss MQ traffic that would mean that this traffic was coming from or going to an IPsec tunnel, i could say for sure when i would have a clue how to get an IP address from something like ip_src = {s_addr = 355576000}. If it really is IPsec traffic then there are no rewrite rules only 10 pf pass rules on the enc0 interface and a "scrub in all" rule. Perhaps it matters that i have these set: net.enc.out.ipsec_bpf_mask=0x00000001 net.enc.out.ipsec_filter_mask=0x00000001 net.enc.in.ipsec_bpf_mask=0x00000002 net.enc.in.ipsec_filter_mask=0x00000002 so that i can filter the "encapsulated" traffic. Thanks, Florian