From owner-freebsd-stable@FreeBSD.ORG Fri Jan 22 14:32:06 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A59F1065670; Fri, 22 Jan 2010 14:32:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id DF6178FC12; Fri, 22 Jan 2010 14:32:05 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 8FBF846B53; Fri, 22 Jan 2010 09:32:05 -0500 (EST) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPA id A558C8A026; Fri, 22 Jan 2010 09:32:04 -0500 (EST) From: John Baldwin To: Florian Smeets Date: Fri, 22 Jan 2010 09:20:13 -0500 User-Agent: KMail/1.12.1 (FreeBSD/7.2-CBSD-20091231; KDE/4.3.1; amd64; ; ) References: <4B58280C.50602@smeets.im> <201001211515.32562.jhb@freebsd.org> <4B595D0D.3060904@smeets.im> In-Reply-To: <4B595D0D.3060904@smeets.im> MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <201001220920.13458.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Fri, 22 Jan 2010 09:32:04 -0500 (EST) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=4.2 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: Luigi Rizzo , Bjoern Zeeb , freebsd-stable@freebsd.org, mlaier@freebsd.org Subject: Re: 7.2-STABLE page fault with kernel from 12.01.2010 / crashinfo available X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2010 14:32:06 -0000 On Friday 22 January 2010 3:08:45 am Florian Smeets wrote: > On 1/21/10 9:15 PM, John Baldwin wrote: > > On Thursday 21 January 2010 2:09:34 pm Florian Smeets wrote: > >> On 1/21/10 8:05 PM, John Baldwin wrote: > >>> On Thursday 21 January 2010 1:33:35 pm Florian Smeets wrote: > >>>> On 1/21/10 6:58 PM, John Baldwin wrote: > >>>>> On Thursday 21 January 2010 8:25:22 am Florian Smeets wrote: > >>>>>> (kgdb) frame 8 > >>>>>> #8 0xc05f8b28 in ip_forward (m=3D0xc23dc900, srcrt=3D0) at > >>>>>> /usr/src/sys/netinet/ip_input.c:1307 > >>>>>> 1307 m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t)); > >>>>>> (kgdb) l > >>>>>> 1302 mcopy =3D NULL; > >>>>>> 1303 } > >>>>>> 1304 if (mcopy !=3D NULL) { > >>>>>> 1305 mcopy->m_len =3D min(ip->ip_len, M_TRAILINGSPACE(mcopy)); > >>>>>> 1306 mcopy->m_pkthdr.len =3D mcopy->m_len; > >>>>>> 1307 m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t)); > >>>>>> 1308 } > >>>>>> 1309=09 > >>>>>> 1310 #ifdef IPSTEALTH > >>>>>> 1311 if (!ipstealth) { > >>>>>> (kgdb) p *m > >>>>>> $1 =3D {m_hdr =3D {mh_next =3D 0x0, mh_nextpkt =3D 0x0, mh_data = =3D 0xc271e80e > >>>>>> "E\020", mh_len =3D 164, mh_flags =3D 3, mh_type =3D 1, pad =3D "\= 000"}, M_dat > > =3D > >>>>>> {MH =3D {MH_pkthdr =3D {rcvif =3D 0xc20a4800, header =3D 0x0, len = =3D 164, > >>>>>> csum_flags =3D 3072, > >>>>>> csum_data =3D 65535, tso_segsz =3D 0, ether_vtag =3D 0= , tags =3D > >>>>>> {slh_first =3D 0xc35bc380}}, MH_dat =3D {MH_ext =3D {ext_buf =3D 0= xc271e800 "", > >>>>>> ext_free =3D 0, ext_args =3D 0x0, ext_size =3D 2048, ref_cnt =3D 0= xc2703ab4, > >>>>>> ext_type =3D 6}, > >>>>>> MH_databuf =3D > >>>>>> "\000?q?\000\000\000\000\000\000\000\000\000\b\000\000?:p? > >>>>> \006\000\000\000dL?\t<+?\202\200\020 > >>>>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l? > > \000\000\001%r??? > >>>>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b= =DF=BC&? > >>>>> > >>> > > \237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Lja= va/l\000\220\032Ae\207\000\002? > >>>>> 36@\210d\021\000\001? > >>> \001B\000!E\000\001@bV\000\000@2\032$W\213\n\034"...}}, > >>>>>> > >>>>>> M_databuf =3D > >>>>>> "\000H\n?\000\000\000\000?\000\000\000\000\f\000\000?? > >>>>> \000\000\000\000\000\000\200?[?\000?q? > >>>>> \000\000\000\000\000\000\000\000\000\b\000\000?:p?\006\000\000\000d= L? > > \t<+? > >>>>> \202\200\020 > >>>>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l? > > \000\000\001%r??? > >>>>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b= =DF=BC&? > >>>>> > >>> > > \237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Lja= va/l\000\220\032Ae\207\000\002? > >>>>> 3"...}} > >>>>> > >>>>> Ok, can you do 'p *m_copy'? > >>>>> > >>>> > >>>> What ever you want :-) > >>>> > >>>> (kgdb) p *m_copy > >>>> No symbol "m_copy" in current context. > >>>> (kgdb) p *m_copydata > >>>> $2 =3D {void (const struct mbuf *, int, int, caddr_t)} > > 0xc0572e10 > >>>> (kgdb) p *mcopy > >>>> $1 =3D {m_hdr =3D {mh_next =3D 0x0, mh_nextpkt =3D 0x0, mh_data =3D = 0xc23cce34 > >>>> "E\020", mh_len =3D 204, mh_flags =3D 2, mh_type =3D 1, pad =3D "\00= 0"}, M_dat =3D > >>>> {MH =3D {MH_pkthdr =3D {rcvif =3D 0xc20a4800, header =3D 0x0, > >>>> len =3D 204, csum_flags =3D 3072, csum_data =3D 65535, ts= o_segsz =3D 0, > >>>> ether_vtag =3D 0, tags =3D {slh_first =3D 0xc23c3e00}}, MH_dat =3D {= MH_ext =3D > >>>> {ext_buf =3D 0x84001045
, > >>> > >>> Hmm, ok. Can you do 'p *ip'? mcopy->m_len (204) is larger than m->m= _len > >>> (164). That shouldn't be the case unless ip->ip_len is somehow larger > > than m- > >>>> m_len. > >>> > >> > >> (kgdb) p *ip > >> $3 =3D {ip_hl =3D 5, ip_v =3D 4, ip_tos =3D 16 '\020', ip_len =3D 3379= 2, ip_id =3D > >> 61492, ip_off =3D 64, ip_ttl =3D 64 '@', ip_p =3D 6 '\006', ip_sum =3D= 34849, > >> ip_src =3D {s_addr =3D 355576000}, ip_dst =3D { > >> s_addr =3D 2251401408}} > > > > Looks like ip_len is in network byte order instead of host byte order a= nd that > > is causing the problem. 33792 =3D=3D 0x8400. Swapping that gives 0x84= =3D=3D 132 > > which would be a reasonable length. Are you using any firewall rules t= hat > > would rewrite packets? I wonder if you are having a packet rewritten a= nd the > > new IP header is written in network byte order, but we swap the IP head= er len > > field to host byte order earlier in ip_input(). Luigi Rizzo may have s= ome > > insight into this. > > >=20 > Well, when looking at MH_databuf i see Jboss MQ traffic that would mean=20 > that this traffic was coming from or going to an IPsec tunnel, i could=20 > say for sure when i would have a clue how to get an IP address from=20 > something like ip_src =3D {s_addr =3D 355576000}. Something like this should show you the IP: (gdb) set $i =3D 355576000 (gdb) printf "%d.%d.%d.%d\n", $i >> 24, $i >> 16 & 0xff, $i >> 8 & 0xff, $i= & 0xff 21.49.168.192 In this case I probably printed it backwards, so it is probably 192.168.49.21. > If it really is IPsec traffic then there are no rewrite rules only 10 pf= =20 > pass rules on the enc0 interface and a "scrub in all" rule. >=20 > Perhaps it matters that i have these set: >=20 > net.enc.out.ipsec_bpf_mask=3D0x00000001 > net.enc.out.ipsec_filter_mask=3D0x00000001 > net.enc.in.ipsec_bpf_mask=3D0x00000002 > net.enc.in.ipsec_filter_mask=3D0x00000002 >=20 > so that i can filter the "encapsulated" traffic. I have no idea, I've cc'd mlaier@ (pf) and bz@ (ipsec) to see if they have any ideas. =2D-=20 John Baldwin