Date: Fri, 22 Jan 2010 09:20:13 -0500 From: John Baldwin <jhb@freebsd.org> To: Florian Smeets <flo@smeets.im> Cc: Luigi Rizzo <luigi@freebsd.org>, Bjoern Zeeb <bz@freebsd.org>, freebsd-stable@freebsd.org, mlaier@freebsd.org Subject: Re: 7.2-STABLE page fault with kernel from 12.01.2010 / crashinfo available Message-ID: <201001220920.13458.jhb@freebsd.org> In-Reply-To: <4B595D0D.3060904@smeets.im> References: <4B58280C.50602@smeets.im> <201001211515.32562.jhb@freebsd.org> <4B595D0D.3060904@smeets.im>
next in thread | previous in thread | raw e-mail | index | archive | help
On Friday 22 January 2010 3:08:45 am Florian Smeets wrote:
> On 1/21/10 9:15 PM, John Baldwin wrote:
> > On Thursday 21 January 2010 2:09:34 pm Florian Smeets wrote:
> >> On 1/21/10 8:05 PM, John Baldwin wrote:
> >>> On Thursday 21 January 2010 1:33:35 pm Florian Smeets wrote:
> >>>> On 1/21/10 6:58 PM, John Baldwin wrote:
> >>>>> On Thursday 21 January 2010 8:25:22 am Florian Smeets wrote:
> >>>>>> (kgdb) frame 8
> >>>>>> #8 0xc05f8b28 in ip_forward (m=3D0xc23dc900, srcrt=3D0) at
> >>>>>> /usr/src/sys/netinet/ip_input.c:1307
> >>>>>> 1307 m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t));
> >>>>>> (kgdb) l
> >>>>>> 1302 mcopy =3D NULL;
> >>>>>> 1303 }
> >>>>>> 1304 if (mcopy !=3D NULL) {
> >>>>>> 1305 mcopy->m_len =3D min(ip->ip_len, M_TRAILINGSPACE(mcopy));
> >>>>>> 1306 mcopy->m_pkthdr.len =3D mcopy->m_len;
> >>>>>> 1307 m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t));
> >>>>>> 1308 }
> >>>>>> 1309=09
> >>>>>> 1310 #ifdef IPSTEALTH
> >>>>>> 1311 if (!ipstealth) {
> >>>>>> (kgdb) p *m
> >>>>>> $1 =3D {m_hdr =3D {mh_next =3D 0x0, mh_nextpkt =3D 0x0, mh_data =
=3D 0xc271e80e
> >>>>>> "E\020", mh_len =3D 164, mh_flags =3D 3, mh_type =3D 1, pad =3D "\=
000"}, M_dat
> > =3D
> >>>>>> {MH =3D {MH_pkthdr =3D {rcvif =3D 0xc20a4800, header =3D 0x0, len =
=3D 164,
> >>>>>> csum_flags =3D 3072,
> >>>>>> csum_data =3D 65535, tso_segsz =3D 0, ether_vtag =3D 0=
, tags =3D
> >>>>>> {slh_first =3D 0xc35bc380}}, MH_dat =3D {MH_ext =3D {ext_buf =3D 0=
xc271e800 "",
> >>>>>> ext_free =3D 0, ext_args =3D 0x0, ext_size =3D 2048, ref_cnt =3D 0=
xc2703ab4,
> >>>>>> ext_type =3D 6},
> >>>>>> MH_databuf =3D
> >>>>>> "\000?q?\000\000\000\000\000\000\000\000\000\b\000\000?:p?
> >>>>> \006\000\000\000dL?\t<+?\202\200\020
> >>>>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l?
> > \000\000\001%r???
> >>>>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b=
=DF=BC&?
> >>>>>
> >>>
> > \237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Lja=
va/l\000\220\032Ae\207\000\002?
> >>>>> 36@\210d\021\000\001?
> >>> \001B\000!E\000\001@bV\000\000@2\032$W\213\n\034"...}},
> >>>>>>
> >>>>>> M_databuf =3D
> >>>>>> "\000H\n?\000\000\000\000?\000\000\000\000\f\000\000??
> >>>>> \000\000\000\000\000\000\200?[?\000?q?
> >>>>> \000\000\000\000\000\000\000\000\000\b\000\000?:p?\006\000\000\000d=
L?
> > \t<+?
> >>>>> \202\200\020
> >>>>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l?
> > \000\000\001%r???
> >>>>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b=
=DF=BC&?
> >>>>>
> >>>
> > \237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Lja=
va/l\000\220\032Ae\207\000\002?
> >>>>> 3"...}}
> >>>>>
> >>>>> Ok, can you do 'p *m_copy'?
> >>>>>
> >>>>
> >>>> What ever you want :-)
> >>>>
> >>>> (kgdb) p *m_copy
> >>>> No symbol "m_copy" in current context.
> >>>> (kgdb) p *m_copydata
> >>>> $2 =3D {void (const struct mbuf *, int, int, caddr_t)}
> > 0xc0572e10<m_copydata>
> >>>> (kgdb) p *mcopy
> >>>> $1 =3D {m_hdr =3D {mh_next =3D 0x0, mh_nextpkt =3D 0x0, mh_data =3D =
0xc23cce34
> >>>> "E\020", mh_len =3D 204, mh_flags =3D 2, mh_type =3D 1, pad =3D "\00=
0"}, M_dat =3D
> >>>> {MH =3D {MH_pkthdr =3D {rcvif =3D 0xc20a4800, header =3D 0x0,
> >>>> len =3D 204, csum_flags =3D 3072, csum_data =3D 65535, ts=
o_segsz =3D 0,
> >>>> ether_vtag =3D 0, tags =3D {slh_first =3D 0xc23c3e00}}, MH_dat =3D {=
MH_ext =3D
> >>>> {ext_buf =3D 0x84001045<Address 0x84001045 out of bounds>,
> >>>
> >>> Hmm, ok. Can you do 'p *ip'? mcopy->m_len (204) is larger than m->m=
_len
> >>> (164). That shouldn't be the case unless ip->ip_len is somehow larger
> > than m-
> >>>> m_len.
> >>>
> >>
> >> (kgdb) p *ip
> >> $3 =3D {ip_hl =3D 5, ip_v =3D 4, ip_tos =3D 16 '\020', ip_len =3D 3379=
2, ip_id =3D
> >> 61492, ip_off =3D 64, ip_ttl =3D 64 '@', ip_p =3D 6 '\006', ip_sum =3D=
34849,
> >> ip_src =3D {s_addr =3D 355576000}, ip_dst =3D {
> >> s_addr =3D 2251401408}}
> >
> > Looks like ip_len is in network byte order instead of host byte order a=
nd that
> > is causing the problem. 33792 =3D=3D 0x8400. Swapping that gives 0x84=
=3D=3D 132
> > which would be a reasonable length. Are you using any firewall rules t=
hat
> > would rewrite packets? I wonder if you are having a packet rewritten a=
nd the
> > new IP header is written in network byte order, but we swap the IP head=
er len
> > field to host byte order earlier in ip_input(). Luigi Rizzo may have s=
ome
> > insight into this.
> >
>=20
> Well, when looking at MH_databuf i see Jboss MQ traffic that would mean=20
> that this traffic was coming from or going to an IPsec tunnel, i could=20
> say for sure when i would have a clue how to get an IP address from=20
> something like ip_src =3D {s_addr =3D 355576000}.
Something like this should show you the IP:
(gdb) set $i =3D 355576000
(gdb) printf "%d.%d.%d.%d\n", $i >> 24, $i >> 16 & 0xff, $i >> 8 & 0xff, $i=
& 0xff
21.49.168.192
In this case I probably printed it backwards, so it is probably
192.168.49.21.
> If it really is IPsec traffic then there are no rewrite rules only 10 pf=
=20
> pass rules on the enc0 interface and a "scrub in all" rule.
>=20
> Perhaps it matters that i have these set:
>=20
> net.enc.out.ipsec_bpf_mask=3D0x00000001
> net.enc.out.ipsec_filter_mask=3D0x00000001
> net.enc.in.ipsec_bpf_mask=3D0x00000002
> net.enc.in.ipsec_filter_mask=3D0x00000002
>=20
> so that i can filter the "encapsulated" traffic.
I have no idea, I've cc'd mlaier@ (pf) and bz@ (ipsec) to see if they have
any ideas.
=2D-=20
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201001220920.13458.jhb>