From owner-freebsd-stable Tue Mar 4 6:52:45 2003 Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 31A1C37B405 for ; Tue, 4 Mar 2003 06:52:41 -0800 (PST) Received: from gandalf.online.bg (gandalf.online.bg [217.75.128.9]) by mx1.FreeBSD.org (Postfix) with SMTP id 2BFE343FCB for ; Tue, 4 Mar 2003 06:52:39 -0800 (PST) (envelope-from roam@ringlet.net) Received: (qmail 14296 invoked from network); 4 Mar 2003 14:48:27 -0000 Received: from office.sbnd.net (HELO straylight.ringlet.net) (217.75.140.130) by gandalf.online.bg with SMTP; 4 Mar 2003 14:48:26 -0000 Received: (qmail 67734 invoked by uid 1000); 4 Mar 2003 14:51:12 -0000 Date: Tue, 4 Mar 2003 16:51:12 +0200 From: Peter Pentchev To: Jonathan Lemon Cc: stable@freebsd.org Subject: Re: 4.8-PRE (and earlier) crash - syncookies? Message-ID: <20030304145112.GE554@straylight.oblivion.bg> References: <200303041440.h24Ee9ha074389@mail.flugsvamp.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UKNXkkdQCYZ6W5l3" Content-Disposition: inline In-Reply-To: <200303041440.h24Ee9ha074389@mail.flugsvamp.com> User-Agent: Mutt/1.5.3i Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --UKNXkkdQCYZ6W5l3 Content-Type: multipart/mixed; boundary="UBnjLfzoMQYIXCvq" Content-Disposition: inline --UBnjLfzoMQYIXCvq Content-Type: text/plain; charset=windows-1251 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 04, 2003 at 08:40:09AM -0600, Jonathan Lemon wrote: > In article you write: > >-=3D-=3D-=3D-=3D-=3D- > >-=3D-=3D-=3D-=3D-=3D- > > > >Hi, > > > >I think this has been going on for a couple of months; unfortunately, I > >cannot remember exactly when it started :( Several machines, which have > >always been updated synchronously at intervals of a couple of weeks or > >so, have been experiencing panics every few days with recent 4.6-STABLE, > >4.7-STABLE, and 4.8-PRERELEASE kernel+world. > > > >Attached is the kernel config and a gdb session of the most recent > >crashdump. Maybe the most interesting part is at the end of the gdb > >session: is there a normal reason why a TAILQ structure would have a > >null start pointer, and an end pointer pointing back to itself? >=20 > [ ... ] >=20 > >#16 0xc01b2768 in syncache_insert (sc=3D0xcb870b80, sch=3D0xc0eeeb7c) > > at /usr/src/sys/netinet/tcp_syncache.c:289 > >289 sc2 =3D TAILQ_FIRST(&sch->sch_bucket); > >(kgdb) print sch > >$1 =3D (struct syncache_head *) 0xc0eeeb7c > >(kgdb) print *sch > >$2 =3D {sch_bucket =3D {tqh_first =3D 0x0, tqh_last =3D 0xc0eeeb7c},=20 > > sch_length =3D 4294967295} >=20 >=20 > It looks to me like the tailq is empty (first =3D=3D null, last =3D=3D &h= ead), > but the code path to remove an entry was taken because the length > counter indicated it was over the bucket limit. sch_length =3D=3D -1. >=20 > So my guess is the code dropped a syncache entry twice, and this is=20 > the resulting mess. Can you print out the contents of tcp_syncache > and tcpstat? See attached; I could provide the whole corefile on request, if it will be helpful. G'luck, Peter --=20 Peter Pentchev roam@ringlet.net roam@sbnd.net roam@FreeBSD.org PGP key: http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 This sentence would be seven words long if it were six words shorter. --UBnjLfzoMQYIXCvq Content-Type: text/plain; charset=windows-1251 Content-Disposition: attachment; filename="pan-pl.tcpstat.txt" Content-Transfer-Encoding: quoted-printable (kgdb) fr 16 #16 0xc01b2768 in syncache_insert (sc=3D0xcb870b80, sch=3D0xc0eeeb7c) at /usr/src/sys/netinet/tcp_syncache.c:289 289 sc2 =3D TAILQ_FIRST(&sch->sch_bucket); (kgdb) print tcp_syncache $1 =3D {hashbase =3D 0xc0eee000, zone =3D 0xc0d94c80, hashsize =3D 512,=20 hashmask =3D 511, bucket_limit =3D 30, cache_count =3D 4294967295,=20 cache_limit =3D 15359, rexmt_limit =3D 3, hash_secret =3D 1416729105,=20 next_reseed =3D 0, timerq =3D {{tqh_first =3D 0x0, tqh_last =3D 0xc02a874= 8}, { tqh_first =3D 0x0, tqh_last =3D 0xc02a8750}, {tqh_first =3D 0x0,=20 tqh_last =3D 0xc02a8758}, {tqh_first =3D 0x0, tqh_last =3D 0xc02a8760= }},=20 tt_timerq =3D {{c_links =3D {sle =3D {sle_next =3D 0xc603a2e8}, tqe =3D { tqe_next =3D 0xc603a2e8, tqe_prev =3D 0xc603c620}}, c_time =3D 55= 852000,=20 c_arg =3D 0x0, c_func =3D 0, c_flags =3D 0}, {c_links =3D {sle =3D { sle_next =3D 0x0}, tqe =3D {tqe_next =3D 0x0, tqe_prev =3D 0xc603= e200}},=20 c_time =3D 55658332, c_arg =3D 0x1, c_func =3D 0, c_flags =3D 0}, {c_= links =3D { sle =3D {sle_next =3D 0xc603a2e8}, tqe =3D {tqe_next =3D 0xc603a2e8= ,=20 tqe_prev =3D 0xc603c130}}, c_time =3D 52482882, c_arg =3D 0x2, c_= func =3D 0,=20 c_flags =3D 0}, {c_links =3D {sle =3D {sle_next =3D 0xc603a2e8}, tqe = =3D { tqe_next =3D 0xc603a2e8, tqe_prev =3D 0xc603cc30}}, c_time =3D 52= 485282,=20 c_arg =3D 0x3, c_func =3D 0, c_flags =3D 0}}} (kgdb) print tcpstat $2 =3D {tcps_connattempt =3D 598918, tcps_accepts =3D 32652, tcps_connects = =3D 612607,=20 tcps_drops =3D 11716, tcps_conndrops =3D 13231, tcps_closed =3D 636232,= =20 tcps_segstimed =3D 1956837, tcps_rttupdated =3D 2499750, tcps_delack =3D = 615940,=20 tcps_timeoutdrop =3D 23, tcps_rexmttimeo =3D 98711, tcps_persisttimeo =3D= 55,=20 tcps_keeptimeo =3D 6166, tcps_keepprobe =3D 204, tcps_keepdrops =3D 5962,= =20 tcps_sndtotal =3D 5368356, tcps_sndpack =3D 1503648, tcps_sndbyte =3D 388= 668829,=20 tcps_sndrexmitpack =3D 13994, tcps_sndrexmitbyte =3D 10990401,=20 tcps_sndacks =3D 2587703, tcps_sndprobe =3D 48, tcps_sndurg =3D 0,=20 tcps_sndwinup =3D 53833, tcps_sndctrl =3D 1209130, tcps_rcvtotal =3D 4122= 112,=20 tcps_rcvpack =3D 2027883, tcps_rcvbyte =3D 301857290, tcps_rcvbadsum =3D = 45,=20 tcps_rcvbadoff =3D 0, tcps_rcvmemdrop =3D 0, tcps_rcvshort =3D 0,=20 tcps_rcvduppack =3D 571696, tcps_rcvdupbyte =3D 1922742,=20 tcps_rcvpartduppack =3D 27, tcps_rcvpartdupbyte =3D 14961,=20 tcps_rcvoopack =3D 23227, tcps_rcvoobyte =3D 24974272,=20 tcps_rcvpackafterwin =3D 637, tcps_rcvbyteafterwin =3D 677422,=20 tcps_rcvafterclose =3D 479, tcps_rcvwinprobe =3D 4, tcps_rcvdupack =3D 61= 2690,=20 tcps_rcvacktoomuch =3D 0, tcps_rcvackpack =3D 2551692,=20 tcps_rcvackbyte =3D 384639652, tcps_rcvwinupd =3D 12215, tcps_pawsdrop = =3D 83,=20 tcps_predack =3D 58265, tcps_preddat =3D 255753, tcps_pcbcachemiss =3D 0,= =20 tcps_cachedrtt =3D 852, tcps_cachedrttvar =3D 852, tcps_cachedssthresh = =3D 282,=20 tcps_usedrtt =3D 11154, tcps_usedrttvar =3D 11154, tcps_usedssthresh =3D = 2053,=20 tcps_persistdrop =3D 0, tcps_badsyn =3D 24, tcps_mturesent =3D 0,=20 tcps_listendrop =3D 0, tcps_sc_added =3D 32716, tcps_sc_retransmitted =3D= 197,=20 tcps_sc_dupsyn =3D 165, tcps_sc_dropped =3D 0, tcps_sc_completed =3D 3265= 2,=20 tcps_sc_bucketoverflow =3D 0, tcps_sc_cacheoverflow =3D 0, tcps_sc_reset = =3D 52,=20 tcps_sc_stale =3D 13, tcps_sc_aborted =3D 0, tcps_sc_badack =3D 0,=20 tcps_sc_unreach =3D 0, tcps_sc_zonefail =3D 0, tcps_sc_sendcookie =3D 0,= =20 tcps_sc_recvcookie =3D 0} (kgdb) q --UBnjLfzoMQYIXCvq-- --UKNXkkdQCYZ6W5l3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQE+ZL1g7Ri2jRYZRVMRAjIvAJ4ntEc3PLrf/5GryfePorrZvldeNACfSasa xFtp5E2D+otJOeDRAD4Goag= =hWsO -----END PGP SIGNATURE----- --UKNXkkdQCYZ6W5l3-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message