From owner-freebsd-stable@FreeBSD.ORG Sat Apr 16 03:18:09 2005 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D27A816A4CE for ; Sat, 16 Apr 2005 03:18:09 +0000 (GMT) Received: from carver.gumbysoft.com (carver.gumbysoft.com [66.220.23.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8EA0E43D58 for ; Sat, 16 Apr 2005 03:18:09 +0000 (GMT) (envelope-from dwhite@gumbysoft.com) Received: by carver.gumbysoft.com (Postfix, from userid 1000) id 836A972DDD; Fri, 15 Apr 2005 20:18:09 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by carver.gumbysoft.com (Postfix) with ESMTP id 7E21272DDB; Fri, 15 Apr 2005 20:18:09 -0700 (PDT) Date: Fri, 15 Apr 2005 20:18:09 -0700 (PDT) From: Doug White To: Michiel Boland In-Reply-To: Message-ID: <20050415201719.R38788@carver.gumbysoft.com> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-stable@freebsd.org Subject: Re: fxp lockups on 5.4-RC1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2005 03:18:09 -0000 On Wed, 13 Apr 2005, Michiel Boland wrote: > Hi. I was using a small test program to generate artificial network load > on 5.4-RC1. The program sets up 1000 sockets that all connect, send a > number of pipelined request, and then read the response from a nearby web > server. > > After a short while the machine locked up, even the console was dead. > Luckily DDB worked, so I was able to gather some backtraces (see below). > Looks like something is stuck in the fxp process. you appear to be running low on kenrel memory. the only msleep in uma_zone_slab is for zonelimit and that gets triggered when kernel memroy runs low. > > Being a bit impatient, I did not check to see if the lockup would go away > after some time. > > The problem went away after I increased nmbclusters to 32768. Probably the right thing to do. > > If more info is needed I can provide this of course. > > Cheers > Michiel > > > tr > Tracing pid 23 tid 100017 td 0xc157da80 > kdb_enter(c05d55bb) at kdb_enter+0x2b > siointr1(c167f000) at siointr1+0xce > siointr(c167f000) at siointr+0x38 > intr_execute_handlers(c05ffac0,d4000bd8,0,c0c46460,c0c45240) at intr_execute_handlers+0x7d > atpic_handle_intr(4) at atpic_handle_intr+0x96 > Xatpic_intr4() at Xatpic_intr4+0x20 > --- interrupt, eip = 0xc057ff87, esp = 0xd4000c1c, ebp = 0xd4000c28 --- > uma_zfree_internal(c0c45240,c4173418,0,0,80) at uma_zfree_internal+0x153 > bucket_free(c4173418,1,80,f3c8,c0c45bd8) at bucket_free+0x22 > uma_zalloc_bucket(c0c45ba0,1) at uma_zalloc_bucket+0x263 > uma_zalloc_arg(c0c45ba0,d4000c9c,1) at uma_zalloc_arg+0x274 > fxp_add_rfabuf(c1616000,c16165d0) at fxp_add_rfabuf+0x2a > fxp_intr_body(c1616000,c1616000,50,ffffffff) at fxp_intr_body+0xd4 > fxp_intr(c1616000) at fxp_intr+0xf4 > ithread_loop(c1576800,d4000d48) at ithread_loop+0x151 > fork_exit(c04838a0,c1576800,d4000d48) at fork_exit+0x74 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0x1, eip = 0, esp = 0xd4000d7c, ebp = 0 --- > db> ps > pid proc uid ppid pgrp flag stat wmesg wchan cmd > 375 c1731388 1001 370 375 0004002 [SLPQ zonelimit 0xc0c1f460][SLP] kqmaps > 370 c1731c5c 1001 369 370 0004002 [SLPQ wait 0xc1731c5c][SLP] bash > 369 c1731e20 0 1 369 0004102 [SLPQ wait 0xc1731e20][SLP] login > 368 c18b5000 0 1 368 0004002 [SLPQ ttyin 0xc1574e10][SLP] getty > 367 c18b51c4 0 1 367 0004002 [SLPQ ttyin 0xc1683010][SLP] getty > 366 c18b5388 0 1 366 0004002 [SLPQ ttyin 0xc1683210][SLP] getty > 365 c18b554c 0 1 365 0004002 [SLPQ ttyin 0xc1683410][SLP] getty > 364 c18b5710 0 1 364 0004002 [SLPQ ttyin 0xc1573010][SLP] getty > 363 c18b58d4 0 1 363 0004002 [SLPQ ttyin 0xc1573210][SLP] getty > 362 c172d54c 0 1 362 0004002 [SLPQ ttyin 0xc1573410][SLP] getty > 361 c172d710 0 1 361 0004002 [SLPQ ttyin 0xc1573610][SLP] getty > 321 c15c7e20 0 1 321 0000100 [SLPQ select 0xc060f364][SLP] sshd > 199 c172d1c4 0 1 199 0000000 [SLPQ select 0xc060f364][SLP] devd > 38 c172d8d4 0 0 0 0000204 [SLPQ - 0xd5445d18][SLP] schedcpu > 37 c172da98 0 0 0 0000204 [SLPQ syncer 0xc060baec][SLP] syncer > 36 c172dc5c 0 0 0 0000204 [SLPQ vlruwt 0xc172dc5c][SLP] vnlru > 35 c172de20 0 0 0 0000204 [SLPQ psleep 0xc060f84c][SLP] bufdaemon > 9 c1731000 0 0 0 000020c [SLPQ pgzero 0xc0616ed4][SLP] pagezero > 8 c15b654c 0 0 0 0000204 [SLPQ psleep 0xc0616f28][SLP] vmdaemon > 7 c15b6710 0 0 0 0000204 [SLPQ psleep 0xc0616ee4][SLP] pagedaemon > 34 c15b68d4 0 0 0 0000204 [IWAIT] swi0: sio > 33 c15b6a98 0 0 0 0000204 [IWAIT] swi6: task queue > 6 c15b6c5c 0 0 0 0000204 [SLPQ - 0xc15e7e00][SLP] kqueue taskq > 32 c15b6e20 0 0 0 0000204 [IWAIT] swi6:+ > 5 c15c7000 0 0 0 0000204 [SLPQ - 0xc15f5000][SLP] thread taskq > 31 c15c71c4 0 0 0 0000204 [IWAIT] swi6:+ > 30 c15c7388 0 0 0 0000204 [SLPQ - 0xc0603a40][SLP] yarrow > 4 c15c754c 0 0 0 0000204 [SLPQ - 0xc0606428][SLP] g_down > 3 c15c7710 0 0 0 0000204 [SLPQ - 0xc0606424][SLP] g_up > 2 c15c78d4 0 0 0 0000204 [SLPQ - 0xc060641c][SLP] g_event > 29 c15801c4 0 0 0 0000204 [IWAIT] swi1: net > 28 c1580388 0 0 0 0000204 [IWAIT] swi4: vm > 27 c158054c 0 0 0 000020c [RUNQ] swi5: clock sio > 26 c1580710 0 0 0 0000204 [IWAIT] irq15: ata1 > 25 c15808d4 0 0 0 0000204 [IWAIT] irq14: ata0 > 24 c1580a98 0 0 0 0000204 [IWAIT] irq13: > 23 c1580c5c 0 0 0 0000204 [CPU 0] irq12: fxp0 > 22 c1580e20 0 0 0 0000204 [IWAIT] irq11: > 21 c15b6000 0 0 0 0000204 [IWAIT] irq10: > 20 c15b61c4 0 0 0 0000204 [IWAIT] irq9: > 19 c15b6388 0 0 0 0000204 [IWAIT] irq8: rtc > 18 c1578000 0 0 0 0000204 [IWAIT] irq7: > 17 c15781c4 0 0 0 0000204 [IWAIT] irq6: > 16 c1578388 0 0 0 0000204 [IWAIT] irq5: > 15 c157854c 0 0 0 0000204 [IWAIT] irq4: sio0 > 14 c1578710 0 0 0 0000204 [IWAIT] irq3: sio1 > 13 c15788d4 0 0 0 0000204 [IWAIT] irq1: atkbd0 > 12 c1578a98 0 0 0 0000204 [IWAIT] irq0: clk > 11 c1578c5c 0 0 0 000020c [Can run] idle > 1 c1578e20 0 0 1 0004200 [SLPQ wait 0xc1578e20][SLP] init > 10 c1580000 0 0 0 0000204 [SLPQ ktrace 0xc0609d18][SLP] ktrace > 0 c06064c0 0 0 0 0000200 [SLPQ sched 0xc06064c0][SLP] swapper > db> tr 375 > Tracing pid 375 tid 100038 td 0xc15b8c00 > sched_switch(c15b8c00,0,1) at sched_switch+0x143 > mi_switch(1,0,c15b8c00,0,c15b8c00) at mi_switch+0x1ba > sleepq_switch(c0c1f460) at sleepq_switch+0xda > sleepq_wait(c0c1f460,0,c0c45240,0,0) at sleepq_wait+0xb > msleep(c0c1f460,c0c1f468,44,c05d48df,0) at msleep+0x2ce > uma_zone_slab(c0c45b40,2,2,80,314c) at uma_zone_slab+0xd2 > uma_zalloc_bucket(c0c45b40,2) at uma_zalloc_bucket+0x14c > uma_zalloc_arg(c0c45b40,c41c1c00,2) at uma_zalloc_arg+0x274 > mb_init_pack(c41c1c00,100,2) at mb_init_pack+0x1d > uma_zalloc_bucket(c0c45ba0,3) at uma_zalloc_bucket+0x1b4 > uma_zalloc_arg(c0c45ba0,d5229c0c,2) at uma_zalloc_arg+0x274 > sosend(c1a05000,0,d5229c54,0,0) at sosend+0x33d > kern_sendit(c15b8c00,3fd,d5229ccc,0,0) at kern_sendit+0x11c > sendit(c15b8c00,3fd,d5229ccc,0,805a000) at sendit+0x145 > sendto(c15b8c00,d5229d14,6,60,202) at sendto+0x4d > syscall(2f,805002f,bfbf002f,1a4,80521b0) at syscall+0x27b > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (133, FreeBSD ELF32, sendto), eip = 0x280c2b9b, esp = 0xbfbfeb4c, ebp = 0xbfbfeb78 --- > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > -- Doug White | FreeBSD: The Power to Serve dwhite@gumbysoft.com | www.FreeBSD.org