From owner-freebsd-hackers@FreeBSD.ORG Sat Nov 20 19:09:43 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0DCB81065670 for ; Sat, 20 Nov 2010 19:09:43 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7E2668FC0A for ; Sat, 20 Nov 2010 19:09:42 +0000 (UTC) Received: by bwz2 with SMTP id 2so5072291bwz.13 for ; Sat, 20 Nov 2010 11:09:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :x-comment-to:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=ZJHsQJResiOBa/6yOk3OxpRO11Msht2/d7K/2+ygkF8=; b=If8gSo59MPbIR4qAqPEy4ZDSmWMALUIAsjdADjY0MxudFM1/FQTWI4tbnOtHOemz7U XcjsXk1MvO0pryzWiWeHSb/VbXU53mRlupp3ZFxWF5oiRBHKceY7JQugHuyUVmx6gzwA ynW/uCwbdQsTd+399j53JA2DzWQn2oQqdE2Qo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=tXU3b0L9fdBqoPeK2q8LlP5EbFyixwRIEMRbNskYgGWAajqC5kbNi3KqPAKnjueQ7Q nui2eHEh+hRycXBp1oxFNLfXSoRqBsTNK9MQK+7SaxAEQFezRA8qQGhmS+n550S46xAg /rSf1Gz4admradkVAKn+NE9W67qTbVznTc3Ws= Received: by 10.204.64.80 with SMTP id d16mr3546348bki.181.1290280180487; Sat, 20 Nov 2010 11:09:40 -0800 (PST) Received: from localhost ([95.69.174.185]) by mx.google.com with ESMTPS id t10sm1507271bkj.4.2010.11.20.11.09.37 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sat, 20 Nov 2010 11:09:39 -0800 (PST) From: Mikolaj Golub To: "Bjoern A. Zeeb" References: <86pqu0nexd.fsf@kopusha.home.net> <20101120165604.T24596@maildrop.int.zabbadoz.net> X-Comment-To: Bjoern A. Zeeb Date: Sat, 20 Nov 2010 21:09:35 +0200 In-Reply-To: <20101120165604.T24596@maildrop.int.zabbadoz.net> (Bjoern A. Zeeb's message of "Sat, 20 Nov 2010 17:03:13 +0000 (UTC)") Message-ID: <868w0nolqo.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Cc: freebsd-hackers@freebsd.org Subject: Re: flowtable_cleaner/flowtable_flush livelock X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Nov 2010 19:09:43 -0000 --=-=-= On Sat, 20 Nov 2010 17:03:13 +0000 (UTC) Bjoern A. Zeeb wrote: BAZ> I think net@ would have been a better initial place but since this BAZ> seems to be a problem when interacting with VIMAGE BAZ> freebsd-virtualization might be better. BAZ> What you could try is: BAZ> http://people.freebsd.org/~bz/20100216-10-ft-cv.diff Ah, I have recalled I had already saw this patch but did not understand what the problem was that it fixed, thus did not associated it with my case (actually, I thought you had committed all these patches to the tree long time ago and I was running the kernel with them already :-). BTW, the patch needs updating: in the current flow_full() wakes up flowcleaner too, and flowcleaner sleeps for flowclean_freq instead of 10*hz (see the attached patch). With the patch I can't reproduce the lock. Only the crash I mentioned in my first letter is observed: (kgdb) bt #0 doadump () at pcpu.h:231 #1 0xc04f2789 in db_fncall (dummy1=1, dummy2=0, dummy3=-1056677760, dummy4=0xc8731860 "") at /usr/src/sys/ddb/db_command.c:548 #2 0xc04f2b81 in db_command (last_cmdp=0xc0e79f7c, cmd_table=0x0, dopager=1) at /usr/src/sys/ddb/db_command.c:445 #3 0xc04f2cda in db_command_loop () at /usr/src/sys/ddb/db_command.c:498 #4 0xc04f4bfd in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_main.c:229 #5 0xc09119be in kdb_trap (type=12, code=0, tf=0xc8731a94) at /usr/src/sys/kern/subr_kdb.c:546 #6 0xc0c3da8f in trap_fatal (frame=0xc8731a94, eva=3735929074) at /usr/src/sys/i386/i386/trap.c:970 #7 0xc0c3e0be in trap (frame=0xc8731a94) at /usr/src/sys/i386/i386/trap.c:361 #8 0xc0c272dc in calltrap () at /usr/src/sys/i386/i386/exception.s:168 #9 0xc0988415 in strncmp (s1=0xc1fee4e0 "epair20b", s2=0xdeadc0f2
, n=16) at /usr/src/sys/libkern/strncmp.c:44 #10 0xc09929d7 in ifunit_ref (name=0xc1fee4e0 "epair20b") at /usr/src/sys/net/if.c:1986 #11 0xc0996982 in ifioctl (so=0xc25649c0, cmd=3223349536, data=0xc1fee4e0 "epair20b", td=0xc286c000) at /usr/src/sys/net/if.c:2475 #12 0xc09307f7 in soo_ioctl (fp=0xc1ff5af0, cmd=3223349536, data=0xc1fee4e0, active_cred=0xc1d83e80, td=0xc286c000) at /usr/src/sys/kern/sys_socket.c:212 #13 0xc092a61d in kern_ioctl (td=0xc286c000, fd=3, com=3223349536, data=0xc1fee4e0 "epair20b") at file.h:254 #14 0xc092a7a4 in ioctl (td=0xc286c000, uap=0xc8731cec) at /usr/src/sys/kern/sys_generic.c:679 #15 0xc091f303 in syscallenter (td=0xc286c000, sa=0xc8731ce4) at /usr/src/sys/kern/subr_trap.c:318 #16 0xc0c3dd2f in syscall (frame=0xc8731d28) at /usr/src/sys/i386/i386/trap.c:1094 #17 0xc0c27371 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:266 #18 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) fr 10 #10 0xc09929d7 in ifunit_ref (name=0xc1fee4e0 "epair20b") at /usr/src/sys/net/if.c:1986 1986 if (strncmp(name, ifp->if_xname, IFNAMSIZ) == 0 && (kgdb) p ifp $1 = (struct ifnet *) 0xdeadc0de I might want to report it to freebsd-virtualization unless I find that this is a known issue. -- Mikolaj Golub --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=20101120-10-ft-cv.diff Index: sys/net/flowtable.c =================================================================== --- sys/net/flowtable.c (revision 215574) +++ sys/net/flowtable.c (working copy) @@ -195,7 +195,8 @@ STATIC_VNET_DEFINE(uma_zone_t, flow_ipv6_zone); #define V_flow_ipv6_zone VNET(flow_ipv6_zone) -static struct cv flowclean_cv; +static struct cv flowclean_f_cv; +static struct cv flowclean_c_cv; static struct mtx flowclean_lock; static uint32_t flowclean_cycles; static uint32_t flowclean_freq; @@ -951,7 +952,7 @@ flow_full(struct flowtable *ft) if ((ft->ft_flags & FL_HASH_ALL) == 0) ft->ft_udp_idle = ft->ft_fin_wait_idle = ft->ft_syn_idle = ft->ft_tcp_idle = 5; - cv_broadcast(&flowclean_cv); + cv_broadcast(&flowclean_c_cv); } else if (!full && ft->ft_full) { flowclean_freq = 20*hz; if ((ft->ft_flags & FL_HASH_ALL) == 0) @@ -1560,14 +1561,14 @@ flowtable_cleaner(void) } VNET_LIST_RUNLOCK(); - flowclean_cycles++; /* * The 10 second interval between cleaning checks * is arbitrary */ mtx_lock(&flowclean_lock); - cv_broadcast(&flowclean_cv); - cv_timedwait(&flowclean_cv, &flowclean_lock, flowclean_freq); + flowclean_cycles++; + cv_broadcast(&flowclean_f_cv); + cv_timedwait(&flowclean_c_cv, &flowclean_lock, 10*hz); mtx_unlock(&flowclean_lock); } } @@ -1580,8 +1581,8 @@ flowtable_flush(void *unused __unused) mtx_lock(&flowclean_lock); start = flowclean_cycles; while (start == flowclean_cycles) { - cv_broadcast(&flowclean_cv); - cv_wait(&flowclean_cv, &flowclean_lock); + cv_broadcast(&flowclean_c_cv); + cv_wait(&flowclean_f_cv, &flowclean_lock); } mtx_unlock(&flowclean_lock); } @@ -1613,7 +1614,8 @@ static void flowtable_init(const void *unused __unused) { - cv_init(&flowclean_cv, "flowcleanwait"); + cv_init(&flowclean_c_cv, "c_flowcleanwait"); + cv_init(&flowclean_f_cv, "f_flowcleanwait"); mtx_init(&flowclean_lock, "flowclean lock", NULL, MTX_DEF); EVENTHANDLER_REGISTER(ifnet_departure_event, flowtable_flush, NULL, EVENTHANDLER_PRI_ANY); --=-=-=--