From owner-freebsd-net@FreeBSD.ORG Tue Jun 14 22:51:13 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC8E5106566B; Tue, 14 Jun 2011 22:51:12 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id 880A68FC14; Tue, 14 Jun 2011 22:51:12 +0000 (UTC) Received: by yic13 with SMTP id 13so2152142yic.13 for ; Tue, 14 Jun 2011 15:51:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:date:x-google-sender-auth :message-id:subject:from:to:cc:content-type; bh=DGk3iY5W4pOeBwMutcHYG6qfjhJ1A8hNlVb2Qr/2X5s=; b=ZoIdeD+rBWjwTkuoL3YKs2aGkwrSB8zstohzDCF9kJg9a7mzp4k3P/V2n/p1qxB6Wz U1NNpA1CeOgV4ZA5mRHs8L9cxg8++zuHYMjhDykmM/FUIJluPm4MrAZUdXVNtkrNv478 4s5dhwT8LeulelvlRq2W8EPUKYu34gC8eNVus= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; b=bpospt+sMT+8ODyOLZ041A01Q3G+egWMT30beElV3vgGLwg9RfmY1Y2i8tS7sgscPl K5nYYdOljHjBXxqNglPlTmcpKSW9oNOVy52wIO0bVJyoI8A9ntThT/qpJtXeNmVLwEYb GMn8BMjmdcSovrtkUI2fXoPxO4aatOaMB0YE4= MIME-Version: 1.0 Received: by 10.236.186.65 with SMTP id v41mr10935634yhm.1.1308090142297; Tue, 14 Jun 2011 15:22:22 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.236.61.73 with HTTP; Tue, 14 Jun 2011 15:22:22 -0700 (PDT) Date: Tue, 14 Jun 2011 15:22:22 -0700 X-Google-Sender-Auth: QfrUYqe8tJz_a23dbAhyXwAdlwc Message-ID: From: Artem Belevich To: FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1 Cc: Rick Macklem Subject: amd + NFS reconnect = ICMP storm + unkillable process. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jun 2011 22:51:13 -0000 Hi, I wonder if someone else ran into this issue before and, maybe, have a solution. I've been running into a problem where access to filesystems mouted with amd wedges processes in an unkillable state and produces ICMP storm on loopback interface.I've managed to narrow down to NFS reconnect, but that's when I ran out of ideas. Usually the problem happens when I abort a parallel build job in an i386 jail on FreeBSD-8/amd64 (r223055). When the build job is killed now and then I end up with one process consuming 100% of CPU time on one of the cores. At the same time I get a lot of messages on the console saying "Limiting icmp unreach response from 49837 to 200 packets/sec" and the loopback traffic goes way up. As far as I can tell here's what's happening: * My setup uses a lot of filesystems mounted by amd. * amd itself pretends to be an NFS server running on the localhost and serving requests for amd mounts. * Now and then amd seems to change the ports it uses. Beats me why. * the problem seems to happen when some process is about to access amd mountpoint when amd instance disappears from the port it used to listen on. In my case it does correlate with interrupted builds, but I have no clue why. * NFS client detects disconnect and tries to reconnect using the same destination port. * That generates ICMP response as port is unreachable and it reconnect call returns almost immediatelly. * We try to reconnect again, and again, and again.... * the process in this state is unkillable Here's what the stack of the 'stuck' process looks like in those rare moments when it gets to sleep: 18779 100511 collect2 - mi_switch+0x176 turnstile_wait+0x1cb _mtx_lock_sleep+0xe1 sleepq_catch_signals+0x386 sleepq_timedwait_sig+0x19 _sleep+0x1b1 clnt_dg_call+0x7e6 clnt_reconnect_call+0x12e nfs_request+0x212 nfs_getattr+0x2e4 VOP_GETATTR_APV+0x44 nfs_bioread+0x42a VOP_READLINK_APV+0x4a namei+0x4f9 kern_statat_vnhook+0x92 kern_statat+0x15 freebsd32_stat+0x2e syscallenter+0x23d * Usually some timeout expires in few minutes, the process dies, ICMP storm stops and the system is usable again. * On occasion the process is stuck forever and I have to reboot the box. I'm not sure who's to blame here. Is the automounter at fault for disappearing from the port it was supposed to listen to? If NFS guilty of trying blindly to reconnect on the same port and not giving up sooner? Should I flog the operator (ALA myself) for misconfiguring something (what?) in amd or NFS? More importantly -- how do I fix it? Any suggestions on fixing/debugging this issue? --Artem