From owner-freebsd-questions@FreeBSD.ORG Thu Oct 16 22:52:54 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B4010567 for ; Thu, 16 Oct 2014 22:52:54 +0000 (UTC) Received: from mail-qg0-x22e.google.com (mail-qg0-x22e.google.com [IPv6:2607:f8b0:400d:c04::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E93DCCB for ; Thu, 16 Oct 2014 22:52:54 +0000 (UTC) Received: by mail-qg0-f46.google.com with SMTP id z60so3322872qgd.33 for ; Thu, 16 Oct 2014 15:52:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsd.com.br; s=capeta; h=date:from:to:subject:message-id:organization:mime-version :content-type:content-transfer-encoding; bh=Gt4Z36lW66sO3wzhr3Xtx4Kd5c4LTFnXYgX4byvmI64=; b=Uo4jIi68FuvrV2iNrbQ27BJ0xKjTLst02Acjsn+Gv3cZJDLUg1jG4uTPHLWyqb8EE6 /eX1UwjVgJcZ5D2++ehi1D6Y7A9HtGgu8pY1XbPlIXWQUjTXsZAq2cTCkeROgl1CRpme Z4jh39SF4yxYX3WAL8eTaPCOcRJOFlKx2FZrE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:subject:message-id:organization :mime-version:content-type:content-transfer-encoding; bh=Gt4Z36lW66sO3wzhr3Xtx4Kd5c4LTFnXYgX4byvmI64=; b=LVpkCatAnJV0wndet2xoGKzSMrWUv+nUY2BeXj0tAty1lEXeX954AEjA0X1VHT/ZWt CVFIpEp3yfnthDUJPA4H/fQfPMnnmA5WP/ry4kGzDcXNJdgW3kKRUIm0qeWaboXoKp8d vG29+4GP9BCSpYz94j40EMp1uAsVNqzInEdzT208RY4DdJav1WGr3qUFXOJHDoIk/iFT PzSHEr9Z7RU0cwMMLGNTMzHEv2ZWkktY3L5E5AFJvIHLCuwM4CLgmDfSsanBZmRbulp5 bNepxt8R81ndXQkBCrF6+N5sK3C+esDW4YdLog9E9eCNCoVXQjXH4WwQUwgHjI83yKu2 3IwQ== X-Gm-Message-State: ALoCoQlogrKzFAuts3BryjtfOtbt+J7YMs9GWKqPW3nEXO+xVkDZvDSnxcOY7G3zVr25OSOlgu3K X-Received: by 10.140.29.134 with SMTP id b6mr6376612qgb.23.1413499973263; Thu, 16 Oct 2014 15:52:53 -0700 (PDT) Received: from Papi ([177.134.200.237]) by mx.google.com with ESMTPSA id h32sm11355576qge.16.2014.10.16.15.52.52 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 16 Oct 2014 15:52:53 -0700 (PDT) Date: Thu, 16 Oct 2014 19:53:13 -0300 From: Mario Lobo To: freebsd-questions@freebsd.org Subject: Network problem Message-ID: <20141016195313.65bdca0f@Papi> Organization: BSD X-Mailer: Claws Mail 3.10.1 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 22:52:54 -0000 Hi; I have 2 links with 2 different ISPs on this firewall. On one link, everything is normal [~]>ping www.google.com PING www.google.com (64.233.185.104): 56 data bytes 64 bytes from 64.233.185.104: icmp_seq=0 ttl=44 time=175.683 ms 64 bytes from 64.233.185.104: icmp_seq=1 ttl=44 time=174.499 ms 64 bytes from 64.233.185.104: icmp_seq=2 ttl=44 time=171.770 ms 64 bytes from 64.233.185.104: icmp_seq=3 ttl=44 time=170.737 ms 64 bytes from 64.233.185.104: icmp_seq=4 ttl=44 time=170.705 ms 64 bytes from 64.233.185.104: icmp_seq=5 ttl=44 time=171.699 ms 64 bytes from 64.233.185.104: icmp_seq=6 ttl=44 time=178.248 ms 64 bytes from 64.233.185.104: icmp_seq=7 ttl=44 time=170.924 ms 64 bytes from 64.233.185.104: icmp_seq=8 ttl=44 time=174.189 ms 64 bytes from 64.233.185.104: icmp_seq=9 ttl=44 time=169.742 ms 64 bytes from 64.233.185.104: icmp_seq=10 ttl=44 time=171.725 ms on the other, I get this: [~]>setfib 1 ping www.google.com PING www.google.com (64.233.185.104): 56 data bytes ping: sendto: No buffer space available 64 bytes from 64.233.185.104: icmp_seq=0 ttl=39 time=2837.234 ms 64 bytes from 64.233.185.104: icmp_seq=2 ttl=39 time=1999.021 ms ping: sendto: No buffer space available 64 bytes from 64.233.185.104: icmp_seq=3 ttl=39 time=2256.407 ms 64 bytes from 64.233.185.104: icmp_seq=4 ttl=39 time=3002.886 ms 64 bytes from 64.233.185.104: icmp_seq=6 ttl=39 time=1570.676 ms 64 bytes from 64.233.185.104: icmp_seq=7 ttl=39 time=1765.401 ms 64 bytes from 64.233.185.104: icmp_seq=8 ttl=39 time=2148.457 ms ping: sendto: No buffer space available 64 bytes from 64.233.185.104: icmp_seq=9 ttl=39 time=2207.230 ms 64 bytes from 64.233.185.104: icmp_seq=10 ttl=39 time=1752.444 ms 64 bytes from 64.233.185.104: icmp_seq=12 ttl=39 time=1619.150 ms re0 - PROD-LAN re1 - LAB-LAN sk0 - ISP1 (fib 0) sk1 - ISP2 (fib 1) FreeBSD FreeBSD 8.3-STABLE #0 r204106 Wed Sep 17 10:28:45 BRT 2014 last pid: 6963; load averages: 0.00, 0.00,0.00 19:47:29 61 processes: 1 running, 60 sleeping CPU: 0.0% user, 0.0% nice, 0.2% system, 0.1% interrupt, 99.7% idle Mem: 210M Active, 3933M Inact, 1559M Wired, 516K Cache, 771M Buf, 1671M Free Swap: 16G Total, 16G Free This was at 7:oo PM but even during peak work ours it doesn't go lower than 88% idle when it peaks usage. I have one squid for each link with exactly the same configs and pf doing the round-robin between the two. This server has never done this before!. I've been googling around and tried a few suggestions but none really solved it. I have already set kern.ipc.nmbclusters from 32768 to 131072. It seemed to delay the "no buffer" message but it's back. My stats seem to me to be well bellow the limits. [~]>netstat -m 2187/2178/4365 mbufs in use (current/cache/total) 1861/1453/3314/131072 mbuf clusters in use (current/cache/total/max) 1861/1339 mbuf+clusters out of packet secondary zone in use (current/cache) 212/321/533/65536 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/32768 9k jumbo clusters in use (current/cache/total/max) 0/0/0/16384 16k jumbo clusters in use (current/cache/total/max) 5169K/4734K/9903K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 3 requests for I/O initiated by sendfile 0 calls to protocol drain routines I already rebooted and switched cables. The only thing I didnt do yet was to switch boards. Should I be looking into that? Any advice? Thanks, -- Mario Lobo http://www.mallavoodoo.com.br FreeBSD since 2.2.8 [not Pro-Audio.... YET!!] (99% winblows FREE) "UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things."