From owner-freebsd-net@freebsd.org Wed Jan 15 22:24:19 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5029C1FF7C3 for ; Wed, 15 Jan 2020 22:24:19 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47yhh21VK8z47Mh for ; Wed, 15 Jan 2020 22:24:18 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pl1-x634.google.com with SMTP id ay11so7438045plb.0 for ; Wed, 15 Jan 2020 14:24:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=82HGVpe2Ffum+GkjHFZTAXSboZ7F+gW8tzf/9oRh/nA=; b=C6ND2dsHS4IYcmMcSeU7cEJ/8ZqiZjvjj6wzc42y5xLx6VITi7YtTY3y6iqxlJ+dNj 5lPoPzIs2G+LGaBDOZd6+gOw2g89sP8g7LN392+StszxgjHlBX8DVen0GUFhNCmGL9zR FR5bl9ijxpO7YNWSEONNxveG2NeMNIi+TqIAkviVnNaAbWJDsHDkwZEIqZn42+zU3eXX BcFpWFvbauVcZdKDPXNl4HwI8BFvhStxg5Hrumnous6sYOLoDfjoSDz2WleIOwvGnwDx 5YIdSVHhvdYmE42qMjQ9ov33bE3GsXiJcr+7oYoFbWHF+p/1fGXESB7jUhWJtd+kSEsY njAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=82HGVpe2Ffum+GkjHFZTAXSboZ7F+gW8tzf/9oRh/nA=; b=WoWkQBiUSCI/IPWHZiQ2cihpl//QQOFQ0eRrtAdxG9e54WHD6DaNBJMOvTQ4Ffxgnr H2SAxu8NNWy9kdgKcebMzL4yn+IVA+Gwor5AHd7dtYQlYwf1f+rbljZFEiP+PWjPj6Y4 Z9XoDHTTmh2FfTahmPolc49NpKeKEY8nT6fD+VZtEDnIFdQq3RnG7dJbU4/TjwFnBdg3 lXkyIKYxL/Yxt4RJ8uWtDtr3nnqUwJ4aWLojGtP/Ai6b6i3wkmtLoRDX8RwQ3fNoDCzj R351bwss8mtNnkAkW9KQm+jHKtFcqtx5p+gtchl7Z4bkTpFE3SQEBM8cztR1/k3qi117 2bPQ== X-Gm-Message-State: APjAAAXMRVzrFM90i5rAWVOv7e+/3n9tjTIm37NoyDbEGGoiHnsOShlh C6iTASfY0FBXa6HhEUY5OCq/lweM X-Google-Smtp-Source: APXvYqzYA+NuWhC3SdC0Wb4uwkOOh3aEAz1pD55xg0OcIgeLQzrVXVa5i8N9gG2jGH/dE+pFP8JqSg== X-Received: by 2002:a17:90a:ca12:: with SMTP id x18mr2653953pjt.66.1579127056576; Wed, 15 Jan 2020 14:24:16 -0800 (PST) Received: from [10.192.166.0] (stargate.chelsio.com. [12.32.117.8]) by smtp.googlemail.com with ESMTPSA id a10sm22185350pgm.81.2020.01.15.14.24.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 15 Jan 2020 14:24:15 -0800 (PST) Subject: Re: unexplained latency, interrupt spikes and loss of throughput on FreeBSD router/firewall system To: John Jasen , FreeBSD Net References: From: Navdeep Parhar Message-ID: Date: Wed, 15 Jan 2020 14:24:14 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47yhh21VK8z47Mh X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=C6ND2dsH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of nparhar@gmail.com designates 2607:f8b0:4864:20::634 as permitted sender) smtp.mailfrom=nparhar@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-8.87), ipnet: 2607:f8b0::/32(-2.09), asn: 15169(-1.83), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.3.6.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jan 2020 22:24:19 -0000 On 1/15/20 6:55 AM, John Jasen wrote: > Executive summary: > > Periodically, load will spike on network interrupts on one of our > firewalls. Latency will quickly climb to the point that things are > unresponsive, sessions will timeout, and bandwidth will plummet. Is this with 9000 MTU? Can you please post "netstat -m" from this system? Assuming this is 9000 MTU, try setting this in /boot/loader.conf and reboot: hw.cxgbe.largest_rx_cluster=4096 > We do not see increases in ethernet pause frames, drops, errors, or > anything else like that from the system. This part is strange. The incoming frames are either being dropped (errors or overflows) or getting throttled via pause frames. I'd have expected "netstat -dI " to show errors or drops or "sysctl dev.cc dev.cxl | grep pause" to show some activity. Can you please double check? Regards, Navdeep > > Usually, the quickest fix is to failover to the backup firewall. At that > time, the backup firewall behaves normally and interrupt load drops on the > afflicted firewall device. > > I'm stumped. Networking says its these systems. I believe its something on > other side. > > Any ideas? > > Background information: > FreeBSD 11.3-RELEASE-p3 > hw.machine: amd64 > hw.model: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz > hw.ncpu: 24 > hw.machine_arch: amd64 > Firewall: pf > failover: CARP > network cards: seen with Chelsio T5-580 and T6 series cards. > other networking information: VLANs are in use. Occasional LAGG usage as > well. > > When this occurs, some of the interrupts dedicated to cxgbe queues spike > to 100%. Latency climbs to the point that TCP timeouts start kicking in, > and users start complaining. Bandwidth drops from 2-3Gbs to ~100-200Mbs > > netstat shows no increase of error or drop packets. sysctl shows no > increase in pause frames. > > I'm happy to provide further information. > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >