From owner-freebsd-net@freebsd.org Mon Apr 30 19:55:07 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08877FB537E for ; Mon, 30 Apr 2018 19:55:07 +0000 (UTC) (envelope-from jason@singular.net) Received: from mail-ot0-x22a.google.com (mail-ot0-x22a.google.com [IPv6:2607:f8b0:4003:c0f::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 99B6F75FAB for ; Mon, 30 Apr 2018 19:55:06 +0000 (UTC) (envelope-from jason@singular.net) Received: by mail-ot0-x22a.google.com with SMTP id i5-v6so10851550oth.1 for ; Mon, 30 Apr 2018 12:55:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=singular-net.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=dO3sLI8xOjU1WhaDdU79z2FLlZqIuEwk6XioAv9PfOc=; b=SacGTRWNxaur8se4zqVI7PsQqq0uWQDpQTz5rP4Y7Ll5LSh4ELMxcmJhj+Ts9NQVLb nP1dcWBRRxitTNo7+1a9xLKWuB0txDhStykC0N8WQIy1xdmWaQC5pbeX6NFGa79v9FLy wdwWwccOCUG+To3/Hk32s485/OyYAPMzTxxctE6CV89/hyhNGt+oZErgRYmic9nXq44Q jsw/i0ye4ZZRNa/hbejoI72apklobEqQwqlWADVBq/SHgxK5JgcLI74d6Jk8goYG55v8 57iSf94UAIX06EyxU3dihmzaET6Fbm1lD67XO3kWRCDY/PDNwGSUw981BunY1kuwFbVh sunA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=dO3sLI8xOjU1WhaDdU79z2FLlZqIuEwk6XioAv9PfOc=; b=kahlBbJ+QBfo23Fm3ZsD9ILWq+VQGb1CYLQtRiWMcoayDK5VhVfQTwjEgXk+jMXFHn fBgx8C1VFvHPI8twQchdj+kd8pgnnH7sHdwnBUpTO+fUqxjz8dBSVarNVduKYieoFbvp URqnj3ECkINWMgCOVhyMN8REMAXGQibL+3MPUy04zJBoJiJz69DYFykxLVvu4sTh6w98 MZpdV6oeczUcWKQjuDIZvGyyMklSQq6QFTDJythd9SRX8869isSTJ+x1zpOlla8nskT9 oFwUqDybfVBCdJfkY1SJGDO/yf0LKA8r0A3FqwlK9wFD+KbwT34C4ULzYq7ZTwQQeyV3 LoTA== X-Gm-Message-State: ALQs6tCWP5bV//LlTZYLjvBUdAERUXUayyvTz+LC6f3zx6Bl5jphuuJh FWu+iQi7gP9D9gFb1CSFfsA3vWt3pSL2mpFAClzu+BQ3 X-Google-Smtp-Source: AB8JxZqeIMvciDHAaLLWtdcUYfLUcd7cYhEUdikVbQlcrYnAGYC55i+9bJJYOZJ2nTMbDlq6yzblcR9QIrRnCRRKJ90= X-Received: by 2002:a9d:2456:: with SMTP id p80-v6mr8708780ota.231.1525118105481; Mon, 30 Apr 2018 12:55:05 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a9d:1dd2:0:0:0:0:0 with HTTP; Mon, 30 Apr 2018 12:55:04 -0700 (PDT) From: Jason Matthews Date: Mon, 30 Apr 2018 12:55:04 -0700 Message-ID: Subject: high interrupts on network load To: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Apr 2018 19:55:07 -0000 I have a pair systems running relayd in DSR mode. The load balancer works well, to a point, but suffers from high interrupts and cpu utilization on eight cores. The monitoring systems show increased latency until it falls over dead at round 330k requests/sec where the CARP seems to stop responding and the partner jumps in only to wash-rinse-repeat the cycle. top -aSCHIP shows 85-90% CPU utilization due to interrupts when running near peak. What can be done to lower the interrupt load? Or spread it across more CPUs? Is the design such that one core is dedicated to a queue on the card? Or can the load be spread across more CPUs? The systems are Intel S5520UR boards with a X540-AT2 card on X5650 CPUs. Any suggestions are appreciated. thanks, j. ---------------- = Internet = ---------------- | ix0 --------------- = relayd = ---------------- | ix1 | -------------------- - web servers - ---------------------