From owner-svn-src-head@freebsd.org Wed Jul 27 19:44:53 2016 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B643BA6833; Wed, 27 Jul 2016 19:44:53 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x233.google.com (mail-io0-x233.google.com [IPv6:2607:f8b0:4001:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D9C3A183B; Wed, 27 Jul 2016 19:44:52 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x233.google.com with SMTP id b62so79980036iod.3; Wed, 27 Jul 2016 12:44:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=H9rhgFC2elknqbUzHBLOGCGBtdJRXdxey2lDayd0Hpc=; b=i66JB/DUsc1AS1RwUzr4HF1WLK5tr/LnrnBlbjdKX4hW9yLREyKhGvIcElHWVk1J7b G6xymaOjy703xgAlToDiUY73zkGUW6HIpT2x4VZyDgciWyOclWXdHPS3Ldv4WCl7aiFK Nh4dCyAscgv4Ty1dIHQxZKGE9G1bzVFAetWxtaLnWUD2P1GMZPDe/LRlJHawek7IDxww znqpJUZu4IgwBSJ+AvBwPbvW8jE3+oimhNY57ioewN9J0rk/Q43P1pOMkqI8qONckej1 X5BWdx7PJdfnSGd02XrRwnQVww51Gaw8CwHayodfd6bik5B4BFswrRx3g/LpAVBpDNc5 M3og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=H9rhgFC2elknqbUzHBLOGCGBtdJRXdxey2lDayd0Hpc=; b=XLJ9IfJ1FtaSrzxgb9R0RN2BY/HvNxzcChsmUj/Ly8RViCteVHNEDDPeiadk8CUUy0 dIB3VAJomxZgUL7q3cD+iKoC5Kj+ceu/2ntlXCVWNdywb3m0EC+++QtS0n37k17hDki0 jKLzr/arrES/hrXVhZpgeSlLM2KXsTJsU57jGaHaUcnU/3NelrJ679i3XekV5AcVQ0k4 iiES1K5HOriFAAQVasI6vnriorCTV3dluOt4YG+fVzq3v10fKjzcBzrjoy/2BVdyjRg+ amGSWvzrv/VbMpwBJZNBQT3IbZ7hIo5DcZRIpa25vdVN2BtyukO1ftvVYjRo9UbzUHc2 RJnQ== X-Gm-Message-State: AEkoout24VdiGUuUZidIa8xHg+qfjjP1KreWLk4sTctRv9X1Bet2Xq9qoRflkJ9h4nB/HBvTep1PEYN9sOjE8w== X-Received: by 10.107.15.157 with SMTP id 29mr33161186iop.123.1469648692155; Wed, 27 Jul 2016 12:44:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.36.141.129 with HTTP; Wed, 27 Jul 2016 12:44:51 -0700 (PDT) In-Reply-To: <3422795.rot3cCl2OH@ralph.baldwin.cx> References: <201607271829.u6RITZlx041710@repo.freebsd.org> <3422795.rot3cCl2OH@ralph.baldwin.cx> From: Adrian Chadd Date: Wed, 27 Jul 2016 12:44:51 -0700 Message-ID: Subject: Re: svn commit: r303405 - in head/sys/dev/cxgbe: . tom To: John Baldwin Cc: "src-committers@freebsd.org" , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jul 2016 19:44:53 -0000 [snip] When we had my kqueue sendfile stuff in tree to handle notifications, I was getting 40g across what, 64k sockets on 8 cores using SHM sendfile + kqueue sendfile completion. it worked pretty well. One core could get 40g if I pre-seeded enough data into it (ie, multiple sendfile transactions on a socket) so I kept the socket buffer full. Otherwise there'd be dead time where the socket buffer was empty. It was very sensitive to the TCP segment size - ie, if for some reason the TCP TX path passed up less than 32k chunks, it would end up chewing way too much CPU in tcp_output(). That was very sensitive to the TX write() size and latency to the receiver. It wasn't running out of data either; it was just some side effect of how big the writes were and how quickly the TX socket buffer was being topped off. I .. well, never got to finish instrumenting the full relationship going on there (between TSO and buffer sizes/latency) to see if that could be better engineered so we always kept the TSO full if we could. -adrian