From owner-freebsd-transport@freebsd.org Thu Jan 21 21:56:59 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4489A8C751 for ; Thu, 21 Jan 2016 21:56:58 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-io0-x234.google.com (mail-io0-x234.google.com [IPv6:2607:f8b0:4001:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BBBA81CB7 for ; Thu, 21 Jan 2016 21:56:58 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-io0-x234.google.com with SMTP id 1so69146807ion.1 for ; Thu, 21 Jan 2016 13:56:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=ie85nspYRdxr+Nbt53L9v74uXlksY2OZsmYtzRSrPmU=; b=GPcXPYLtqxRO3Wknr/dRBk6Sq5TuRx7/WZYbXekKai7jX5bDRvwkNHTg2FXf6kHpCD tvpjhLMCILg14t96zKVLbkdzNp6q7pMgtwr6hOLbUxuhq9t+DA3PWZrBFRIl7k6hOPK1 3EAxF1+izI0gR2pionKiYFMVFn/umTA5i+vdaZxJAS7cIum/dTQG0GE6Kkmp+9Yp4/kD BLXmVS2fhX44QaLFr8p+XeghRpTTz+oyTZGuRgsuN2B5ByRRbn5vyg3vXnXlM/u7toFC WQv45ZxXyFxmqTyjCaA2UfpRvJa2aMvdsNQZ4Q0/LjB2nwX39+Vv6f47YlDU63sIsGan //rA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=ie85nspYRdxr+Nbt53L9v74uXlksY2OZsmYtzRSrPmU=; b=kQP8p9xHQGl/wPonlWO6ViQKKBMbXQK/Z8wbXe6ufWTES3iP5CNKkYrSr1S68TR9KO bxW6OrOwtUwDI6QCJZKKpEJVxxaFdL6MUjggOkqyrNbJUuBXmcIiMfI0VBI0gTR8XYaQ ML0L7To+IEf1vBwo1aFWfOTaoQLt4rPy948SsBFIKu1TEfMhY4k4sWxLEjHyHm459RjW ALCxRbAGH9GMKBwOyZMGACI86n+0ZUTz2wob+WhjktGpJuRQMh0IGudxbh0QwmcEl1Ao 1GxVM91Epgyz82s0oM7zTa9AQ5NQBksJeENXuZOF9FBu9zpglzq2hHhO+42p9daj9jWq T1Xg== X-Gm-Message-State: AG10YORHJovebOIStwbhk4I9bpyIFEsPQtkK16KR+T08Ms2Z3w3uIJoiyy8aG7xjSqlDbBW8XRFfuA428hkIjQ== MIME-Version: 1.0 X-Received: by 10.107.153.140 with SMTP id b134mr145385ioe.113.1453413418013; Thu, 21 Jan 2016 13:56:58 -0800 (PST) Received: by 10.107.178.193 with HTTP; Thu, 21 Jan 2016 13:56:57 -0800 (PST) Date: Thu, 21 Jan 2016 16:56:57 -0500 Message-ID: Subject: Flamegraph of 8KB message bottlenecks From: Ryan Stone To: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 21:56:59 -0000 I brought up on the -transport call last week that I was having trouble receiving 8KB packets at 40Gbps. I blamed "TCP stack" peformance, although what I really meant was "kernel performance" -- I just assumed that the problem was in the stack. From looking at a flamegraph, I'm not so sure this was the case. The flamegraph was taken on a "FreeBSD 10.1-ish" system. Unfortunately running stock FreeBSD head in this environment is quite difficult, so I probably wouldn't be able to check if anything on head has alleviated things. You can see the flamegraph for youself here: https://people.freebsd.org/~rstone/incast_8kb_flame.svg You'll note that my underland application is way off in the bottom left, taking 3% of all cycles. Meanwhile top reports that the system is completely bogged down in interrupt and system time.