From owner-freebsd-transport@freebsd.org Thu Jan 21 21:56:59 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4489A8C751 for ; Thu, 21 Jan 2016 21:56:58 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-io0-x234.google.com (mail-io0-x234.google.com [IPv6:2607:f8b0:4001:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BBBA81CB7 for ; Thu, 21 Jan 2016 21:56:58 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-io0-x234.google.com with SMTP id 1so69146807ion.1 for ; Thu, 21 Jan 2016 13:56:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=ie85nspYRdxr+Nbt53L9v74uXlksY2OZsmYtzRSrPmU=; b=GPcXPYLtqxRO3Wknr/dRBk6Sq5TuRx7/WZYbXekKai7jX5bDRvwkNHTg2FXf6kHpCD tvpjhLMCILg14t96zKVLbkdzNp6q7pMgtwr6hOLbUxuhq9t+DA3PWZrBFRIl7k6hOPK1 3EAxF1+izI0gR2pionKiYFMVFn/umTA5i+vdaZxJAS7cIum/dTQG0GE6Kkmp+9Yp4/kD BLXmVS2fhX44QaLFr8p+XeghRpTTz+oyTZGuRgsuN2B5ByRRbn5vyg3vXnXlM/u7toFC WQv45ZxXyFxmqTyjCaA2UfpRvJa2aMvdsNQZ4Q0/LjB2nwX39+Vv6f47YlDU63sIsGan //rA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=ie85nspYRdxr+Nbt53L9v74uXlksY2OZsmYtzRSrPmU=; b=kQP8p9xHQGl/wPonlWO6ViQKKBMbXQK/Z8wbXe6ufWTES3iP5CNKkYrSr1S68TR9KO bxW6OrOwtUwDI6QCJZKKpEJVxxaFdL6MUjggOkqyrNbJUuBXmcIiMfI0VBI0gTR8XYaQ ML0L7To+IEf1vBwo1aFWfOTaoQLt4rPy948SsBFIKu1TEfMhY4k4sWxLEjHyHm459RjW ALCxRbAGH9GMKBwOyZMGACI86n+0ZUTz2wob+WhjktGpJuRQMh0IGudxbh0QwmcEl1Ao 1GxVM91Epgyz82s0oM7zTa9AQ5NQBksJeENXuZOF9FBu9zpglzq2hHhO+42p9daj9jWq T1Xg== X-Gm-Message-State: AG10YORHJovebOIStwbhk4I9bpyIFEsPQtkK16KR+T08Ms2Z3w3uIJoiyy8aG7xjSqlDbBW8XRFfuA428hkIjQ== MIME-Version: 1.0 X-Received: by 10.107.153.140 with SMTP id b134mr145385ioe.113.1453413418013; Thu, 21 Jan 2016 13:56:58 -0800 (PST) Received: by 10.107.178.193 with HTTP; Thu, 21 Jan 2016 13:56:57 -0800 (PST) Date: Thu, 21 Jan 2016 16:56:57 -0500 Message-ID: Subject: Flamegraph of 8KB message bottlenecks From: Ryan Stone To: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 21:56:59 -0000 I brought up on the -transport call last week that I was having trouble receiving 8KB packets at 40Gbps. I blamed "TCP stack" peformance, although what I really meant was "kernel performance" -- I just assumed that the problem was in the stack. From looking at a flamegraph, I'm not so sure this was the case. The flamegraph was taken on a "FreeBSD 10.1-ish" system. Unfortunately running stock FreeBSD head in this environment is quite difficult, so I probably wouldn't be able to check if anything on head has alleviated things. You can see the flamegraph for youself here: https://people.freebsd.org/~rstone/incast_8kb_flame.svg You'll note that my underland application is way off in the bottom left, taking 3% of all cycles. Meanwhile top reports that the system is completely bogged down in interrupt and system time. From owner-freebsd-transport@freebsd.org Thu Jan 21 22:13:13 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8AA4EA8CDFD for ; Thu, 21 Jan 2016 22:13:13 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 565681577 for ; Thu, 21 Jan 2016 22:13:13 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-ig0-x22b.google.com with SMTP id z14so136289944igp.1 for ; Thu, 21 Jan 2016 14:13:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=GBTKXE31ywVz8OHxS0B7VkJGT2W1CgjuyntX8twE2GY=; b=mHGLL+L0bSRFDZdbFL7+tykaxfavYDdBiZ6lgw3s9NOQfWgIw75YaBOCFJSaEhcHRe 9VD65EYnyurqpi7WpmIXVPv/8j3R7GhW/nQkNlY9Ger2h4F+ESQGQI0xNSMSic6dBahi Dt5cHyamIRxSNWm+3zaqnkkUo83db3KMvQRjvGWtJu047ZGTSZTGjHdg4YoeXs6w8kRA Y9fipHIbkZ+rDcXFYX3GucReY4ENt6qhnfgcgjXFtOzSlz5Xh4oZn/mi93A7FTC3/HWm vMNzBYazabqpqOOW3lcXKV1SX1ZqoZbZTmaSgGl1L15W6KiVcejs6bYcsvne3w+tA7gD vsKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=GBTKXE31ywVz8OHxS0B7VkJGT2W1CgjuyntX8twE2GY=; b=RidKO39CvGFIvjJpQ9+1BB9LFATSGHTXmV1LnIlI1bUOYwlyfxpWOTHYATsUH8rpjg v5UBRF8QE/l+tGUkWlxdIOBjvPsoY9jkfvc/IDsXSMkHv0e0TDredL0RmjoHJ6Mc7vHc ug8xJGy0XbW933LNJxsZhD5Gid65iQe0r32nANsCM0KtU9MCxL29ke/69IFJTLzJh/xx 3CpWEIRdmt5VdKx8uTqESsM5857pCSdzO9/Usg2lWvzq7c9PdND77SPrissDWyoXPEuT bkY9t16FeavRsLSuo2ndJ8iqimEiZTaL7BBrPpOpT2SpjBKQHgEc2eDilJKkfRtw47B4 YAWQ== X-Gm-Message-State: AG10YOQWSu7NUZQHykx4iYKwSO4H74jIfwKSvOcGJg+xvP5Mc2CMPb+mz31tjMP93LDY+iNDYvO0kygMHAuPpQ== MIME-Version: 1.0 X-Received: by 10.50.160.11 with SMTP id xg11mr12007361igb.83.1453414392643; Thu, 21 Jan 2016 14:13:12 -0800 (PST) Received: by 10.107.178.193 with HTTP; Thu, 21 Jan 2016 14:13:12 -0800 (PST) In-Reply-To: References: Date: Thu, 21 Jan 2016 17:13:12 -0500 Message-ID: Subject: Re: Flamegraph of 8KB message bottlenecks From: Ryan Stone To: Adrian Chadd Cc: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 22:13:13 -0000 A lot. Over 100K per second I think? But the time spent in the syscalls seems to be dominating the time handling interrupts. On Thu, Jan 21, 2016 at 5:11 PM, Adrian Chadd wrote: > Well, how many interrupts/second is this seeing? > > > -a > From owner-freebsd-transport@freebsd.org Thu Jan 21 22:11:35 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16E19A8CCAE for ; Thu, 21 Jan 2016 22:11:35 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x231.google.com (mail-io0-x231.google.com [IPv6:2607:f8b0:4001:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D8BEA1379 for ; Thu, 21 Jan 2016 22:11:34 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x231.google.com with SMTP id 77so69408748ioc.2 for ; Thu, 21 Jan 2016 14:11:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bpLwqIIAcwjv+2EmsJm/n+gNovoHzlUg2hXVAqZbzn4=; b=gJqnEjIDQTeXB1gTS8ueSbE93wxp9O5xvJf8yyL1SQto5d4kJ8Hn/iXfihmjU0YFy/ p/NbRSsLYk+vMRW5xu0/CGFQv+lj1rwIsO32Sw20al9vfkVOzMN4lQMv0VVo5zSfcXx6 R2+snL3mgmz3/YkOMcqZNNJUZ4u0mKx6TdZHFg7uKOJU6D0pOl7x8YMMv9B/4bvgovX6 OMyJIzIwVURMAl1HMsZhT20KGfiAWQ7p3PEbfqeH4VbgTJYXWlIMMJEe55JGUZmUgT/b wLTwIYjEr8XobkVNw/0xteRcddXSFhMAbgDzlERs8AquZVs+UlWWbI43aUR4AfEBnfnK oC3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=bpLwqIIAcwjv+2EmsJm/n+gNovoHzlUg2hXVAqZbzn4=; b=frehsvi/ab5h8uDQ1T7xTq2khRm6CZ0pc8oRfoKexGKStNLZRIpzgIobJVYPx5d0pV 5h3YQc9SZOJAXckVc6v9gaXlqCfHhtGKRPenuHwDXurpFn9XEE71lohWLv0g2Q1v1uIZ K9jm2VM/M02HlONTRyc9tufuTesMWC69crFsbNp2MdZm/kSjDPsUJ9VOvjcwuD8/BIdo M4SohAIG8X4YCUa6ECx5zXSQPXyivdFRqxnEwKFMGRJgNok3TGUT1EUiClZo/8q8d+Z7 m4b/7134rWWZJyF4WIWfEDBKTP6l7ef2vS8MR6YplajalIapPDLGjjZY/0KK8c0N4Rrh Vevg== X-Gm-Message-State: AG10YOR/hznR+TbImO/r1u1+CxE4Lxtd0Uo/RfKrkh4qCAb4nrPWAGqJx9SNP2Gd1BLkE3RgcMjS/WXSib4A/w== MIME-Version: 1.0 X-Received: by 10.107.162.146 with SMTP id l140mr191008ioe.123.1453414294281; Thu, 21 Jan 2016 14:11:34 -0800 (PST) Received: by 10.36.121.16 with HTTP; Thu, 21 Jan 2016 14:11:34 -0800 (PST) In-Reply-To: References: Date: Thu, 21 Jan 2016 14:11:34 -0800 Message-ID: Subject: Re: Flamegraph of 8KB message bottlenecks From: Adrian Chadd To: Ryan Stone Cc: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Mailman-Approved-At: Thu, 21 Jan 2016 22:32:05 +0000 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 22:11:35 -0000 Well, how many interrupts/second is this seeing? -a From owner-freebsd-transport@freebsd.org Thu Jan 21 22:34:22 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2AB2FA8C680 for ; Thu, 21 Jan 2016 22:34:22 +0000 (UTC) (envelope-from mmacy@nextbsd.org) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1E3701272 for ; Thu, 21 Jan 2016 22:34:21 +0000 (UTC) (envelope-from mmacy@nextbsd.org) Received: from mail.zoho.com by mx.zohomail.com with SMTP id 145341565274672.66706930303178; Thu, 21 Jan 2016 14:34:12 -0800 (PST) Date: Thu, 21 Jan 2016 14:34:12 -0800 From: Matthew Macy To: "Adrian Chadd" Cc: "Ryan Stone" , "" Message-ID: <1526652dd58.117a1e7e938176.477104883701987065@nextbsd.org> In-Reply-To: References: Subject: Re: Flamegraph of 8KB message bottlenecks MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Priority: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Zoho-Virus-Status: 1 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 22:34:22 -0000 ~170k interrupts / second for ~800kpps, but at this point clearly all the time is spent on syscall overhead ---- On Thu, 21 Jan 2016 14:11:34 -0800 Adrian Chadd wrote ---- > Well, how many interrupts/second is this seeing? > > > -a > _______________________________________________ > freebsd-transport@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-transport > To unsubscribe, send any mail to "freebsd-transport-unsubscribe@freebsd.org" > From owner-freebsd-transport@freebsd.org Thu Jan 21 22:16:33 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83E8BA8CEE0 for ; Thu, 21 Jan 2016 22:16:33 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22a.google.com (mail-io0-x22a.google.com [IPv6:2607:f8b0:4001:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5A10217A6 for ; Thu, 21 Jan 2016 22:16:33 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x22a.google.com with SMTP id g73so71230482ioe.3 for ; Thu, 21 Jan 2016 14:16:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=EWFVvt+f7V+HdpX0Zm4rv8MD+Gw4gLTiE1P+4PE051Y=; b=EkLZS1+v2tRk8+5r4pnFdTAfOIoaEmEfVKsZrN9a6gYHvoG3OR50yc4WZ63QMT4sif T7l03XvYR7V9bQJGvGCgOBiSzUs42it61B+p2mmlaclviwlh6UUH1ZfhVmJqloiKHj2B 5AwLg5B9XPcKQZUG95c+iSRAEkJOBsDwBcbyR8TG66Zij8n+SAFqcNKfR5oMN6ttIkh9 d1IkEwvrEJAj105kLEuriso4hG5f1aJfJRZFN/4+Lk3F8kl7TX7z1r9rIdQkC3Q7J3g6 VP0r8SCReXCVaTxBIHJguuV1lXgRi8gp375UvQCk1xjUHxR7jPJf9f8P8WojQpA0ak/Q ds5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=EWFVvt+f7V+HdpX0Zm4rv8MD+Gw4gLTiE1P+4PE051Y=; b=AKBKScjhcx7SunFg1AJeBJwROqkRChWfxR3d0Gg9fp3AP47uGiS3p1T0A9g6H9Ppum /ULK47oFYiXDStid0EtKnTVZDC2IgW6eVdksmpeONkE7PvSRSYMd5Fs7FJACY1BPQE3V tpF2z5xW3Eup1nG2pBkMhG+51wTCUTQObhx/6fD55UynGVXcLTvKy8cO0MfYzfsMS7Lc qteNjPXDBp7ezRJnaS65Bg6rREri9BRatIbdrfgbTBliqSV2q2i0271V140I5WCcHKQJ B+0+ExUvtTRlAaRMEo8T3e/NGdcAA3A3FnBnSSQNhl4uLWZTYrNsGnOlWHsPl6N68bU1 srKQ== X-Gm-Message-State: AG10YORqPo3Lo5NeYNsYAALiiYh65cDwSDDjycCn9cWGmodyNplEEhjHxBVCKnPywbKXXsICvD0lBul+jMbUbQ== MIME-Version: 1.0 X-Received: by 10.107.10.217 with SMTP id 86mr234307iok.75.1453414592784; Thu, 21 Jan 2016 14:16:32 -0800 (PST) Received: by 10.36.121.16 with HTTP; Thu, 21 Jan 2016 14:16:32 -0800 (PST) In-Reply-To: References: Date: Thu, 21 Jan 2016 14:16:32 -0800 Message-ID: Subject: Re: Flamegraph of 8KB message bottlenecks From: Adrian Chadd To: Ryan Stone Cc: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Mailman-Approved-At: Thu, 21 Jan 2016 23:46:04 +0000 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 22:16:33 -0000 A lot of that syscall time is in kevent, doing fget locking. What kind of filedescriptor is being used here? Just a normal socket, or is it kqueue on some /dev entry or something? What's that "idb_" stuff? -a From owner-freebsd-transport@freebsd.org Thu Jan 21 22:42:41 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B4769A8C955 for ; Thu, 21 Jan 2016 22:42:41 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22b.google.com (mail-io0-x22b.google.com [IPv6:2607:f8b0:4001:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A6861979 for ; Thu, 21 Jan 2016 22:42:41 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x22b.google.com with SMTP id g73so71845839ioe.3 for ; Thu, 21 Jan 2016 14:42:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=NK6Mkis+BUBGo3TdPsEY05QAb8IpvC8GH6xZN95KOzQ=; b=vx2Yu1SEIUe75bH+F58qZNZmjoBhKr8eAR7Dzzpax0QMlBpZRpVx1gj/gCt2ViYMtD xyi1qIiFrM4qhfvp4y4DfKN1vF+IJuu7xVfLJCChp/mnjNvo6vofdtC3dvPo85oE0Hep q2Y58Lj64t84xMvS1DtiyxF+C5GyGzg5zWwiZvtssOplkzAgrkKW+nbZwyLEDJP8bbw1 zAaL06FabcPmrtDHt2m2xGY8apLjBmc3xr38raRP3/FOpnAcMoybNXwUs6prFi/badod GbZ0gtP02ZLfWps9IDihOI6tJ9MACvLhDGG+vnvuLcoJi27144k83WwksPo1gZi2u2mf yIOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=NK6Mkis+BUBGo3TdPsEY05QAb8IpvC8GH6xZN95KOzQ=; b=jJuW1GqdQJUfbIn2f3kauj/I9atEE9MJ/rEaecgInSH/YP7pg25VgHRjMobfsxR97g 7qheasR2xuXJ+2NnCeEIMMqek8hvMDmTXFIFx3kZHS6JSDvb64YcIoOXS5hjonTsPTE4 cIlPcGgcbO0kUVyxZxoc4eORk/MCL31SQxspJIOsipTcoGm0EvOxXKTJAxNgXp21eJQF uo14ck/coTyV8pBz+HY3i06smrvlO66YpYGpWNO1+kECQyLpi6wlt+30w1jywergnUpS VusEqwR2osLkHE3D0q1PgJZ2DV/GcTZEB0Pk3/WJPeXbaN8Q/wzQzCHXNT2pjHipJVM6 K8eA== X-Gm-Message-State: AG10YOSOltF2MNDI3bvwKJKVPdDLTpa2rMffzuwjoElBKeRwUVgg3887OlK6spvV0ciIFyBZOa7UPokFtwFYDA== MIME-Version: 1.0 X-Received: by 10.107.11.162 with SMTP id 34mr299469iol.165.1453416160982; Thu, 21 Jan 2016 14:42:40 -0800 (PST) Received: by 10.36.121.16 with HTTP; Thu, 21 Jan 2016 14:42:40 -0800 (PST) In-Reply-To: <1526652dd58.117a1e7e938176.477104883701987065@nextbsd.org> References: <1526652dd58.117a1e7e938176.477104883701987065@nextbsd.org> Date: Thu, 21 Jan 2016 14:42:40 -0800 Message-ID: Subject: Re: Flamegraph of 8KB message bottlenecks From: Adrian Chadd To: Matthew Macy Cc: Ryan Stone , "" Content-Type: text/plain; charset=UTF-8 X-Mailman-Approved-At: Fri, 22 Jan 2016 00:13:38 +0000 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 22:42:41 -0000 On 21 January 2016 at 14:34, Matthew Macy wrote: > ~170k interrupts / second for ~800kpps, but at this point clearly all the time is spent on syscall overhead Sure, that's why I asked. I see the fgets stuff looking lock contend-y, and I see a sleeplock kicking in via kern_sendit -> sx_xlock_hard; so something's blocking on a lock. The kqueue locking too is a problem - kqueue doesn't handle lots of fast producer/consumer things and it contends badly. :( -a > > > > ---- On Thu, 21 Jan 2016 14:11:34 -0800 Adrian Chadd wrote ---- > > Well, how many interrupts/second is this seeing? > > > > > > -a > > _______________________________________________ > > freebsd-transport@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-transport > > To unsubscribe, send any mail to "freebsd-transport-unsubscribe@freebsd.org" > > > From owner-freebsd-transport@freebsd.org Fri Jan 22 00:00:50 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76AADA8AD19 for ; Fri, 22 Jan 2016 00:00:50 +0000 (UTC) (envelope-from benno@jeamland.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C0651106 for ; Fri, 22 Jan 2016 00:00:50 +0000 (UTC) (envelope-from benno@jeamland.net) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 572F720C76 for ; Thu, 21 Jan 2016 19:00:49 -0500 (EST) Received: from frontend2 ([10.202.2.161]) by compute2.internal (MEProxy); Thu, 21 Jan 2016 19:00:49 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=jeamland.net; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-sasl-enc :x-sasl-enc; s=mesmtp; bh=3peO/6BeGH6kSl8LkZ5YzxWa+vI=; b=jzPjK1 k7JZ7CXaGsDZsqXOxioUs/+nNbgm4sbJ2P4+0semlmK2hy4RxSaAEAJNPt4CYveU 263M/4UKFQXSuTK5fAHhIOLtC1+/3uaV+1lF6trMVvO03BBThlP7g2ELJ8O8qJQw AmctakCjQe0V7mSRnShW7EGVzpBPQ7vR1J/rs= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=3peO/6BeGH6kSl8 LkZ5YzxWa+vI=; b=oa1hCVXR5DldNgPAn3yWErIT3nbfaiAOPTaY24hcGKydXAe MAmQJb1TzlfCw+/TS1hJBbQkk1bqlOJdulqoKKadCrEz4y2QYA8s16OKBpzBT5Re zcq0cTOXy7dTAdG/Kogy8YOVquKA/gzy0ACzlx6myI/ZhH9DJXKjEsLd49WE= X-Sasl-enc: 7NXzI8PBJUPbr/xTuFooqbNyutHNb7wFjdWIIDUTysnn 1453420848 Received: from mittlerweile.west.isilon.com (c-67-182-131-225.hsd1.wa.comcast.net [67.182.131.225]) by mail.messagingengine.com (Postfix) with ESMTPA id 8077B6801EE; Thu, 21 Jan 2016 19:00:48 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: Flamegraph of 8KB message bottlenecks From: Benno Rice In-Reply-To: Date: Thu, 21 Jan 2016 16:00:46 -0800 Cc: Ryan Stone , freebsd-transport@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Adrian Chadd X-Mailer: Apple Mail (2.3112) X-Mailman-Approved-At: Fri, 22 Jan 2016 00:31:50 +0000 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 00:00:50 -0000 The idb stuff is an Isilon-internal thing. It=E2=80=99s a different = storage system for the file descriptor tables. > On Jan 21, 2016, at 14:16, Adrian Chadd = wrote: >=20 > A lot of that syscall time is in kevent, doing fget locking. What kind > of filedescriptor is being used here? Just a normal socket, or is it > kqueue on some /dev entry or something? >=20 > What's that "idb_" stuff? >=20 >=20 > -a > _______________________________________________ > freebsd-transport@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-transport > To unsubscribe, send any mail to = "freebsd-transport-unsubscribe@freebsd.org" From owner-freebsd-transport@freebsd.org Fri Jan 22 02:02:19 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F21BA89F82 for ; Fri, 22 Jan 2016 02:02:19 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22f.google.com (mail-ig0-x22f.google.com [IPv6:2607:f8b0:4001:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 381E21033 for ; Fri, 22 Jan 2016 02:02:19 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x22f.google.com with SMTP id z14so1648047igp.0 for ; Thu, 21 Jan 2016 18:02:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=DbdQg45yBsso2fUbimigHTf/z9c6a025aiD1At3tKT4=; b=EIXLUR2z1vijwhIrNEDAGC/z3CLYef2FTd/pEW96W6WLo6xJysQYgaoyedLKTt3lg/ I0hlKiwLnNchoOEPiROgt552T3n6fy89NS6dhFfubT29hsAs+YbnxR3XAnF20OcQAD45 Wc0cMmzkcrvuqnLdkLOt95jx5UVpaxhHOR95Xy2RZATKtnt9uK+rTx3Cx9ja+5LNUTD4 eL4csvUhiBIkwdoaF60mMR62ohq4IgAa3PiZw1I+/onJABSL7f91gfUtSWZogMP7kaFG RjlZ9oWVO+l75OB26w1W28Xk/dxSetEwjYaU+WCaEIWfQFjj2XplYu4I6fB2ae+GFIF5 Z2/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=DbdQg45yBsso2fUbimigHTf/z9c6a025aiD1At3tKT4=; b=bxCg5VBJ+a/ew3bVx7OpaBn52teR0Wtd7DmmtC7bYelrT/nW9eMLUMHZM1ZKFDcdD/ O8SApYSY3kAMI5am5zUKJ0mOmy9HppnhVO4v7RSSwMnzcL32EIGkZS4wnCEtavrw9Q1J 1mtK9T2RQXi3DOlsFw0HbDxaDA3ez5nIkt0nxZ3eJ3PdbA1xyWFFTYirsVjZ9Q5bSqKr 7DAf8v/H+FPEnufvP3mkWaEXrBek7XPVZX+v0L+imGLsBN1RmRgP8L5iF1ARY2rZQOCM E6dxs+dba1I1fZSbuLudUMaT0tCJ4LHWnoWnvHEDh4LB4w0THgtDAsx2xrFttOOAjd6K /8qg== X-Gm-Message-State: AG10YOT50fKTZ5E2enkkaN+MbLNAt31KGt+O0IIn32jY8bY3za7lS5OBAVuEfEdjcP2KH97dZL3O7TBiNizmJA== MIME-Version: 1.0 X-Received: by 10.50.178.178 with SMTP id cz18mr907115igc.37.1453428138269; Thu, 21 Jan 2016 18:02:18 -0800 (PST) Received: by 10.36.121.16 with HTTP; Thu, 21 Jan 2016 18:02:18 -0800 (PST) In-Reply-To: References: Date: Thu, 21 Jan 2016 18:02:18 -0800 Message-ID: Subject: Re: Flamegraph of 8KB message bottlenecks From: Adrian Chadd To: Benno Rice Cc: Ryan Stone , "" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailman-Approved-At: Fri, 22 Jan 2016 03:04:58 +0000 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 02:02:19 -0000 On 21 January 2016 at 16:00, Benno Rice wrote: > The idb stuff is an Isilon-internal thing. It=E2=80=99s a different stora= ge system for the file descriptor tables. Ok. So hm, looking at that, i'd look at the syscall count breakdown to see if you're doing very inefficient kqueue (eg add, trigger/remove, read/write, add, etc.) rather than keeping it in the kqueue list. But yeah, that fgets in the kqueue path and sleep lock contention in the send path is hurting. :( Maybe get a syscall breakdown, and maybe a lock profile trace? -a > >> On Jan 21, 2016, at 14:16, Adrian Chadd wrote: >> >> A lot of that syscall time is in kevent, doing fget locking. What kind >> of filedescriptor is being used here? Just a normal socket, or is it >> kqueue on some /dev entry or something? >> >> What's that "idb_" stuff? >> >> >> -a >> _______________________________________________ >> freebsd-transport@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-transport >> To unsubscribe, send any mail to "freebsd-transport-unsubscribe@freebsd.= org" > From owner-freebsd-transport@freebsd.org Fri Jan 22 05:18:22 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16709A8C5A7 for ; Fri, 22 Jan 2016 05:18:22 +0000 (UTC) (envelope-from mmacy@nextbsd.org) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 044D41B99 for ; Fri, 22 Jan 2016 05:18:21 +0000 (UTC) (envelope-from mmacy@nextbsd.org) Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1453439898485793.8275248779895; Thu, 21 Jan 2016 21:18:18 -0800 (PST) Date: Thu, 21 Jan 2016 21:18:18 -0800 From: Matthew Macy To: "adrian.chadd" Cc: "Benno Rice" , , "Ryan Stone" Message-ID: <15267c4d33c.1135ce4d741465.1101206746379007944@nextbsd.org> In-Reply-To: References: Subject: Re: Flamegraph of 8KB message bottlenecks MIME-Version: 1.0 User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Zoho-Virus-Status: 1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 05:18:22 -0000