From owner-freebsd-transport@freebsd.org Wed Jul 20 17:29:31 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7F53B9F7E6 for ; Wed, 20 Jul 2016 17:29:31 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-io0-x231.google.com (mail-io0-x231.google.com [IPv6:2607:f8b0:4001:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 752D71A9B for ; Wed, 20 Jul 2016 17:29:31 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-io0-x231.google.com with SMTP id 38so53931472iol.0 for ; Wed, 20 Jul 2016 10:29:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=wyIC6X34Q7tJP2mgPd2oIj8Wy1k7YjNqFKxkFDByROU=; b=B/Y2+Ovle7tWK+Z3bQk+ofRNVsVFk/L7tIYOXImkXOsUXL1ygwRyni7RflW4jTr0VS +rLls390JNhN1DqgUruHt2ufur6fii0CXRLBrEkvnpAXNsjHWrQsKrfouJJWRbYEgag6 rtYm09lnjdvwpmUvRWeEBH9yf83oi7lVvTYGmr+lXHDrsmmlnn4T/aP3pGgzYG+yupGE Vrdh4ksXva61zQQm7BtJvzTn5jshjvURN3hS/jt+93wRb+By2R6B2rLPeR67/COyB9G1 ApI2k98ZGt7p9mzQmZxuWj/nI1BodxPZNzHbujxNaQGG4gGfvwuhAzCQlmGQ88OTBeV5 SSYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=wyIC6X34Q7tJP2mgPd2oIj8Wy1k7YjNqFKxkFDByROU=; b=japIhuie+8Q2NShdilnJztwnzNkgS0dmuC76Kc1mOKgfIwUtE7L6sja/bCAjx82Ij5 YqVb3TGY7LmS5i2qPKixXvU4lstUX/25TljBmI7/AZflX6KS6gOa4qVT50wcHtmmCrR8 dRXpPB9t5F5FzUg8HqGFvFAg5gQpBbQMVUEqBapViGwUxhDHjqP1czm028X1hjaooEjO zrHkZPF2vkkiksFnovPsvlSqSpQ40quAQw3l5gcX2uk5SNgPcNXhVNrm6/xe0ZP8vXcU muXLERtqT1GurE7T4kFaon6e9HPYD6GTlRRp3hhPOliiTv6YaU9YVi4/4V5Xg/6huUja Ka1A== X-Gm-Message-State: ALyK8tLFGVlbe4aNeXimt2m3KMnpxalos9SPgQ7Iw+JW3ajTpE8AIFDC+Jjq92xZo4k1PgAfZ7ROfPlld4891Q== X-Received: by 10.107.129.97 with SMTP id c94mr47032700iod.102.1469035770549; Wed, 20 Jul 2016 10:29:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.200.71 with HTTP; Wed, 20 Jul 2016 10:29:30 -0700 (PDT) From: Ryan Stone Date: Wed, 20 Jul 2016 13:29:30 -0400 Message-ID: Subject: in_broadcast() called for almost every packet in ip_output() To: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 17:29:31 -0000 I've hit a dilemma in a fix I have in review[1]. Currently, in_broadcast() iterates over the ifnet address list without holding any locks. I can pretty easily panic the kernel by constantly adding and removing addresses while passing traffic over the interface. The fix is to acquire the appropriate rlock, but the catch is that according to dtrace, in_broadcast() is called on almost every packet that passes through ip_output(). I'm concerned about adding additional locking operations to the transmit path. Do we actually need to call in_broadcast() so often? It seems silly to check whether we are sending to a broadcast address on a per-packet basis. [1] https://reviews.freebsd.org/D7227 From owner-freebsd-transport@freebsd.org Wed Jul 20 19:00:52 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AAF7AB9F56E for ; Wed, 20 Jul 2016 19:00:52 +0000 (UTC) (envelope-from gallatin@netflix.com) Received: from mail-it0-x229.google.com (mail-it0-x229.google.com [IPv6:2607:f8b0:4001:c0b::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 767B5105A for ; Wed, 20 Jul 2016 19:00:52 +0000 (UTC) (envelope-from gallatin@netflix.com) Received: by mail-it0-x229.google.com with SMTP id j124so59224613ith.1 for ; Wed, 20 Jul 2016 12:00:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=wPyr2t3wYDveshK11H2cClRAPtxuumW187ON3kdfVck=; b=GUOtjcqascREHrPH/2O0XEWPZsqxKd2unSgGFl6+lsspxX6kQdC5kJ2Dy6djQC/rMd iG0stLaiPQfcQGzIt1UUDlZYLJrWgiRAswKwnktUfm7Hw0KnEKWKl/43yWAyKOSJpixe T1EFaWTRucs3Jf0rhnYsI9GLo5HgnBBfZYKGA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=wPyr2t3wYDveshK11H2cClRAPtxuumW187ON3kdfVck=; b=Y3/QMRPUc5Z71wMpjWexeqkEd+cEQCFrUaN4zpm7+WPmOVWxQYlxM3W7sFdi5O0kt9 IDJeo4RWkQw9LQnM4DeAeUjolcDnvAEf6P1VNZHV8/grRyJU2klDM87s7i6D6W5edxXS Maht+CnvHSXJMRoJxJmtMHRNXd1ZXkrVGFE8K9PB+A7O/pkQJo8bP97r+jDfG6Ggz64C nj7rni9Tr3CnzYs6U6y6ijd5eNxnqrJcbMfXcWcpt/4d7oobvsoeYEC9ZBEBey05vgQt 2DK5E2ddkLEDJ3Rd5b78AMpp/CEZeVJQV94BbFusUqq+/yuUEBIQpwwJ/i0V5s+1UIyZ l58g== X-Gm-Message-State: ALyK8tLYAsl1ccKSlKF01ZZbmxMEJ80V9uW8fKYFtWClNGsbTEUY1YQXAeFsfM4u3quDTe9RYy0/MGLpSyBqsrxZ X-Received: by 10.36.19.16 with SMTP id 16mr63155577itz.41.1469041251871; Wed, 20 Jul 2016 12:00:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.41.163 with HTTP; Wed, 20 Jul 2016 12:00:51 -0700 (PDT) In-Reply-To: References: From: Drew Gallatin Date: Wed, 20 Jul 2016 15:00:51 -0400 Message-ID: Subject: Re: in_broadcast() called for almost every packet in ip_output() To: Ryan Stone Cc: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 19:00:52 -0000 I'd certainly prefer not to add any overhead. Properly managing this (and other similar) lists is a job for some lightweight lifecycle based mechanism like concurrency kit or rcu. Unless you have a solid reason to fix it, I'd suggest just adding the locking commented out (the way rwatson did with IN_IFADDR_RLOCK in ip_input(), so that the next person to trip over it will know what's going on. Drew On Wed, Jul 20, 2016 at 1:29 PM, Ryan Stone wrote: > I've hit a dilemma in a fix I have in review[1]. Currently, in_broadcast() > iterates over the ifnet address list without holding any locks. I can > pretty easily panic the kernel by constantly adding and removing addresses > while passing traffic over the interface. The fix is to acquire the > appropriate rlock, but the catch is that according to dtrace, > in_broadcast() is called on almost every packet that passes through > ip_output(). I'm concerned about adding additional locking operations to > the transmit path. > > Do we actually need to call in_broadcast() so often? It seems silly to > check whether we are sending to a broadcast address on a per-packet basis. > > > [1] https://reviews.freebsd.org/D7227 > _______________________________________________ > freebsd-transport@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-transport > To unsubscribe, send any mail to " > freebsd-transport-unsubscribe@freebsd.org" > From owner-freebsd-transport@freebsd.org Wed Jul 20 19:09:36 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B9D7B9F7F7 for ; Wed, 20 Jul 2016 19:09:36 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-it0-x236.google.com (mail-it0-x236.google.com [IPv6:2607:f8b0:4001:c0b::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 152751729 for ; Wed, 20 Jul 2016 19:09:36 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-it0-x236.google.com with SMTP id f6so58662728ith.0 for ; Wed, 20 Jul 2016 12:09:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=xYNeUbasf66NuWbP1g4BVnra6eiuMLOJKZxS1yFyPJc=; b=An+9+fVsTHJqDNR35yhJoZ+9aWk1dcxvXylVl90zvwpTAfGN9tMUaXJV+6rUBGdJeX IlSHtkjYiXdowew3xWt2IeHO569hVSZXgIf8VVQHF9N3Bl7xgejJHEGTuW32nxpr76Kr UgsTh14JkFRF7CsTDeQAkDvMolB0PglexQa1HLqtIoKVfMpI9ieq/SxC/o9fdA17oOx4 cWDq59KqABVIrVOJaZnYoYrJYyDXF9OE67Su/5AJB2ekp8XOvs8YOWchqknXIiXX4yYT voS5yHSJ67fy/3fmnCUGPwQ5NaOHbY9kCl91HvWTRTMh27wqMy4NZovJofuKq+X8GYLJ OnmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=xYNeUbasf66NuWbP1g4BVnra6eiuMLOJKZxS1yFyPJc=; b=MRv1B06S9z47Ppjfydb810ZiewgyPdD9H1PvI9R6+bfE8sqO9NVJdH0aQ8E0t+YaKB SSvdW0AN9693S+id5cWZi9dkUnZnrAkf/WoJ29ukU2uCTM4oBE54xRevaX1k4S03q/AW pamEJfK7EP7kTDFisMxuli3U2qQVu9QKj8tWrEnR+Q6F4Z3oxGkvNC1rGzaldUeDNCdq 50UMWJXtjZUyf2KIe5Muq0Z4H3eIRnLH54uSWQKXBG/XnIB0jBLE80GXIW7wR2GqYIVS MvSvHUNQ+eYMAXLrpSJny/TmHeBR5i5C5kZn7m6ENGQhqSH0XXZOEuGkRlGLfx+XGA9p 2XAA== X-Gm-Message-State: ALyK8tI5302AUL28AVWHZihiECqcXYsHWmyWVPOTCTnGzYYyTEugpbPrfLIb7TxDhz2dBgsQBoNFSpfZ+aTY6w== X-Received: by 10.36.21.65 with SMTP id 62mr11821594itq.20.1469041774635; Wed, 20 Jul 2016 12:09:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.200.71 with HTTP; Wed, 20 Jul 2016 12:09:34 -0700 (PDT) In-Reply-To: References: From: Ryan Stone Date: Wed, 20 Jul 2016 15:09:34 -0400 Message-ID: Subject: Re: in_broadcast() called for almost every packet in ip_output() To: Drew Gallatin Cc: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 19:09:36 -0000 On Wed, Jul 20, 2016 at 3:00 PM, Drew Gallatin wrote: > I'd certainly prefer not to add any overhead. Properly managing this (and > other similar) lists is a job for some lightweight lifecycle based > mechanism like concurrency kit or rcu. > > Unless you have a solid reason to fix it, I'd suggest just adding the > locking commented out (the way rwatson did with IN_IFADDR_RLOCK in > ip_input(), so that the next person to trip over it will know what's going > on. > > Drew > I'm not clear on the details, but we have some internal tests that were provoking this exact crash after a couple of days of stress testing. I believe that the actual fix would involve caching the result in the pcb. When the endpoint is already known, there's no reason to check for a broadcast packet on every single packet. On a system that uses a large number of IP addresses on an interface (e.g. for jails) the overhead of just iterating over the list is going to add up no matter what synchronization mechanism you use. From owner-freebsd-transport@freebsd.org Wed Jul 20 19:22:23 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FA54B9E132 for ; Wed, 20 Jul 2016 19:22:23 +0000 (UTC) (envelope-from gallatin@netflix.com) Received: from mail-pa0-x232.google.com (mail-pa0-x232.google.com [IPv6:2607:f8b0:400e:c03::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F54F1660 for ; Wed, 20 Jul 2016 19:22:23 +0000 (UTC) (envelope-from gallatin@netflix.com) Received: by mail-pa0-x232.google.com with SMTP id ks6so21072095pab.0 for ; Wed, 20 Jul 2016 12:22:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=3puGC85BXxhFT0/pHgJyJgiPr7g7SvkCRnkY1D61G2g=; b=f+SM1QWH1rTowvfu5uZsUcaZl5dgR8KUdGMlCz+X32KKW0nJMMjYQYYVOiQQi36dmk oxDRcS8NDlC/KYymuE2ygj0VEspeXG+RDLqF560PTjMMA0orlKJ46Dz+rgVJRdRhTACB 1OV7uKYrZonBs8p/Gq7opB2V+cDzn0QV8LRGI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=3puGC85BXxhFT0/pHgJyJgiPr7g7SvkCRnkY1D61G2g=; b=GVJ+Ow8jp8+e1xMMgpfeAi0Th614781eU+uJx4oAmnJ5FX/yakp0J/yFiIFtgV0cGp VMqxn13Daov5SYZnR/qOTr8c5pVIKPJuP1ebHwtsFYgveffNWAgxtu4DTylKEKOwGti7 jtLdqXEcs+9IIeEhRBspM7kIgnz+JptekYUD/S0mHuYVeqez+uuKrerdyJ+qGXE2zafr gUZM+LuuwZMyPHwXRXfqzqFPUK0FUFkecz8PMllj7V7+qqcrZpoI1fV/43RGIWNPnPOb GY6Rj37kXC2wSagtBKB+2CQy7KRYCcTjQW4ulvsc9JSh2t7Gj8WeqVuDS1YPo3bsWCak VQqg== X-Gm-Message-State: ALyK8tLjI9QvJvhgngxNYz3tHtcpJDAO5aQ4nHlDKKBi+M/ycFOndVwWNaNirPsFs2JKZsYi X-Received: by 10.66.242.201 with SMTP id ws9mr77017243pac.7.1469042542477; Wed, 20 Jul 2016 12:22:22 -0700 (PDT) Received: from [192.168.200.3] (c-73-147-115-187.hsd1.va.comcast.net. [73.147.115.187]) by smtp.gmail.com with ESMTPSA id h66sm6432208pfe.58.2016.07.20.12.22.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Jul 2016 12:22:21 -0700 (PDT) Subject: Re: in_broadcast() called for almost every packet in ip_output() To: Ryan Stone References: Cc: freebsd-transport@freebsd.org From: Andrew Gallatin Message-ID: <559f1d0e-9d35-f458-bff5-b69a03297cf0@netflix.com> Date: Wed, 20 Jul 2016 15:22:20 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 19:22:23 -0000 On 07/20/2016 15:09, Ryan Stone wrote: > On Wed, Jul 20, 2016 at 3:00 PM, Drew Gallatin > wrote: > > I'd certainly prefer not to add any overhead. Properly managing > this (and other similar) lists is a job for some lightweight > lifecycle based mechanism like concurrency kit or rcu. > > Unless you have a solid reason to fix it, I'd suggest just adding > the locking commented out (the way rwatson did with IN_IFADDR_RLOCK > in ip_input(), so that the next person to trip over it will know > what's going on. > > Drew > > > I'm not clear on the details, but we have some internal tests that were > provoking this exact crash after a couple of days of stress testing. > > I believe that the actual fix would involve caching the result in the > pcb. When the endpoint is already known, there's no reason to check for > a broadcast packet on every single packet. On a system that uses a > large number of IP addresses on an interface (e.g. for jails) the > overhead of just iterating over the list is going to add up no matter > what synchronization mechanism you use. Caching sounds reasonable, but.. won't the lock still add overhead to paths like unconnected udp and packet forwarding? From owner-freebsd-transport@freebsd.org Wed Jul 20 19:50:06 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F31C6B9E9BC for ; Wed, 20 Jul 2016 19:50:06 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-io0-x231.google.com (mail-io0-x231.google.com [IPv6:2607:f8b0:4001:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB65518A9 for ; Wed, 20 Jul 2016 19:50:06 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-io0-x231.google.com with SMTP id q83so57315735iod.1 for ; Wed, 20 Jul 2016 12:50:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=DvhbfvdPhjnV4SVTGL0w86RMKPhk6niejJun6mexl2U=; b=u3z4PoQ/d1pd9/25+Dd0j9ejXgf6u/M2IjE2gjF5OsGSuBaHoN1ev6dQA+7VkOu1sJ KCyepATLhkXuKPL8w6WJGfaXAGl42A6t+6qFuOhsVYlWe84P3QjECkU/MEEcTufjSm6l 38fjXBpWfJ8i5zsvC5RNQj48tb80mjZ34zeyjp3B8C4zOKViLzRAahZMh4+x7bJp4p5B Qa9qKrma4sC+VIyctLB7YC2OYCyW2ajq+A/Po3A2DLXG2DCfNBXbyTE1XyltJ8Fr+CHD WzBEjG+iZdUsdu2JvHg+OZyMqZpzDWhFcl+5l4qjucFCIFRGQoF7yZT0VotYbaTVq+TB wP7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=DvhbfvdPhjnV4SVTGL0w86RMKPhk6niejJun6mexl2U=; b=fR0hzpnmBg7GHd403GGKoQ+rMTEkTgyv9U05Dm+VSuSt28oUYjAoMLxZxdy9ciBf/H NJ2+KygMmGNzKPcYPRFcNYolxNdTid8vHYHdivyI9sp9vdWCnVugFyINjDsYNLLIgYk3 xgUIc1OX4ydzPI6xon/9wQzviU0IhNwQRNsO0PAEA8dPkq548zJ3lFszT4b0EatPaE47 9KXqUPFhHlEPgV4vPp0VmRBz9WO9WleECBD/BepZG8Mp5JtqZVSfVB7jDZ2RXHNPABKG xwbj9Pki9FVzqj+Oux3yvOejlrWJoutrR5b9N2tp9MT+D1pyp0TokAYoQ/d2NO5JYwSR ahKA== X-Gm-Message-State: ALyK8tIpdIulTAmoNLvcxzMEzWTQa4UhbjQRrE9akTFfYJ0BrmTaUWOxrUx+/kytm+e+hsNX/tk2YyICFcg5Nw== X-Received: by 10.107.159.147 with SMTP id i141mr17681132ioe.29.1469044206271; Wed, 20 Jul 2016 12:50:06 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.200.71 with HTTP; Wed, 20 Jul 2016 12:50:05 -0700 (PDT) In-Reply-To: <559f1d0e-9d35-f458-bff5-b69a03297cf0@netflix.com> References: <559f1d0e-9d35-f458-bff5-b69a03297cf0@netflix.com> From: Ryan Stone Date: Wed, 20 Jul 2016 15:50:05 -0400 Message-ID: Subject: Re: in_broadcast() called for almost every packet in ip_output() To: Andrew Gallatin Cc: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 19:50:07 -0000 On Wed, Jul 20, 2016 at 3:22 PM, Andrew Gallatin wrote: > Caching sounds reasonable, but.. won't the lock still add overhead > to paths like unconnected udp and packet forwarding? > Forwarded packets will never be broadcast, so checking for a broadcast address would just be redundant. Unconnected UDP would get overhead, but I'm honestly not sure how many people care about the performance of unconnected UDP. From owner-freebsd-transport@freebsd.org Wed Jul 20 20:28:59 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61733B9F823 for ; Wed, 20 Jul 2016 20:28:59 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-io0-x231.google.com (mail-io0-x231.google.com [IPv6:2607:f8b0:4001:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 24DD2199D for ; Wed, 20 Jul 2016 20:28:59 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-io0-x231.google.com with SMTP id b62so57978741iod.3 for ; Wed, 20 Jul 2016 13:28:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=bGNbzuhqDKpVnkIqJYkAEszjlCgpx4RDp8eeMWHFmKc=; b=RRwHpWK6gCRoYys6M1zO8XT/lNHBXiUYZLLAuIGy75kZAtgyjrXhn2P+VBivs3QclY MPT8LX3cQxKgyxTggRY3BVxda4X1SfKrDtbc2bR30tTR4KzXvgtnwGXk9+1vWCF9aIBJ 7o19DxnNtsm1eqm/IroFHPa+NrieYSqye5ZDPOy/4m8dIp0sfwOQmD40nzxgsNOtMRV1 XJdQ4yBA2V7AeaeG8K6qmZJjBjXiNH2RGx5N3UQTcp96YhOqU2gaeir71iJ9TSvwLt3c 5nSUjDtLFWJ/1555iSbPlT2w/+bOFBZp+gdopHvhhgUNz73R24F/lDz0L/up9Of85ml1 dFyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=bGNbzuhqDKpVnkIqJYkAEszjlCgpx4RDp8eeMWHFmKc=; b=kqT2y0LB05j5HQxwGn5mcrBzz0bPjjr7YWLLUWF/qV4R3agLDFF74uxlZGhvtEaY0Y iFVvRmW2TLbU7hq7lmC+6JiAMu/mR1DK+Up10W69pOpVJJpndxbLdJ76654jQdmsRlbP Q0YziUIXczKZAuQbMZ0G35NJ3miXYrSU28DcIgqop4xCyJscdIcMO1z4R858z6GhCmJ3 wHPDU8/sTwA9W7Kb+4lznqfiEae/FPrn+IgFga4/7rFwHrZlt4axqgx7PTaMjRMqG0nI TbZCzw1B6ov3usENqURpyCowqY/W7go2Zz9lYTpP+sIG7r6rplcDGqxL6ff4cC03yxib EPRQ== X-Gm-Message-State: ALyK8tJKzGc9QDnM4N0ZihT5khBWsHCx6NweZYGNV/00W0UL/9ItRDC9lkt+d/SD0NYOTOjOsKuNZuTljpmwZQ== X-Received: by 10.107.159.147 with SMTP id i141mr17830341ioe.29.1469046538613; Wed, 20 Jul 2016 13:28:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.200.71 with HTTP; Wed, 20 Jul 2016 13:28:58 -0700 (PDT) In-Reply-To: References: <559f1d0e-9d35-f458-bff5-b69a03297cf0@netflix.com> From: Ryan Stone Date: Wed, 20 Jul 2016 16:28:58 -0400 Message-ID: Subject: Re: in_broadcast() called for almost every packet in ip_output() To: Andrew Gallatin Cc: freebsd-transport@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 20:28:59 -0000 Hang on a sec. At this point in ip_output(), we already have the route. Can't we just calculate the broadcast address using the route and then skip in_broadcast() entirely? From owner-freebsd-transport@freebsd.org Wed Jul 20 20:41:20 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5AE13B9FECE for ; Wed, 20 Jul 2016 20:41:20 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x230.google.com (mail-io0-x230.google.com [IPv6:2607:f8b0:4001:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2272F143F for ; Wed, 20 Jul 2016 20:41:20 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x230.google.com with SMTP id m101so58305865ioi.2 for ; Wed, 20 Jul 2016 13:41:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=D1g+WhcgJAeqr4pWX5Co/+Hk22vFlC+SphVgHG8sanE=; b=iVZrZhEYy9H8JjTHhW+xrgZ+/EA52WUxTTJFR7S1WBI7oKntLsA9qeNL8DaQa7io6T MyoSdGzTAaCj17kkykPCQp9+QWvURLfJz8KraSW7O81RxZxQ0Ja5+PopK7g5I4L7xfQR 1crCLJXXIMO9maFD4e0b3My6pCS0b+cHQ76x4xBV1lQduxO/8eZeMOPcd5Db3HXGVl1j swI+ofNeDlYc0Gkb+0jeiaMBeamZUm+ZUJ1OH3qTCeiEkC+yP8UFJAzqwz9N+teivsSB VHApAkM1iatSJzsFF1VehsLcc74uHvog8jbe66FNl8HRBwOd6p93cWgYefKtfvk4tTJz FjNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=D1g+WhcgJAeqr4pWX5Co/+Hk22vFlC+SphVgHG8sanE=; b=USAStYkcHwjqvVy2LxQiJBHN3rd0j4Yzpyf7YE5oShEpkTAerYqbBYKS2waonPWK66 5zNx7B8IIS02p4mmV9g+dL9PghMJb1Ec1I1EI2waMOXpvb91cu4h6EbsFf6E3HfC95Ua WQLFhS+pinfytI7SEWdR7TtaT9T3Nsr/x4doxZsPc5SMoRjshWaNgWe3RduAihc+Z5gO kByRAEUuh7jL3NuNrjl+DyJ51S/JN4Ilur3AYJbmOfZMd4AcJDlPECMrReC7UxASJ5yD fAk/JPBA/EOo6AbRzfzQMLB7k1lpQyG4oJG4NW1z1j8UXxLx47kFkUZvD1Ru6NJB7GyZ QoTg== X-Gm-Message-State: ALyK8tK5QbJGYIYAZ0SKVugvJEkmQraQlrGFRWroTlynt+kxe1bUjztEZxRD6UIaq5xvfrPkbn5JyBf+/eyXSw== X-Received: by 10.107.15.157 with SMTP id 29mr28091403iop.123.1469047279491; Wed, 20 Jul 2016 13:41:19 -0700 (PDT) MIME-Version: 1.0 Received: by 10.36.141.129 with HTTP; Wed, 20 Jul 2016 13:41:18 -0700 (PDT) In-Reply-To: References: <559f1d0e-9d35-f458-bff5-b69a03297cf0@netflix.com> From: Adrian Chadd Date: Wed, 20 Jul 2016 13:41:18 -0700 Message-ID: Subject: Re: in_broadcast() called for almost every packet in ip_output() To: Ryan Stone Cc: Andrew Gallatin , "" Content-Type: text/plain; charset=UTF-8 X-Mailman-Approved-At: Wed, 20 Jul 2016 20:58:39 +0000 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 20:41:20 -0000 Hi, I care about the performance of unconnected UDP; it's how a lot of UDP network services work to arbitrary clients. It works fine and parallelises fine in the RSS case (as in, linear scaling to num-queues; so I've done millions of pps through that path.) The in_broadcast() overhead sucked; I found that out when using IP aliases on an interface and discovered the in_broadcast() check sucked time if you didn't bind the local address (forcing it to an address lookup/determination each time, for reasons I forgot.) So, ideally we'd just do some two-stage caching mechanism using atomics, rcu, generation count, etc where we'd do a cheap hash lookup to see if we have to do the slow broadcast check, and then the broadcast check itself would be a hash rather than a linear O(n) list walk. -adrian From owner-freebsd-transport@freebsd.org Wed Jul 20 23:57:40 2016 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0D26B9F7EC for ; Wed, 20 Jul 2016 23:57:40 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-it0-x235.google.com (mail-it0-x235.google.com [IPv6:2607:f8b0:4001:c0b::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 965E81C84 for ; Wed, 20 Jul 2016 23:57:40 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-it0-x235.google.com with SMTP id f6so2300707ith.1 for ; Wed, 20 Jul 2016 16:57:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=cCpjHNs6CHIiGjrCLmRsYXn3QDhw+VyQ5MF6K9zwmnk=; b=JopOF1FxiHQtY4dFJSAabzb9u/oRbbyTfv40uBtfKp4ORHoE/KMfoq9/hQ3hbjmplj iw9nXL3dNSXp789A996NmIeEqrAZ3HqQKirskVYwzkZ1fWsfJ/9ZVbJPDPZ5Z1BRCbV9 r6ZRrc0tw5PYfATP+kv4G7i5pJT25ppcULqMcUAImBM0fGxoQCMlJcUCtAEWY2+WKTmp yNfr9aYWr2Kyir8TQJGK4U8cwhPFnn351/QV/oPyVjEVNKHrgKj6hIMzUT2gmDtjxMa2 2N3X6pYqwCNMp0Fs3PT7tyAlj3rZSzzCiCXxRg3XeZZy2f+9I6ySNL/TK+eLlmnXM23v XT9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=cCpjHNs6CHIiGjrCLmRsYXn3QDhw+VyQ5MF6K9zwmnk=; b=kB2aWcSaysZ3nmil+49m4X2kvGcrSmPcNSptlm5a77McdLrXhrPPC9iKwA7oN/sjSo MVYhhszi1RusWlK9CV/kUXWxqlQGFIC6gSP2L4ikhaStE4UPkfVPb0W6sfWKJoRJhtkE wuA71zMpTO9z3lCrW61mj/Lx9UhgHQ93/BgVTw1vchcwfiArTO2ffjmq4ASbdjGF5c1J ASuso24DDe5cSsw2iy1i23pO7sIl87s4QHp3lnFxDguYA2J1uEt71odEeR6aO9lT6GHn tt5WRXL/tGPdPS8bTNg3+ubUrKGuHd+uApm1vCq84qqpyZAWPQbUcmt6bWPdGpcf3Blr 4uqQ== X-Gm-Message-State: ALyK8tIkejx4oHU6bW8oGdBYHPOQ7eQIQyHTb18IPb5crki1BxNFfFOq1l8NoKYSP4Tc0p01+KP8lArLQ1Rh2g== X-Received: by 10.36.21.65 with SMTP id 62mr12918399itq.20.1469059060109; Wed, 20 Jul 2016 16:57:40 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.200.71 with HTTP; Wed, 20 Jul 2016 16:57:39 -0700 (PDT) In-Reply-To: References: <559f1d0e-9d35-f458-bff5-b69a03297cf0@netflix.com> From: Ryan Stone Date: Wed, 20 Jul 2016 19:57:39 -0400 Message-ID: Subject: Re: in_broadcast() called for almost every packet in ip_output() To: Adrian Chadd Cc: Andrew Gallatin , "" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 23:57:40 -0000 Ok, this should address the issue in ip_output() for everything: TCP, UDP, forwarded packets, raw sockets. I've tested UDP and icmp traffic with both broadcast and unicast addresses and everything seems correct. https://reviews.freebsd.org/D7266 Unfortunately, I also notice that udp_input() calls in_broadcast() on every input UDP packet: https://svnweb.freebsd.org/base/head/sys/netinet/udp_usrreq.c?revision=301717&view=markup#l524 Would it be correct to check for M_BCAST on the packet before checking for for a broadcast IP address? I don't believe that there would be any security concerns with that approach. If somebody injected a UDP packet with a broadcast IP address but a unicast MAC address, we would try to look up a pcb that matched, fail to find anything, and then drop the packet.