From owner-freebsd-transport@freebsd.org Sun Dec 20 01:36:46 2015 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 436E19D3E67 for ; Sun, 20 Dec 2015 01:36:46 +0000 (UTC) (envelope-from jtl@freebsd.org) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0114.outbound.protection.outlook.com [157.56.111.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B679F1DD9; Sun, 20 Dec 2015 01:36:44 +0000 (UTC) (envelope-from jtl@freebsd.org) Received: from BY1PR0501CA0031.namprd05.prod.outlook.com (10.162.139.41) by CY1PR0501MB1386.namprd05.prod.outlook.com (10.160.148.140) with Microsoft SMTP Server (TLS) id 15.1.355.16; Sun, 20 Dec 2015 01:36:36 +0000 Received: from BL2FFO11FD021.protection.gbl (2a01:111:f400:7c09::197) by BY1PR0501CA0031.outlook.office365.com (2a01:111:e400:4821::41) with Microsoft SMTP Server (TLS) id 15.1.361.13 via Frontend Transport; Sun, 20 Dec 2015 01:36:35 +0000 Authentication-Results: spf=softfail (sender IP is 66.129.239.19) smtp.mailfrom=freebsd.org; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=freebsd.org; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning freebsd.org discourages use of 66.129.239.19 as permitted sender) Received: from p-emfe01b-sac.jnpr.net (66.129.239.19) by BL2FFO11FD021.mail.protection.outlook.com (10.173.161.100) with Microsoft SMTP Server (TLS) id 15.1.355.15 via Frontend Transport; Sun, 20 Dec 2015 01:36:34 +0000 Received: from magenta.juniper.net (172.17.27.123) by p-emfe01b-sac.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.123.3; Sat, 19 Dec 2015 17:36:33 -0800 Received: from [172.29.32.216] (farena-sslvpn-nc.jnpr.net [172.29.32.216]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id tBK1aUD44362; Sat, 19 Dec 2015 17:36:31 -0800 (PST) (envelope-from jtl@freebsd.org) User-Agent: Microsoft-MacOutlook/14.5.9.151119 Date: Sat, 19 Dec 2015 20:35:45 -0500 Subject: Re: Extending FIBs to support multi-tenancy From: "Jonathan T. Looney" Sender: Jonathan Looney To: Patrick Kelsey , Ryan Stone CC: "freebsd-transport@freebsd.org" , "Gleb Smirnoff" Message-ID: Thread-Topic: Extending FIBs to support multi-tenancy References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; BL2FFO11FD021; 1:snAa+MILYgMUWQ/smycFrS6X7UeMYB206eVnd2ehmo2qrr+vChQHhWx5N4ih3QANlTBqYuwv97lgapcObO5T3erxXtoC0mQdiAnJbYV5NfByjkY2/gtA7Nl8ivJQ/b/4znqrSTn56ngnJVvMhOCsXkJtibWd7bolRmw0yMqofflbpx805Q3KRbXWb61NaoBOAOnUmtpnNSQB6zVpObjL24itHcRvnPCX9dCSQ6iuzqlO/0aSWdLSLDZNkHTL9f1V2pIeG2fJF+UrYof+nOTGYpERDq1LlA4l8/tGkXdorCUvdqWao4XSKzcJlXAPInPOVSR+sUDxAMQ2VFGKmIHvaXtAn5rz8gQLW8lc2mcDpkg3tQZun7aUmmxOHcerxizi X-Forefront-Antispam-Report: CIP:66.129.239.19; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(2980300002)(189002)(199003)(24454002)(377454003)(46406003)(4001350100001)(83506001)(69596002)(19580405001)(50986999)(47776003)(19580395003)(92566002)(77096005)(2950100001)(586003)(86362001)(6806005)(76176999)(81156007)(5001960100002)(105596002)(97736004)(16796002)(189998001)(230700001)(36756003)(87936001)(106466001)(1220700001)(50466002)(5001770100001)(23726003)(1096002)(54356999)(42262002); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR0501MB1386; H:p-emfe01b-sac.jnpr.net; FPR:; SPF:SoftFail; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; CY1PR0501MB1386; 2:khiGsmX4Bv8VkQLXpzV7HefEoIo0r9BqLZginNirNmlIKZgk3BeSCNpe3dwHtZK3xPyxM8NdufM9kYjwBQl9YNt395vgy/rVTpR4IS6yLCbktQYff5BIS9S9zantTvjaaxKoOnbeLdqmKzeT9OpnMQ==; 3:hoNLGmvyT9bhnzyELWmm0QkvOcsdD4FNe4rjj4xNO7Q7vR9BgxD8QYnX2O3JpbRWuhqsokwO/4eMMmyfi/NRM3Q70lAsmiPmGuFryhK41R8Q9aI6ZSwF+AWCKNXg0uq+4IUzLb2llJ5++Oc5vBesKSIHDWn6dMv7lxE9ipnXw4fk9+IdxPAAnbuJPQ6DojEnPF4bXN5fn/G6WYDyj97dY2dC3d5yJGiFTZufULtDFJE=; 25:PFQ9oFTOQjKzXm6dCtPJ92fXEw5VZzZKM2/vW1b65lor/rpuN8mRoZcIJZ4NiPEkfPLUC+0GRzYuOjaGrx6ak5ZxH7F8xfl4o21YMH98g3vzhVFozBuiwtm1kLj/23Tdbwl7KpfqbDvQxSAy651gJzdz8+SKQNncHUe+UO3yTH8YwUBCUnl4qM+vwJzcrOzn96q3B6wXObKbVR2jTTlR5zUIdbAj95xD2J6XEUQHUGk9YBLuy2NCm+3eYRX6RUIz X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY1PR0501MB1386; X-Microsoft-Exchange-Diagnostics: 1; CY1PR0501MB1386; 20:/KsL6K4+tqv1+ASN5py9dnz48EEj9WKTNokzvQAJC/l0PcNXVIlVPqYTJ2IEZOSFvrv54YI9cTep/fuqCs5KH8K6YyNKohD1V5FO80mz3k2VPrJJjd/8Rn+0N9RYXYRa3I6ef1sjMqu2/Z9jRXnB7P2u+FtuzB84ni5aoeBmRBxNpgi7Ty/1V/0giX4RI/aMowNnTpz6X8WH0TxxQD6QSkcvt9JPyZBpXU7YhchYp5oO6pdUU3MVkeaVj3XsK7jHGofwRJp3iTAISTgMDnIumxuYw0oRrdZG3BfRum8AlZWUEOgVkW/Pkq0d96KyyF7EfTRO+cgu5geE2cVMQB1/RDPwCizLsKlgA4u6wPTGAqgA3tWVS8wtLRWS6A6X7dsdqAP1dwC6XWqPaNOEwj5Il+8R7DjIJul2IIknc8RNT9CPlSPTVkYyGjOhpgcBtwfyNEGzmx9OvI5A1cGnX91aZgTRpDFa6lv5PnT4AtBqdCkSwl1h6kMt52qavjqgk7z0; 4:PtvN95opHWsMtmecVXOHqcITZ+EzQrxKl6I2C93VkN+tr8PNgaHwzr+hXliz2onBn3FHtqx4MRpGGzpMb3ToG4IaTt8Mcdxtwy9uxzDNN46C/tL0aL4wjxRlBedjr6PgbMz/+2C9oi1r41FZbaYUfsrVkaRUbdj2RpdlPO6dpQyLE2dqbx4SImuJT6kcu82XQ9Jr39mRxBAfld/InWDIGp/KskMdlJKxwm4VCj0hTVj+MAT3jBd3wexnzyU9ryeitXrJDlY89T3wKJ+FrdAbUKdn6Poua05W1sHaWgpZ61rs5WgTDXi5ESaIdBEd9FBzo9GcS6kEO/l5l/Hh7HmLiXWmvb6BtE8kovWR9h9viWYk93foj8sEmUz8hrk+mtAu X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(3002001)(10201501046); SRVR:CY1PR0501MB1386; BCL:0; PCL:0; RULEID:; SRVR:CY1PR0501MB1386; X-Forefront-PRVS: 0796EBEDE1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY1PR0501MB1386; 23:2Xxb/jDVZHFJSRbSMnMs5CgBJdp0x8hyP8J7mIz?= =?us-ascii?Q?QD4o4KDLG+smaR+w8a2ihmYIxzXKpwVkuib8P3T4BjqxMycpzU9iMyp1HGPc?= =?us-ascii?Q?0zFHHWV5q9A5WUnXg7DLT0SjYKMH8cp7IAbuDF2vlRmq5JWrlpkGOnMGEOQa?= =?us-ascii?Q?U71cfwaHzXtVO5knB7CPwAlkl0UDOYIu3lhy/eMAz6rTbaOZMOWp8MLCboJu?= =?us-ascii?Q?jHiAKQRBJvyISyDgaKgz+czLU9ZPAoQNf4wKu0Zy8kuse4jBrTzlkylzFYZn?= =?us-ascii?Q?qDlk86ABQ1/ZAZg7RX7DDOe3lcmPr22WsdvxJtkKSF0UgXsKc/a1ZX2Yszsi?= =?us-ascii?Q?raHqsia4LHUieGG/bFjMnm9QRb+jjgKSesDCrYZa5EhpQRcSleJP573U9roo?= =?us-ascii?Q?Tg0Jsk1zXTvhi78vNLvgD+aSaFZMYP1hGP6gB3tXEfP3KSEboAnbrMS3/7rr?= =?us-ascii?Q?6afv6uINOvE5IGVO2ozeMUHm+9ii7StiSFxh/n3lO7rwzHcoi0iN9ZVaJ/Oe?= =?us-ascii?Q?ZkeFw3CShQ7TY+qUpUaH8u+ZGQYX8Nq1W1o6EeRkSkIa8cBHdfZ7ZEBeE2Ta?= =?us-ascii?Q?1DxAueKLaZP/IR1uhTHh7g5VmV/+KsNM/OBCDtE/HkbaIqwJd+h3H1+s/qFz?= =?us-ascii?Q?fDUt0pXOr3wvHB1MSeeAe8hSqkMOyUpApjlKJs2mqRJ3FA9BgtYtEjxGA9Wl?= =?us-ascii?Q?NnghqG26xvl39i6rHnLTVcB3mrYcdU1uLUWM5kAN1kL093j64bDEVD/ftkJH?= =?us-ascii?Q?XUnMYVya7Lfr5bOr3Pmau4LTqgmwHklRPUaL7Ghr57+HUYn2TGqMR0kU1wbf?= =?us-ascii?Q?YooUY9AssuXus5x3fZxAPPT4fOipCRhCDH3UW2ye11D/NbQY5lmLXLQqQmzk?= =?us-ascii?Q?G9UoCu62jpES0hlN1iVCpjzFn2gH224hl4aon+zq8yXOVfvlR9SjD9J/73Bi?= =?us-ascii?Q?MNxPN2V3Z18pfhO2p4IdEFaOjzHpPx2AVkVVoZMm5EXY8FcMBYyieGTz7vzM?= =?us-ascii?Q?dt/lgzkOSVxcTN8LZMWILbhByegyqtH+z3In3fpDSRzh7Gw=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; CY1PR0501MB1386; 5:vx/cRMhy4aBrz61+DZYtcZWEsIxwH6iZASF8L1+HxaeD8fEq8hL7IJ722TOAKUjC10QXbJdj7Ba9vaOIC7sK9WrkzS3AW2a60x43QLMVY9gFMJgM7lXOA1aHPEdrPs7P7k0t22K3C0+g7huaW1T+vg==; 24:60lBEi27AG3NB1aTLC7eXFLxcn+10Zouhy0PPgTgajAzqHvx055QfCMJgKGWo3ALTyva51w+r4vCDeLTxu44KRNdravs5EYuKgDlwkU3mZg= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2015 01:36:34.9909 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.19]; Helo=[p-emfe01b-sac.jnpr.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0501MB1386 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Dec 2015 01:36:46 -0000 On Fri, Dec 18, 2015 at 9:07 PM, Patrick Kelsey > wrote: > > Further, per-thread FIBs may not solve your scaling problem. You >initially > > stated that your objection to VNET was that you would need a minimum >of "A > > * B * C threads to ensure that any given service on any single tenant > > network could fully utilize the system's resources to process >requests". > > If you assign threads to a particular FIB, then you are back in the A >* B > > * C scaling model that you didn't want. > > I think it would be reduced to A * C threads, where A was the number of > services and C the number of CPUs - what you would drop is the B >dimension > (replication of service connections across all tenant networks). I may not have stated this clearly, so I'll restate it. I see two main alternatives: 1. There is a stable relationship between threads and FIBs. This requires dedicating threads to a particular FIB (and potentially leads back to the A * B * C scaling model, although I admit that depends on some of the implementation details). 2. Threads service requests from multiple FIBs. This requires changing the default FIB for a thread every time it switches to servicing a different request. This could be quite complex. If Ryan is thinking about a different model, he can correct me. :-) Jonathan