From owner-freebsd-net@freebsd.org Sun Apr 11 16:54:42 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CE9FD5D55DB for ; Sun, 11 Apr 2021 16:54:42 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on20628.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eae::628]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FJHz55fqRz3t6H; Sun, 11 Apr 2021 16:54:40 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dKRwUoLVjFoTrCtFoTL8GuhHxlMBukPzdhsKIvz53bXy2QhjqgU2/VqVznUl5TTbT4zXBTUcSlXXRPWnPmCoIkc54CO7wkNXXi50OclDCLq6ls5KUy+uxxl7WLrhjRwuK8Llk28tsOTC9hf2P0eesQFA+S+45DI4SdXV+a08Jd3rGnwJJQM1Ym718wYPpwZvsOCo8fVS5UrqEBSHTTHz4DNjex+0j0XRXc09UPc1GDrlxXZK4p49kJ4MwF+IZ6BNzyEs8RW2ysfeRFjXsEDxi49MmeEP6XTtXYJdaawh8TFpFORsdvfUFjMHyynz3zn2zhwOp2s05RB1Zz+80Tzi7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Y2Sl5Y4jNkyykm5bHA81Qm9LtBhU2zB48SAPyDmC5PU=; b=G3AADgLiRd8yMoPV++fh42dW9kEDwAzl3HDw8UoYmQWUrNWA4ajn2JAWgg4s6r5AmjUaORyDyp4QBgQu1O+ffTowckwUOylJ0wdqytKabEDosZT0QrrtH+wmQDaT8RkVF3qDazxRBNsc96fdy/oIlOMCCH4FnexNZyGsRbLYaW10/SYNU6lhCfZvZTOXT1FE11WwckxZQElrIGHlUOidn0Wa1RSAsfcZhVB7tIrqiCIs6rX5wsiFhD1RpfAXZAEbnGxy+0ajKXiOKVcA48DQXbbIn6Zl3fF1updqA76m2BImBuj+f5BgkTkkrH7q35ylipIWODPi9Savql5KypWXxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netapp.com; dmarc=pass action=none header.from=netapp.com; dkim=pass header.d=netapp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Y2Sl5Y4jNkyykm5bHA81Qm9LtBhU2zB48SAPyDmC5PU=; b=h8+n/hvmSLNz+8Tgs4T8uRN2LxW7CvScXUHytEXOqLYJOfWbRx1Xa3l7oosr0QZFNCpsgjBQ6kgawGgloL1T0NQLcTKlghp09LCMhnth9W/mEPyV8GNf4q//CJuF+5ZB/QWY10ujAf3nXzgTH0KRs7iF99cvBmyv33hqAUHv/0iZU553LG1eYBIRDq1K8hgxa7Gu70WbmhMpP6GuBDrvwgSpMKTCjlXlpYVvzDikSTSExN4/OT9aXmaNTOnJMQSi9rDXFUa8M2fs4CZNqadpTb5YLznEUt1AnHxnwNmVd7z/akRYRzG0NUFPQDvmo99DHZX4GKun7Q+2Uk07VNZhjw== Received: from SN4PR0601MB3728.namprd06.prod.outlook.com (2603:10b6:803:51::24) by SN7PR06MB7184.namprd06.prod.outlook.com (2603:10b6:806:10f::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.17; Sun, 11 Apr 2021 16:54:37 +0000 Received: from SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef]) by SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::ccb:944d:e270:63ef%6]) with mapi id 15.20.4020.021; Sun, 11 Apr 2021 16:54:37 +0000 From: "Scheffenegger, Richard" To: "tuexen@freebsd.org" , Rick Macklem CC: Youssef GHORBAL , "freebsd-net@freebsd.org" Subject: Re: NFS Mount Hangs Thread-Topic: NFS Mount Hangs Thread-Index: AQHXG1GB6agsoGWN0UqRoZFo/qoHTaqMDIkAgAL97ICACMXzgIAAsfOAgAfvbwCAAQ5PAIAAWDiAgAKBMZWAAD3WgIAAFNIAgAA/e4CAABvaAIAAEe2AgAEE0ACAAJCpAIAAgu0AgAXcwwCAAH0H4IAARSaAgAAmg3iAABY/gIAABIEAgABhAYCAAPMogIAAR//T Date: Sun, 11 Apr 2021 16:54:37 +0000 Message-ID: References: <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> <8E745920-1092-4312-B251-B49D11FE8028@pasteur.fr> <765CE1CD-6AAB-4BEF-97C6-C2A1F0FF4AC5@freebsd.org> <2B189169-C0C9-4DE6-A01A-BE916F10BABA@freebsd.org> <077ECE2B-A84C-440D-AAAB-00293C841F14@freebsd.org> <3980F368-098D-4EE4-B213-4113C2CAFE7D@freebsd.org> , <23F49FD9-A8B6-460F-9CD2-BBC3181A058F@freebsd.org> In-Reply-To: <23F49FD9-A8B6-460F-9CD2-BBC3181A058F@freebsd.org> Accept-Language: de-AT, en-US Content-Language: de-AT X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [2001:4bb8:100:49d6:e51c:e977:d324:dfc1] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 080b1e98-1cbe-420a-f49d-08d8fd0a7d9f x-ms-traffictypediagnostic: SN7PR06MB7184: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2201; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: +U9tdABXJVdQObsg312DliO9rQY8nNg+00mT/EqfHp7//ZymYiB+fb8kxSeTyMu322WNstIOXkhWlySeHE+XPjh5kFNQRvILNSFhGMmxFTTyQoy6zTzQkPvB8b3OEXoBkBvYMtg9ORhXV1sDSB7Y15eLbn0PIaz2c1W0418cX4njskNFMmv2BxP7nUNZgu4+c5pQ/EAl/QiWLJUAybrh9kqcCb0wGd+84jEGTy4M3dlDSvyrYgMyBG4HqHtuzYfi2FP+7j7kH6DRMJTA03GdnmhSPPtfuRsPk9QT9cOmv8eq5OwmRLgdEZIJ8jAXYLJwqU5j6634IEONZnM3XLG7mX78abkGxHEfb9ja4WxWwsi2PXw/aPOEsaoyJ5ZMPkG/v/buNuLCRtxRMrxoIrjiBIb8a2AsHFsUyocsYu7jAugQ4NG0INE0RBipq8ilVLGeHObxETFs54oT+5kWZ+Z/GA3NdzU1KtgDkte1XryiTjlzZuPQ5gtdM2odHsVwUPjuL5c2dMQH3Ef3yC3yWar5720bnY1VKnhk06Q3MME0AzcF0Upx9c4j5VJ7w8jt06VteDvlzBQy2HEb02eyIH0aAMm7y9PcpbPUsxzTOnspvg3hEbTuW9P7EPZICB3kYmz0YZmLalYIbcLsbig19yKE1Ld8Dh0+WcEHDwBnHmZBziaAJ/KzpFB4UGKB+jm7RIg7 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN4PR0601MB3728.namprd06.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(396003)(346002)(39860400002)(136003)(366004)(71200400001)(8676002)(7116003)(7696005)(54906003)(52536014)(5660300002)(86362001)(8936002)(166002)(38100700002)(2906002)(66946007)(9686003)(966005)(4326008)(316002)(110136005)(91956017)(3480700007)(66476007)(76116006)(66446008)(66556008)(64756008)(478600001)(83380400001)(186003)(84040400003)(55016002)(33656002)(6506007); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?Windows-1252?Q?etJpTSr/TQJBRFRvQ4mKYi7bmCepGDXyscDWHMziiMIK3OSvt14aMvzf?= =?Windows-1252?Q?mSz+OsxwosIdWFT5FDV84Gmr4ihDhOcB1sUARn1nWYaUN3RT9i/OB9GD?= =?Windows-1252?Q?mgoHhDtfLF+H4NFaYuNbfDP88S6J1kDXj/hflsr6b0Jl7Khok+dk9/OK?= =?Windows-1252?Q?sJThePxwNeAoOBBh1V69k9vH+Dgse4bA+e2XX1mJGDOn/6b0eRzjoxD1?= =?Windows-1252?Q?W5Lgrqt6o+mJyseWkPojCvCKygsh6X6hpdDZO0cIaP5wdN4SKRL9/bCY?= =?Windows-1252?Q?CUOQALCnaWguDVRvcgkrgG+e4VNJbkXPGJQzPLXCelaiF/z94Hfme4Az?= =?Windows-1252?Q?Fae1NSbMHz/wxQ0l+KUOc+ILtLn7G+sKplytXHZLwkMiAV9tLi7aL+Hn?= =?Windows-1252?Q?FLfltO6ZIhxqDZXdEgcwAAP7qp4D6kJHDgEzJrZxv4zIlBDSb9/pHjDV?= =?Windows-1252?Q?/pEElVCw1KEFxY6Mu7EMAoIzbxLOhPq1GgoNiOk9VpFNVcmhn5Rt15HN?= =?Windows-1252?Q?UjrwK3VGr5p8ZpoMj+LeZacTZZPNMP6brZ4nMjCs7pHeDtmofP9OngCU?= =?Windows-1252?Q?5YzSaKla+2i7AemqBHN09b0WiaiqNlsKFSgblZxuDJRqXX7xYr8N6SuA?= =?Windows-1252?Q?aJtnADY+0b9kwRu2NFWxXTgn0V1/LbcaW4yKoefCd+LEQcwyC5LUAMj9?= =?Windows-1252?Q?xnq8WojTk6g2bDNsjStdO8jtAthic+YgOuEVmcVAkV3dnUJshuPBMw8Y?= =?Windows-1252?Q?II8AGhbkIXlgJK/neh8w8G2viT+sYOyRi3MPswugwiPb34FhguX6f7LM?= =?Windows-1252?Q?iOT3mPBvgCWieEvCMjRd8dIWVn/dReDIcfOzt9pID602SJoRnZJ/5CaH?= =?Windows-1252?Q?W2/9uFwkHLGG27RDLPPgyZYUFb4Fk3rK/vu2Xi+so7hxKQvhYSX4y62S?= =?Windows-1252?Q?4CHEzPb46CSuNdReXnvbRnrlP83oiUNoM6HxjFbZaOuWYT5pSZKe6tzj?= =?Windows-1252?Q?apAw/g50zJQlYFOsQaHxZMUKZCGelrZ7qGVQuOICPdAQszfrM0x0Q9eL?= =?Windows-1252?Q?oucC58FLQRtt97xFfgnlIj/Ztz09Eel/nlXvnFRSojhZOhKaEpVXGvs3?= =?Windows-1252?Q?GCiarbuNEC3OltGisuJA53PLZYa+S6Qxr72kOzl8264Hl++RxIND0yVv?= =?Windows-1252?Q?qQN0UC/J/7yykNrVYJUcLtPVHxNSRSad2rz9p1Sq5c5YB6XTNA+ZrFKW?= =?Windows-1252?Q?boH7khb4VFPNIsowkqhNBJVcoAk0Hq5InKStA0UGv1P+4l2wlfXk8oV7?= =?Windows-1252?Q?ayOff8RJzpdam0Fngw21vka17N0y9qP8IBBZKhsEGqewQudOD4Y3stBI?= =?Windows-1252?Q?XiHAKW4mkxaVL6vmCeFyuPbW/UdSGwpYgEA8yId8AuGgJ2n6763uw57R?= =?Windows-1252?Q?KYg0hPetC/53MVyV5HV1Qx1vJpgOr3HCHKYAQA9UCRy9uwqPWAzlR3+G?= =?Windows-1252?Q?shmjfS8uOJkJy942aOUPF2d7BZ0gHQ=3D=3D?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: netapp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN4PR0601MB3728.namprd06.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 080b1e98-1cbe-420a-f49d-08d8fd0a7d9f X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Apr 2021 16:54:37.1030 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: BmFO8MNUFMEqdOQKwmOXzetHYNydlyi8xvzYLZqKvmbD+ZJDiZUPDnGxzMcZz4Vb/eDIFbTsTSLzQUqddFfQQA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR06MB7184 X-Rspamd-Queue-Id: 4FJHz55fqRz3t6H X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netapp.com header.s=selector1 header.b=h8+n/hvm; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=netapp.com; spf=pass (mx1.freebsd.org: domain of Richard.Scheffenegger@netapp.com designates 2a01:111:f400:7eae::628 as permitted sender) smtp.mailfrom=Richard.Scheffenegger@netapp.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:7eae::628:from]; R_DKIM_ALLOW(-0.20)[netapp.com:s=selector1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[netapp.com:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:7eae::628:from:127.0.2.255]; DKIM_TRACE(0.00)[netapp.com:+]; DMARC_POLICY_ALLOW(-0.50)[netapp.com,none]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-net] Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Apr 2021 16:54:42 -0000 >From what i understand rick stating around the socket state changing before= the upcall, i can only speculate that the rst fight is for the new sessios= the client tries with the same 5tuple, while server side the old original = session persists, as the nfs server never closes /shutdown the session . But a debug logged version of the socket upcall used by the nfs server shou= ld reveal any differences in socket state at the time of upcall. I would very much like to know if d29690 addresses that problem (if it was = due to releasing the lock before the upcall), or if that still shows differ= ences between prior to my central upcall change, post that change and with = d29690 ... ________________________________ Von: tuexen@freebsd.org Gesendet: Sunday, April 11, 2021 2:30:09 PM An: Rick Macklem Cc: Scheffenegger, Richard ; Youssef GHOR= BAL ; freebsd-net@freebsd.org Betreff: Re: NFS Mount Hangs NetApp Security WARNING: This is an external email. Do not click links or o= pen attachments unless you recognize the sender and know the content is saf= e. > On 10. Apr 2021, at 23:59, Rick Macklem wrote: > > tuexen@freebsd.org wrote: >> Rick wrote: > [stuff snipped] >>>> With r367492 you don't get the upcall with the same error state? Or yo= u don't get an error on a write() call, when there should be one? >> If Send-Q is 0 when the network is partitioned, after healing, the krpc = sees no activity on >> the socket (until it acquires/processes an RPC it will not do a sosend()= ). >> Without the 6minute timeout, the RST battle goes on "forever" (I've neve= r actually >> waited more than 30minutes, which is close enough to "forever" for me). >> --> With the 6minute timeout, the "battle" stops after 6minutes, when th= e timeout >> causes a soshutdown(..SHUT_WR) on the socket. >> (Since the soshutdown() patch is not yet in "main". I got comments, = but no "reviewed" >> on it, the 6minute timer won't help if enabled in main. The soclose= () won't happen >> for TCP connections with the back channel enabled, such as Linux 4.= 1/4.2 ones.) >> I'm confused. So you are saying that if the Send-Q is empty when you par= tition the >> network, and the peer starts to send SYNs after the healing, FreeBSD res= ponds >> with a challenge ACK which triggers the sending of a RST by Linux. This = RST is >> ignored multiple times. >> Is that true? Even with my patch for the the bug I introduced? > Yes and yes. > Go take another look at linuxtofreenfs.pcap > ("fetch https://people.freebsd.org/~rmacklem/linuxtofreenfs.pcap" if you = don't > already have it.) > Look at packet #1949->2069. I use wireshark, but you'll have your favouri= te. > You'll see the "RST battle" that ends after > 6minutes at packet#2069. If there is no 6minute timeout enabled in the > server side krpc, then the battle just continues (I once let it run for a= bout > 30minutes before giving up). The 6minute timeout is not currently enabled > in main, etc. Hmm. I don't understand why r367492 can impact the processing of the RST, w= hich basically destroys the TCP connection. Richard: Can you explain that? Best regards Michael > >> What version of the kernel are you using? > "main" dated Dec. 23, 2020 + your bugfix + assorted NFS patches that > are not relevant + 2 small krpc related patches. > --> The two small krpc related patches enable the 6minute timeout and > add a soshutdown(..SHUT_WR) call when the 6minute timeout is > triggered. These have no effect until the 6minutes is up and, witho= ut > them the "RTS battle" goes on forever. > > Add to the above a revert of r367492 and the RST battle goes away and thi= ngs > behave as expected. The recovery happens quickly after the network is > unpartitioned, with either 0 or 1 RSTs. > > rick > ps: Once the irrelevant NFS patches make it into "main", I will upgrade t= o > main bits-de-jur for testing. > > Best regards > Michael >> >> If Send-Q is non-empty when the network is partitioned, the battle will = not happen. >> >>> >>> My understanding is that he needs this error indication when calling sh= utdown(). >> There are several ways the krpc notices that a TCP connection is no long= er functional. >> - An error return like EPIPE from either sosend() or soreceive(). >> - A return of 0 from soreceive() with no data (normal EOF from other end= ). >> - A 6minute timeout on the server end, when no activity has occurred on = the >> connection. This timer is currently disabled for NFSv4.1/4.2 mounts in "= main", >> but I enabled it for this testing, to stop the "RST battle goes on forev= er" >> during testing. I am thinking of enabling it on "main", but this crude b= andaid >> shouldn't be thought of as a "fix for the RST battle". >> >>>> >>>> From what you describe, this is on writes, isn't it? (I'm asking, at t= he original problem that was fixed with r367492, occurs in the read path (d= raining of ths so_rcv buffer in the upcall right away, which subsequently i= nfluences the ACK sent by the stack). >>>> >>>> I only added the so_snd buffer after some discussion, if the WAKESOR s= houldn't have a symmetric equivalent on WAKESOW.... >>>> >>>> Thus a partial backout (leaving the WAKESOR part inside, but reverting= the WAKESOW part) would still fix my initial problem about erraneous DSACK= s (which can also lead to extremely poor performance with Linux clients), b= ut possible address this issue... >>>> >>>> Can you perhaps take MAIN and apply https://reviews.freebsd.org/D29690= for the revert only on the so_snd upcall? >> Since the krpc only uses receive upcalls, I don't see how reverting the = send side would have >> any effect? >> >>> Since the release of 13.0 is almost done, can we try to fix the issue i= nstead of reverting the commit? >> I think it has already shipped broken. >> I don't know if an errata is possible, or if it will be broken until 13.= 1. >> >> --> I am much more concerned with the otis@ stuck client problem than th= is RST battle that only >> occurs after a network partitioning, especially if it is 13.0 speci= fic. >> I did this testing to try to reproduce Jason's stuck client (with c= onnection in CLOSE_WAIT) >> problem, which I failed to reproduce. >> >> rick >> >> Rs: agree, a good understanding where the interaction btwn stack, socket= and in kernel tcp user breaks is needed; >> >>> >>> If this doesn't help, some major surgery will be necessary to prevent N= FS sessions with SACK enabled, to transmit DSACKs... >> >> My understanding is that the problem is related to getting a local error= indication after >> receiving a RST segment too late or not at all. >> >> Rs: but the move of the upcall should not materially change that; i don= =92t have a pc here to see if any upcall actually happens on rst... >> >> Best regards >> Michael >>> >>> >>>> I know from a printf that this happened, but whether it caused the RST= battle to not happen, I don't know. >>>> >>>> I can put r367492 back in and do more testing if you'd like, but I thi= nk it probably needs to be reverted? >>> >>> Please, I don't quite understand why the exact timing of the upcall wou= ld be that critical here... >>> >>> A comparison of the soxxx calls and errors between the "good" and the "= bad" would be perfect. I don't know if this is easy to do though, as these = calls appear to be scattered all around the RPC / NFS source paths. >>> >>>> This does not explain the original hung Linux client problem, but does= shed light on the RST war I could create by doing a network partitioning. >>>> >>>> rick >>> >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"