From owner-freebsd-hackers@freebsd.org Sat Aug 18 10:11:09 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F5AB108BAF4 for ; Sat, 18 Aug 2018 10:11:09 +0000 (UTC) (envelope-from doubleble@outlook.com) Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-oln040092001048.outbound.protection.outlook.com [40.92.1.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 98AC78C070; Sat, 18 Aug 2018 10:11:08 +0000 (UTC) (envelope-from doubleble@outlook.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5dmO3CSm7OuoVDqP2HEqo1cUB80V79j8PXr58l9NQW0=; b=fUbQJqKifF8pgFrNLYZZsymJkHulrj8nTiZc3+Ht0UiRgTcnv9C6ySIlA88wPr3UgLTgsiIdgW+sFQ1FAoLzeX7/b/bbOUfib93cd/soV9ZKj2nlitK/ZRFKxDBJQlcK6xUko38crxYX5I3ER6bMPsQKNflMLk27mtpGjhvrKnPqbItMsw0OhrjwQhGCEWlS+9dUl0FDwGINoO67iFUp3cLeApyVHrJfN9mRCwdwN2exl9qz/YENumiprtbvLRvapNuMlURtfGD+We7AgIcjBBh1KsP92DnMjo3WXf6VAHzA148uBuSsl3YWJQYalW/jJWfhuKwxfwJbuRACUKtRmA== Received: from BN3NAM01FT008.eop-nam01.prod.protection.outlook.com (10.152.66.55) by BN3NAM01HT182.eop-nam01.prod.protection.outlook.com (10.152.67.98) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.20.1080.9; Sat, 18 Aug 2018 10:11:06 +0000 Received: from BN6PR2201MB1523.namprd22.prod.outlook.com (10.152.66.55) by BN3NAM01FT008.mail.protection.outlook.com (10.152.67.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.20.1080.9 via Frontend Transport; Sat, 18 Aug 2018 10:11:06 +0000 Received: from BN6PR2201MB1523.namprd22.prod.outlook.com ([fe80::4c9a:37fe:d25e:45e]) by BN6PR2201MB1523.namprd22.prod.outlook.com ([fe80::4c9a:37fe:d25e:45e%2]) with mapi id 15.20.1059.023; Sat, 18 Aug 2018 10:11:06 +0000 From: Double Tong To: Alan Somers CC: "freebsd-hackers@freebsd.org" Subject: Re: Create a thread with a separate file descriptor table (set RFFDG flag) Thread-Topic: Create a thread with a separate file descriptor table (set RFFDG flag) Thread-Index: AQHUNXm4sY9W87Z5LU6r9yREhH5aIaTClEaAgAK1FYA= Date: Sat, 18 Aug 2018 10:11:06 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-CA, zh-CN, en-US Content-Language: en-CA X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:E63E6D51D1013AC45B0A4A89EFE21AE84D4F05E4436621FDA7D97C236769FE6E; UpperCasedChecksum:052208DBD2331FEC3871F976E99EAF8CD468817E6625B845AD9DD43DC9761CDB; SizeAsReceived:7321; Count:47 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [7BnyNqIlIvvoxNnkIl20zTmLuo4UE9vC] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; BN3NAM01HT182; 6:fQp2XLcosY1MgOcZruc6pkMh8NdEF2CbpWXA+Lu2AD0N6QI3yIpOEFBHPrVWLMsQ3WRuSonbqEiG4vkYK+DU/DuER49h0fJpAHi4kuPmGrCH+ja8NIG54M7UUWKl6DVQl1bxLtOIpq1Y1fMWPPNfqQ6eAMmb7LWisUe/gnMxpvos4PPtXXwt9ondO+1vWYadJH0Y8aQC1Ou8DTUwLNHY4atJBXRD9mli/1VERfdT6c74tXfzvFv5W3bfF/61V1i98MfQ6WQBfniGTd63gsmh5vB9V7ZPWWVcxX2b7Bne2YPiCtHA4jYVySpV7AtPS7znhzJXq5r67HN1QymUp4STw7Ep6aOcW+f5UwhwYf1nkAdFdUgIjvcpguXNpOcT9MeIhB/ZIzkGsCZLamvaDII6k+xMXZChIX29gXHpRK+Ut3nMOr0rpzHuCak7LiLCLHyDitRdYFaESwicjl4OEO6gVA==; 5:awjh9Q4wBZxUW80Tk27LCeuYRSCIkFNDEM4/XmrIjU97pGFWFsZeT5UnNbNtYzAblkLbjJ3o0O3aDR2krikCkfs7MddSM8MHTE5wT+syK5ib6gxBhc+LX158HLdsJvW18nK4cCjfgwXXvhPt85/6OU4FoXp+BVengWQ3qtjTBts=; 7:eahmRIM6AS/BOGhCZC/RNWbJeXFd4ofDQDKUiOlXoeNWisUjdMht0QJ1xPm80a9HnN0br2GhQpoFfOGM4VEjbjSrQfNK2jJY5jJnhaZ8FOKOjm7KEbhtNrvaFiVcpcedHRvrQfauBdzsdAMIW/TY7g3UEfGNIZgeeOOEek/kL0MuACLL4wtQ2/Ui3cklXMp3XRdFauJsqRm5BBYFE/cRi2RP9I9iiJ/ab+aq3XUmq0VvQTEnyZ2lA5cipwTTy81w x-incomingheadercount: 47 x-eopattributedmessage: 0 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(201702061078)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1603101475)(1601125500)(1701031045); SRVR:BN3NAM01HT182; x-ms-traffictypediagnostic: BN3NAM01HT182: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(4566010)(82015058); SRVR:BN3NAM01HT182; BCL:0; PCL:0; RULEID:; SRVR:BN3NAM01HT182; x-forefront-prvs: 076804FE30 x-forefront-antispam-report: SFV:NSPM; SFS:(7070007)(199004)(189003)(501624003)(6916009)(5250100002)(82202002)(99286004)(5660300001)(8676002)(8936002)(2900100001)(81156014)(345774005)(54896002)(55016002)(6306002)(486006)(256004)(14444005)(33656002)(86362001)(5024004)(19627405001)(87572001)(236005)(476003)(104016004)(229853002)(20460500001)(68736007)(6436002)(606006)(25786009)(105586002)(106356001)(76176011)(97736004)(6246003)(4326008)(450100002)(7696005)(11346002)(446003)(426003)(45080400002)(105004)(14454004)(102836004)(6346003)(53546011)(26005)(56003)(74316002); DIR:OUT; SFP:1901; SCL:1; SRVR:BN3NAM01HT182; H:BN6PR2201MB1523.namprd22.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: outlook.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=doubleble@outlook.com; x-microsoft-antispam-message-info: n3b26mDbGYUJIr6stgY9K/gsse+SYIAhtq7CVdE9ROQ02DB2cCbN3nG4NQv1n8JFfC5ix7C9wVX+XTNslE0m8yG8QrLwSr5wa4+jPmJ4F5bCeWSiAGnxmlgzgwNNvZuxvhhQC1yzfwHC/mVc5jlyKFtboQz4YZKVHl5OsAKr1YAKRGG05hWue60oU8Jffj6I+Qlw8+o6UNuDWXGvVg2lwYVDIV579nmndYg1Nw17G4g= MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 9a4e3081-9524-43cf-bfc3-dcaef82d5da1 X-MS-Exchange-CrossTenant-Network-Message-Id: 302e4b8a-414f-4e0e-0d6d-08d604f2e96a X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 9a4e3081-9524-43cf-bfc3-dcaef82d5da1 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Aug 2018 10:11:06.2735 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3NAM01HT182 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Aug 2018 10:11:09 -0000 Thank you Alan! As the program's main structure is mostly done, using shared memory object = to help coping with this issue requires too much work for us. We did not us= e any select calls. I was told the performance gain can be greater if these= connections are short-lived (i.e. one connection transmit one request). I = will try it later to get some data. And on Linux it is easy to do, simply a= unshare(2) syscall would do it, which is why we are considering doing this= on FreeBSD as well. Anyway it is something nice to have, not something tha= t would block the project. I am still interested my question 2 and 3, as theoretically, it feels like = the idea should work. Thanks, -Shuangyi Tong ________________________________ From: asomers@gmail.com on behalf of Alan Somers Sent: August 16, 2018 12:38 To: Double Tong Cc: freebsd-hackers@freebsd.org Subject: Re: Create a thread with a separate file descriptor table (set RFF= DG flag) On Thu, Aug 16, 2018 at 10:28 AM, Double Tong > wrote: Hello, I want to create a thread with a separate file descriptor table to have bet= ter performance with kevent(2). On Linux, I was using unshare(2) syscall to= achieve this, which as far as I know there is no equivalent or similar sys= call in FreeBSD. I have posted on freebsd forums (https://forums.freebsd.org/threads/create-= a-thread-with-a-separate-file-descriptor-table-set-rffdg-flag.67143/), and = now I understood the following: rfork_thread(3) is deprecated in favor of = pthread_create(3). rfork_thread(3) is written in assembly language to perfo= rm stack swapping, which means if rfork_thread(3) no longer exists in the b= uild, it can damage our program's portability if it relies on rfork_thread. With the above consideration, I still have the following questions: 1. Is there an elegant way to create a thread with a separate file descr= iptor table? Sort of. The usual way to do this is to create a separate process, not a s= eparate thread. If you also need a shared address space, then you can prob= ably satisfy that by creating a shared memory object, and using that for wh= ichever data structures actually need to be shared. See shm_open(2) for de= tails. 2. If you are thinking about using rfork_thread(3) to do this, I am work= ing on this direction. I am using waitpid to join these "threads", and the = thread exits in the middle of execution with status 0x8b collected by waitp= id. I guess this status means invalid page access. I wrote a tiny program (= attached below) to reflect the code I am using in my program, I appreciate = if you would like to take a look at it to see if there is anything I was no= t doing correctly. 3. As I was reading the code of pthread_create, it allocates a pthread s= truct on the top of thread, and then calls clone, which freebsd implemented= its version of clone that actually calls rfork (I did not find the source = of freebsd's clone, can someone provides a link?). So I believe theoretical= ly there should be a way to achieve this in the user space. And if I am not= using pthread related APIs, then missing pthread struct should be fine as = well? 4. On Linux, after calling unshare(CLONE_FLIES), I got performance incre= ase around 10% with 1000 concurrent TCP connections. I am instructed to imp= lement this by my supervisor, and I do not have much details about why the = performance would increase. Would this also works for freebsd as well (keve= nt calls)? Does your Linux program use select(2), epoll(2), or something else? select= (2) has well-known performance problems with large file-descriptor tables, = but epoll(2) and kevent(2) do not. If your program is using select(2), the= n you should convert it to use epoll(2)/kevent(2) and ditch unshare(2). Thank you for any help, comments in advance! -Alan