Date: Sat, 18 Aug 2018 10:11:06 +0000 From: Double Tong <doubleble@outlook.com> To: Alan Somers <asomers@freebsd.org> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: Create a thread with a separate file descriptor table (set RFFDG flag) Message-ID: <BN6PR2201MB1523A83CCB94F28E72ED822BA23C0@BN6PR2201MB1523.namprd22.prod.outlook.com> In-Reply-To: <CAOtMX2j8p_ZMd=4nfSe2jY-ZuBnqVzw_yv=Hs6uRCJC4zX6mug@mail.gmail.com> References: <BN6PR2201MB1523127E9938E9CC1FADB944A23E0@BN6PR2201MB1523.namprd22.prod.outlook.com>, <CAOtMX2j8p_ZMd=4nfSe2jY-ZuBnqVzw_yv=Hs6uRCJC4zX6mug@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Thank you Alan! As the program's main structure is mostly done, using shared memory object = to help coping with this issue requires too much work for us. We did not us= e any select calls. I was told the performance gain can be greater if these= connections are short-lived (i.e. one connection transmit one request). I = will try it later to get some data. And on Linux it is easy to do, simply a= unshare(2) syscall would do it, which is why we are considering doing this= on FreeBSD as well. Anyway it is something nice to have, not something tha= t would block the project. I am still interested my question 2 and 3, as theoretically, it feels like = the idea should work. Thanks, -Shuangyi Tong ________________________________ From: asomers@gmail.com <asomers@gmail.com> on behalf of Alan Somers <asome= rs@freebsd.org> Sent: August 16, 2018 12:38 To: Double Tong Cc: freebsd-hackers@freebsd.org Subject: Re: Create a thread with a separate file descriptor table (set RFF= DG flag) On Thu, Aug 16, 2018 at 10:28 AM, Double Tong <doubleble@outlook.com<mailto= :doubleble@outlook.com>> wrote: Hello, I want to create a thread with a separate file descriptor table to have bet= ter performance with kevent(2). On Linux, I was using unshare(2) syscall to= achieve this, which as far as I know there is no equivalent or similar sys= call in FreeBSD. I have posted on freebsd forums (https://forums.freebsd.org/threads/create-= a-thread-with-a-separate-file-descriptor-table-set-rffdg-flag.67143/), and = now I understood the following: rfork_thread(3) is deprecated in favor of = pthread_create(3). rfork_thread(3) is written in assembly language to perfo= rm stack swapping, which means if rfork_thread(3) no longer exists in the b= uild, it can damage our program's portability if it relies on rfork_thread. With the above consideration, I still have the following questions: 1. Is there an elegant way to create a thread with a separate file descr= iptor table? Sort of. The usual way to do this is to create a separate process, not a s= eparate thread. If you also need a shared address space, then you can prob= ably satisfy that by creating a shared memory object, and using that for wh= ichever data structures actually need to be shared. See shm_open(2) for de= tails. 2. If you are thinking about using rfork_thread(3) to do this, I am work= ing on this direction. I am using waitpid to join these "threads", and the = thread exits in the middle of execution with status 0x8b collected by waitp= id. I guess this status means invalid page access. I wrote a tiny program (= attached below) to reflect the code I am using in my program, I appreciate = if you would like to take a look at it to see if there is anything I was no= t doing correctly. 3. As I was reading the code of pthread_create, it allocates a pthread s= truct on the top of thread, and then calls clone, which freebsd implemented= its version of clone that actually calls rfork (I did not find the source = of freebsd's clone, can someone provides a link?). So I believe theoretical= ly there should be a way to achieve this in the user space. And if I am not= using pthread related APIs, then missing pthread struct should be fine as = well? 4. On Linux, after calling unshare(CLONE_FLIES), I got performance incre= ase around 10% with 1000 concurrent TCP connections. I am instructed to imp= lement this by my supervisor, and I do not have much details about why the = performance would increase. Would this also works for freebsd as well (keve= nt calls)? Does your Linux program use select(2), epoll(2), or something else? select= (2) has well-known performance problems with large file-descriptor tables, = but epoll(2) and kevent(2) do not. If your program is using select(2), the= n you should convert it to use epoll(2)/kevent(2) and ditch unshare(2). Thank you for any help, comments in advance! -Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BN6PR2201MB1523A83CCB94F28E72ED822BA23C0>