From owner-freebsd-hackers@freebsd.org Thu Aug 16 16:38:55 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9E2F106D8AF for ; Thu, 16 Aug 2018 16:38:54 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3FA918C4C2 for ; Thu, 16 Aug 2018 16:38:54 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-x134.google.com with SMTP id o72-v6so1708691lfk.1 for ; Thu, 16 Aug 2018 09:38:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=m3oGd90k2enUrq4EIrLzEeGCv+RlUb64U02SPHobXR4=; b=Bbzkoqenr/LkbuIdGLwhh0i3vLWsrKjTVvDU/sqVm0PJVNLL5vC1migr0Y9/gFsIQS A2y+/4NzN2rjeSz91m1SJCZtW/eyQjrU1fldFnaALng6FfakLF47bvbOmrjrF3xCPUXN bMrKxqtZs/19UTsM4KzjbYgOl3YDpr4EHJdSRegxRtRk8/lq90xsuT66jp4i0VvaI9UQ yQyB/1DD0RjumBYNi/STFpmb1/Ca6Gxha4X5lWAqjQbzlj8fgEzOMHNYVJa9Y3krgr83 Yev6coL3LCNzQDmZCE0LkmyJ+V7JDPAknejL+kXFhS+chKUuzj7wZMRWONj9gBUMHEzX ggKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=m3oGd90k2enUrq4EIrLzEeGCv+RlUb64U02SPHobXR4=; b=TR1mirwIJi+28isGTAOHv797x4nwQ/JIeOX3VyRcA1K3B6WckMTWolQRbJzX+FedUY H8Y4UEMrtCp65ljeSV1KKDGybORaZ62ZyyShg8MN9VQ8UqS1KxJxRuQEsUCoG8GBsTKV L9iYlIqhnnXNj+sfnr9gqU7uJFegtaoRcFDaXNC+yRX6R5sIWWuCJg4QiVmG1n2lPCzX //58EAtbPhZhM3xIuTZvT6GCU6PgIEF1CZW8vZfZ8p/cs+BGgbJHkOAQj1oK8CWdi3CN cd4IDs+tqrYasgGXomUGRskeAYGJrluS+dtNFoBXrKnA9J7OiDzxgMDHA+t6xXZCxWdO jzDg== X-Gm-Message-State: AOUpUlFwcccVIjVsitBXwIrMlKgHAeOeiEPSMvNGxAJF0zfW9Is85jHo HiohQwD3AdCs7QRdKiOzN405cHPvYsWkSdt+0GszWA== X-Google-Smtp-Source: AA+uWPwgEzLgN43OJNhMhqy7sxgx9TjZcsuZHRNyQZ4pwJaG3/UzdMzLYSWCSz74tHI/+W86jRvtnxPHmFTSqd/zfGU= X-Received: by 2002:a19:d942:: with SMTP id q63-v6mr19174480lfg.39.1534437532514; Thu, 16 Aug 2018 09:38:52 -0700 (PDT) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 2002:ab3:7851:0:0:0:0:0 with HTTP; Thu, 16 Aug 2018 09:38:51 -0700 (PDT) In-Reply-To: References: From: Alan Somers Date: Thu, 16 Aug 2018 10:38:51 -0600 X-Google-Sender-Auth: JVTRMJOZv1fW7gPIoAR-DaSdSgA Message-ID: Subject: Re: Create a thread with a separate file descriptor table (set RFFDG flag) To: Double Tong Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2018 16:38:55 -0000 On Thu, Aug 16, 2018 at 10:28 AM, Double Tong wrote: > Hello, > > I want to create a thread with a separate file descriptor table to have > better performance with kevent(2). On Linux, I was using unshare(2) syscall > to achieve this, which as far as I know there is no equivalent or similar > syscall in FreeBSD. > > I have posted on freebsd forums (https://forums.freebsd.org/ > threads/create-a-thread-with-a-separate-file-descriptor- > table-set-rffdg-flag.67143/), and now I understood the following: > rfork_thread(3) is deprecated in favor of pthread_create(3). > rfork_thread(3) is written in assembly language to perform stack swapping, > which means if rfork_thread(3) no longer exists in the build, it can damage > our program's portability if it relies on rfork_thread. > > With the above consideration, I still have the following questions: > > 1. Is there an elegant way to create a thread with a separate file > descriptor table? > Sort of. The usual way to do this is to create a separate process, not a separate thread. If you also need a shared address space, then you can probably satisfy that by creating a shared memory object, and using that for whichever data structures actually need to be shared. See shm_open(2) for details. > 2. If you are thinking about using rfork_thread(3) to do this, I am > working on this direction. I am using waitpid to join these "threads", and > the thread exits in the middle of execution with status 0x8b collected by > waitpid. I guess this status means invalid page access. I wrote a tiny > program (attached below) to reflect the code I am using in my program, I > appreciate if you would like to take a look at it to see if there is > anything I was not doing correctly. > 3. As I was reading the code of pthread_create, it allocates a pthread > struct on the top of thread, and then calls clone, which freebsd > implemented its version of clone that actually calls rfork (I did not find > the source of freebsd's clone, can someone provides a link?). So I believe > theoretically there should be a way to achieve this in the user space. And > if I am not using pthread related APIs, then missing pthread struct should > be fine as well? > 4. On Linux, after calling unshare(CLONE_FLIES), I got performance > increase around 10% with 1000 concurrent TCP connections. I am instructed > to implement this by my supervisor, and I do not have much details about > why the performance would increase. Would this also works for freebsd as > well (kevent calls)? Does your Linux program use select(2), epoll(2), or something else? select(2) has well-known performance problems with large file-descriptor tables, but epoll(2) and kevent(2) do not. If your program is using select(2), then you should convert it to use epoll(2)/kevent(2) and ditch unshare(2). > > Thank you for any help, comments in advance! > > -Alan