Date: Tue, 27 Dec 2016 11:27:36 -0800 From: John Baldwin <jhb@freebsd.org> To: freebsd-arch@freebsd.org Cc: Xin LI <delphij@gmail.com>, Alfred Perlstein <alfred@freebsd.org>, Hongjiang Zhang <honzhan@microsoft.com>, =?utf-8?B?55ub5oWn5Y2O?= <hhsheng@corp.netease.com>, Jilles Tjoelker <jilles@stack.nl> Subject: Re: question about fopen fd limit Message-ID: <2792667.4rqir5N98G@ralph.baldwin.cx> In-Reply-To: <CAGMYy3tQ0=7wEegf0W6Fq-Av%2BDTquc0su7Eg8o4OfEAmq3rCrg@mail.gmail.com> References: <2016122223570929089978@corp.netease.com> <db32b02b-2eb3-cee8-98cd-0940223f04d9@freebsd.org> <CAGMYy3tQ0=7wEegf0W6Fq-Av%2BDTquc0su7Eg8o4OfEAmq3rCrg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tuesday, December 27, 2016 09:52:27 AM Xin LI wrote: > -freebsd-net (bcc), +freebsd-arch >=20 > This would work but also comes with a lot of pain. >=20 > (But really - why do we implement these accessors, like fileno() and > friends as macros with knowledge of the sFILE layout? Will it be > reasonable to start converting these to functions as a first step, wh= ich > would not break ABI but allow us to do it in the future? Otherwise w= e > would be just kicking the same can along the street forever...) >=20 > FILE is supposed to be fully opaque to application writers in my opin= ion. It should be, but it's a pain. There are various things that know abou= t FILE internals to manipulate the ungetc() state, etc. That is in thing= s like the gzip code and copied into umpteen different places in various ports. I have an older set of branches that attempt to make FILE parti= ally opaque and then add a new int-sized _file. However, I haven't fixed th= e last round of fallout from trying to make FILE more opaque. The PR for this (hiding most of FILE) is here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205029 The associated GIT branch is here. I have done some updates to it sinc= e the build failures in the PR, but haven't verified that I've fixed all of the reported bugs via my own port builds before submitting an updated patch to the PR to try for a new exp-run. https://github.com/freebsd/freebsd/compare/master...bsdjhb:stdio_hide The other PR related to exp-runs for expanding _file is: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D172332 The GIT branch that includes the fixes for this (including some compat stubs to account for old perl binaries that predate fdclose()) is here:= https://github.com/freebsd/freebsd/compare/master...bsdjhb:stdio_file I still think trying to finish the stdio_hide branch and that approach first is the right path before returning to _file itself. > On Sat, Dec 24, 2016 at 5:19 PM, Alfred Perlstein <alfred@freebsd.org= > > wrote: >=20 > > Hello =E7=9B=9B=E6=85=A7=E5=8D=8E > > > > Here's another trick that may work. > > > > Use funopen(3) and provide your own read/write/seek and close funct= ions > > for the high fds. > > > > You can basically make "cookie" a struct that contains your "int si= zed" > > fds. > > > > FILE * > > funopen(const void *cookie, int (*readfn)(void *, char *, int)= , int > > (*writefn)(void *, const char *, int), > > fpos_t (*seekfn)(void *, fpos_t, int), int (*closefn)(void= *)); > > > > > > If you need more help please make sure to email me directly so I ca= n see > > your question. > > > > -Alfred > > > > > > > > On 12/23/16 12:48 AM, =E7=9B=9B=E6=85=A7=E5=8D=8E wrote: > > > >> hi all, > >> > >> Thank you for your advice ~ > >> solution 2 definitly broaden my horizons ~~but may be not a good= choice > >> for my project ~~LoL > >> i will try to mail freebsd-current mail list, if libc is as your > >> description , may be i should modify it by myself ~~ > >> Thank you so much~ > >> Are u KingSoft's Dr Zhang ? nice to meet you !!!!! > >> > >> > >> winson sheng > >> > >> > >> winson sheng > >> From: Hongjiang Zhang > >> Date: 2016-12-23 11:44 > >> To: =E7=9B=9B=E6=85=A7=E5=8D=8E; freebsd-net > >> Subject: RE: RE: question about fopen fd limit > >> Ok. I know. > >> There are two possible solutions: > >> Quick solution for short term: modify short to int in libc by your= self, > >> buildworld and installworld. Pushing to modify libc may take a lon= g time, > >> especially only few people encounter this issue. You=E2=80=99d bet= ter send email to > >> freebsd-current to confirm whether they accept your suggestion. > >> Work around: You can first reserve a series of fd before opening T= CP > >> connections. For example, invoke open(=E2=80=9C/dev/null=E2=80=9D)= for 10000 times to get > >> 10000 fds. Those fd values are small enough to be held by =E2=80=9C= short=E2=80=9D. After > >> that, start TCP connections. Once you need to fopen a file, please= call > >> open(=E2=80=9Cxxx=E2=80=9D) instead, and then use dup2(old_fd, new= _fd) to exchange the two > >> fd. The old_fd value is the one obtained by open(=E2=80=9Cxxx=E2=80= =9D), and new_fd is one > >> in your reserved fd fields, and next please use fdopen(fd, mode). = Here, you > >> have to manage the reserved fds by yourself including open/close. > >> In my eyes: > >> is the quick method, and there is no modifications in your logic. > >> Needs you to maintain the reserved consecutive fields for fd by yo= urself, > >> which increased the complexity of your logic. > >> Thanks > >> Hongjiang Zhang > >> From: =E7=9B=9B=E6=85=A7=E5=8D=8E [mailto:hhsheng@corp.netease.c= om] > >> Sent: Friday, December 23, 2016 11:02 AM > >> To: Hongjiang Zhang <honzhan@microsoft.com>; freebsd-net < > >> freebsd-net@freebsd.org> > >> Subject: Re: RE: question about fopen fd limit > >> hi all, > >> not map TCP to FILE, you misunderstanding my meaning~ > >> for example, if my server tcp already holds 32000 connection > >> fopen only has 767 fd to use > >> the problem has no bussiness with tcp fd, BUT fopen ... > >> in some particular situlations , my server will open 1k+ FILE= , that > >> will exceed the fileno limit, and overflow occur > >> my server can't open any file more ,that's the problem ~ > >> so i felt if bsd official could change FILE struct's fileno t= o a > >> UNSIGNED SHORT that may be an effecient and convenient solution ju= st for my > >> case ? > >> UNSIGNED SHORT fileno is enough for me, and i don't wanna chang= e a lot > >> of FILE function that take FILE * as its argument ~ > >> Thank you ~~~ > >> winson sheng > >> > >> > >> winson sheng > >> From: Hongjiang Zhang > >> Date: 2016-12-23 10:17 > >> To: =E7=9B=9B=E6=85=A7=E5=8D=8E; freebsd-net > >> Subject: RE: question about fopen fd limit > >> Why do you need to map TCP fd to FILE? > >> It is difficult to modify FILE structure. If it is possible, let= us > >> figure out some new designs to meet your requirement. > >> -----Original Message----- > >> From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@free= bsd.org] > >> On Behalf Of ??? > >> Sent: Thursday, December 22, 2016 11:57 PM > >> To: freebsd-net <freebsd-net@freebsd.org> > >> Subject: question about fopen fd limit > >> hi all, > >> hi~ > >> we are from Chinese Game Develop Corp, Netease. > >> and One of our product using FreeBsd as its OS platform. > >> This Game has Millions of players online , and Each Server may= holds > >> 25000+ tcp connection at the same time.Thanks to BSD and kqueue :)= > >> for example, it's one of our server , netstat cmd to list > >> connections overall... > >> netstat -an | grep 13396 (it's our listening port) | wc -l > >> 23221 > >> recently we do some performance optimize and promote this c= onnect > >> limit to 28000+ or 30000+. > >> But we find Freebsd has a limit that this huge online number wi= ll take > >> 28000+ fd, and bsd FILE * struct's > >> fd only support to SHORT . such as .. > >> struct __sFILE { > >> ... > >> short _file; /* (*) fileno, if Unix descriptor, else -1 */ ... > >> so if our server want to fopen some file when we still hold t= his > >> online number, the fd amount may easily exceed 32767, and fopen de= finitely > >> return a err code. then the server will appear some fataly ERROR. > >> we do a simple test and confirm this situation. > >> then in fopen's code , we notice that we can use open to retu= rn a fd > >> instread of fopen to avoid this overflow, > >> as below > >> 68 /* > >> 1 * File descriptors are a full int, but _file is only a short. > >> 2 * If we get a valid file descriptor that is greater than > >> 3 * SHRT_MAX, then the fd will get sign-extended into an > >> 4 * invalid file descriptor. Handle this case by failing the > >> 5 * open. > >> 6 */ > >> BUT ... so many c lib FILE series function needs a FILE * p= ointer > >> as input argument, we can't convert all of them to fd, or it will = be a > >> rather suffering things to us. > >> and even in BSD 10 , it seems this short limit still there , = but > >> other OS as debian , FILE strucnt's fileno is a int . > >> we found an unoffical patch easily change this fileno to uns= igned , > >> but we are a very stready project, we can't afford the risk to use= an > >> unoffical patch. > >> so, do you have any plan to change this fopen fd limit to UNS= IGNED > >> SHORT or int in the future ? ushort is enough for us . > >> if you do , we are really glad and excited~~~~~~~if you don't ,= it > >> donen't matter, plz give us a reply so that we may need to > >> find some other plan to resolved this suffering thing. > >> LoL, thank you !!!!! > >> yours sincerely > >> winson sheng > >> winson sheng > >> _______________________________________________ > >> freebsd-net@freebsd.org mailing list > >> https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3A% > >> 2F%2Flists.freebsd.org%2Fmailman%2Flistinfo%2Ffreebsd > >> -net&data=3D02%7C01%7Chonzhan%40microsoft.com%7C4a9dfccbccd4 > >> 46be2f4a08d42a833fb0%7C72f988bf86f141af91ab2d7cd011db47%7C1% > >> 7C0%7C636180190584478890&sdata=3DPAwJP5IXHy0WJwxbV7MB%2B8 > >> zvKheZUYjhHx3ohFRSPZM%3D&reserved=3D0 > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.= org" > >> _______________________________________________ > >> freebsd-net@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-net > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.= org" > >> > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.o= rg" > > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.or= g" --=20 John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2792667.4rqir5N98G>