Date: Tue, 27 Dec 2016 13:19:01 -0500 From: Lewis Donzis <lew@perftech.com> To: freebsd-arch@freebsd.org Cc: Xin LI <delphij@gmail.com> Subject: Re: question about fopen fd limit Message-ID: <7A472344-E4DA-452A-AC81-9CA67CD0B26C@perftech.com> In-Reply-To: <CAGMYy3tQ0=7wEegf0W6Fq-Av%2BDTquc0su7Eg8o4OfEAmq3rCrg@mail.gmail.com> References: <2016122223570929089978@corp.netease.com> <CY1PR03MB1517A33F8F9E0BAC00C2E345B5950@CY1PR03MB1517.namprd03.prod.outlook.com> <2016122311014089280414@corp.netease.com> <CY1PR03MB151796BFAEA5DDB1CD6DB752B5950@CY1PR03MB1517.namprd03.prod.outlook.com> <2016122316484066524625@corp.netease.com> <db32b02b-2eb3-cee8-98cd-0940223f04d9@freebsd.org> <CAGMYy3tQ0=7wEegf0W6Fq-Av%2BDTquc0su7Eg8o4OfEAmq3rCrg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I agree, and, in my humble opinion, the FILE structure should never have =
made any assumptions about the value of an fd, especially that it would =
fit in a short when an fd is defined as an int. For that matter, =
fileno() is defined as returning an int, but the implementation is =
technically returning a short in a non-threaded environment.
I would support changing the __sFILE structure=E2=80=99s _file member to =
an int and probably also changing _flags to int or unsigned since it =
would end up getting padded for alignment anyway. It would also be more =
efficient.
> On Dec 27, 2016, at 12:52 PM, Xin LI <delphij@gmail.com> wrote:
>=20
> -freebsd-net (bcc), +freebsd-arch
>=20
> This would work but also comes with a lot of pain.
>=20
> (But really - why do we implement these accessors, like fileno() and
> friends as macros with knowledge of the sFILE layout? Will it be
> reasonable to start converting these to functions as a first step, =
which
> would not break ABI but allow us to do it in the future? Otherwise we
> would be just kicking the same can along the street forever...)
>=20
> FILE is supposed to be fully opaque to application writers in my =
opinion.
>=20
> On Sat, Dec 24, 2016 at 5:19 PM, Alfred Perlstein <alfred@freebsd.org>
> wrote:
>=20
>> Hello =E7=9B=9B=E6=85=A7=E5=8D=8E
>>=20
>> Here's another trick that may work.
>>=20
>> Use funopen(3) and provide your own read/write/seek and close =
functions
>> for the high fds.
>>=20
>> You can basically make "cookie" a struct that contains your "int =
sized"
>> fds.
>>=20
>> FILE *
>> funopen(const void *cookie, int (*readfn)(void *, char *, int), =
int
>> (*writefn)(void *, const char *, int),
>> fpos_t (*seekfn)(void *, fpos_t, int), int (*closefn)(void =
*));
>>=20
>>=20
>> If you need more help please make sure to email me directly so I can =
see
>> your question.
>>=20
>> -Alfred
>>=20
>>=20
>>=20
>> On 12/23/16 12:48 AM, =E7=9B=9B=E6=85=A7=E5=8D=8E wrote:
>>=20
>>> hi all,
>>>=20
>>> Thank you for your advice ~
>>> solution 2 definitly broaden my horizons ~~but may be not a good =
choice
>>> for my project ~~LoL
>>> i will try to mail freebsd-current mail list, if libc is as your
>>> description , may be i should modify it by myself ~~
>>> Thank you so much~
>>> Are u KingSoft's Dr Zhang ? nice to meet you !!!!!
>>>=20
>>>=20
>>> winson sheng
>>>=20
>>>=20
>>> winson sheng
>>> From: Hongjiang Zhang
>>> Date: 2016-12-23 11:44
>>> To: =E7=9B=9B=E6=85=A7=E5=8D=8E; freebsd-net
>>> Subject: RE: RE: question about fopen fd limit
>>> Ok. I know.
>>> There are two possible solutions:
>>> Quick solution for short term: modify short to int in libc by =
yourself,
>>> buildworld and installworld. Pushing to modify libc may take a long =
time,
>>> especially only few people encounter this issue. You=E2=80=99d =
better send email to
>>> freebsd-current to confirm whether they accept your suggestion.
>>> Work around: You can first reserve a series of fd before opening TCP
>>> connections. For example, invoke open(=E2=80=9C/dev/null=E2=80=9D) =
for 10000 times to get
>>> 10000 fds. Those fd values are small enough to be held by =
=E2=80=9Cshort=E2=80=9D. After
>>> that, start TCP connections. Once you need to fopen a file, please =
call
>>> open(=E2=80=9Cxxx=E2=80=9D) instead, and then use dup2(old_fd, =
new_fd) to exchange the two
>>> fd. The old_fd value is the one obtained by open(=E2=80=9Cxxx=E2=80=9D=
), and new_fd is one
>>> in your reserved fd fields, and next please use fdopen(fd, mode). =
Here, you
>>> have to manage the reserved fds by yourself including open/close.
>>> In my eyes:
>>> is the quick method, and there is no modifications in your logic.
>>> Needs you to maintain the reserved consecutive fields for fd by =
yourself,
>>> which increased the complexity of your logic.
>>> Thanks
>>> Hongjiang Zhang
>>> From: =E7=9B=9B=E6=85=A7=E5=8D=8E [mailto:hhsheng@corp.netease.com]
>>> Sent: Friday, December 23, 2016 11:02 AM
>>> To: Hongjiang Zhang <honzhan@microsoft.com>; freebsd-net <
>>> freebsd-net@freebsd.org>
>>> Subject: Re: RE: question about fopen fd limit
>>> hi all,
>>> not map TCP to FILE, you misunderstanding my meaning~
>>> for example, if my server tcp already holds 32000 connection
>>> fopen only has 767 fd to use
>>> the problem has no bussiness with tcp fd, BUT fopen ...
>>> in some particular situlations , my server will open 1k+ FILE , =
that
>>> will exceed the fileno limit, and overflow occur
>>> my server can't open any file more ,that's the problem ~
>>> so i felt if bsd official could change FILE struct's fileno to a
>>> UNSIGNED SHORT that may be an effecient and convenient solution just =
for my
>>> case ?
>>> UNSIGNED SHORT fileno is enough for me, and i don't wanna change a =
lot
>>> of FILE function that take FILE * as its argument ~
>>> Thank you ~~~
>>> winson sheng
>>>=20
>>>=20
>>> winson sheng
>>> From: Hongjiang Zhang
>>> Date: 2016-12-23 10:17
>>> To: =E7=9B=9B=E6=85=A7=E5=8D=8E; freebsd-net
>>> Subject: RE: question about fopen fd limit
>>> Why do you need to map TCP fd to FILE?
>>> It is difficult to modify FILE structure. If it is possible, let us
>>> figure out some new designs to meet your requirement.
>>> -----Original Message-----
>>> From: owner-freebsd-net@freebsd.org =
[mailto:owner-freebsd-net@freebsd.org]
>>> On Behalf Of ???
>>> Sent: Thursday, December 22, 2016 11:57 PM
>>> To: freebsd-net <freebsd-net@freebsd.org>
>>> Subject: question about fopen fd limit
>>> hi all,
>>> hi~
>>> we are from Chinese Game Develop Corp, Netease.
>>> and One of our product using FreeBsd as its OS platform.
>>> This Game has Millions of players online , and Each Server may =
holds
>>> 25000+ tcp connection at the same time.Thanks to BSD and kqueue :)
>>> for example, it's one of our server , netstat cmd to list
>>> connections overall...
>>> netstat -an | grep 13396 (it's our listening port) | wc -l
>>> 23221
>>> recently we do some performance optimize and promote this =
connect
>>> limit to 28000+ or 30000+.
>>> But we find Freebsd has a limit that this huge online number will =
take
>>> 28000+ fd, and bsd FILE * struct's
>>> fd only support to SHORT . such as ..
>>> struct __sFILE {
>>> ...
>>> short _file; /* (*) fileno, if Unix descriptor, else -1 */ ...
>>> so if our server want to fopen some file when we still hold this
>>> online number, the fd amount may easily exceed 32767, and fopen =
definitely
>>> return a err code. then the server will appear some fataly ERROR.
>>> we do a simple test and confirm this situation.
>>> then in fopen's code , we notice that we can use open to return =
a fd
>>> instread of fopen to avoid this overflow,
>>> as below
>>> 68 /*
>>> 1 * File descriptors are a full int, but _file is only a short.
>>> 2 * If we get a valid file descriptor that is greater than
>>> 3 * SHRT_MAX, then the fd will get sign-extended into an
>>> 4 * invalid file descriptor. Handle this case by failing the
>>> 5 * open.
>>> 6 */
>>> BUT ... so many c lib FILE series function needs a FILE * =
pointer
>>> as input argument, we can't convert all of them to fd, or it will be =
a
>>> rather suffering things to us.
>>> and even in BSD 10 , it seems this short limit still there , but
>>> other OS as debian , FILE strucnt's fileno is a int .
>>> we found an unoffical patch easily change this fileno to =
unsigned ,
>>> but we are a very stready project, we can't afford the risk to use =
an
>>> unoffical patch.
>>> so, do you have any plan to change this fopen fd limit to =
UNSIGNED
>>> SHORT or int in the future ? ushort is enough for us .
>>> if you do , we are really glad and excited~~~~~~~if you don't ,it
>>> donen't matter, plz give us a reply so that we may need to
>>> find some other plan to resolved this suffering thing.
>>> LoL, thank you !!!!!
>>> yours sincerely
>>> winson sheng
>>> winson sheng
>>> _______________________________________________
>>> freebsd-net@freebsd.org mailing list
>>> https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3A%
>>> 2F%2Flists.freebsd.org%2Fmailman%2Flistinfo%2Ffreebsd
>>> -net&data=3D02%7C01%7Chonzhan%40microsoft.com%7C4a9dfccbccd4
>>> 46be2f4a08d42a833fb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%
>>> 7C0%7C636180190584478890&sdata=3DPAwJP5IXHy0WJwxbV7MB%2B8
>>> zvKheZUYjhHx3ohFRSPZM%3D&reserved=3D0
>>> To unsubscribe, send any mail to =
"freebsd-net-unsubscribe@freebsd.org"
>>> _______________________________________________
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to =
"freebsd-net-unsubscribe@freebsd.org"
>>>=20
>>=20
>> _______________________________________________
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to =
"freebsd-net-unsubscribe@freebsd.org"
>>=20
> _______________________________________________
> freebsd-arch@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to =
"freebsd-arch-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7A472344-E4DA-452A-AC81-9CA67CD0B26C>