Date: Thu, 08 Apr 2004 12:32:11 -0600 From: Brandon Erhart <berhart@ErhartGroup.COM> To: freebsd-hackers@freebsd.org Subject: KQueue dropping events? Message-ID: <6.0.2.0.2.20040408123017.01cecec0@mx1.erhartgroup.com>
next in thread | raw e-mail | index | archive | help
Hi,
I am writing a web sucker downer (mirror) for a project on indexing the web
(got myself a 1TB raid, just gonna d/l text ..). I am using the KQueue API
in FreeBSD 4.9-REL to take care of watching over my sockets. I seem to be
running into a nasty problem, however.
Here's a scenario. I set the outgoing connections to, say, 5000. The
problem is, the amount of connections my program shows as being connected
is roughly 1/5 to sometimes even 1/8th of what is actually connected. I see
what is "actually connected" by doing a netstat. The program would say
750/5000 connections, while a netstat would show 4500 connections in the
ESTABLISHED state.
You may be saying, "it must be a bug in your connection tracking logic". I
honestly don't think that's it. I have only TWO places in my code where I
check if the connection was successful by calling getpeername(), if it
wasnt, i return back to the main loops, otherwise I know it connected (and
therefore increasing the global connection counter) -- in the callback
function for data read() from KQueue-monitored fds (all of the sockets),
and then also in my main loop (I check for read, write and connect timeouts
there, right after my call to kevent()).
Basically the main loop looks like, in psuedo-code of course:
while (there_is_still_events)
{
if (kevent()) <-- i pull 16 events from the kqueue
{
execute_the_callback_function;
}
check_for_read()_write()_and_connect()_timeouts;
check_for_"client_descriptors(basically just a structure that holds info
on the kqueue event)"_that_need_to_be_connected_to_the_next_server_and_do_so;
}
It's pretty straight forward. I have no idea why my program would be
reporting a smaller amount. Is it possible that my program is not getting
ALL the information it needs from kevent()? Perhaps the KQueue is becoming
"full"? Is this possible? Should I be pulling more than the 16 events off
the kqueue at a time?
I have been up ALL NIGHT trying to debug this, and cannot figure it out.
Any and all help is appreciated!
Thanks,
Brandon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6.0.2.0.2.20040408123017.01cecec0>