Date: Mon, 13 May 2013 21:14:24 +0300 From: Eugen-Andrei Gavriloaie <shiretu@gmail.com> To: Adrian Chadd <adrian@freebsd.org> Cc: freebsd-hackers@freebsd.org Subject: Re: Managing userland data pointers in kqueue/kevent Message-ID: <BCDD5783-72F7-40CD-85D0-5D96D64B93AC@gmail.com> In-Reply-To: <CAJ-VmomV=3%2BryHpxzFEe_Yb3WK1MThDQ_CZ8KUhQFRDn0TBd_w@mail.gmail.com> References: <CCE4FFC4-F846-4F81-85EE-776B753C63C6@gmail.com> <CAJ-VmokBfBvdLa_Wf2EajF%2BvecVntLDaxdVeNvhAOiPp6HkjNA@mail.gmail.com> <84DCA050-99D4-4B22-A031-35E0928709E0@gmail.com> <CAJ-Vmo=CVeXpf9WNOegD3yG9Q0NwUWaLadVrv1RgeyAaHYADiQ@mail.gmail.com> <985C1C3F-3F70-47D2-8F43-F3D6CCA4482C@gmail.com> <CAJ-VmomV=3%2BryHpxzFEe_Yb3WK1MThDQ_CZ8KUhQFRDn0TBd_w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_5AA28570-F534-489B-94D6-2357ED428192 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 ------ Eugen-Andrei Gavriloaie Web: http://www.rtmpd.com On May 13, 2013, at 9:02 PM, Adrian Chadd <adrian@freebsd.org> wrote: > Hi, >=20 > The reason I tend to suggest this is for portability and debugging > reasons. (Before and even since libevent came into existence.) >=20 > If you do it right, you can stub / inline out all of the wrapper > functions in userland and translate them to straight system or library > calls. >=20 > Anyway. I'm all for making kqueue better. I just worry that adding > little hacks here and there isn't the right way to do it. If you want > to guarantee specific behaviours with kqueue, you should likely define > how it should work in its entirety and see if it will cause > architectural difficulties down the track. And it caused some so far. We have workarounds for it, no problem. > Until that is done, I think > you have no excuse to get your code working as needed. Yes. I agree. But when I look at the user space code without that = feature, and when thinking how it would have been with that feature, it = kinda makes me cry. A little more pain and I will make that patch = myself. I'm just hoping that kq kernel side code will be handled by more = capable hands before me. Ideally, by the creators. >=20 > Don't blame kqueue because what (iirc) is not defined behaviour isn't > defined in a way that makes you happy :) Nobody blamed kqueue. I'm just saying that it would be better for me = (and I'm not the only one) who could use a liiiitle more help from it. = It was born from needs, it evolved because of needs, why stop now? I = dare to say it will become a standard on linux and other OSs very soon. = It is the best fd reactor. Hands down! Best regards, Andrei >=20 >=20 >=20 > Adrian >=20 > On 13 May 2013 09:36, Eugen-Andrei Gavriloaie <shiretu@gmail.com> = wrote: >> Hi Adrian, >>=20 >> All the tricks, work arounds, paradigms suggested/implemented by us, = the kq users, are greatly simplified by simply adding that thing that = Paul is suggesting. What you are saying here is to basically do = not-so-natural things to overcome a real problem which can be very easy = and non-intrusivly solved at lower levels. Seriously, if you truly = believe that you can put the equal sign between the complexity of the = user space code and the wanted patch in kqueue kernel side, than I = simply shut up. >>=20 >> Besides, one of the important points in kq philosophy is simplifying = things. I underline the "one of". It is not the goal, of course. Complex = things are complex things no matter how hard you try to simplify them. = But this is definitely (should) not falling into that category. >>=20 >> ------ >> Eugen-Andrei Gavriloaie >> Web: http://www.rtmpd.com >>=20 >> On May 13, 2013, at 6:47 PM, Adrian Chadd <adrian@freebsd.org> wrote: >>=20 >>> ... holy crap. >>>=20 >>> On 13 May 2013 08:37, Eugen-Andrei Gavriloaie <shiretu@gmail.com> = wrote: >>>> Hi, >>>>=20 >>>> Well, Paul already asked this question like 3-4 times now. Even = insisting on it. I will also ask it again: >>>> If user code is responsible of tracking down the data associated = with the signalled entity, what is the point of having user data? >>>> Is rendered completely useless=85 >>>=20 >>> .. why does everything have to have a well defined purpose that is >>> also suited for use in _all_ situations? >> That is called perfection. I know we can't achieve it, but I like to = walk in that direction at least. >>=20 >>>=20 >>>> Not to mention, that your suggestion with FD index is a definite = no-go. The FD values are re-used. Especially in MT environments. Imagine = one kqueue call taking place in thread A and another one in thread B. = Both threads waiting for events. >>>=20 >>> .. so don't do that. I mean, you're already having to write your = code >>> to _not_ touch FDs in other threads. I've done this before, it isn't >>> that hard and it doesn't hurt performance. >> Why not? This is how you achieve natural load balancing for multiple = kevent() calls from multiple threads over the same kq fd. Otherwise, = again, you have to write complex code to manually balance the threads. = That brings locking again=85. >> Why people always think that locking is cheap? Excessive locking = hurts. A lot! >>=20 >>>=20 >>>> When A does his magic, because of internal business rules, it = decides to close FD number 123. It closes it and it connects somewhere = else by opening a new one. Surprise, we MAY get the value 123 again as = a new socket, we put it on our index, etc. Now, thread B comes in and it = has stale/old events for the old 123 FD. Somethings bad like EOF for the = OLD version of FD number 123 (the one we just closed anyway). Guess = what=85 thread B will deallocate the perfectly good thingy inside the = index associated with 123. >>>=20 >>> So you just ensure that nothing at all calls a close(123); but calls >>> fd_close(123) which will in turn close(123) and free all the state >>> associated with it. >> Once threads A and B returned from their kevent() calls, all bets are = off. In between, you get the the behaviour I just described from threads = A and B racing towards FD123 to either close it or create a new one. How = is wrapping close() going to help? Is not like you have any control over = what the socket() function is going to return. (That gave me another = token idea btw=85 I will explain in another email, perhaps you care to = comment) >> Mathematically speaking, the fd-to-data association is not bijective. >>=20 >>=20 >>>=20 >>> You have fd_close() either grab a lock, or you ensure that only the >>> owning thread can call fd_close(123) and if any other thread calls = it, >>> the behaviour is undefined. >> As I said, that adds up to the user-space code complexity. Just don't = forget that Paul's suggestion solves all this problems in a ridiculously = simple manner. All our ideas of keeping track who is owning who and = indexes are going to be put to rest. kq will notify us when the udata is = out of scope from kq perspective. That is all we ask. >>=20 >>>=20 >>>> And regarding the "thread happiness", that is not happiness at all = IMHO=85 >>>=20 >>> Unless you're writing a high connection throughput web server, the >>> overhead of grabbing a lock in userland during the fd shutdown = process >>> is trivial. Yes, I've written those. It doesn't hurt you that much. >> That "that much" is subjective. And a streaming server is a few = orders of magnitude more complex than a web server. Remember, a web = server is bound to request/response paradigm. While a streaming server = is a full duplex (not request/response based) animal for most of = connections. I strongly believe that becomes a real problem. (I would = love to be wrong on this one!) >>=20 >>>=20 >>> I'm confused as to why this is still an issue. Sure, fix the kqueue >>> semantics and do it in a way that doesn't break backwards >>> compatibility. >> Than, if someone has time and pleasure, it would be nice to have it. = Is a neat solution. Is one thing saying, hey, we don't have time, do it = yourself. And another thing in trying to offer "better" solutions by = defending such an obvious caveat. >>=20 >>> But please don't claim that it's stopping you from >>> getting real work done. >> I didn't and I won't. I promise! >>=20 >>> I've written network apps with kqueue that >>> scales to 8+ cores and (back in mid-2000's) gigabit + of small HTTP >>> transactions. >> Good for you. How is this relevant to or discussion of simplifying = things? Of course is possible. But let's make things simpler and more = efficient. It really pays off in the long run. Hell, this is how kq was = born in the first place: getting rid of all garbage that one was = supposed to do to achieve what kq does with a few lines of code. Let's = make that even better than it currently is. >>=20 >>> This stuff isn't at all problematic. >>>=20 >>>=20 >>> Adrian >>=20 --Apple-Mail=_5AA28570-F534-489B-94D6-2357ED428192 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIO5jCCBJ0w ggOFoAMCAQICEDQ96SusJzT/j8s0lPvMcFQwDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMCU0Ux FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAgTmV0 d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0wNTA2MDcwODA5MTBa Fw0yMDA1MzAxMDQ4MzhaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNh bHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0 dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0 aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsjmF pPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVNNRm5pELlzkniii8efNIxB8dOtINknS4p1aJk xIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQylbsMTzC9mKALi+VuG6JG+ni8om+rWV6lL8/K2m2q L+usobNqqrcuZzWLeeEeaYji5kbNoKXqvgvOdjp6Dpvq/NonWz1zHyLmSGHGTPNpsaguG7bUMSAs vIKKjqQOpdeJQ/wWWq8dcdcRWdq6hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7NlyP0e03RiqhjKaJMe oYV+9Udly/hNVyh00jT/MLbu9mIwFIws6wIDAQABo4H0MIHxMB8GA1UdIwQYMBaAFK29mHo0tCb3 +sQmVO8DveAky1QaMB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNVHQ8BAf8EBAMC AQYwDwYDVR0TAQH/BAUwAwEB/zARBgNVHSAECjAIMAYGBFUdIAAwRAYDVR0fBD0wOzA5oDegNYYz aHR0cDovL2NybC51c2VydHJ1c3QuY29tL0FkZFRydXN0RXh0ZXJuYWxDQVJvb3QuY3JsMDUGCCsG AQUFBwEBBCkwJzAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG 9w0BAQUFAAOCAQEAAbyc42MosPMxAcLfe91ioAGdIzEPnJJzU1HqH0z61p/Eyi9nfngzD3QWuZGH kfWKJvpkcADYHvkLBGJQh5OB1Nr1I9s0u4VWtHA0bniDNx6FHMURFZJfhxe9rGr98cLRzIlfsXzw PlHyNfN87GCYazor4O/fs32G67Ub9VvsonyYE9cAULnRLXPeA3h04QWFMV7LmrmdlMa5lDd1ctxE +2fo8PolHlKn2iXpR+CgxzygTrEKNvt3SJ/vl4r7tP7jlBSog7xcLT/SYHFg7sJxggzpiDbj2iC0 o6BsqpZLuICOdcpJB/Y7FLrf3AXZn9vgsuZNoHgm5+ctbn9fxh6IFTCCBRowggQCoAMCAQICEG0Z 6qcZT2ozIuYiMnqqcd4wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJV VDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29y azEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZp cnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTEwNDI4MDAwMDAwWhcNMjAw NTMwMTA0ODM4WjCBkzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQ MA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENP TU9ETyBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBAJKEhFtLV5jUXi+LpOFAyKNTWF9mZfEyTvefMn1V0HhMVbdC lOD5J3EHxcZppLkyxPFAGpDMJ1Zifxe1cWmu5SAb5MtjXmDKokH2auGj/7jfH0htZUOMKi4rYzh3 37EXrMLaggLW1DJq1GdvIBOPXDX65VSAr9hxCh03CgJQU2yVHakQFLSZlVkSMf8JotJM3FLb3uJA AVtIaN3FSrTg7SQfOq9xXwfjrL8UO7AlcWg99A/WF1hGFYE8aIuLgw9teiFX5jSw2zJ+40rhpVJy ZCaRTqWSD//gsWD9Gm9oUZljjRqLpcxCm5t9ImPTqaD8zp6Q30QZ9FxbNboW86eb/8ECAwEAAaOC AUswggFHMB8GA1UdIwQYMBaAFImCZ33EnSZwAEu0UEh83j2uBG59MB0GA1UdDgQWBBR6E04AdFvG eGNkJ8Ev4qBbvHnFezAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIBADARBgNVHSAE CjAIMAYGBFUdIAAwWAYDVR0fBFEwTzBNoEugSYZHaHR0cDovL2NybC51c2VydHJ1c3QuY29tL1VU Ti1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwdAYIKwYBBQUHAQEE aDBmMD0GCCsGAQUFBzAChjFodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVROQWRkVHJ1c3RDbGll bnRfQ0EuY3J0MCUGCCsGAQUFBzABhhlodHRwOi8vb2NzcC51c2VydHJ1c3QuY29tMA0GCSqGSIb3 DQEBBQUAA4IBAQCF1r54V1VtM39EUv5C1QaoAQOAivsNsv1Kv/avQUn1G1rF0q0bc24+6SZ85kyY wTAo38v7QjyhJT4KddbQPTmGZtGhm7VNm2+vKGwdr+XqdFqo2rHA8XV6L566k3nK/uKRHlZ0sviN 0+BDchvtj/1gOSBH+4uvOmVIPJg9pSW/ve9g4EnlFsjrP0OD8ODuDcHTzTNfm9C9YGqzO/761Mk6 PB/tm/+bSTO+Qik5g+4zaS6CnUVNqGnagBsePdIaXXxHmaWbCG0SmYbWXVcHG6cwvktJRLiQfsrR eTjrtDP6oDpdJlieYVUYtCHVmdXgQ0BCML7qpeeU0rD+83X5f27nMIIFIzCCBAugAwIBAgIQY7lP +MJT4QzHh5dD4LWVRzANBgkqhkiG9w0BAQUFADCBkzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdy ZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExp bWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBF bWFpbCBDQTAeFw0xMjA4MjEwMDAwMDBaFw0xMzA4MjEyMzU5NTlaMCIxIDAeBgkqhkiG9w0BCQEW EXNoaXJldHVAZ21haWwuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAziBH0E99 6jx0UrKapQcbxgWLgux1j9/Ec7K3QkkFtzCgxtF9ohpfsiAVOkYuCzVxoxoC0eNKO4qbZPYNbsN6 gtVZMrQnphgG3Wzmf28azn60TtCEQNoFXdW4On+cZgcvUxZS09DaPuTrHB5BuOmYG7b93ZjvJuov KCYAjYNrg0S9TKTQdRJAu95E2OuoVzin+QwIfEVwOA+PSRTPbNnjHvHUMg2G98CXFKIYPy9Pz8Kk rd54q7eSanjNiQK4Z9UYFjjNokx0nhymUAlnzWDAPj8jlh5i6gPrWsr9mYmQ6Y9wm2RHyf7QA6nM BIO43JNPklqjUT5LnB47rxVnLKnYmQIDAQABo4IB4TCCAd0wHwYDVR0jBBgwFoAUehNOAHRbxnhj ZCfBL+KgW7x5xXswHQYDVR0OBBYEFFoYqkq01lUdSiemIAjKHTuJiRj9MA4GA1UdDwEB/wQEAwIF oDAMBgNVHRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgB hvhCAQEEBAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0 cHM6Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwVwYDVR0fBFAwTjBMoEqgSIZGaHR0cDovL2NybC5j b21vZG9jYS5jb20vQ09NT0RPQ2xpZW50QXV0aGVudGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNy bDCBiAYIKwYBBQUHAQEEfDB6MFIGCCsGAQUFBzAChkZodHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9D T01PRE9DbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3J0MCQGCCsGAQUFBzAB hhhodHRwOi8vb2NzcC5jb21vZG9jYS5jb20wHAYDVR0RBBUwE4ERc2hpcmV0dUBnbWFpbC5jb20w DQYJKoZIhvcNAQEFBQADggEBAHOQHTMzyiauVSuvCNgN2laY11tdDegdalfDK8UIFpdv08TE1viQ Q9gQfJpGNiO4wxpK8NeB8Qkh+AlL9ygq3jNwuAhk+TXMJ4Jr/I7LkTyJFnW45ADLZs/ufAbc+gdK lxdIxRJBOHmEpA51h0beSlfgHZcvvGshESzwLF290LuEdmLWncIWbhuO4vOJRK/7wJ/KXD8bZdx5 LGvA/QPPi5u95zFoQPRLSDl7lqJailBJZe6YFs7zBK2lm3u0w7exJw/rYeXFjMR0g5BPQCiihMq5 77k2T7uKmQqzapeCqLcd0sFNi94lzgSJxaQDyiDZYeYjWX0lSNVEG0z/fCIVJj4xggOrMIIDpwIB ATCBqDCBkzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE BxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBD bGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIQY7lP+MJT4QzHh5dD4LWV RzAJBgUrDgMCGgUAoIIB1zAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEP Fw0xMzA1MTMxODE0MjVaMCMGCSqGSIb3DQEJBDEWBBRxCr3wVIjYpCFf4PcmEN+KnQ2DOzCBuQYJ KwYBBAGCNxAEMYGrMIGoMIGTMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVz dGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE5MDcGA1UE AxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhBjuU/4 wlPhDMeHl0PgtZVHMIG7BgsqhkiG9w0BCRACCzGBq6CBqDCBkzELMAkGA1UEBhMCR0IxGzAZBgNV BAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RP IENBIExpbWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNl Y3VyZSBFbWFpbCBDQQIQY7lP+MJT4QzHh5dD4LWVRzANBgkqhkiG9w0BAQEFAASCAQAIGSIxfpvu 44gzMPwF+m5sexjox5N4wiBhtfnAXi/UxEiRGmLipGM8cqHD9PckQeI4UIOf3G5HTMJdbB6pCSde AYLWdWsTopHDAXXYImqJzWzR7vwpHXAnzRw+9YnjPtUDG3Y2a0M9K5ZGIl47Bt1P7YaScKPGXP1x nC4sWaPAFr3iN3ldAdKP9i9mmH4HtEYgdxqep+oA3S4pFGi68aqRrgxwSY1HISc8p1F6Vlqw65R0 TiHPW/6Ntc+sjSxI1VMGDAVfyyebRlhlB6Mcr/BRHjg2+6ZKCMZPrxOQ0YFJhhg5y/yDieXJOKtX HsNFWJM0VVTzCBMAtq3bRSaFNMEtAAAAAAAA --Apple-Mail=_5AA28570-F534-489B-94D6-2357ED428192--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BCDD5783-72F7-40CD-85D0-5D96D64B93AC>