Date: Fri, 8 Jan 2016 20:59:59 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Jeff Darcy <jdarcy@redhat.com> Cc: Raghavendra G <raghavendra@gluster.com>, freebsd-fs <freebsd-fs@freebsd.org>, Hubbard Jordan <jkh@ixsystems.com>, Xavier Hernandez <xhernandez@datalab.es>, Gluster Devel <gluster-devel@gluster.org> Subject: Re: [Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage Message-ID: <981529129.154244852.1452304799182.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <1924941590.6473225.1452248249994.JavaMail.zimbra@redhat.com> References: <571237035.145690509.1451437960464.JavaMail.zimbra@uoguelph.ca> <20151230103152.GS13942@ndevos-x240.usersys.redhat.com> <2D8C2729-D556-479B-B4E2-66E1BB222F41@ixsystems.com> <1083933309.146084334.1451517977647.JavaMail.zimbra@uoguelph.ca> <CADRNtgStOg8UZfxNt-SzvvPf7d1J7CC_gi49ww3BbixU0Ey-rg@mail.gmail.com> <568F6D07.6070500@datalab.es> <CADRNtgRM17Eg3Z=LWifVNo=ai72dMiEVRKS3RwNfQ-dK7Pspew@mail.gmail.com> <1924941590.6473225.1452248249994.JavaMail.zimbra@redhat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_154244850_659138303.1452304799181 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Jeff Darcy wrote: > > > I don't know anything about gluster's poll implementation so I may > > > be totally wrong, but would it be possible to use an eventfd (or a > > > pipe if eventfd is not supported) to signal the need to add more > > > file descriptors to the poll call ? > > > > > > > > > The poll call should listen on this new fd. When we need to change > > > the fd list, we should simply write to the eventfd or pipe from > > > another thread. This will cause the poll call to return and we will > > > be able to change the fd list without having a short timeout nor > > > having to decide on any trade-off. > > > > > > Thats a nice idea. Based on my understanding of why timeouts are being > > used, this approach can work. > > The own-thread code which preceded the current poll implementation did > something similar, using a pipe fd to be woken up for new *outgoing* > messages. That code still exists, and might provide some insight into > how to do this for the current poll code. I took a look at event-poll.c and found something interesting... - A pipe called "breaker" is already set up by event_pool_new_poll() and closed by event_pool_destroy_poll(), however it never gets used for anything. So, I added a few lines of code that writes a byte to it whenever the list of file descriptors is changed and read when poll() returns, if its revents is set. I also changed the timeout to -1 (infinity) and it seems to work for a trivial test. --> Btw, I also noticed the "changed" variable gets set to 1 on a change, but never reset to 0. I didn't change this, since it looks "racey". (ie. I think you could easily get a race between a thread that clears it and one that adds a new fd.) A slightly safer version of the patch would set a long (100msec ??) timeout instead of -1. Anyhow, I've attached the patch in case anyone would like to try it and will create a bug report for this after I've had more time to test it. (I only use a couple of laptops, so my testing will be minimal.) Thanks for all the help, rick > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > ------=_Part_154244850_659138303.1452304799181 Content-Type: text/x-patch; name=gluster-poll.patch Content-Disposition: attachment; filename=gluster-poll.patch Content-Transfer-Encoding: base64 LS0tIGdsdXN0ZXJmcy0zLjcuNi9saWJnbHVzdGVyZnMvc3JjL2V2ZW50LXBvbGwuYy5zYXYJMjAx Ni0wMS0wNiAxNTo1ODowMy41MjIyODYwMDAgLTA1MDAKKysrIGdsdXN0ZXJmcy0zLjcuNi9saWJn bHVzdGVyZnMvc3JjL2V2ZW50LXBvbGwuYwkyMDE2LTAxLTA4IDE3OjQ1OjI4LjA3ODAyNDAwMCAt MDUwMApAQCAtMTgwLDYgKzE4MCwxNiBAQCBldmVudF9wb29sX25ld19wb2xsIChpbnQgY291bnQs IGludCBldmVuCiAgICAgICAgIHJldHVybiBldmVudF9wb29sOwogfQogCitzdGF0aWMgdm9pZAor ZXZlbnRfcG9vbF9jaGFuZ2VkIChzdHJ1Y3QgZXZlbnRfcG9vbCAqZXZlbnRfcG9vbCkKK3sKKwor ICAgICAgICBldmVudF9wb29sLT5jaGFuZ2VkID0gMTsKKyAgICAgICAgLyogV3JpdGUgYSBieXRl IGludG8gdGhlIGJyZWFrZXIgcGlwZSB0byB3YWtlIHVwIHBvbGwoKS4gKi8KKyAgICAgICAgaWYg KGV2ZW50X3Bvb2wtPmJyZWFrZXJbMV0gPj0gMCkKKyAgICAgICAgICAgICAgICB3cml0ZShldmVu dF9wb29sLT5icmVha2VyWzFdLCAiWCIsIDEpOworfQorCiAKIHN0YXRpYyBpbnQKIGV2ZW50X3Jl Z2lzdGVyX3BvbGwgKHN0cnVjdCBldmVudF9wb29sICpldmVudF9wb29sLCBpbnQgZmQsCkBAIC0y NDQsNyArMjU0LDcgQEAgZXZlbnRfcmVnaXN0ZXJfcG9sbCAoc3RydWN0IGV2ZW50X3Bvb2wgKgog ICAgICAgICAgICAgICAgICAgICAgICAgYnJlYWs7CiAgICAgICAgICAgICAgICAgfQogCi0gICAg ICAgICAgICAgICAgZXZlbnRfcG9vbC0+Y2hhbmdlZCA9IDE7CisgICAgICAgICAgICAgICAgZXZl bnRfcG9vbF9jaGFuZ2VkKGV2ZW50X3Bvb2wpOwogCiAgICAgICAgIH0KIHVubG9jazoKQEAgLTI3 NSw3ICsyODUsNyBAQCBldmVudF91bnJlZ2lzdGVyX3BvbGwgKHN0cnVjdCBldmVudF9wb29sCiAg ICAgICAgICAgICAgICAgfQogCiAgICAgICAgICAgICAgICAgZXZlbnRfcG9vbC0+cmVnW2lkeF0g PSAgZXZlbnRfcG9vbC0+cmVnWy0tZXZlbnRfcG9vbC0+dXNlZF07Ci0gICAgICAgICAgICAgICAg ZXZlbnRfcG9vbC0+Y2hhbmdlZCA9IDE7CisgICAgICAgICAgICAgICAgZXZlbnRfcG9vbF9jaGFu Z2VkKGV2ZW50X3Bvb2wpOwogICAgICAgICB9CiB1bmxvY2s6CiAgICAgICAgIHB0aHJlYWRfbXV0 ZXhfdW5sb2NrICgmZXZlbnRfcG9vbC0+bXV0ZXgpOwpAQCAtMzUwLDcgKzM2MCw3IEBAIGV2ZW50 X3NlbGVjdF9vbl9wb2xsIChzdHJ1Y3QgZXZlbnRfcG9vbCAKICAgICAgICAgICAgICAgICB9CiAK ICAgICAgICAgICAgICAgICBpZiAocG9sbF9pbiArIHBvbGxfb3V0ID4gLTIpCi0gICAgICAgICAg ICAgICAgICAgICAgICBldmVudF9wb29sLT5jaGFuZ2VkID0gMTsKKyAgICAgICAgICAgICAgICAg ICAgICAgIGV2ZW50X3Bvb2xfY2hhbmdlZChldmVudF9wb29sKTsKICAgICAgICAgfQogdW5sb2Nr OgogICAgICAgICBwdGhyZWFkX211dGV4X3VubG9jayAoJmV2ZW50X3Bvb2wtPm11dGV4KTsKQEAg LTQ0OCw2ICs0NTgsNyBAQCBldmVudF9kaXNwYXRjaF9wb2xsIChzdHJ1Y3QgZXZlbnRfcG9vbCAq CiAgICAgICAgIGludCAgICAgICAgICAgICAgc2l6ZSA9IDA7CiAgICAgICAgIGludCAgICAgICAg ICAgICAgaSA9IDA7CiAgICAgICAgIGludCAgICAgICAgICAgICAgcmV0ID0gLTE7CisgICAgICAg IGNoYXIgICAgICAgICAgICAgeDsKIAogICAgICAgICBHRl9WQUxJREFURV9PUl9HT1RPICgiZXZl bnQiLCBldmVudF9wb29sLCBvdXQpOwogCkBAIC00NzIsNyArNDgzLDcgQEAgZXZlbnRfZGlzcGF0 Y2hfcG9sbCAoc3RydWN0IGV2ZW50X3Bvb2wgKgogICAgICAgICAgICAgICAgIHNpemUgPSBldmVu dF9kaXNwYXRjaF9wb2xsX3Jlc2l6ZSAoZXZlbnRfcG9vbCwgdWZkcywgc2l6ZSk7CiAgICAgICAg ICAgICAgICAgdWZkcyA9IGV2ZW50X3Bvb2wtPmV2Y2FjaGU7CiAKLSAgICAgICAgICAgICAgICBy ZXQgPSBwb2xsICh1ZmRzLCBzaXplLCAxKTsKKyAgICAgICAgICAgICAgICByZXQgPSBwb2xsICh1 ZmRzLCBzaXplLCAtMSk7CiAKICAgICAgICAgICAgICAgICBpZiAocmV0ID09IDApCiAgICAgICAg ICAgICAgICAgICAgICAgICAvKiB0aW1lb3V0ICovCkBAIC00ODIsNyArNDkzLDEzIEBAIGV2ZW50 X2Rpc3BhdGNoX3BvbGwgKHN0cnVjdCBldmVudF9wb29sICoKICAgICAgICAgICAgICAgICAgICAg ICAgIC8qIHN5cyBjYWxsICovCiAgICAgICAgICAgICAgICAgICAgICAgICBjb250aW51ZTsKIAot ICAgICAgICAgICAgICAgIGZvciAoaSA9IDA7IGkgPCBzaXplOyBpKyspIHsKKyAgICAgICAgICAg ICAgICBpZiAodWZkc1swXS5yZXZlbnRzICE9IDAgJiYgZXZlbnRfcG9vbC0+YnJlYWtlclswXSA+ PSAwKSB7CisgICAgICAgICAgICAgICAgICAgICAgICAvKiBKdXN0IHJlYWQgYWxsIHRoZSBqdW5r IGluIHRoZSBicmVha2VyIHBpcGUuICovCisgICAgICAgICAgICAgICAgICAgICAgICB3aGlsZSAo cmVhZChldmVudF9wb29sLT5icmVha2VyWzBdLCAmeCwgMSkgPiAwKQorICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICA7CisgICAgICAgICAgICAgICAgfQorCisgICAgICAgICAgICAgICAg Zm9yIChpID0gMTsgaSA8IHNpemU7IGkrKykgewogICAgICAgICAgICAgICAgICAgICAgICAgaWYg KCF1ZmRzW2ldLnJldmVudHMpCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRp bnVlOwogCg== ------=_Part_154244850_659138303.1452304799181--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?981529129.154244852.1452304799182.JavaMail.zimbra>