From owner-freebsd-fs@freebsd.org Sat Jan 9 02:00:07 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1FEFDA67E29 for ; Sat, 9 Jan 2016 02:00:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id AF7B315E3 for ; Sat, 9 Jan 2016 02:00:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:D2KIhRWK+QvKr+JgPuwVClqbM+HV8LGtZVwlr6E/grcLSJyIuqrYZhKCt8tkgFKBZ4jH8fUM07OQ6PC+HzRYqb+681k8M7V0HycfjssXmwFySOWkMmbcaMDQUiohAc5ZX0Vk9XzoeWJcGcL5ekGA6ibqtW1aJBzzOEJPK/jvHcaK1oLsh770o8WbSj4LrQT+SIs6FA+xowTVu5teqqpZAYF19CH0pGBVcf9d32JiKAHbtR/94sCt4MwrqHwI6LoJvvRNWqTifqk+UacQTHF/azh0t4XXskzhQAWBrlcVSG4H2k5KDwHf5wDSRJr9siLm8OF63X/JE9fxSOUOWD+hp4JiQxzshSJPYyQ8+WrUjsF1pL9crw+sowR/hYXdNtLGfMFid7/QKItJDVFKWdxcAmkYWtux X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CtBAA7aZBW/61jaINehAxtBohTs1mBZBgKhSNKAoFWEgEBAQEBAQEBgQmCLYIIAQEEAQEBIAQnIAsFCwIBCA4KERkCAgIlAQkmAgQIBwQBHASIDg6xNJA2AQEBAQEBAQEBAQEBAQEBAQEBEgUEgi+EJ4R/hDcBARsZgwaBSQWOMIhdgnSCT4UphEiHbYUyjkwCKQQ3ghEcgXsgNAeEGDqBCAEBAQ X-IronPort-AV: E=Sophos;i="5.20,541,1444708800"; d="scan'208";a="260364885" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 08 Jan 2016 21:00:00 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id E4DF615F55D; Fri, 8 Jan 2016 20:59:59 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 7fOr1uN6_XPd; Fri, 8 Jan 2016 20:59:59 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 5557615F565; Fri, 8 Jan 2016 20:59:59 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id HHb_4WZqdWC7; Fri, 8 Jan 2016 20:59:59 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 342EA15F55D; Fri, 8 Jan 2016 20:59:59 -0500 (EST) Date: Fri, 8 Jan 2016 20:59:59 -0500 (EST) From: Rick Macklem To: Jeff Darcy Cc: Raghavendra G , freebsd-fs , Hubbard Jordan , Xavier Hernandez , Gluster Devel Message-ID: <981529129.154244852.1452304799182.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <1924941590.6473225.1452248249994.JavaMail.zimbra@redhat.com> References: <571237035.145690509.1451437960464.JavaMail.zimbra@uoguelph.ca> <20151230103152.GS13942@ndevos-x240.usersys.redhat.com> <2D8C2729-D556-479B-B4E2-66E1BB222F41@ixsystems.com> <1083933309.146084334.1451517977647.JavaMail.zimbra@uoguelph.ca> <568F6D07.6070500@datalab.es> <1924941590.6473225.1452248249994.JavaMail.zimbra@redhat.com> Subject: Re: [Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_154244850_659138303.1452304799181" X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF43 (Win)/8.0.9_GA_6191) Thread-Topic: FreeBSD port of GlusterFS racks up a lot of CPU usage Thread-Index: nzgvBLPPgcBXuRsf6GlZU17foVfxCE+M3nAj X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jan 2016 02:00:07 -0000 ------=_Part_154244850_659138303.1452304799181 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Jeff Darcy wrote: > > > I don't know anything about gluster's poll implementation so I may > > > be totally wrong, but would it be possible to use an eventfd (or a > > > pipe if eventfd is not supported) to signal the need to add more > > > file descriptors to the poll call ? > > > > > > > > > The poll call should listen on this new fd. When we need to change > > > the fd list, we should simply write to the eventfd or pipe from > > > another thread. This will cause the poll call to return and we will > > > be able to change the fd list without having a short timeout nor > > > having to decide on any trade-off. > > > > > > Thats a nice idea. Based on my understanding of why timeouts are being > > used, this approach can work. > > The own-thread code which preceded the current poll implementation did > something similar, using a pipe fd to be woken up for new *outgoing* > messages. That code still exists, and might provide some insight into > how to do this for the current poll code. I took a look at event-poll.c and found something interesting... - A pipe called "breaker" is already set up by event_pool_new_poll() and closed by event_pool_destroy_poll(), however it never gets used for anything. So, I added a few lines of code that writes a byte to it whenever the list of file descriptors is changed and read when poll() returns, if its revents is set. I also changed the timeout to -1 (infinity) and it seems to work for a trivial test. --> Btw, I also noticed the "changed" variable gets set to 1 on a change, but never reset to 0. I didn't change this, since it looks "racey". (ie. I think you could easily get a race between a thread that clears it and one that adds a new fd.) A slightly safer version of the patch would set a long (100msec ??) timeout instead of -1. Anyhow, I've attached the patch in case anyone would like to try it and will create a bug report for this after I've had more time to test it. (I only use a couple of laptops, so my testing will be minimal.) Thanks for all the help, rick > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > ------=_Part_154244850_659138303.1452304799181 Content-Type: text/x-patch; name=gluster-poll.patch Content-Disposition: attachment; filename=gluster-poll.patch Content-Transfer-Encoding: base64 LS0tIGdsdXN0ZXJmcy0zLjcuNi9saWJnbHVzdGVyZnMvc3JjL2V2ZW50LXBvbGwuYy5zYXYJMjAx Ni0wMS0wNiAxNTo1ODowMy41MjIyODYwMDAgLTA1MDAKKysrIGdsdXN0ZXJmcy0zLjcuNi9saWJn bHVzdGVyZnMvc3JjL2V2ZW50LXBvbGwuYwkyMDE2LTAxLTA4IDE3OjQ1OjI4LjA3ODAyNDAwMCAt MDUwMApAQCAtMTgwLDYgKzE4MCwxNiBAQCBldmVudF9wb29sX25ld19wb2xsIChpbnQgY291bnQs IGludCBldmVuCiAgICAgICAgIHJldHVybiBldmVudF9wb29sOwogfQogCitzdGF0aWMgdm9pZAor ZXZlbnRfcG9vbF9jaGFuZ2VkIChzdHJ1Y3QgZXZlbnRfcG9vbCAqZXZlbnRfcG9vbCkKK3sKKwor ICAgICAgICBldmVudF9wb29sLT5jaGFuZ2VkID0gMTsKKyAgICAgICAgLyogV3JpdGUgYSBieXRl IGludG8gdGhlIGJyZWFrZXIgcGlwZSB0byB3YWtlIHVwIHBvbGwoKS4gKi8KKyAgICAgICAgaWYg KGV2ZW50X3Bvb2wtPmJyZWFrZXJbMV0gPj0gMCkKKyAgICAgICAgICAgICAgICB3cml0ZShldmVu dF9wb29sLT5icmVha2VyWzFdLCAiWCIsIDEpOworfQorCiAKIHN0YXRpYyBpbnQKIGV2ZW50X3Jl Z2lzdGVyX3BvbGwgKHN0cnVjdCBldmVudF9wb29sICpldmVudF9wb29sLCBpbnQgZmQsCkBAIC0y NDQsNyArMjU0LDcgQEAgZXZlbnRfcmVnaXN0ZXJfcG9sbCAoc3RydWN0IGV2ZW50X3Bvb2wgKgog ICAgICAgICAgICAgICAgICAgICAgICAgYnJlYWs7CiAgICAgICAgICAgICAgICAgfQogCi0gICAg ICAgICAgICAgICAgZXZlbnRfcG9vbC0+Y2hhbmdlZCA9IDE7CisgICAgICAgICAgICAgICAgZXZl bnRfcG9vbF9jaGFuZ2VkKGV2ZW50X3Bvb2wpOwogCiAgICAgICAgIH0KIHVubG9jazoKQEAgLTI3 NSw3ICsyODUsNyBAQCBldmVudF91bnJlZ2lzdGVyX3BvbGwgKHN0cnVjdCBldmVudF9wb29sCiAg ICAgICAgICAgICAgICAgfQogCiAgICAgICAgICAgICAgICAgZXZlbnRfcG9vbC0+cmVnW2lkeF0g PSAgZXZlbnRfcG9vbC0+cmVnWy0tZXZlbnRfcG9vbC0+dXNlZF07Ci0gICAgICAgICAgICAgICAg ZXZlbnRfcG9vbC0+Y2hhbmdlZCA9IDE7CisgICAgICAgICAgICAgICAgZXZlbnRfcG9vbF9jaGFu Z2VkKGV2ZW50X3Bvb2wpOwogICAgICAgICB9CiB1bmxvY2s6CiAgICAgICAgIHB0aHJlYWRfbXV0 ZXhfdW5sb2NrICgmZXZlbnRfcG9vbC0+bXV0ZXgpOwpAQCAtMzUwLDcgKzM2MCw3IEBAIGV2ZW50 X3NlbGVjdF9vbl9wb2xsIChzdHJ1Y3QgZXZlbnRfcG9vbCAKICAgICAgICAgICAgICAgICB9CiAK ICAgICAgICAgICAgICAgICBpZiAocG9sbF9pbiArIHBvbGxfb3V0ID4gLTIpCi0gICAgICAgICAg ICAgICAgICAgICAgICBldmVudF9wb29sLT5jaGFuZ2VkID0gMTsKKyAgICAgICAgICAgICAgICAg ICAgICAgIGV2ZW50X3Bvb2xfY2hhbmdlZChldmVudF9wb29sKTsKICAgICAgICAgfQogdW5sb2Nr OgogICAgICAgICBwdGhyZWFkX211dGV4X3VubG9jayAoJmV2ZW50X3Bvb2wtPm11dGV4KTsKQEAg LTQ0OCw2ICs0NTgsNyBAQCBldmVudF9kaXNwYXRjaF9wb2xsIChzdHJ1Y3QgZXZlbnRfcG9vbCAq CiAgICAgICAgIGludCAgICAgICAgICAgICAgc2l6ZSA9IDA7CiAgICAgICAgIGludCAgICAgICAg ICAgICAgaSA9IDA7CiAgICAgICAgIGludCAgICAgICAgICAgICAgcmV0ID0gLTE7CisgICAgICAg IGNoYXIgICAgICAgICAgICAgeDsKIAogICAgICAgICBHRl9WQUxJREFURV9PUl9HT1RPICgiZXZl bnQiLCBldmVudF9wb29sLCBvdXQpOwogCkBAIC00NzIsNyArNDgzLDcgQEAgZXZlbnRfZGlzcGF0 Y2hfcG9sbCAoc3RydWN0IGV2ZW50X3Bvb2wgKgogICAgICAgICAgICAgICAgIHNpemUgPSBldmVu dF9kaXNwYXRjaF9wb2xsX3Jlc2l6ZSAoZXZlbnRfcG9vbCwgdWZkcywgc2l6ZSk7CiAgICAgICAg ICAgICAgICAgdWZkcyA9IGV2ZW50X3Bvb2wtPmV2Y2FjaGU7CiAKLSAgICAgICAgICAgICAgICBy ZXQgPSBwb2xsICh1ZmRzLCBzaXplLCAxKTsKKyAgICAgICAgICAgICAgICByZXQgPSBwb2xsICh1 ZmRzLCBzaXplLCAtMSk7CiAKICAgICAgICAgICAgICAgICBpZiAocmV0ID09IDApCiAgICAgICAg ICAgICAgICAgICAgICAgICAvKiB0aW1lb3V0ICovCkBAIC00ODIsNyArNDkzLDEzIEBAIGV2ZW50 X2Rpc3BhdGNoX3BvbGwgKHN0cnVjdCBldmVudF9wb29sICoKICAgICAgICAgICAgICAgICAgICAg ICAgIC8qIHN5cyBjYWxsICovCiAgICAgICAgICAgICAgICAgICAgICAgICBjb250aW51ZTsKIAot ICAgICAgICAgICAgICAgIGZvciAoaSA9IDA7IGkgPCBzaXplOyBpKyspIHsKKyAgICAgICAgICAg ICAgICBpZiAodWZkc1swXS5yZXZlbnRzICE9IDAgJiYgZXZlbnRfcG9vbC0+YnJlYWtlclswXSA+ PSAwKSB7CisgICAgICAgICAgICAgICAgICAgICAgICAvKiBKdXN0IHJlYWQgYWxsIHRoZSBqdW5r IGluIHRoZSBicmVha2VyIHBpcGUuICovCisgICAgICAgICAgICAgICAgICAgICAgICB3aGlsZSAo cmVhZChldmVudF9wb29sLT5icmVha2VyWzBdLCAmeCwgMSkgPiAwKQorICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICA7CisgICAgICAgICAgICAgICAgfQorCisgICAgICAgICAgICAgICAg Zm9yIChpID0gMTsgaSA8IHNpemU7IGkrKykgewogICAgICAgICAgICAgICAgICAgICAgICAgaWYg KCF1ZmRzW2ldLnJldmVudHMpCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRp bnVlOwogCg== ------=_Part_154244850_659138303.1452304799181--