Date: Wed, 18 Aug 2010 23:11:44 +0400 From: pluknet <pluknet@gmail.com> To: Kostik Belousov <kostikbel@gmail.com> Cc: FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: LOR on nfs: vfs_vnops.c:301 kern_descrip.c:1580 Message-ID: <AANLkTi=V8bumpGKgi1aLXxhJkdKdpG1jfyrcXbMyc3Yw@mail.gmail.com> In-Reply-To: <20100818134623.GU2396@deviant.kiev.zoral.com.ua> References: <AANLkTimJ=d06D2z24QyRQ98zEa1Pemk4=vkNGLNiX90N@mail.gmail.com> <AANLkTi=22WWy8Z5VWqri1p70iRzyocD9bWvNHwJavZ%2Bj@mail.gmail.com> <20100816185456.GU2396@deviant.kiev.zoral.com.ua> <AANLkTimfZ=UCiX9eo9XCwcROwViFCxJoB-rD1JxdAt2B@mail.gmail.com> <20100817160445.GO2396@deviant.kiev.zoral.com.ua> <AANLkTimyk0ZUnHpxxJuJtTya2U3npj_5Bm8brmwHfzr3@mail.gmail.com> <AANLkTimXxF0US60NsbOV_HvRrib7SJWmrLqpEpznPTxB@mail.gmail.com> <20100818134623.GU2396@deviant.kiev.zoral.com.ua>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
On 18 August 2010 17:46, Kostik Belousov <kostikbel@gmail.com> wrote:
> On Wed, Aug 18, 2010 at 02:43:19PM +0400, pluknet wrote:
>> On 18 August 2010 12:07, pluknet <pluknet@gmail.com> wrote:
>> > On 17 August 2010 20:04, Kostik Belousov <kostikbel@gmail.com> wrote:
>> >
>> >>
>> >> Also please take a note of the John' suggestion to use the taskqueue.
>> >
>> > I decided to go this road. Thank you both.
>> > Now I do nfs buildkernel survive and prepare some benchmark results.
>> >
>>
>> So, I modified the patch to defer proc_create() with taskqueue(9).
>> Below is `time make -j5 buildkernel WITHOUT_MODULES=yes` perf. evaluation.
>> Done on 4-way CPU on clean /usr/obj with /usr/src & /usr/obj both
>> nfs-mounted over 1Gbit LAN.
>>
>> clean old
>> 1137.985u 239.411s 7:42.15 298.0% 6538+2133k 87+43388io 226pf+0w
>>
>> clean new
>> 1134.755u 240.032s 7:41.25 298.0% 6553+2133k 87+43367io 224pf+0w
>>
>> Patch needs polishing, though it generally works.
>> Not sure if shep_chan (or whatever name it will get) needs locking.
> As I said yesterday, if several requests to create nfsiod coming one
> after another, you would loose all but the last.
>
> You should put the requests into the list, probably protected by
> nfs_iod_mtx.
>
How about this patch? Still several things to ask.
1) I used malloc instance w/ M_NOWAIT, since it's called with nfs_iod_mtx held.
2) Probably busy/done gymnastics is a wrong mess. Your help is appreciated.
3) if (1) is fine, is it right to use fail: logic (i.e. set
NFSIOD_NOT_AVAILABLE)
on memory shortage? Not tested.
There are debug printf() left intentionally to see how 3 contexts run under load
to each other. I attached these messages as well if that makes sense.
--
wbr,
pluknet
[-- Attachment #2 --]
new: nip: 0xffffff00027dec60
new: ni_inst: 0xffffffff80e6df60
new: ni_iod: 0
tq: SLIST nip: 0xffffff00027dec60
tq: nip: 0xffffff00027dec60
tq: ni_inst: 0xffffffff80e6df60
tq: ni_iod: 0
instance: 0xffffffff80e6df60
myiod: 0
new: nip: 0xffffff000250c040
new: ni_inst: 0xffffffff80e6df60
new: ni_iod: 0
tq: SLIST nip: 0xffffff000250c040
tq: nip: 0xffffff000250c040
tq: ni_inst: 0xffffffff80e6df60
tq: ni_iod: 0
instance: 0xffffffff80e6df60
myiod: 0
new: nip: 0xffffff00027dec40
new: ni_inst: 0xffffffff80e6df60
new: ni_iod: 0
tq: SLIST nip: 0xffffff00027dec40
tq: nip: 0xffffff00027dec40
tq: ni_inst: 0xffffffff80e6df60
tq: ni_iod: 0
instance: 0xffffffff80e6df60
myiod: 0
new: nip: 0xffffff0002fea000
new: ni_inst: 0xffffffff80e6df64
new: ni_iod: 1
tq: SLIST nip: 0xffffff0002fea000
tq: nip: 0xffffff0002fea000new: nip: 0xffffff000296fc80
tq: ni_inst: 0xffffffff80e6df64
new: ni_inst: 0xffffffff80e6df68
tq: ni_iod: 1
new: ni_iod: 2
instance: 0xffffffff80e6df64
myiod: 1
tq: SLIST nip: 0xffffff000296fc80
tq: nip: 0xffffff000296fc80new: nip: 0xffffff000296fc60
tq: ni_inst: 0xffffffff80e6df68
new: ni_inst: 0xffffffff80e6df6c
tq: ni_iod: 2
new: ni_iod: 3
instance: 0xffffffff80e6df68
myiod: 2
tq: SLIST nip: 0xffffff000296fc60
tq: nip: 0xffffff000296fc60
tq: ni_inst: 0xffffffff80e6df6c
tq: ni_iod: 3
instance: 0xffffffff80e6df6c
myiod: 3
new: nip: 0xffffff000250d680
new: ni_inst: 0xffffffff80e6df70
new: ni_iod: 4
tq: SLIST nip: 0xffffff000250d680
tq: nip: 0xffffff000250d680
tq: ni_inst: 0xffffffff80e6df70
tq: ni_iod: 4
instance: 0xffffffff80e6df70
myiod: 4
new: nip: 0xffffff000250c020
new: ni_inst: 0xffffffff80e6df74
new: ni_iod: 5
tq: SLIST nip: 0xffffff000250c020
tq: nip: 0xffffff000250c020
tq: ni_inst: 0xffffffff80e6df74
tq: ni_iod: 5
instance: 0xffffffff80e6df74
myiod: 5
new: nip: 0xffffff00027dec00
new: ni_inst: 0xffffffff80e6df74
new: ni_iod: 5
tq: SLIST nip: 0xffffff00027dec00
tq: nip: 0xffffff00027dec00
tq: ni_inst: 0xffffffff80e6df74
tq: ni_iod: 5
instance: 0xffffffff80e6df74
myiod: 5
new: nip: 0xffffff000250c000
new: ni_inst: 0xffffffff80e6df78
new: ni_iod: 6
tq: SLIST nip: 0xffffff000250c000
tq: nip: 0xffffff000250c000
tq: ni_inst: 0xffffffff80e6df78
tq: ni_iod: 6
instance: 0xffffffff80e6df78
myiod: 6
new: nip: 0xffffff000296fc40
new: ni_inst: 0xffffffff80e6df7c
new: ni_iod: 7
tq: SLIST nip: 0xffffff000296fc40
tq: nip: 0xffffff000296fc40new: nip: 0xffffff000296fc20
tq: ni_inst: 0xffffffff80e6df7c
new: ni_inst: 0xffffffff80e6df80
new: ni_iod: 8
tq: ni_iod: 7
tq: SLIST nip: 0xffffff000296fc20
tq: nip: 0xffffff000296fc20instance: 0xffffffff80e6df7c
tq: ni_inst: 0xffffffff80e6df80
myiod: 7
tq: ni_iod: 8
instance: 0xffffffff80e6df80
myiod: 8
new: nip: 0xffffff00027debe0
new: ni_inst: 0xffffffff80e6df84
new: ni_iod: 9
tq: SLIST nip: 0xffffff00027debe0
tq: nip: 0xffffff00027debe0
tq: ni_inst: 0xffffffff80e6df84
tq: ni_iod: 9
instance: 0xffffffff80e6df84
myiod: 9
new: nip: 0xffffff00027debc0
new: ni_inst: 0xffffffff80e6df88
new: ni_iod: 10
tq: SLIST nip: 0xffffff00027debc0
tq: nip: 0xffffff00027debc0
tq: ni_inst: 0xffffffff80e6df88
tq: ni_iod: 10
instance: 0xffffffff80e6df88
myiod: 10
new: nip: 0xffffff00027dec00
new: ni_inst: 0xffffffff80e6df8c
new: ni_iod: 11
tq: SLIST nip: 0xffffff00027dec00
tq: nip: 0xffffff00027dec00
tq: ni_inst: 0xffffffff80e6df8c
tq: ni_iod: 11
instance: 0xffffffff80e6df8c
myiod: 11
new: nip: 0xffffff000296fc00
new: ni_inst: 0xffffffff80e6df90
new: ni_iod: 12
tq: SLIST nip: 0xffffff000296fc00
tq: nip: 0xffffff000296fc00
tq: ni_inst: 0xffffffff80e6df90
tq: ni_iod: 12
instance: 0xffffffff80e6df90
myiod: 12
new: nip: 0xffffff000296fbe0
new: ni_inst: 0xffffffff80e6df94
new: ni_iod: 13
tq: SLIST nip: 0xffffff000296fbe0
tq: nip: 0xffffff000296fbe0
tq: ni_inst: 0xffffffff80e6df94
tq: ni_iod: 13
instance: 0xffffffff80e6df94
myiod: 13
new: nip: 0xffffff000250d680
new: ni_inst: 0xffffffff80e6df98
new: ni_iod: 14
tq: SLIST nip: 0xffffff000250d680
tq: nip: 0xffffff000250d680
tq: ni_inst: 0xffffffff80e6df98
tq: ni_iod: 14
instance: 0xffffffff80e6df98
myiod: 14
new: nip: 0xffffff00027deba0
new: ni_inst: 0xffffffff80e6df9c
new: ni_iod: 15
tq: SLIST nip: 0xffffff00027deba0
tq: nip: 0xffffff00027deba0
tq: ni_inst: 0xffffffff80e6df9c
tq: ni_iod: 15
instance: 0xffffffff80e6df9c
myiod: 15
new: nip: 0xffffff000296fbc0
new: ni_inst: 0xffffffff80e6dfa0
new: ni_iod: 16
tq: SLIST nip: 0xffffff000296fbc0
tq: nip: 0xffffff000296fbc0
tq: ni_inst: 0xffffffff80e6dfa0
tq: ni_iod: 16
instance: 0xffffffff80e6dfa0
myiod: 16
new: nip: 0xffffff00027dec60
new: ni_inst: 0xffffffff80e6dfa4
new: ni_iod: 17
tq: SLIST nip: 0xffffff00027dec60
tq: nip: 0xffffff00027dec60new: nip: 0xffffff00027deb80
tq: ni_inst: 0xffffffff80e6dfa4
new: ni_inst: 0xffffffff80e6dfa8
tq: ni_iod: 17
new: ni_iod: 18
instance: 0xffffffff80e6dfa4
myiod: 17
tq: SLIST nip: 0xffffff00027deb80
tq: nip: 0xffffff00027deb80new: nip: 0xffffff000250c060
tq: ni_inst: 0xffffffff80e6dfa8
new: ni_inst: 0xffffffff80e6dfac
tq: ni_iod: 18
new: ni_iod: 19
instance: 0xffffffff80e6dfa8
myiod: 18
tq: SLIST nip: 0xffffff000250c060
tq: nip: 0xffffff000250c060
tq: ni_inst: 0xffffffff80e6dfac
tq: ni_iod: 19
instance: 0xffffffff80e6dfac
myiod: 19
[-- Attachment #3 --]
Index: sys/nfsclient/nfs_subs.c
===================================================================
--- sys/nfsclient/nfs_subs.c (revision 211279)
+++ sys/nfsclient/nfs_subs.c (working copy)
@@ -59,6 +59,7 @@
#include <sys/sysent.h>
#include <sys/syscall.h>
#include <sys/sysproto.h>
+#include <sys/taskqueue.h>
#include <vm/vm.h>
#include <vm/vm_object.h>
@@ -117,6 +118,7 @@
struct nfs_bufq nfs_bufq;
static struct mtx nfs_xid_mtx;
+struct task nfs_nfsiodnew_task;
/*
* and the reverse mapping from generic to Version 2 procedure numbers
@@ -354,6 +356,7 @@
*/
mtx_init(&nfs_iod_mtx, "NFS iod lock", NULL, MTX_DEF);
mtx_init(&nfs_xid_mtx, "NFS xid lock", NULL, MTX_DEF);
+ TASK_INIT(&nfs_nfsiodnew_task, 0, nfs_nfsiodnew_tq, NULL);
nfs_pbuf_freecnt = nswbuf / 2 + 1;
Index: sys/nfsclient/nfs_nfsiod.c
===================================================================
--- sys/nfsclient/nfs_nfsiod.c (revision 211279)
+++ sys/nfsclient/nfs_nfsiod.c (working copy)
@@ -59,6 +59,7 @@
#include <sys/fcntl.h>
#include <sys/lockf.h>
#include <sys/mutex.h>
+#include <sys/taskqueue.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
@@ -75,6 +76,17 @@
static void nfssvc_iod(void *);
+struct nfsiod_str {
+ SLIST_ENTRY(nfsiod_str) ni_list;
+ int *ni_inst;
+ int ni_iod;
+ int ni_error;
+ int ni_busy;
+ int ni_done;
+};
+static SLIST_HEAD(, nfsiod_str) nfsiodhead =
+ SLIST_HEAD_INITIALIZER(nfsiodhead);
+
static int nfs_asyncdaemon[NFS_MAXASYNCDAEMON];
SYSCTL_DECL(_vfs_nfs);
@@ -159,11 +171,34 @@
sizeof (nfs_iodmax), sysctl_iodmax, "IU",
"Max number of nfsiod kthreads");
+void
+nfs_nfsiodnew_tq(__unused void *arg, int pending)
+{
+ struct nfsiod_str *nip;
+
+ mtx_lock(&nfs_iod_mtx);
+ SLIST_FOREACH(nip, &nfsiodhead, ni_list) {
+ printf("tq: SLIST nip: %p\n", nip);
+ if (nip->ni_busy == 0) {
+ nip->ni_busy = 1;
+ break;
+ }
+ }
+ mtx_unlock(&nfs_iod_mtx);
+ KASSERT(nip != NULL, ("nfs_nfsiodnew_tq: nip is NULL"));
+ printf("tq: nip: %p\n", nip);
+ printf("tq: ni_inst: %p\n", nip->ni_inst);
+ printf("tq: ni_iod: %d\n", nip->ni_iod);
+ nip->ni_error = kproc_create(nfssvc_iod, nip->ni_inst, NULL,
+ RFHIGHPID, 0, "nfsiod %d", nip->ni_iod);
+ nip->ni_done = 1;
+}
+
int
nfs_nfsiodnew(int set_iodwant)
{
- int error, i;
- int newiod;
+ int i, newiod, error;
+ struct nfsiod_str *nip, *nip_temp;
if (nfs_numasync >= nfs_iodmax)
return (-1);
@@ -178,17 +213,34 @@
return (-1);
if (set_iodwant > 0)
nfs_iodwant[i] = NFSIOD_CREATED_FOR_NFS_ASYNCIO;
+ nip = malloc(sizeof(*nip), M_TEMP, M_NOWAIT | M_ZERO);
+ if (nip == NULL)
+ goto fail;
+ nip->ni_inst = nfs_asyncdaemon + i;
+ nip->ni_iod = newiod;
+ SLIST_INSERT_HEAD(&nfsiodhead, nip, ni_list);
mtx_unlock(&nfs_iod_mtx);
- error = kproc_create(nfssvc_iod, nfs_asyncdaemon + i, NULL, RFHIGHPID,
- 0, "nfsiod %d", newiod);
+ printf("new: nip: %p\n", nip);
+ printf("new: ni_inst: %p\n", nip->ni_inst);
+ printf("new: ni_iod: %d\n", nip->ni_iod);
+ taskqueue_enqueue(taskqueue_thread, &nfs_nfsiodnew_task);
mtx_lock(&nfs_iod_mtx);
- if (error) {
- if (set_iodwant > 0)
- nfs_iodwant[i] = NFSIOD_NOT_AVAILABLE;
- return (-1);
+ error = nip->ni_error;
+ SLIST_FOREACH_SAFE(nip, &nfsiodhead, ni_list, nip_temp) {
+ if (nip->ni_busy != 0 && nip->ni_done != 0) {
+ SLIST_REMOVE(&nfsiodhead, nip, nfsiod_str, ni_list);
+ free(nip, M_TEMP);
+ break;
+ }
}
+ if (error)
+ goto fail;
nfs_numasync++;
return (newiod);
+fail:
+ if (set_iodwant > 0)
+ nfs_iodwant[i] = NFSIOD_NOT_AVAILABLE;
+ return (-1);
}
static void
@@ -231,6 +283,8 @@
mtx_lock(&nfs_iod_mtx);
myiod = (int *)instance - nfs_asyncdaemon;
+ printf("instance: %p\n", instance);
+ printf("myiod: %d\n", myiod);
/*
* Main loop
*/
Index: sys/nfsclient/nfs.h
===================================================================
--- sys/nfsclient/nfs.h (revision 211279)
+++ sys/nfsclient/nfs.h (working copy)
@@ -125,6 +125,7 @@
extern struct nfsstats nfsstats;
extern struct mtx nfs_iod_mtx;
+extern struct task nfs_nfsiodnew_task;
extern int nfs_numasync;
extern unsigned int nfs_iodmax;
@@ -253,6 +254,7 @@
struct ucred *cred, struct thread *td);
int nfs_readdirrpc(struct vnode *, struct uio *, struct ucred *);
int nfs_nfsiodnew(int);
+void nfs_nfsiodnew_tq(__unused void *, int);
int nfs_asyncio(struct nfsmount *, struct buf *, struct ucred *, struct thread *);
int nfs_doio(struct vnode *, struct buf *, struct ucred *, struct thread *);
void nfs_doio_directwrite (struct buf *);
help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTi=V8bumpGKgi1aLXxhJkdKdpG1jfyrcXbMyc3Yw>