From owner-freebsd-stable@freebsd.org Fri Sep 16 09:16:31 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9F80BDC078 for ; Fri, 16 Sep 2016 09:16:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 540362EA; Fri, 16 Sep 2016 09:16:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u8G9GI0V092376 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 16 Sep 2016 12:16:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u8G9GI0V092376 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u8G9GHBO092375; Fri, 16 Sep 2016 12:16:17 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 16 Sep 2016 12:16:17 +0300 From: Konstantin Belousov To: John Baldwin Cc: Slawa Olhovchenkov , freebsd-stable@freebsd.org Subject: Re: nginx and FreeBSD11 Message-ID: <20160916091617.GT38409@kib.kiev.ua> References: <20160907191348.GD22212@zxy.spb.ru> <1823460.vTm8IvUQsF@ralph.baldwin.cx> <20160915174948.GW2840@zxy.spb.ru> <15357564.UUkYdbUZHc@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <15357564.UUkYdbUZHc@ralph.baldwin.cx> User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Sep 2016 09:16:31 -0000 On Thu, Sep 15, 2016 at 11:54:12AM -0700, John Baldwin wrote: > On Thursday, September 15, 2016 08:49:48 PM Slawa Olhovchenkov wrote: > > On Thu, Sep 15, 2016 at 10:28:11AM -0700, John Baldwin wrote: > > > > > On Thursday, September 15, 2016 05:41:03 PM Slawa Olhovchenkov wrote: > > > > On Wed, Sep 07, 2016 at 10:13:48PM +0300, Slawa Olhovchenkov wrote: > > > > > > > > > I am have strange issuse with nginx on FreeBSD11. > > > > > I am have FreeBSD11 instaled over STABLE-10. > > > > > nginx build for FreeBSD10 and run w/o recompile work fine. > > > > > nginx build for FreeBSD11 crushed inside rbtree lookups: next node > > > > > totaly craped. > > > > > > > > > > I am see next potential cause: > > > > > > > > > > 1) clang 3.8 code generation issuse > > > > > 2) system library issuse > > > > > > > > > > may be i am miss something? > > > > > > > > > > How to find real cause? > > > > > > > > I find real cause and this like show-stopper for RELEASE. > > > > I am use nginx with AIO and AIO from one nginx process corrupt memory > > > > from other nginx process. Yes, this is cross-process memory > > > > corruption. > > > > > > > > Last case, core dumped proccess with pid 1060 at 15:45:14. > > > > Corruped memory at 0x860697000. > > > > I am know about good memory at 0x86067f800. > > > > Dumping (form core) this region to file and analyze by hexdump I am > > > > found start of corrupt region -- offset 0000c8c0 from 0x86067f800. > > > > 0x86067f800+0xc8c0 = 0x86068c0c0 > > > > > > > > I am preliminary enabled debuggin of AIO started operation to nginx > > > > error log (memory address, file name, offset and size of transfer). > > > > > > > > grep -i 86068c0c0 error.log near 15:45:14 give target file. > > > > grep ce949665cbcd.hls error.log near 15:45:14 give next result: > > > > > > > > 2016/09/15 15:45:13 [notice] 1055#0: *11659936 AIO_RD 000000082065DB60 start 000000086068C0C0 561b0 2646736 ce949665cbcd.hls > > > > 2016/09/15 15:45:14 [notice] 1060#0: *10998125 AIO_RD 000000081F1FFB60 start 000000086FF2C0C0 6cdf0 140016832 ce949665cbcd.hls > > > > 2016/09/15 15:45:14 [notice] 1055#0: *11659936 AIO_RD 00000008216B6B60 start 000000086472B7C0 7ff70 2999424 ce949665cbcd.hls > > > > > > Does nginx only use AIO for regular files or does it also use it with sockets? > > > > Only for regular files. > > > > > You can try using this patch as a diagnostic (you will need to > > > run with INVARIANTS enabled, > > > > How much debugs produced? > > I am have about 5-10K aio's per second. > > > > > or at least enabled for vfs_aio.c): > > > > How I can do this (enable INVARIANTS for vfs_aio.c)? > > Include INVARIANT_SUPPORT in your kernel and add a line with: > > #define INVARIANTS > > at the top of sys/kern/vfs_aio.c. > > > > > > Index: vfs_aio.c > > > =================================================================== > > > --- vfs_aio.c (revision 305811) > > > +++ vfs_aio.c (working copy) > > > @@ -787,6 +787,8 @@ aio_process_rw(struct kaiocb *job) > > > * aio_aqueue() acquires a reference to the file that is > > > * released in aio_free_entry(). > > > */ > > > + KASSERT(curproc->p_vmspace == job->userproc->p_vmspace, > > > + ("%s: vmspace mismatch", __func__)); > > > if (cb->aio_lio_opcode == LIO_READ) { > > > auio.uio_rw = UIO_READ; > > > if (auio.uio_resid == 0) > > > @@ -1054,6 +1056,8 @@ aio_switch_vmspace(struct kaiocb *job) > > > { > > > > > > vmspace_switch_aio(job->userproc->p_vmspace); > > > + KASSERT(curproc->p_vmspace == job->userproc->p_vmspace, > > > + ("%s: vmspace mismatch", __func__)); > > > } > > > > > > If this panics, then vmspace_switch_aio() is not working for > > > some reason. > > > > This issuse caused rare, this panic produced with issuse or on any aio > > request? (this is production server) > > It would panic in the case that we are going to write into the wrong > process (so about as rare as your issue). > vmspace_switch_aio() allows context switching with old curpmap and new proc->p_vmspace. This is a weird condition, where curproc->p_vmspace->vm_pmap is not equal to curcpu->pc_curpmap. I do not see an obvious place which would immediately break, e.g. even for context switch between assignment of newvm to p_vmspace and pmap_activate(), the context-switch call to pmap_activate_sw() seems to do right thing. Still, just in case, try this diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c index a23468e..fbaa6c1 100644 --- a/sys/vm/vm_map.c +++ b/sys/vm/vm_map.c @@ -481,6 +481,7 @@ vmspace_switch_aio(struct vmspace *newvm) if (oldvm == newvm) return; + critical_enter(); /* * Point to the new address space and refer to it. */ @@ -489,6 +490,7 @@ vmspace_switch_aio(struct vmspace *newvm) /* Activate the new mapping. */ pmap_activate(curthread); + critical_exit(); /* Remove the daemon's reference to the old address space. */ KASSERT(oldvm->vm_refcnt > 1,