From owner-freebsd-stable@freebsd.org Tue Sep 20 01:05:54 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD2E6BE0B83 for ; Tue, 20 Sep 2016 01:05:54 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mail.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F2D7CF7 for ; Tue, 20 Sep 2016 01:05:54 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by mail.baldwin.cx (Postfix) with ESMTPSA id 0BB1F10AF8C; Mon, 19 Sep 2016 21:05:53 -0400 (EDT) From: John Baldwin To: Slawa Olhovchenkov Cc: freebsd-stable@freebsd.org Subject: Re: nginx and FreeBSD11 Date: Mon, 19 Sep 2016 18:05:46 -0700 Message-ID: <2122051.7RxZBKUSFc@ralph.baldwin.cx> User-Agent: KMail/4.14.10 (FreeBSD/11.0-PRERELEASE; KDE/4.14.10; amd64; ; ) In-Reply-To: <20160918162241.GE2960@zxy.spb.ru> References: <20160907191348.GD22212@zxy.spb.ru> <1823460.vTm8IvUQsF@ralph.baldwin.cx> <20160918162241.GE2960@zxy.spb.ru> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mail.baldwin.cx); Mon, 19 Sep 2016 21:05:53 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.99.2 at mail.baldwin.cx X-Virus-Status: Clean X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Sep 2016 01:05:54 -0000 On Sunday, September 18, 2016 07:22:41 PM Slawa Olhovchenkov wrote: > On Thu, Sep 15, 2016 at 10:28:11AM -0700, John Baldwin wrote: > > > On Thursday, September 15, 2016 05:41:03 PM Slawa Olhovchenkov wrote: > > > On Wed, Sep 07, 2016 at 10:13:48PM +0300, Slawa Olhovchenkov wrote: > > > > > > > I am have strange issuse with nginx on FreeBSD11. > > > > I am have FreeBSD11 instaled over STABLE-10. > > > > nginx build for FreeBSD10 and run w/o recompile work fine. > > > > nginx build for FreeBSD11 crushed inside rbtree lookups: next node > > > > totaly craped. > > > > > > > > I am see next potential cause: > > > > > > > > 1) clang 3.8 code generation issuse > > > > 2) system library issuse > > > > > > > > may be i am miss something? > > > > > > > > How to find real cause? > > > > > > I find real cause and this like show-stopper for RELEASE. > > > I am use nginx with AIO and AIO from one nginx process corrupt memory > > > from other nginx process. Yes, this is cross-process memory > > > corruption. > > > > > > Last case, core dumped proccess with pid 1060 at 15:45:14. > > > Corruped memory at 0x860697000. > > > I am know about good memory at 0x86067f800. > > > Dumping (form core) this region to file and analyze by hexdump I am > > > found start of corrupt region -- offset 0000c8c0 from 0x86067f800. > > > 0x86067f800+0xc8c0 = 0x86068c0c0 > > > > > > I am preliminary enabled debuggin of AIO started operation to nginx > > > error log (memory address, file name, offset and size of transfer). > > > > > > grep -i 86068c0c0 error.log near 15:45:14 give target file. > > > grep ce949665cbcd.hls error.log near 15:45:14 give next result: > > > > > > 2016/09/15 15:45:13 [notice] 1055#0: *11659936 AIO_RD 000000082065DB60 start 000000086068C0C0 561b0 2646736 ce949665cbcd.hls > > > 2016/09/15 15:45:14 [notice] 1060#0: *10998125 AIO_RD 000000081F1FFB60 start 000000086FF2C0C0 6cdf0 140016832 ce949665cbcd.hls > > > 2016/09/15 15:45:14 [notice] 1055#0: *11659936 AIO_RD 00000008216B6B60 start 000000086472B7C0 7ff70 2999424 ce949665cbcd.hls > > > > Does nginx only use AIO for regular files or does it also use it with sockets? > > > > You can try using this patch as a diagnostic (you will need to > > run with INVARIANTS enabled, or at least enabled for vfs_aio.c): > > > > Index: vfs_aio.c > > =================================================================== > > --- vfs_aio.c (revision 305811) > > +++ vfs_aio.c (working copy) > > @@ -787,6 +787,8 @@ aio_process_rw(struct kaiocb *job) > > * aio_aqueue() acquires a reference to the file that is > > * released in aio_free_entry(). > > */ > > + KASSERT(curproc->p_vmspace == job->userproc->p_vmspace, > > + ("%s: vmspace mismatch", __func__)); > > if (cb->aio_lio_opcode == LIO_READ) { > > auio.uio_rw = UIO_READ; > > if (auio.uio_resid == 0) > > @@ -1054,6 +1056,8 @@ aio_switch_vmspace(struct kaiocb *job) > > { > > > > vmspace_switch_aio(job->userproc->p_vmspace); > > + KASSERT(curproc->p_vmspace == job->userproc->p_vmspace, > > + ("%s: vmspace mismatch", __func__)); > > } > > > > If this panics, then vmspace_switch_aio() is not working for > > some reason. > > I am try using next DTrace script: > ==== > #pragma D option dynvarsize=64m > > int req[struct vmspace *, void *]; > self int trace; > > syscall:freebsd:aio_read:entry > { > this->aio = *(struct aiocb *)copyin(arg0, sizeof(struct aiocb)); > req[curthread->td_proc->p_vmspace, this->aio.aio_buf] = curthread->td_proc->p_pid; > } > > fbt:kernel:aio_process_rw:entry > { > self->job = args[0]; > self->trace = 1; > } > > fbt:kernel:aio_process_rw:return > /self->trace/ > { > req[self->job->userproc->p_vmspace, self->job->uaiocb.aio_buf] = 0; > self->job = 0; > self->trace = 0; > } > > fbt:kernel:vn_io_fault:entry > /self->trace && !req[curthread->td_proc->p_vmspace, args[1]->uio_iov[0].iov_base]/ > { > this->buf = args[1]->uio_iov[0].iov_base; > printf("%Y vn_io_fault %p:%p pid %d\n", walltimestamp, curthread->td_proc->p_vmspace, this->buf, req[curthread->td_proc->p_vmspace, this->buf]); > } > === > > And don't got any messages near nginx core dump. > What I can check next? > May be check context/address space switch for kernel process? Which CPU are you using? Perhaps try disabling PCID support (I think vm.pmap.pcid_enabled=0 from loader prompt or loader.conf)? (Wondering if pmap_activate() is somehow not switching) -- John Baldwin