From owner-freebsd-stable@freebsd.org Wed Oct 11 10:25:12 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E898E27BEC for ; Wed, 11 Oct 2017 10:25:12 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-vk0-x22b.google.com (mail-vk0-x22b.google.com [IPv6:2607:f8b0:400c:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1CF913571 for ; Wed, 11 Oct 2017 10:25:12 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by mail-vk0-x22b.google.com with SMTP id g69so662947vke.5 for ; Wed, 11 Oct 2017 03:25:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=ynbPhq6n/lxAkY4dyBOrdqkhbbCbIk5jdgY6MsrsaPU=; b=jYnWlq6icjxZqVWUxaFn/gV/yLS3T4knet9RhGw9wnwBaa4f6dWpe4k+CJ9RFrxbJS tlaVjGYrPskHi96UqLKbN6KqPJuRfmiRQLagpeCNEkfvrH0cJ1GPBBQOBgqOBhkBh2V6 kkVBnaeCeH8Uzj2LT3qTA+TqWucoIG8IMZdoNH85OPidGuNPqCVc3yiwgtVwNaVlJYZv 2BA4GhD+UcoBc9KoA2I2aimYWqOBJIHTTnWcQbkvT3dvPe7U+8xyacqeTdygQoX0dQt1 Z4tnJvgRrZte/dFLX5knnOctzjjP03mLygoWPHeM6vIzxRAXMRjltDlnMvVIdkf49Ib5 sNvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=ynbPhq6n/lxAkY4dyBOrdqkhbbCbIk5jdgY6MsrsaPU=; b=dpkF9z6h5NUXSS5iMNADkNWot/NzBF3BrbcC4Fvz9D5saJFbix5UWqGIY5eQOvrO16 OqXgS0ASJgeMPxwtXci+BY4daESCMbEpnbsFgrQ1HujGVsGpbHQOeskFDrTijoveK56o Mf2Clsf7tJgZ8c5KAZhV3bxwXGCpDnp3bwqsFk560Cbh7duIctcIDNfQXSDCR1HLBC7d NAGFIES7pN8PvMOW6VCF+J0lp4AuizRtL1rYKpjGKicPxKvhrmPUbDy98J30k5NhpP7b gD7aGcdhSIVlQOlgVYUw8WLTaOc9/faCdYJuZx3S8t6fm6mcRJCmozjpKNAMqUG5IWDh kWDQ== X-Gm-Message-State: AMCzsaXLXYcno6NB4N9jR17L33ZTEA2rbrBSGwYBwdQKtelW8WpeV8V/ 3yxMgB18sTOpqlR3Woc3FmGARNeiEc//GOCPEWA= X-Google-Smtp-Source: AOwi7QBp+gZ5JwnktyZBTVTMgwIh3LtqX8ZwIGIdbRtX8hKOqUQnqdUfr4KHOVpaAaWlCuZFwlusMl7r1IvLDy69nqk= X-Received: by 10.31.230.5 with SMTP id d5mr8478319vkh.64.1507717511105; Wed, 11 Oct 2017 03:25:11 -0700 (PDT) MIME-Version: 1.0 Sender: etnapierala@gmail.com Received: by 10.176.66.67 with HTTP; Wed, 11 Oct 2017 03:25:10 -0700 (PDT) In-Reply-To: References: From: Edward Napierala Date: Wed, 11 Oct 2017 11:25:10 +0100 X-Google-Sender-Auth: cO9F9NcT73LlIZV7JcQ9T3QhciY Message-ID: Subject: Re: zfs, iSCSI and volmode=dev To: "Eugene M. Zheganin" Cc: FreeBSD Stable Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Oct 2017 10:25:12 -0000 2017-10-09 17:38 GMT+01:00 Eugene M. Zheganin : > Hi, > > On 27.09.2017 16:07, Edward Napierala wrote: > >> 2017-08-30 11:45 GMT+02:00 Eugene M. Zheganin > emz@norma.perm.ru>>: >> >> Hi, >> >> >> I have an iSCSI production system that exports a large number of >> zvols as the iSCSI targets. System is running FreeBSD >> 11.0-RELEASE-p7 and initially all of the zvols were confugured >> with default volmode. I've read that it's recommended to use them >> in dev mode, so the system isn't bothered with all of these geom >> structures, so I've switched all of the zvols to dev mode, then I >> exported/imported the pools back. Surprisingly, the performance >> has fallen down like 10 times (200-300 Mbits/sec against 3-4 >> Gbits/sec previously). After observing for 5 minutes the ESXes >> trying to boot up, and doing this extremely slowly, I switched the >> volmode back to default, then again exported/imported the pools. >> The performance went back to normal. >> >> >> So... why did this happen ? The result seems to be >> counter-intuitive. At least not obvious to me. >> >> >> I don't really have an answer - mav@ would be the best person to ask. >> Based >> on his description, "ZVOLs in GEOM mode don't support DPO/FUA cache >> control >> bits, had to chunk large I/Os into MAXPHYS-sized pieces and go through >> GEOM." >> There also used to be so that TRIM was only supported in the "dev" mode, >> but >> that changed a while ago. >> >> Yeah, but you mean dev is faster by design. So was my first thought too, > but it seems like the opposite. Default volmode is geom, and it's much > faster than dev. I'd expect it to be faster, but it might be it interferes with something. For example, if Windows forces direct media access (bypassing the disk cache), and going through GEOM made the target ignore this bit, that could hurt performance. (Note that it's just an idea; I have no idea what's actually happening there.) My first suspect would be TRIM, but then it should work in both modes. Might be worth checking to be sure, by disabling the "unmap" option in ctl.conf (option "unmap" "off") to see if it makes a difference.