From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 19 16:25:46 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EAC18E82 for ; Thu, 19 Sep 2013 16:25:45 +0000 (UTC) (envelope-from guy.helmer@gmail.com) Received: from mail-ye0-x234.google.com (mail-ye0-x234.google.com [IPv6:2607:f8b0:4002:c04::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AD192295D for ; Thu, 19 Sep 2013 16:25:45 +0000 (UTC) Received: by mail-ye0-f180.google.com with SMTP id m15so3454608yen.39 for ; Thu, 19 Sep 2013 09:25:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:subject:message-id:date:to:mime-version; bh=fjl7vzflLDuuLfPyxx+n9fzaXUy+eleKyDy6fIGXRVs=; b=VxbknW8B0XJvvYQPqLs4YBXayBMfx4/tHirTDR+G6796B4Yawgz+hyIXvNqOG9khgk vD6Fm5nv3Se2KPhp4qLvr/Ky8yR2igAH7JfgRrDk21JumSSjRVG4y1751xyTIPKrowwd KWADRZPxhKZuloJfUKO1A0pV5lvGGBQfAcFtwG1qQJuLiVxr88Shgb6s1mCn2kQr3DGF N9B+A6FdYt9IUZyCfFDtO6RVZJyg0evfHt9KspiLZwqNoWqombfBml7OoJAlSpGkIJ8e nGS8SDnNfNOa/SnRSqkmzj0fKg3QM2u7O70ByEYTeQZBpjuO6DUxjd0Uq7WTUdmXmSbh gGYw== X-Received: by 10.236.133.193 with SMTP id q41mr1852368yhi.51.1379607944654; Thu, 19 Sep 2013 09:25:44 -0700 (PDT) Received: from u5m21307ghelmer.dyn.palisadesys.com ([192.119.231.58]) by mx.google.com with ESMTPSA id e39sm11796568yhq.15.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 19 Sep 2013 09:25:43 -0700 (PDT) From: Guy Helmer Content-Type: multipart/signed; boundary="Apple-Mail=_AA5E5FCA-97F8-4583-9505-7A8B5C122F2E"; protocol="application/pgp-signature"; micalg=pgp-sha512 Subject: IO Performance under VMware on LSI RAID controller Message-Id: Date: Thu, 19 Sep 2013 11:25:40 -0500 To: freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) X-Mailer: Apple Mail (2.1510) X-Mailman-Approved-At: Thu, 19 Sep 2013 17:59:37 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Sep 2013 16:25:46 -0000 --Apple-Mail=_AA5E5FCA-97F8-4583-9505-7A8B5C122F2E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Normally I build VMware ESXi servers with enterprise-class WD SATA = drives and I/O performance in FreeBSD VMs on the servers is fine. Whenever I build a VMware ESXi server with a RAID controller, IO = performance is awful in FreeBSD VMs. I've previously seen this effect = with VMware ESXi 3ware 9690SA-8I and 9650 RAID controllers, and now I'm = seeing similar performance with a Dell 6/iR controller. Any suggestions would be appreciated. Guy Details of the current environment: VMware ESXi 5.1 on Dell R610 4GB = RAM, SAS 6/iR controller, 2x500GB disks in RAID1 set (default stripe = size) and 1x1TB (no RAID). =46rom VMware's client, I see I/O rates in = the sub-MBps range and latencies peaking occasionally at 80 ms. FreeBSD 9.2 (RC2) amd64 in a VM with 2GB RAM assigned, virtual disks = assigned from both the RAID1 set and 1TB (no RAID) drive, UFS+soft = updates file systems. The virtual drives show up in FreeBSD attached to an mpt virtual = controller: mpt0: port 0x1400-0x14ff mem = 0xd0040000-0xd005ffff,0xd0020000-0xd003ffff irq 17 at device 16.0 on = pci0 mpt0: MPI Version=3D1.2.0.0 I don't see anything else sharing the interrupt - vmstat -i shows: irq17: mpt0 77503 27 gstat is showing an abysmal 6 to 16 ops/s for requests on the virtual = disks. I've used gpart to setup the GPT partition table on the virtual disk = assigned from the 1TB drive with alignment for the first UFS partition = at 1MB to try to optimize alignment: Geom name: da0 modified: false state: OK fwheads: 255 fwsectors: 63 last: 268435422 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da0p1 Mediasize: 524288 (512k) Sectorsize: 512 Stripesize: 0 Stripeoffset: 20480 Mode: r0w0e0 rawuuid: d9e6e3e8-1bdb-11e3-b7c5-000c29cbf143 rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f label: gpboot length: 524288 offset: 20480 type: freebsd-boot index: 1 end: 1063 start: 40 2. Name: da0p2 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 1048576 Mode: r1w1e2 rawuuid: fbd6cf40-1bdb-11e3-b7c5-000c29cbf143 rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b label: gprootfs length: 2147483648 offset: 1048576 type: freebsd-ufs index: 2 end: 4196351 start: 2048 3. Name: da0p3 Mediasize: 4294967296 (4.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2148532224 Mode: r1w1e1 rawuuid: 0658208d-1bdc-11e3-b7c5-000c29cbf143 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: gpswap length: 4294967296 offset: 2148532224 type: freebsd-swap index: 3 end: 12584959 start: 4196352 4. Name: da0p4 Mediasize: 130995437056 (122G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2148532224 Mode: r1w1e2 rawuuid: 0ca5bc32-1bdc-11e3-b7c5-000c29cbf143 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: gpusrfs length: 130995437056 offset: 6443499520 type: freebsd-swap index: 4 end: 268435422 start: 12584960 Consumers: 1. Name: da0 Mediasize: 137438953472 (128G) Sectorsize: 512 Mode: r3w3e8 sysctl vfs shows: vfs.ufs.dirhash_reclaimage: 5 vfs.ufs.dirhash_lowmemcount: 179 vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 0 vfs.ufs.dirhash_maxmem: 3481600 vfs.ufs.dirhash_minsize: 2560 vfs.ufs.rename_restarts: 0 vfs.nfs.downdelayinitial: 12 vfs.nfs.downdelayinterval: 30 vfs.nfs.keytab_enctype: 1 vfs.nfs.skip_wcc_data_onerr: 1 vfs.nfs.nfs3_jukebox_delay: 10 vfs.nfs.reconnects: 0 vfs.nfs.bufpackets: 4 vfs.nfs.debuglevel: 0 vfs.nfs.callback_addr:=20 vfs.nfs.realign_count: 0 vfs.nfs.realign_test: 0 vfs.nfs.nfs_directio_allow_mmap: 1 vfs.nfs.nfs_keep_dirty_on_error: 0 vfs.nfs.nfs_directio_enable: 0 vfs.nfs.clean_pages_on_close: 1 vfs.nfs.commit_on_close: 0 vfs.nfs.prime_access_cache: 0 vfs.nfs.access_cache_timeout: 60 vfs.nfs.diskless_rootpath:=20 vfs.nfs.diskless_valid: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.defect: 0 vfs.nfs.iodmax: 20 vfs.nfs.iodmin: 0 vfs.nfs.iodmaxidle: 120 vfs.devfs.rule_depth: 1 vfs.devfs.generation: 113 vfs.nfsd.disable_checkutf8: 0 vfs.nfsd.server_max_nfsvers: 4 vfs.nfsd.server_min_nfsvers: 2 vfs.nfsd.nfs_privport: 0 vfs.nfsd.async: 0 vfs.nfsd.enable_locallocks: 0 vfs.nfsd.issue_delegations: 0 vfs.nfsd.commit_miss: 0 vfs.nfsd.commit_blks: 0 vfs.nfsd.mirrormnt: 1 vfs.nfsd.minthreads: 1 vfs.nfsd.maxthreads: 1 vfs.nfsd.threads: 0 vfs.nfsd.request_space_used: 0 vfs.nfsd.request_space_used_highest: 0 vfs.nfsd.request_space_high: 13107200 vfs.nfsd.request_space_low: 8738133 vfs.nfsd.request_space_throttled: 0 vfs.nfsd.request_space_throttle_count: 0 vfs.nfsd.fha.enable: 1 vfs.nfsd.fha.bin_shift: 22 vfs.nfsd.fha.max_nfsds_per_fh: 8 vfs.nfsd.fha.max_reqs_per_nfsd: 0 vfs.nfsd.fha.fhe_stats: No file handle entries. vfs.pfs.trace: 0 vfs.pfs.vncache.misses: 0 vfs.pfs.vncache.hits: 0 vfs.pfs.vncache.maxentries: 0 vfs.pfs.vncache.entries: 0 vfs.acl_nfs4_old_semantics: 0 vfs.flushwithdeps: 0 vfs.unmapped_buf_allowed: 1 vfs.barrierwrites: 1 vfs.notbufdflashes: 0 vfs.flushbufqtarget: 100 vfs.mappingrestarts: 0 vfs.getnewbufrestarts: 337501 vfs.getnewbufcalls: 349444 vfs.hifreebuffers: 1524 vfs.lofreebuffers: 762 vfs.numfreebuffers: 13601 vfs.dirtybufthresh: 3084 vfs.hidirtybuffers: 3427 vfs.lodirtybuffers: 1713 vfs.numdirtybuffers: 11 vfs.recursiveflushes: 341175 vfs.altbufferflushes: 0 vfs.bdwriteskip: 0 vfs.dirtybufferflushes: 0 vfs.hirunningspace: 3538944 vfs.lorunningspace: 2359296 vfs.bufdefragcnt: 0 vfs.buffreekvacnt: 339965 vfs.bufreusecnt: 347292 vfs.hibufspace: 222625792 vfs.lobufspace: 222560256 vfs.maxmallocbufspace: 11131289 vfs.bufmallocspace: 0 vfs.maxbufspace: 223281152 vfs.unmapped_bufspace: 290652160 vfs.bufspace: 291602432 vfs.runningbufspace: 131072 vfs.vmiodirenable: 1 vfs.cache.numfullpathfound: 47 vfs.cache.numfullpathfail4: 0 vfs.cache.numfullpathfail2: 0 vfs.cache.numfullpathfail1: 0 vfs.cache.numfullpathcalls: 47 vfs.cache.numupgrades: 32 vfs.cache.numneghits: 2853 vfs.cache.numnegzaps: 16 vfs.cache.numposhits: 356750 vfs.cache.numposzaps: 282 vfs.cache.nummisszap: 10 vfs.cache.nummiss: 35615 vfs.cache.numchecks: 377860 vfs.cache.dotdothits: 19 vfs.cache.dothits: 146 vfs.cache.numcalls: 395710 vfs.cache.numcache: 29909 vfs.cache.numneg: 336 vfs.ncsizefactor: 2 vfs.ncnegfactor: 16 vfs.read_min: 1 vfs.read_max: 64 vfs.write_behind: 1 vfs.typenumhash: 1 vfs.lookup_shared: 1 vfs.usermount: 0 vfs.worklist_len: 3 vfs.timestamp_precision: 0 vfs.reassignbufcalls: 515051 vfs.vlru_allow_cache_src: 0 vfs.freevnodes: 27829 vfs.wantfreevnodes: 27833 vfs.numvnodes: 29507 vfs.ffs.doreallocblks: 1 vfs.ffs.doasyncfree: 1 vfs.ffs.compute_summary_at_mount: 0 --Apple-Mail=_AA5E5FCA-97F8-4583-9505-7A8B5C122F2E Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJSOyWEAAoJEJmJzK+PHOvEhgsH/0qm0ZD7qZ0CHhIVxSBn9f4Y eY5HpTRbjXWEtG0En3V8ZcLjk/oXF6CSC6olwvczim6pscq3XJ2EDK+6zWZFHYq8 KczXW++3T0U5HxK83ghjFZidsgo64lKFKpQb35KOZFd0MIWHHYYRx9EL+uFdyxXn BycgjQral+gZqwRELYwOtA5oqGXkMQpAIh18X1qH/ZVPD3XFwewsDe0VZuf4jHqi rZFMozmvDyqfP2ytYFWNHuO3U1OHbn6yRrwfCuLIHmD498ms/FYTwtEVTZ7xQr0n lxofZVQcI+t9XLe36n7Jgci6+YVL1N2lI2KZ5/t8QPwLV5Po1N2/nbGTI+3Rjac= =tq1b -----END PGP SIGNATURE----- --Apple-Mail=_AA5E5FCA-97F8-4583-9505-7A8B5C122F2E--