Date: Sun, 2 Aug 2015 15:21:41 +0300 From: Max Gurtovoy <maxg@mellanox.com> To: <freebsd-scsi@freebsd.org> Subject: PIM_UNMAPPED question Message-ID: <55BE0B55.8070807@mellanox.com> In-Reply-To: <VI1PR05MB1440A7BED8C6434EE01360CFDE880@VI1PR05MB1440.eurprd05.prod.outlook.com> References: <VI1PR05MB1440A7BED8C6434EE01360CFDE880@VI1PR05MB1440.eurprd05.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, I am working on a new iscsi RDMA transport implementation. In order to avoid kernel buffer allocation in the IO path I added support for unmapped buffers in my driver (PIM_UNMAPPED hba_misc flag). This indeed helped performance by improving the IOPs By a factor of 4x vs. KVAs. However, I noticed that the bio_ma vector consists of physically contiguous addresses. I expected that user-space allocations would not be physically contiguous. Is it possible that I'm still not seeing the user IO buffers? I tested using dd/fio traffic generators and the phenomenon still persists. 128k Read output to show the phenomenon: kernel: dma_address 0x40608000 len 4096 kernel: dma_address 0x40609000 len 4096 kernel: dma_address 0x4060a000 len 4096 kernel: dma_address 0x4060b000 len 4096 kernel: dma_address 0x4060c000 len 4096 kernel: dma_address 0x4060d000 len 4096 kernel: dma_address 0x4060e000 len 4096 kernel: dma_address 0x4060f000 len 4096 kernel: dma_address 0x40610000 len 4096 kernel: dma_address 0x40611000 len 4096 kernel: dma_address 0x40612000 len 4096 kernel: dma_address 0x40613000 len 4096 kernel: dma_address 0x40614000 len 4096 kernel: dma_address 0x40615000 len 4096 kernel: dma_address 0x40616000 len 4096 kernel: dma_address 0x40617000 len 4096 kernel: dma_address 0x40618000 len 4096 kernel: dma_address 0x40619000 len 4096 kernel: dma_address 0x4061a000 len 4096 kernel: dma_address 0x4061b000 len 4096 kernel: dma_address 0x4061c000 len 4096 kernel: dma_address 0x4061d000 len 4096 kernel: dma_address 0x4061e000 len 4096 kernel: dma_address 0x4061f000 len 4096 kernel: dma_address 0x40620000 len 4096 kernel: dma_address 0x40621000 len 4096 kernel: dma_address 0x40622000 len 4096 kernel: dma_address 0x40623000 len 4096 kernel: dma_address 0x40624000 len 4096 kernel: dma_address 0x40625000 len 4096 kernel: dma_address 0x40626000 len 4096 kernel: dma_address 0x40627000 len 4096 Initiator machine info: - OS 11-current r284921 - 32G RAM - 32 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz CPUs The code is available at: https://github.com/sagigrimberg/iser-freebsd/tree/iser-rebase-11-current-r284921 Thanks in advanced, Max Gurtovoy.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55BE0B55.8070807>