Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 2 Aug 2015 15:21:41 +0300
From:      Max Gurtovoy <maxg@mellanox.com>
To:        <freebsd-scsi@freebsd.org>
Subject:   PIM_UNMAPPED question
Message-ID:  <55BE0B55.8070807@mellanox.com>
In-Reply-To: <VI1PR05MB1440A7BED8C6434EE01360CFDE880@VI1PR05MB1440.eurprd05.prod.outlook.com>
References:  <VI1PR05MB1440A7BED8C6434EE01360CFDE880@VI1PR05MB1440.eurprd05.prod.outlook.com>

next in thread | previous in thread | raw e-mail | index | archive | help

Hi,

I am working on a new iscsi RDMA transport implementation. In order to 
avoid kernel buffer allocation in the IO path
I added support for unmapped buffers in my driver (PIM_UNMAPPED hba_misc 
flag).
This indeed helped performance by improving the IOPs By a factor of 4x 
vs. KVAs.

However, I noticed that the bio_ma vector consists of physically 
contiguous addresses. I expected that user-space
allocations would not be physically contiguous. Is it possible that I'm 
still not seeing the user IO buffers?

I tested using dd/fio traffic generators and the phenomenon still persists.

128k Read output to show the phenomenon:

kernel: dma_address 0x40608000 len 4096
kernel: dma_address 0x40609000 len 4096
kernel: dma_address 0x4060a000 len 4096
kernel: dma_address 0x4060b000 len 4096
kernel: dma_address 0x4060c000 len 4096
kernel: dma_address 0x4060d000 len 4096
kernel: dma_address 0x4060e000 len 4096
kernel: dma_address 0x4060f000 len 4096
kernel: dma_address 0x40610000 len 4096
kernel: dma_address 0x40611000 len 4096
kernel: dma_address 0x40612000 len 4096
kernel: dma_address 0x40613000 len 4096
kernel: dma_address 0x40614000 len 4096
kernel: dma_address 0x40615000 len 4096
kernel: dma_address 0x40616000 len 4096
kernel: dma_address 0x40617000 len 4096
kernel: dma_address 0x40618000 len 4096
kernel: dma_address 0x40619000 len 4096
kernel: dma_address 0x4061a000 len 4096
kernel: dma_address 0x4061b000 len 4096
kernel: dma_address 0x4061c000 len 4096
kernel: dma_address 0x4061d000 len 4096
kernel: dma_address 0x4061e000 len 4096
kernel: dma_address 0x4061f000 len 4096
kernel: dma_address 0x40620000 len 4096
kernel: dma_address 0x40621000 len 4096
kernel: dma_address 0x40622000 len 4096
kernel: dma_address 0x40623000 len 4096
kernel: dma_address 0x40624000 len 4096
kernel: dma_address 0x40625000 len 4096
kernel: dma_address 0x40626000 len 4096
kernel: dma_address 0x40627000 len 4096

Initiator machine info:
- OS 11-current r284921
- 32G RAM
- 32 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz CPUs

The code is available at:
https://github.com/sagigrimberg/iser-freebsd/tree/iser-rebase-11-current-r284921

Thanks in advanced,

Max Gurtovoy.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55BE0B55.8070807>