Date: Wed, 03 May 2023 12:34:59 +0200 From: Dmitry Salychev <dsl@freebsd.org> To: Mitchell Horne <mhorne@freebsd.org> Cc: freebsd-arm@freebsd.org Subject: Re: About PHYS_TO_DMAP Message-ID: <86r0rxllrw.fsf@peasant.tower.home> In-Reply-To: <08ff62fd-89db-d429-9ef3-04f213118f83@freebsd.org> References: <86ildyucuv.fsf@peasant.tower.home> <eec51f08-8f20-df60-d736-a9cf37f8b11f@freebsd.org> <86zg78s2bj.fsf@peasant.tower.home> <08ff62fd-89db-d429-9ef3-04f213118f83@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Mitchell Horne <mhorne@freebsd.org> writes: > On 4/16/23 04:06, Dmitry Salychev wrote: >> Mitchell Horne <mhorne@freebsd.org> writes: >> >>> On 4/14/23 04:31, Dmitry Salychev wrote: >>>> Hi, >>>> I'm struggling to understand which KVA will be returned by >>>> PHYS_TO_DMAP >>>> on arm64. For example, if I'll create a DMA tag this way: >>>> bus_dma_tag_create( >>>> bus_get_dma_tag(dev), >>>> sc->buf_align, 0, /* alignment, boundary */ >>>> DMAP_MAX_PHYSADDR, /* low restricted addr */ >>>> DMAP_MIN_PHYSADDR, /* high restricted addr */ >>> >>> I think you are confused about the purpose of lowaddr and >>> highaddr. They specify the window of bus-space addresses that are >>> *inaccessible* *to the device* for DMA. It does not prevent you from >>> providing a buffer backed by physical memory within this range, but in >>> this case bounce pages will be used as an intermediate to perform the >>> DMA transaction. >>> >>> Most commonly, if the device can only address a 32-bit range, then the >>> values lowaddr=BUS_SPACE_MAXADDR_32BIT and highaddr=BUS_SPACE_MAXADDR >>> will be given. If the entire bus range is accessible to the device for >>> DMA, a zero-sized window will be given: lowaddr=BUS_SPACE_MAXADDR, >>> highaddr=BUS_SPACE_MAXADDR. >>> >>> This is all described in adequate detail in busdma(9), but it is still >>> not easily understood without a lot of code study, in my experience. >>> >> According to busdma(9), the window contains all addresses *greater >> than* >> lowaddr and *less than or equal to* highaddr, i.e. your example with the >> 32-bit range looks like: >> allowed prohibited >> ------------------------------(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> >> lowaddr highaddr >> BUS_SPACE_MAXADDR_32BIT BUS_SPACE_MAXADDR >> But my example looks a bit different: >> prohibited allowed prohibited >> xxxxxxxxxxxxxxxx]----------------------------(xxxxxxxxxxxxxxxxxxx> >> highaddr lowaddr >> DMAP_MIN_PHYSADDR DMAP_MAX_PHYSADDR >> > > Okay I see, you are using MAX as lowaddr and MIN as highaddr. I read > it backwards the first time. > > However, I looked at the code and I'm not sure that it will handle the > case where lowaddr > highaddr correctly. Just a warning to you. > >> I've found the only example of such DMA tag configuration in >> src/sys/dev/vnic/nicvf_queues.c at line 379 (rbdr_buff_dmat). >> > I think this example is misleading, if not incorrect. All DMA > transactions for this tag will have a destination (physical memory > address) that falls within the [DMAP_MIN_PHYSADDR,DMAP_MAX_PHYSADDR] > range. So, the specified boundaries are not meaningful, and don't > represent a real constraint of the device. At least, this is my > understanding of it. I don't know the bus_dma implementation details yet unfortunately and thought that DMA tag created with [DMAP_MIN_PHYSADDR,DMAP_MAX_PHYSADDR] window is required to calculate KVA by PHYS_TO_DMAP properly. However, PHYS_TO_DMAP gave me an address to access frame annotation even with DMA tag created without a restriction window at all, i.e. BUS_SPACE_MAXADDR used for both lowaddr and highaddr. I've opened a review https://reviews.freebsd.org/D39946 if you'll be interested in details. > >> >>>> NULL, NULL, /* filter, filterarg */ >>>> BUF_SIZE, 1, /* maxsize, nsegments */ >>>> BUF_SIZE, 0, /* maxsegsize, flags */ >>>> NULL, NULL, /* lockfunc, lockarg */ >>>> &dmat); >>>> in order to restrict any physical addresses but a window defined by >>>> DMAP_MIN_PHYSADDR and DMAP_MAX_PHYSADDR. Later on when I'll be >>>> mapping my mbuf (BUF_SIZE) with >>>> bus_dmamap_load_mbuf_sg(dmat, dmap, >>>> m, &segs, &nsegs, BUS_DMA_NOWAIT); >>>> I expect that >>>> m->m_data == PHYS_TO_DMAP(segs[0].ds_addr) >>> >>> Why do you expect or need this to be the case? >>> >>> busdma is not responsible for setting or modifying m_data, it comes >>> from wherever you allocated the mbuf. Calling >>> bus_dmamap_load_mbuf_sg() will prepare a DMA mapping where the m_data >>> buffer is used as the source/destination for the DMA transaction, but >>> busdma does not allocate the buffer itself. >>> >> I don't need this to be the case exactly, but I'd like to be able to >> access a "frame annotation" (64 bytes long) which is located exactly at >> the start of m_data buffer having that physical address is provided, i.e. >> fa = (struct dpaa2_fa *) mbuf->m_data; >> /* ... fa populated ... */ >> /* ... DMA transaction of the Tx frame (together with fa) ... */ >> /* ... Tx confirmation from HW (bus address of the frame >> only) ...*/ >> fa = (struct dpaa2_fa *) PHYS_TO_DMAP(paddr); >> > > The short answer is that you are using PHYS_TO_DMAP correctly. Every > real paddr should have a corresponding VA in the DMAP that is valid > for read/write access to that memory. However, it is unusual to see > DMAP usage in driver code. > > It is hard for me to say exactly what you should be doing without more > information. > > In general, the m_data pointer remains valid after the DMA > transaction. If your updated frame annotation is located at the > beginning of this buffer, then you want to use the same m_data pointer > to access it. > > I guess the challenge is that the TX confirmation occurs in a separate > context (interrupt handler?) where the mbuf reference is lost? I > believe the usual practice is to maintain some kind of ring or queue > of transmitted mbuf+dma_map_t pairs, which can be handled together in > the TX interrupt handler (unload DMA map, free and/or reassign mbuf). This is correct. However, HW (firmware in my case) provides a frame descriptor which carries physical address of the Rx/Tx buffer only. This is why I needed a way to access the buffer itself from the kernel address space. > > Of course, you must use appropriate sync calls before accessing the > buffer again on the host, probably: bus_dmamap_sync(tag, map, > BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE). I'm sure you know this. > >>>> but it isn't true. Could somebody explain what exactly is returned >>>> by >>>> PHYS_TO_DMAP in this case and whether it's possible to translate >>>> physical address to KVA as fast as possible (O(1) ideally). >>>> >>> >>> PHYS_TO_DMAP is always a linear calculation of: physaddr + DMAP_MIN_ADDRESS. >>> >>> I do not think PHYS_TO_DMAP is in use at all in this example, or >>> anywhere within busdma really. >>> >> Is there any other way to obtain KVA of a buffer mapped for DMA >> transaction by the physical address? I've been crawling source code for >> sometime already, but DMAP is the only thing I managed to find. >> >>> Regards, >>>> Dmitry >>>> -- >>>> Open source software/hardware enthusiast >>>> hackaday.io/dsl | github.com/mcusim | patreon.com/salychev >>>> >> Regards, >> Dmitry -- Open source software/hardware enthusiast hackaday.io/dsl | github.com/mcusim | patreon.com/salychev
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?86r0rxllrw.fsf>