Date: Wed, 12 Nov 1997 03:06:46 +0000 (WET) From: Antonio Luis Alves <aalves@dei.uc.pt> To: freebsd-atm@FreeBSD.ORG Subject: ATM driver & udp stream Message-ID: <Pine.BSF.3.96.971112005854.20573A-100000@zorg.dei.uc.pt>
next in thread | raw e-mail | index | archive | help
I have been working on a port to FreeBSD of a major manufacturer atm implementation which includes classical ip and lan emulation. The most difficult part was the development of the low level code to drive the pci adapters as this was my first project working inside the kernel and also on FreeBSD. At this time most of the work as been done and I have only the lan emu code and ILMI to finish the port. We have been running the port for several months on Pentium machines with FreeBSD 2.2.2 Release without any problems, until two weeks ago when I started to do some performance tests. One of tests ( a UDP stream ) was crashing the driver allays at the same place in the pdu receive routine. The driver allocs several buffers at init time to be used by the card to write the received pdu's. The supply routine allocs an mbuf for each buffer and supplies the card with the dma address of the buffer and the mbuf handle. When the card interrupts, the pdu receive routine reads the receive pdu queue. Each entry in this queue is a receive pdu descriptor. Each descriptor contains several information related to the received pdu, including several segments to form the pdu. Each segment contains the mbuf handle ( allocated by the supply routine ) and the number of bytes in the buffer written by the card. At this time I use 4k buffers and each pdu descriptor with max 16 segments to be able to receive a max pdu of 64k. The mbuf handles are read by the card from the supply queue and written to the segment descriptors after the buffers have been filled with cells payloads. The mbuf chain forming the received pdu is formed and the pdu is send upstairs. Each mbuf in the chain as the ext_buf pointing to the buffer, and pointers to the external_free_routine and an external_ref_function. The buffers will return to the free buffer pool on the driver as soon as the external_free_routine is called and the reference count allows it. These buffers are then used again by the supply routine. When the card is low on buffers, the pdu receive routine does not send the external buffers upstairs, instead it copies the data to mbufs with clusters, so the buffers are made available again to the card faster. All was working well until I made a UDP stream test. The sender was a Pentium 233mmx and on the receive side a Pentium 166. The test is a normal netperf UDP_STREAM with default values for the datagram, which is the max udp datagram 9216. The driver crashes at the pdu receive routine when accessing the segments which form the pdu. With the debugger I can see that the mbuf handles are correct but the data on the mbuf is not. Some times the m_type is MT_FREE and m->next as got a valid pointer on it, and other times all the members of the mbuf are zero. On a normal situation the mbufs would be of type MT_DATA, and correctly initialized with the pointers to the external free and reference function, m_len, ext_size and ext_buffer. >From the firmware guide it says the card never touches the mbuf, it just pass the mbuf handle received from the supply routine to the segments on the received pdu descriptor. These problem only happens when there is fragmentation and reassembly of packets. If I make the test with the udp datagram below the mtu of the interface ( 9188 ) it goes well without any problems. Also if I pace the transfer with delays and small burst it also works for the max datagram. I made also other tests from a sun as the sending machine ( which is very slow compared with the Pentiums we have here ) and it works ok even with the max datagram of 9216. So I suppose there is something related to the full blast of the Pentium 233mmx and the reassembly of the packets at the ip layer. I also think that at this speed ( 155MPS cards ) most of the fragments ( mbuf w/ external buffer ) are kept at the ip fragment queue and then as soon as the card gets low on buffers , the copy to mbufs w/ clusters routine enters the game and starts asking the system for a lot of mbufs and clusters. At this time I am running out of ideas of where to look to find the problem, so I decided to do a post here and ask for some ideas. I have made already lots of modifications in the driver with lots of paranoid checks but without any success. Sorry for the long post, but I tried to explain as much as possible for a better evaluation of this problem. Thank you. Antonio Alves
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.3.96.971112005854.20573A-100000>