From owner-freebsd-atm Wed Nov 12 08:50:59 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id IAA16986 for atm-outgoing; Wed, 12 Nov 1997 08:50:59 -0800 (PST) (envelope-from owner-freebsd-atm) Received: from galileo.ravel.ufrj.br (galileo.ravel.ufrj.br [146.164.32.68]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id IAA16893 for ; Wed, 12 Nov 1997 08:49:58 -0800 (PST) (envelope-from rodolfo@galileo.ravel.ufrj.br) Received: (from rodolfo@localhost) by galileo.ravel.ufrj.br (8.8.7/8.8.7) id OAA00538; Wed, 12 Nov 1997 14:41:43 -0200 (EDT) From: Rodolfo Heitor Gevaerd de Faria Message-Id: <199711121641.OAA00538@galileo.ravel.ufrj.br> Subject: Re: ATM driver & udp stream In-Reply-To: from Antonio Luis Alves at "Nov 12, 97 03:06:46 am" To: aalves@dei.uc.pt Date: Wed, 12 Nov 1997 14:41:43 -0200 (EDT) Cc: freebsd-atm@FreeBSD.ORG X-Mailer: ELM [version 2.4ME+ PL32 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-atm@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk Antonio Luis Alves was saying that, ^ ^ I have been working on a port to FreeBSD of a major manufacturer atm If you don't mind, what is the "major manufacturer" which you're porting ? ^ implementation which includes classical ip and lan emulation. The most ^ difficult part was the development of the low level code to drive the pci ^ adapters as this was my first project working inside the kernel and also ^ on FreeBSD. At this time most of the work as been done and I have only the ^ lan emu code and ILMI to finish the port. ^ ^ We have been running the port for several months on Pentium machines with ^ FreeBSD 2.2.2 Release without any problems, until two weeks ago when I ^ started to do some performance tests. One of tests ( a UDP stream ) was ^ crashing the driver allays at the same place in the pdu receive routine. ^ ^ The driver allocs several buffers at init time to be used by the card to ^ write the received pdu's. The supply routine allocs an mbuf for each ^ buffer and supplies the card with the dma address of the buffer and ^ the mbuf handle. When the card interrupts, the pdu receive routine reads ^ the receive pdu queue. Each entry in this queue is a receive pdu ^ descriptor. Each descriptor contains several information related to the ^ received pdu, including several segments to form the pdu. Each segment ^ contains the mbuf handle ( allocated by the supply routine ) and the ^ number of bytes in the buffer written by the card. At this time I use 4k ^ buffers and each pdu descriptor with max 16 segments to be able to receive ^ a max pdu of 64k. The mbuf handles are read by the card from the supply ^ queue and written to the segment descriptors after the buffers have been ^ filled with cells payloads. ^ The mbuf chain forming the received pdu is formed and the pdu is send ^ upstairs. Each mbuf in the chain as the ext_buf pointing to the buffer, ^ and pointers to the external_free_routine and an external_ref_function. ^ The buffers will return to the free buffer pool on the driver as soon as ^ the external_free_routine is called and the reference count allows it. ^ These buffers are then used again by the supply routine. ^ When the card is low on buffers, the pdu receive routine does not send the ^ external buffers upstairs, instead it copies the data to mbufs with ^ clusters, so the buffers are made available again to the card faster. ^ ^ All was working well until I made a UDP stream test. The sender was a ^ Pentium 233mmx and on the receive side a Pentium 166. The test is a normal ^ netperf UDP_STREAM with default values for the datagram, which is the max ^ udp datagram 9216. The driver crashes at the pdu receive routine when ^ accessing the segments which form the pdu. With the debugger I can see that ^ the mbuf handles are correct but the data on the mbuf is not. Some times ^ the m_type is MT_FREE and m->next as got a valid pointer on it, and other ^ times all the members of the mbuf are zero. On a normal situation the ^ mbufs would be of type MT_DATA, and correctly initialized with the ^ pointers to the external free and reference function, m_len, ext_size and ^ ext_buffer. ^ >From the firmware guide it says the card never touches the mbuf, it just ^ pass the mbuf handle received from the supply routine to the segments on ^ the received pdu descriptor. ^ ^ These problem only happens when there is fragmentation and reassembly of ^ packets. If I make the test with the udp datagram below the mtu of the ^ interface ( 9188 ) it goes well without any problems. Also if I pace the ^ transfer with delays and small burst it also works for the max datagram. I ^ made also other tests from a sun as the sending machine ( which is very ^ slow compared with the Pentiums we have here ) and it works ok even with ^ the max datagram of 9216. ^ ^ So I suppose there is something related to the full blast of the Pentium ^ 233mmx and the reassembly of the packets at the ip layer. I also think ^ that at this speed ( 155MPS cards ) most of the fragments ( mbuf w/ ^ external buffer ) are kept at the ip fragment queue and then as soon as ^ the card gets low on buffers , the copy to mbufs w/ clusters routine ^ enters the game and starts asking the system for a lot of mbufs and ^ clusters. ^ ^ At this time I am running out of ideas of where to look to find the ^ problem, so I decided to do a post here and ask for some ideas. ^ I have made already lots of modifications in the driver with lots of ^ paranoid checks but without any success. ^ ^ Sorry for the long post, but I tried to explain as much as possible for a ^ better evaluation of this problem. ^ ^ Thank you. ^ ^ ^ Antonio Alves ^ Rodolfo H G Faria