From owner-freebsd-hackers@FreeBSD.ORG Thu Jul 29 14:43:30 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ABC0A106566B for ; Thu, 29 Jul 2010 14:43:29 +0000 (UTC) (envelope-from pebu3op@googlemail.com) Received: from mail.net.t-labs.tu-berlin.de (mail.net.t-labs.tu-berlin.de [130.149.220.252]) by mx1.freebsd.org (Postfix) with ESMTP id 253028FC17 for ; Thu, 29 Jul 2010 14:43:28 +0000 (UTC) Received: from raven.net.t-labs.tu-berlin.de (raven.net.t-labs.tu-berlin.de [130.149.220.18]) by mail.net.t-labs.tu-berlin.de (Postfix) with ESMTP id B680570015BA for ; Thu, 29 Jul 2010 16:13:16 +0200 (CEST) From: Alexander Fiveg Organization: Google To: freebsd-hackers@freebsd.org Date: Thu, 29 Jul 2010 16:13:15 +0200 User-Agent: KMail/1.9.10 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201007291613.15719.pebu3op@googlemail.com> Subject: coherence-problem on the mapped memory buffer X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pebu3op@googlemail.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2010 14:43:30 -0000 Hello hackers, while working on the "ringmap"-project I've faced a problem of "no coherency in the memory regions mapped from kernel into the user-space". Details: While integrating ringmap with the ixgbe-driver, I've made some changes to the ixgbe: 1. The mbufs for received packets will be only allocated once. 2. Allocated mbufs will be reused as in ring-buffer one after the other (no new mbufs will be allocated again). 3. Packet buffers (mbuf->m_data) will mapped into the user-space. So, the user-space process has access to the packets after those DMA-transfer from the network adapter into the RAM Problem: Sometimes the user-space process sees not new DMAed data in the mapped packet-buffer, but the OLD data that was previously stored in the same packet buffer. If I try to monitor the received data in the kernel, the kernel sees the data correctly. But sometimes it is vice versa: the user-space process sees the correct new data and the kernel sees the old data in the buffer. It seems to be that the memory-buffer for packets is not synchronized with all CPU's caches. Probably [user|kernel]-thread tries sometimes to reads the old dirty data from the cache of the CPU the thread running on. (In the same time the other thread sees the new data in the same mapped buffer). Can you please provide me with some information that would be helpful for avoiding this unexpected coherence-problem. Alex P.S. Details about hardware and used software: 1. /var/run/dmesg.boot : ... CPU: Dual Core AMD Opteron(tm) Processor 865 (1800.01-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x20f10 Family = f Model = 21 Stepping = 0 Features=0x178bfbff Features2=0x1 AMD Features=0xe2500800 AMD Features2=0x3 real memory = 3758030848 (3583 MB) avail memory = 3677495296 (3507 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 4 package(s) x 2 core(s) ... 2. uname -v FreeBSD 9.0-CURRENT #3 3. sysctl kern.osreldate kern.osreldate: 900014 4. //depot/projects/soc2010/ringmap/