From owner-freebsd-xen@freebsd.org Wed Aug 24 08:45:24 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C7E5BC025C for ; Wed, 24 Aug 2016 08:45:24 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: from mail-pf0-x243.google.com (mail-pf0-x243.google.com [IPv6:2607:f8b0:400e:c00::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0DC4E19C3; Wed, 24 Aug 2016 08:45:24 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: by mail-pf0-x243.google.com with SMTP id h186so805556pfg.2; Wed, 24 Aug 2016 01:45:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id; bh=cDO9L6dURh+hNM4hoJgmuL4DjQ1PAi0wVupVZbMRLm4=; b=k//5g2VC7jcj7Pz09l7/fDYvx35xzNQqdjL9rajz3Why8HCc8dHpv9yeE1XukqENxj o8OPusJWFi472nFATgUWgC1wJXUyNRE4cQn2S9BfRprueINH92ZvrWIGlrxg554whp9f 3KrZOFmRwIHFVzLXYObkz3bMQy1GZ4pz4YLSzy1imAr4n7IGVFw/Spdw2GvAIm2PG4Ok aq3Ly42PEsKDMLTHaRKM0V0USw0kvyisJE3wj7RpukRyC3H3OV8K0oZU4WtylA2U/mqs WcO+DWwZyaRqj95u8NvttnUG6atinGEs5aovToLfjRbMAjVVg1M+Tb50nhLOjVMMI0nl 8hGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=cDO9L6dURh+hNM4hoJgmuL4DjQ1PAi0wVupVZbMRLm4=; b=KNtWd72WG7yLFZxJqUMTB8L8tS8VNDY8XPqFTPvVaapGvLBB4BssW3r6dO2UV1P6t9 gb5He4TXfQezOkuADnuy9cUxkIqeG2La2QJZNmv41tCzIOVFksHSHg/haA6bdD716N0E GicmoSVn981ej4m6xfUaxkJ57TYv0l3v4YiEqC/GsJQ+f7vCCWxjdALjY7AuwyPGbwGR 7INb96cjqHd3YG05T3s1csBuT6VsxvgUZ3LOudyW/q6T4C+rYuE3aQm20htnkSVv/HAH L0xzVsts/bpYIsQkofmfO1dkJzgzEOSHoZKBrEx5ACSpZ9wQ+Je+ocUPCsHVweb+lIWa DGvQ== X-Gm-Message-State: AE9vXwOEbutaooTNvDtmXo90YlBtgbDd7X6qrhasEWlU5LU6sY0rc8qFtjJJlQlRHFdKmA== X-Received: by 10.98.20.201 with SMTP id 192mr3524178pfu.144.1472028323335; Wed, 24 Aug 2016 01:45:23 -0700 (PDT) Received: from localhost.localdomain ([182.69.122.203]) by smtp.gmail.com with ESMTPSA id vy9sm11492410pab.22.2016.08.24.01.45.18 (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 24 Aug 2016 01:45:22 -0700 (PDT) From: Akshay Jaggi To: xen-devel@lists.xenproject.org Cc: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Andrew Cooper , George Dunlap , Ian Jackson , Konrad Rzeszutek Wilk , Stefano Stabellini , Tim Deegan , FreeBSD-Xen , Akshay Jaggi Subject: [PATCH v3] gnttab: Add gntdev device mappings for FreeBSD Date: Wed, 24 Aug 2016 14:14:46 +0530 Message-Id: <1472028286-11560-1-git-send-email-akshay1994.leo@gmail.com> X-Mailer: git-send-email 2.8.2 X-Mailman-Approved-At: Wed, 24 Aug 2016 11:14:27 +0000 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Aug 2016 08:45:24 -0000 Add grant table userspace device mappings for FreeBSD (enables support for qdisk backend on FreeBSD Dom0). Signed-off-by: Akshay Jaggi --- Changed since v1: * fix coding style * remove O_CLOEXEC * remove SET_MAX_GRANTS ioctl * update freebsd/gntdev.h to latest version * replace alloca with malloc Changed since v2: * fix overflow bug --- tools/include/xen-sys/FreeBSD/gntdev.h | 191 ++++++++++++++++++++ tools/libs/gnttab/Makefile | 2 +- tools/libs/gnttab/freebsd.c | 318 +++++++++++++++++++++++++++++++++ 3 files changed, 510 insertions(+), 1 deletion(-) create mode 100644 tools/include/xen-sys/FreeBSD/gntdev.h create mode 100644 tools/libs/gnttab/freebsd.c diff --git a/tools/include/xen-sys/FreeBSD/gntdev.h b/tools/include/xen-sys/FreeBSD/gntdev.h new file mode 100644 index 0000000..5f31e21 --- /dev/null +++ b/tools/include/xen-sys/FreeBSD/gntdev.h @@ -0,0 +1,191 @@ +/*- + * Copyright (c) 2016 Akshay Jaggi + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * gntdev.h + * + * Interface to /dev/xen/gntdev. + * + * This device provides the user with two kinds of functionalities: + * 1. Grant Allocation + * Allocate a page of our own memory, and share it with a foreign domain. + * 2. Grant Mapping + * Map a grant allocated by a foreign domain, into our own memory. + * + * + * Grant Allocation + * + * Steps to allocate a grant: + * 1. Do an `IOCTL_GNTDEV_ALLOC_GREF ioctl`, with + * - `domid`, as the domain-id of the foreign domain + * - `flags`, ORed with GNTDEV_ALLOC_FLAG_WRITABLE if you want the foreign + * domain to have write access to the shared memory + * - `count`, with the number of pages to share with the foreign domain + * + * Ensure that the structure you allocate has enough memory to store + * all the allocated grant-refs, i.e., you need to allocate + * (sizeof(struct ioctl_gntdev_alloc_gref) + (count - 1)*sizeof(uint32_t)) + * bytes of memory. + * + * 2. Mmap the address given in `index` after a successful ioctl. + * This will give you access to the granted pages. + * + * Note: + * 1. The grant is not removed until all three of the following conditions + * are met + * - The region is not mmaped. That is, munmap() has been called if + * the region was mmapped previously. + * - IOCTL_GNTDEV_DEALLOC_GREF ioctl has been performed. After you + * perform this ioctl, you can no longer mmap or set notify on + * the grant. + * - The foreign domain has stopped using the grant. + * 2. Granted pages can only belong to one mmap region. + * 3. Every page of granted memory is a unit in itself. What this means + * is that you can set a unmap notification for each of the granted + * pages, individually; you can mmap and dealloc-ioctl a contiguous + * range of allocated grants (even if alloc-ioctls were performed + * individually), etc. + * + * + * Grant Mapping + * + * Steps to map a grant: + * 1. Do a `IOCTL_GNTDEV_MAP_GRANT_REF` ioctl, with + * - `count`, as the number of foreign grants to map + * - `refs[i].domid`, as the domain id of the foreign domain + * - `refs[i].ref`, as the grant-ref for the grant to be mapped + * + * 2. Mmap the address given in `index` after a successful ioctl. + * This will give you access to the mapped pages. + * + * Note: + * 1. The map hypercall is not made till the region is mmapped. + * 2. The unit is defined by the map ioctl. This means that only one + * unmap notification can be set on a group of pages that were + * mapped together in one ioctl, and also no single mmaping of contiguous + * grant-maps is possible. + * 3. You can mmap the same grant-map region multiple times. + * 4. The grant is not unmapped until both of the following conditions are met + * - The region is not mmaped. That is, munmap() has been called for + * as many times as the grant was mmapped. + * - IOCTL_GNTDEV_UNMAP_GRANT_REF ioctl has been called. + * 5. IOCTL_GNTDEV_GET_OFFSET_FOR_VADDR ioctl gives index and count of + * a grant-map from the virtual address of the location where the grant + * is mmapped. + * + * + * IOCTL_GNTDEV_SET_UNMAP_NOTIFY + * This ioctl allows us to set notifications to be made when the grant is + * either unmapped (in case of a mapped grant), or when it is ready to be + * deallocated by us, ie, the grant is no more mmapped, and the dealloc + * ioctl has been called (in case of an allocated grant). OR `action` with + * the required notification masks, and fill in the appropriate fields. + * - UNMAP_NOTIFY_CLEAR_BYTE clears the byte at `index`, where index is + * the address of the byte in file address space. + * - UNMAP_NOTIFY_SEND_EVENT sends an event channel notification on + * `event_channel_port` + * In case of multiple notify ioctls, only the last one survives. + * + */ + +#ifndef __XEN_GNTDEV_H__ +#define __XEN_GNTDEV_H__ + +#include + +#define IOCTL_GNTDEV_SET_UNMAP_NOTIFY \ + _IOW('E', 0, struct ioctl_gntdev_unmap_notify) +struct ioctl_gntdev_unmap_notify { + /* IN parameters */ + uint64_t index; + uint32_t action; + uint32_t event_channel_port; +}; + +#define UNMAP_NOTIFY_CLEAR_BYTE 0x1 +#define UNMAP_NOTIFY_SEND_EVENT 0x2 + +/*-------------------- Grant Allocation IOCTLs ------------------------------*/ + +#define IOCTL_GNTDEV_ALLOC_GREF \ + _IOWR('E', 1, struct ioctl_gntdev_alloc_gref) +struct ioctl_gntdev_alloc_gref { + /* IN parameters */ + uint16_t domid; + uint16_t flags; + uint32_t count; + /* OUT parameters */ + uint64_t index; + /* Variable OUT parameter */ + uint32_t gref_ids[1]; +}; + +#define GNTDEV_ALLOC_FLAG_WRITABLE 1 + +#define IOCTL_GNTDEV_DEALLOC_GREF \ + _IOW('E', 2, struct ioctl_gntdev_dealloc_gref) +struct ioctl_gntdev_dealloc_gref { + /* IN parameters */ + uint64_t index; + uint32_t count; +}; + +/*-------------------- Grant Mapping IOCTLs ---------------------------------*/ + +struct ioctl_gntdev_grant_ref { + uint32_t domid; + uint32_t ref; +}; + +#define IOCTL_GNTDEV_MAP_GRANT_REF \ + _IOWR('E', 3, struct ioctl_gntdev_map_grant_ref) +struct ioctl_gntdev_map_grant_ref { + /* IN parameters */ + uint32_t count; + uint32_t pad0; + /* OUT parameters */ + uint64_t index; + /* Variable IN parameter */ + struct ioctl_gntdev_grant_ref refs[1]; +}; + +#define IOCTL_GNTDEV_UNMAP_GRANT_REF \ + _IOW('E', 4, struct ioctl_gntdev_unmap_grant_ref) +struct ioctl_gntdev_unmap_grant_ref { + /* IN parameters */ + uint64_t index; + uint32_t count; +}; + +#define IOCTL_GNTDEV_GET_OFFSET_FOR_VADDR \ + _IOWR('E', 5, struct ioctl_gntdev_get_offset_for_vaddr) +struct ioctl_gntdev_get_offset_for_vaddr { + /* IN parameters */ + uint64_t vaddr; + /* OUT parameters */ + uint64_t offset; + uint32_t count; +}; + +#endif /* __XEN_GNTDEV_H__ */ \ No newline at end of file diff --git a/tools/libs/gnttab/Makefile b/tools/libs/gnttab/Makefile index af64542..69bb207 100644 --- a/tools/libs/gnttab/Makefile +++ b/tools/libs/gnttab/Makefile @@ -14,7 +14,7 @@ SRCS-GNTSHR += gntshr_core.c SRCS-$(CONFIG_Linux) += $(SRCS-GNTTAB) $(SRCS-GNTSHR) linux.c SRCS-$(CONFIG_MiniOS) += $(SRCS-GNTTAB) gntshr_unimp.c minios.c -SRCS-$(CONFIG_FreeBSD) += gnttab_unimp.c gntshr_unimp.c +SRCS-$(CONFIG_FreeBSD) += $(SRCS-GNTTAB) $(SRCS-GNTSHR) freebsd.c SRCS-$(CONFIG_SunOS) += gnttab_unimp.c gntshr_unimp.c SRCS-$(CONFIG_NetBSD) += gnttab_unimp.c gntshr_unimp.c diff --git a/tools/libs/gnttab/freebsd.c b/tools/libs/gnttab/freebsd.c new file mode 100644 index 0000000..fc5580e --- /dev/null +++ b/tools/libs/gnttab/freebsd.c @@ -0,0 +1,318 @@ +/* + * Copyright (c) 2007-2008, D G Murray + * Copyright (c) 2016-2017, Akshay Jaggi + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; If not, see . + * + * Split out from linux.c + */ + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include + +#include "private.h" + +#define DEVXEN "/dev/xen/gntdev" + +#define ROUNDUP(_x,_w) (((unsigned long)(_x)+(1UL<<(_w))-1) & ~((1UL<<(_w))-1)) + +#define GTERROR(_l, _f...) xtl_log(_l, XTL_ERROR, errno, "gnttab", _f) +#define GSERROR(_l, _f...) xtl_log(_l, XTL_ERROR, errno, "gntshr", _f) + +#define PAGE_SHIFT 12 +#define PAGE_SIZE (1UL << PAGE_SHIFT) +#define PAGE_MASK (~(PAGE_SIZE-1)) + +int osdep_gnttab_open(xengnttab_handle *xgt) +{ + int fd = open(DEVXEN, O_RDWR|O_CLOEXEC); + + if ( fd == -1 ) + return -1; + xgt->fd = fd; + + return 0; +} + +int osdep_gnttab_close(xengnttab_handle *xgt) +{ + + if ( xgt->fd == -1 ) + return 0; + + return close(xgt->fd); +} + +int osdep_gnttab_set_max_grants(xengnttab_handle *xgt, uint32_t count) +{ + + return 0; +} + +void *osdep_gnttab_grant_map(xengnttab_handle *xgt, + uint32_t count, int flags, int prot, + uint32_t *domids, uint32_t *refs, + uint32_t notify_offset, + evtchn_port_t notify_port) +{ + uint32_t i; + int fd = xgt->fd; + struct ioctl_gntdev_map_grant_ref *map; + void *addr = NULL; + int domids_stride; + unsigned int map_size = ROUNDUP((sizeof(*map) + (count - 1) * + sizeof(struct ioctl_gntdev_map_grant_ref)), + PAGE_SHIFT); + + domids_stride = ( flags & XENGNTTAB_GRANT_MAP_SINGLE_DOMAIN ) ? 0 : 1; + if ( map_size <= PAGE_SIZE ) + map = malloc(sizeof(*map) + + (count - 1) * sizeof(struct ioctl_gntdev_map_grant_ref)); + else + { + map = mmap(NULL, map_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANON, -1, 0); + if ( map == MAP_FAILED ) + { + GTERROR(xgt->logger, "anon mmap of map failed"); + return NULL; + } + } + + for ( i = 0; i < count; i++ ) + { + map->refs[i].domid = domids[i * domids_stride]; + map->refs[i].ref = refs[i]; + } + + map->count = count; + + if ( ioctl(fd, IOCTL_GNTDEV_MAP_GRANT_REF, map) ) + { + GTERROR(xgt->logger, "ioctl MAP_GRANT_REF failed"); + goto out; + } + + addr = mmap(NULL, PAGE_SIZE * count, prot, MAP_SHARED, fd, + map->index); + if ( addr != MAP_FAILED ) + { + int rv = 0; + struct ioctl_gntdev_unmap_notify notify; + + notify.index = map->index; + notify.action = 0; + if ( notify_offset < PAGE_SIZE * count ) + { + notify.index += notify_offset; + notify.action |= UNMAP_NOTIFY_CLEAR_BYTE; + } + if ( notify_port != -1 ) { + notify.event_channel_port = notify_port; + notify.action |= UNMAP_NOTIFY_SEND_EVENT; + } + if ( notify.action ) + rv = ioctl(fd, IOCTL_GNTDEV_SET_UNMAP_NOTIFY, ¬ify); + if ( rv ) + { + GTERROR(xgt->logger, "ioctl SET_UNMAP_NOTIFY failed"); + munmap(addr, count * PAGE_SIZE); + addr = MAP_FAILED; + } + } + if ( addr == MAP_FAILED ) + { + int saved_errno = errno; + struct ioctl_gntdev_unmap_grant_ref unmap_grant; + + /* Unmap the driver slots used to store the grant information. */ + GTERROR(xgt->logger, "mmap failed"); + unmap_grant.index = map->index; + unmap_grant.count = count; + ioctl(fd, IOCTL_GNTDEV_UNMAP_GRANT_REF, &unmap_grant); + errno = saved_errno; + addr = NULL; + } + + out: + if ( map_size > PAGE_SIZE ) + munmap(map, map_size); + else + free(map); + + return addr; +} + +int osdep_gnttab_unmap(xengnttab_handle *xgt, + void *start_address, + uint32_t count) +{ + int rc; + int fd = xgt->fd; + struct ioctl_gntdev_unmap_grant_ref unmap_grant; + struct ioctl_gntdev_get_offset_for_vaddr get_offset; + + if ( start_address == NULL ) + { + errno = EINVAL; + return -1; + } + + /* + * First, it is necessary to get the offset which was initially used to + * mmap() the pages. + */ + get_offset.vaddr = (unsigned long)start_address; + if ( (rc = ioctl(fd, IOCTL_GNTDEV_GET_OFFSET_FOR_VADDR, + &get_offset)) ) + return rc; + + if ( get_offset.count != count ) + { + errno = EINVAL; + return -1; + } + + /* Next, unmap the memory. */ + if ( (rc = munmap(start_address, count * PAGE_SIZE)) ) + return rc; + + /* Finally, unmap the driver slots used to store the grant information. */ + unmap_grant.index = get_offset.offset; + unmap_grant.count = count; + if ( (rc = ioctl(fd, IOCTL_GNTDEV_UNMAP_GRANT_REF, &unmap_grant)) ) + return rc; + + return 0; +} + +int osdep_gntshr_open(xengntshr_handle *xgs) +{ + + int fd = open(DEVXEN, O_RDWR); + if ( fd == -1 ) + return -1; + xgs->fd = fd; + + return 0; +} + +int osdep_gntshr_close(xengntshr_handle *xgs) +{ + + if ( xgs->fd == -1 ) + return 0; + + return close(xgs->fd); +} + +void *osdep_gntshr_share_pages(xengntshr_handle *xgs, + uint32_t domid, int count, + uint32_t *refs, int writable, + uint32_t notify_offset, + evtchn_port_t notify_port) +{ + int err; + int fd = xgs->fd; + void *area = NULL; + struct ioctl_gntdev_unmap_notify notify; + struct ioctl_gntdev_dealloc_gref gref_drop; + struct ioctl_gntdev_alloc_gref *gref_info = NULL; + + gref_info = malloc(sizeof(*gref_info) + count * sizeof(uint32_t)); + if ( gref_info == NULL ) + return NULL; + gref_info->domid = domid; + gref_info->flags = writable ? GNTDEV_ALLOC_FLAG_WRITABLE : 0; + gref_info->count = count; + + err = ioctl(fd, IOCTL_GNTDEV_ALLOC_GREF, gref_info); + if ( err ) + { + GSERROR(xgs->logger, "ioctl failed"); + goto out; + } + + area = mmap(NULL, count * PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, gref_info->index); + + if ( area == MAP_FAILED ) + { + area = NULL; + GSERROR(xgs->logger, "mmap failed"); + goto out_remove_fdmap; + } + + notify.index = gref_info->index; + notify.action = 0; + if ( notify_offset < PAGE_SIZE * count ) + { + notify.index += notify_offset; + notify.action |= UNMAP_NOTIFY_CLEAR_BYTE; + } + if ( notify_port != -1 ) + { + notify.event_channel_port = notify_port; + notify.action |= UNMAP_NOTIFY_SEND_EVENT; + } + if ( notify.action ) + err = ioctl(fd, IOCTL_GNTDEV_SET_UNMAP_NOTIFY, ¬ify); + if ( err ) + { + GSERROR(xgs->logger, "ioctl SET_UNMAP_NOTIFY failed"); + munmap(area, count * PAGE_SIZE); + area = NULL; + } + + memcpy(refs, gref_info->gref_ids, count * sizeof(uint32_t)); + + out_remove_fdmap: + /* + * Removing the mapping from the file descriptor does not cause the + * pages to be deallocated until the mapping is removed. + */ + gref_drop.index = gref_info->index; + gref_drop.count = count; + ioctl(fd, IOCTL_GNTDEV_DEALLOC_GREF, &gref_drop); + out: + free(gref_info); + + return area; +} + +int osdep_gntshr_unshare(xengntshr_handle *xgs, + void *start_address, uint32_t count) +{ + + return munmap(start_address, count * PAGE_SIZE); +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ -- 2.8.2