From nobody Tue Jul 23 23:09:42 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WTCY93hCyz5R721 for ; Tue, 23 Jul 2024 23:10:21 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WTCY86rFkz4dQs for ; Tue, 23 Jul 2024 23:10:20 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=UxBARvBY; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-2cb81c0ecb4so245121a91.0 for ; Tue, 23 Jul 2024 16:10:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721776219; x=1722381019; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=TxCorSKr3bwJYIDIuHjKSC8BUTwZoAnSboGb6av2hpI=; b=UxBARvBYBwYvvDqZP9e4FXoH9O2UYR+Mo1jgywZT1qCiUGsVfvXWjONLfImdBPXlC6 QodjZxjQrtQv3zTgaI95yvfotfV2L1F9trj6uSiK9S39TaZypD4p/JppcHjYvrB02o2S DlE8AXeITQPjkkXtR1mkLa7v6pP+0PYngoLXHPlyrNlrGNEphifdvBgbBififLSUJ6Sq TQu253vZgvtKWndafrjFCpfx04a+qucqeO7vK/6P8J303o4ehPqPTlVlGQamOm9x/7b7 AvlpI/XWYca8w7Zjh7IJhdxvfSHrHKvRnWKX13flwgpeqhBUo4F/FTbGR96PFFEIkiuL /5AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721776219; x=1722381019; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=TxCorSKr3bwJYIDIuHjKSC8BUTwZoAnSboGb6av2hpI=; b=e88xPDJb4DpzE3uW6px4zaSq3H5v3PSCXsuS9Li+DYhfFmlbCzPED+1ff1/xaj0tvF TZw8etbE8KAZrQlv2irt9oCYEF2pgaUNcj1RvmeJ7d0hLPU9A+8NzoWef23xkvOvjgbe jE+8pUEOiEQwuKkKiu7ek57CQE3RGLPaSWqkrSvNSbdVWW7CTvZwZK7uQTLmpEMh0tnm iT0J9Z8htnuKtXEjxmbBQuJ5RMmYfwntjTn32T6J+02zIJ2dPYERRejKH7YRf9c9pctj sxx5xy6VgdANZhlzn5TNAet+tkQG7GkvWsEC+Wiu3yf/w5KUzuMak9A0y/APSIBFguyl MZHQ== X-Gm-Message-State: AOJu0Yzxey1iwe5OpulaTmlDHzVIcJqY5tfDhAc/t6JyMwThdWpglxiz 5n3ZVPxl72qIb+Y3smSnTmuxGje9D3RxOl1C4A75WivL16FgbaeVw1QdWVRlBZslpQQBt1lfp9O fs1ddH1GOqEzcAVYUgDc9zpJc5b9jcqL0F/oKLw== X-Google-Smtp-Source: AGHT+IHzcTH6tKu+y2yogR8VE/FyJD3eORdV621Q2mlydxAtPJLxaqPbCno9radcx3L6AWBSxz/1kZC9zMa0gtMGGrw= X-Received: by 2002:a17:90a:fa01:b0:2cb:5883:8fb0 with SMTP id 98e67ed59e1d1-2cdb9442ae1mr184417a91.14.1721776219227; Tue, 23 Jul 2024 16:10:19 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 From: Mario Marietto Date: Wed, 24 Jul 2024 01:09:42 +0200 Message-ID: Subject: Problems with wayland and kde 6 To: freebsd-hackers Content-Type: multipart/alternative; boundary="000000000000ee0e4d061df24352" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.85 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.85)[-0.845]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102a:from] X-Rspamd-Queue-Id: 4WTCY86rFkz4dQs --000000000000ee0e4d061df24352 Content-Type: text/plain; charset="UTF-8" Hello. We (at https://forums.freebsd.org) are trying to configure and run KDE 6 with Wayland on FreeBSD 14.1-RELEASE p2. After a lot of research and trial and error I've found that the parameters below are able to launch correctly Wayland + Wayfire and KDE 6 on Xorg,but NOT Wayland + KDE 6. The parameters are the following : #!/bin/sh #export QT_DEBUG_PLUGINS=1 #export WAYLAND_DEBUG=1 export XDG_CONFIG_HOME="$HOME/.config" export XDG_CACHE_HOME="$HOME/.cache" export XDG_DATA_HOME="$HOME/.local/share" export XDG_RUNTIME_DIR=/var/run/user/"$(id -u)" export LIBSEAT_BACKEND=consolekit2 export QT_WAYLAND_SHELL_INTEGRATION=xdg-shell export XDG_SESSION_TYPE=xdg-shell export QT_QPA_PLATFORMTHEME=qt5ct #export __GLX_VENDOR_LIBRARY_NAME=nvidia #export CLUTTER_BACKEND=wayland #export SDL_VIDEODRIVER=wayland #export LIBGL_DRI3_ENABLE=1 #export XKB_DEFAULT_RULES=evdev export QT_QPA_PLATFORM=minimal #export QT_WAYLAND_DISABLE_WINDOWDECORATION=1 #export BEMENU_BACKEND=wayland #export WLR_DRM_NO_ATOMIC=1 #export XCURSOR_THEME=whiteglass export WLR_NO_HARDWARE_CURSORS=1 #mkdir -p /var/run/user/"$(id -u)" #echo 1 #chown -R "${USER}":wheel /var/run/user/"$(id -u)" #echo 2 #chmod 700 /var/run/user/"$(id -u)" #echo 3 ck-launch-session dbus-run-session startplasma-wayland 2> startplasma.log #ck-launch-session dbus-run-session wayfire This is the log file. Can you give us some suggestions to fix some of the problems that you see below ? I want to exclude some reasons that could produce some errors that don't make KDE 6 to start. org.kde.startup: not a reply org.freedesktop.locale1 QDBusMessage(type=Error, service="", error name="org.freedesktop.DBus.Error.ServiceUnknown", error message="The name org.freedesktop.locale1 was not provided by any .service files", signature="s", contents=("The name org.freedesktop.locale1 was not provided by any .service files") ) dbus-daemon[1142]: [session uid=1001 pid=1142] Activating service name='org.kde.KSplash' requested by ':1.0' (uid=1001 pid=1143 comm="") dbus-daemon[1142]: [session uid=1001 pid=1142] Successfully activated service 'org.kde.KSplash' Initializing "/usr/local/lib/qt6/plugins/plasma/kcms/systemsettings/kcm_fonts.so" dbus-daemon[1142]: [session uid=1001 pid=1142] Activating service name='org.freedesktop.portal.Desktop' requested by ':1.5' (uid=1001 pid=1148 comm="") dbus-daemon[1142]: [session uid=1001 pid=1142] Activating service name='org.freedesktop.portal.Documents' requested by ':1.6' (uid=1001 pid=1156 comm="") dbus-daemon[1142]: [session uid=1001 pid=1142] Activating service name='org.freedesktop.impl.portal.PermissionStore' requested by ':1.7' (uid=1001 pid=1158 comm="") dbus-daemon[1142]: [session uid=1001 pid=1142] Successfully activated service 'org.freedesktop.impl.portal.PermissionStore' dbus-daemon[1142]: [session uid=1001 pid=1142] Successfully activated service 'org.freedesktop.portal.Documents' fuse: unknown option(s): `-o auto_unmount' error: fuse init failed: Can't create fuse session kf.kirigami.platform: Failed to find a Kirigami platform plugin for style "" (/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.365: Failed to load RealtimeKit property: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.RealtimeKit1 was not provided by any .service files (/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.365: Failed to load RealtimeKit property: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.RealtimeKit1 was not provided by any .service files (/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.366: Failed to load RealtimeKit property: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.RealtimeKit1 was not provided by any .service files dbus-daemon[1142]: [session uid=1001 pid=1142] Activating service name='org.freedesktop.impl.portal.desktop.kde' requested by ':1.6' (uid=1001 pid=1156 comm="") No backend specified, automatically choosing drm dbus-daemon[1142]: [session uid=1001 pid=1142] Successfully activated service 'org.freedesktop.impl.portal.desktop.kde' dbus-daemon[1142]: [session uid=1001 pid=1142] Activating service name='org.gtk.vfs.Daemon' requested by ':1.6' (uid=1001 pid=1156 comm="") dbus-daemon[1142]: [session uid=1001 pid=1142] Successfully activated service 'org.gtk.vfs.Daemon' dbus-daemon[1142]: [session uid=1001 pid=1142] Activating service name='org.gtk.vfs.UDisks2VolumeMonitor' requested by ':1.6' (uid=1001 pid=1156 comm="") dbus-daemon[1142]: [session uid=1001 pid=1142] Successfully activated service 'org.gtk.vfs.UDisks2VolumeMonitor' dbus-daemon[1142]: [session uid=1001 pid=1142] Activating service name='org.gtk.vfs.MTPVolumeMonitor' requested by ':1.6' (uid=1001 pid=1156 comm="") dbus-daemon[1142]: [session uid=1001 pid=1142] Successfully activated service 'org.gtk.vfs.MTPVolumeMonitor' dbus-daemon[1142]: [session uid=1001 pid=1142] Activating service name='org.gtk.vfs.GPhoto2VolumeMonitor' requested by ':1.6' (uid=1001 pid=1156 comm="") dbus-daemon[1142]: [session uid=1001 pid=1142] Successfully activated service 'org.gtk.vfs.GPhoto2VolumeMonitor' (/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.463: Failed connect to PipeWire: Couldn't connect to PipeWire (/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.468: Choosing kwallet.portal for org.freedesktop.impl.portal.Secret via the deprecated UseIn key (/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.468: The preferred method to match portal implementations to desktop environments is to use the portals.conf(5) configuration file dbus-daemon[1142]: [session uid=1001 pid=1142] Successfully activated service 'org.freedesktop.portal.Desktop' Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context No backend specified, automatically choosing drm Unable to determine system time zone: please check your system configuration. kwin_screencast: Failed to connect PipeWire context org.kde.startup: "kdeinit5_shutdown" QList() exited with code 255 Initializing "/usr/local/lib/qt6/plugins/plasma/kcms/systemsettings/kcm_style.so" startplasma-wayland: Shutting down... startplasmacompositor: Shutting down... startplasmacompositor: Done. A connection to the bus can't be made /usr/local/bin/xrdb: Connection refused /usr/local/bin/xrdb: Can't open display ':0' /usr/local/bin/xsetroot: unable to open display ':0' Initializing "/usr/local/lib/qt6/plugins/plasma/kcms/systemsettings/kcm_mouse.so" kcm_mouse: Not able to select appropriate backend. -- Mario. --000000000000ee0e4d061df24352 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello.

We (at https://forums.freebsd.org) are trying to configure and run KDE 6 with Wayland on FreeBSD=20 14.1-RELEASE p2. After a lot of research and trial and error I've found= =20 that the parameters below are able to launch correctly Wayland + Wayfire and KDE 6 on Xorg,but NOT Wayland + KDE 6. The parameters=20 are the following :


#!/bin/sh
#export QT_DEBUG_PLUGINS=3D1
#export WAYLAND_DEBUG=3D1

export X= DG_CONFIG_HOME=3D"$HOME/.config"
export XDG_CACHE_HOME=3D"= ;$HOME/.cache"
export XDG_DATA_HOME=3D"$HOME/.local/share"= ;
export XDG_RUNTIME_DIR=3D/var/run/user/"$(id -u)"
export = LIBSEAT_BACKEND=3Dconsolekit2

export QT_WAYLAND_SHELL_INTEGRATION=3D= xdg-shell
export XDG_SESSION_TYPE=3Dxdg-shell

export QT_QPA_PLATF= ORMTHEME=3Dqt5ct

#export __GLX_VENDOR_LIBRARY_NAME=3Dnvidia
#expo= rt CLUTTER_BACKEND=3Dwayland
#export SDL_VIDEODRIVER=3Dwayland
#expor= t LIBGL_DRI3_ENABLE=3D1
#export XKB_DEFAULT_RULES=3Devdev

export = QT_QPA_PLATFORM=3Dminimal

#export QT_WAYLAND_DISABLE_WINDOWDECORATIO= N=3D1
#export BEMENU_BACKEND=3Dwayland
#export WLR_DRM_NO_ATOMIC=3D1<= br>#export XCURSOR_THEME=3Dwhiteglass

export WLR_NO_HARDWARE_CURSORS= =3D1

#mkdir -p /var/run/user/"$(id -u)"
#echo 1
#cho= wn -R "${USER}":wheel /var/run/user/"$(id -u)"
#echo= 2
#chmod 700 /var/run/user/"$(id -u)"
#echo 3

ck-la= unch-session dbus-run-session startplasma-wayland 2> startplasma.log
=
#ck-launch-session dbus-run-session wayfire

This is the log file. Can you give us some suggestions to fix = some of the problems that you see below ? I want to exclude some reasons=20 that could produce some errors that don't make KDE 6 to start.


org.kde.startup: not a reply org.freed= esktop.locale1 QDBusMessage(type=3DError, service=3D"", error nam= e=3D"org.freedesktop.DBus.Error.ServiceUnknown", error message=3D"The name org.freedesktop.locale1 was not provided by= any .service files", signature=3D"s", contents=3D("The nam= e=20 org.freedesktop.locale1 was not provided by any .service files") )
= dbus-daemon[1142]: [session uid=3D1001 pid=3D1142] Activating service name=3D'org.kde.KSp= lash'=20 requested by ':1.0' (uid=3D1001 pid=3D1143 comm=3D"")
= dbus-daemon[1142]: [session uid=3D1001 pid=3D1142] Successfully activated s= ervice 'org.kde.KSplash'
Initializing =C2=A0"/usr/local/lib= /qt6/plugins/plasma/kcms/systemsettings/kcm_fonts.so"
dbus-daemon[1= 142]: [session uid=3D1001 pid=3D1142] Activating service name=3D'org.fr= eedesktop.portal.Desktop' requested by ':1.5' (uid=3D1001 pid= =3D1148 comm=3D"")
dbus-daemon[1142]: [session uid=3D1001 pid= =3D1142] Activating service name=3D'org.freedesktop.portal.Documents= 9; requested by ':1.6' (uid=3D1001 pid=3D1156 comm=3D"")<= br>dbus-daemon[1142]: [session uid=3D1001 pid=3D1142] Activating service na= me=3D'org.freedesktop.impl.portal.PermissionStore' requested by = 9;:1.7' (uid=3D1001 pid=3D1158 comm=3D"")
dbus-daemon[1142= ]: [session uid=3D1001 pid=3D1142] Successfully activated service 'org.= freedesktop.impl.portal.PermissionStore'
dbus-daemon[1142]: [session= uid=3D1001 pid=3D1142] Successfully activated service 'org.freedesktop= .portal.Documents'
fuse: unknown option(s): `-o auto_unmount'error: fuse init failed: Can't create fuse session
kf.kirigami.plat= form: Failed to find a Kirigami platform plugin for style ""
<= br>(/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.365: Failed to load RealtimeKit property: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name = org.freedesktop.RealtimeKit1 was not provided by any .service files

= (/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.365: Failed to load RealtimeKit property: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name = org.freedesktop.RealtimeKit1 was not provided by any .service files

= (/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.366: Failed to load RealtimeKit property: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name = org.freedesktop.RealtimeKit1 was not provided by any .service files
dbus= -daemon[1142]: [session uid=3D1001 pid=3D1142] Activating service name=3D&#= 39;org.freedesktop.impl.portal.desktop.kde' requested by ':1.6'= (uid=3D1001 pid=3D1156 comm=3D"")
No backend specified, autom= atically choosing drm
dbus-daemon[1142]: [session uid=3D1001 pid=3D1142]= Successfully activated service 'org.freedesktop.impl.portal.desktop.kd= e'
dbus-daemon[1142]: [session uid=3D1001 pid=3D1142] Activating service=20 name=3D'org.gtk.vfs.Daemon' requested by ':1.6' (uid=3D1001= pid=3D1156=20 comm=3D"")
dbus-daemon[1142]: [session uid=3D1001 pid=3D1142] = Successfully activated service 'org.gtk.vfs.Daemon'
dbus-daemon[= 1142]: [session uid=3D1001 pid=3D1142] Activating service name=3D'org.g= tk.vfs.UDisks2VolumeMonitor' requested by ':1.6' (uid=3D1001 pi= d=3D1156 comm=3D"")
dbus-daemon[1142]: [session uid=3D1001 pid= =3D1142] Successfully activated service 'org.gtk.vfs.UDisks2VolumeMonit= or'
dbus-daemon[1142]: [session uid=3D1001 pid=3D1142] Activating se= rvice name=3D'org.gtk.vfs.MTPVolumeMonitor' requested by ':1.6&= #39; (uid=3D1001 pid=3D1156 comm=3D"")
dbus-daemon[1142]: [ses= sion uid=3D1001 pid=3D1142] Successfully activated service 'org.gtk.vfs= .MTPVolumeMonitor'
dbus-daemon[1142]: [session uid=3D1001 pid=3D1142= ] Activating service name=3D'org.gtk.vfs.GPhoto2VolumeMonitor' requ= ested by ':1.6' (uid=3D1001 pid=3D1156 comm=3D"")
dbus= -daemon[1142]: [session uid=3D1001 pid=3D1142] Successfully activated servi= ce 'org.gtk.vfs.GPhoto2VolumeMonitor'

(/usr/local/libexec/xd= g-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.463: Failed= connect to PipeWire: Couldn't connect to PipeWire

(/usr/local/l= ibexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.46= 8: Choosing kwallet.portal for org.freedesktop.impl.portal.Secret via the d= eprecated UseIn key

(/usr/local/libexec/xdg-desktop-portal:1156): xdg-desktop-portal-WARNING **: 23:07:33.468: The preferred method to=20 match portal implementations to desktop environments is to use the=20 portals.conf(5) configuration file
dbus-daemon[1142]: [session uid=3D100= 1 pid=3D1142] Successfully activated service 'org.freedesktop.portal.De= sktop'

Unable to determine system time zone: p= lease check your system configuration.
kwin_screencast: Failed to connec= t PipeWire context
No backend specified, automatically choosing drm
U= nable to determine system time zone: please check your system configuration= .
kwin_screencast: Failed to connect PipeWire context
No backend spec= ified, automatically choosing drm
Unable to determine system time zone: = please check your system configuration.
kwin_screencast: Failed to conne= ct PipeWire context
No backend specified, automatically choosing drm
= Unable to determine system time zone: please check your system configuratio= n.
kwin_screencast: Failed to connect PipeWire context
No backend spe= cified, automatically choosing drm
Unable to determine system time zone:= please check your system configuration.
kwin_screencast: Failed to conn= ect PipeWire context
No backend specified, automatically choosing drmUnable to determine system time zone: please check your system configurati= on.
kwin_screencast: Failed to connect PipeWire context
No backend sp= ecified, automatically choosing drm
Unable to determine system time zone= : please check your system configuration.
kwin_screencast: Failed to con= nect PipeWire context
No backend specified, automatically choosing drmUnable to determine system time zone: please check your system configurat= ion.
kwin_screencast: Failed to connect PipeWire context
No backend s= pecified, automatically choosing drm
Unable to determine system time zon= e: please check your system configuration.
kwin_screencast: Failed to co= nnect PipeWire context
No backend specified, automatically choosing drm<= br>Unable to determine system time zone: please check your system configura= tion.
kwin_screencast: Failed to connect PipeWire context
No backend = specified, automatically choosing drm
Unable to determine system time zo= ne: please check your system configuration.
kwin_screencast: Failed to c= onnect PipeWire context

org.kde.startup: "kde= init5_shutdown" QList() exited with code 255
Initializing =C2=A0&qu= ot;/usr/local/lib/qt6/plugins/plasma/kcms/systemsettings/kcm_style.so"=
startplasma-wayland: Shutting down...
startplasmacompositor: Shuttin= g down...
startplasmacompositor: Done.
A connection to the bus can= 9;t be made
/usr/local/bin/xrdb: Connection refused
/usr/local/bin/xr= db: Can't open display ':0'
/usr/local/bin/xsetroot: =C2=A0u= nable to open display ':0'
Initializing =C2=A0"/usr/local/l= ib/qt6/plugins/plasma/kcms/systemsettings/kcm_mouse.so"
kcm_mouse: = Not able to select appropriate backend.

--
Mario.
--000000000000ee0e4d061df24352-- From nobody Thu Jul 25 09:07:34 2024 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WV4m14YH0z5Rfrw for ; Thu, 25 Jul 2024 09:07:45 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WV4lz5v4Pz3xTQ for ; Thu, 25 Jul 2024 09:07:43 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=oQWPb7UP; dkim=none ("invalid DKIM record") header.d=cse.huji.ac.il header.s=57791128 header.b=vJwy8xCH; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=To:Date:Message-Id:Subject:Mime-Version:Content-Transfer-Encoding:Content-Type:From; bh=nnyt0IYo14zphcgNRzl/mbvIypp1HvHW8Xs0u6AGK74=; b=oQWPb7UPcI3Q8BGjPuGeqOUXbSu3i7mK/L5FjuMu57xb+ukesNlPQ/TeOWU6cYYUVlMoF618iJwkYl1QzkM6jcw7BrzQtV8sKiG7jTN/N8uNZoYoMXPoVoYDCGD+mhr7hmtPFPZnJN2IwYLalUsQ93md495huFnEhqyA7YCYQScxKlmYZCPfXsto33geKQxNuAiS/gkpFQFW3r15hjo1IT4CQuU0Hevb3WWDO6AfJnuUT9LYmrz0mdv9jCuWLl6KuJKODN8oTfMvUNvN2GzA1imRghv2ldzbujWv3etxB3GU4N+ZxCTmQg63L+/F7/15PbtUBqtxc3sEFBAnzXDx0w==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cse.huji.ac.il; s=57791128; h=To:Date:Message-Id:Subject:Mime-Version:Content-Transfer-Encoding:Content-Type:From; bh=nnyt0IYo14zphcgNRzl/mbvIypp1HvHW8Xs0u6AGK74=; b=vJwy8xCHB+Dtn5xI9HnMAMXROKOwSbSApqGkBm92bOuH+khZz8yUNAXgv1RP/n70OnzBKyJ3SpLfMVWqBdutWyqjnnzhOsmgFgulsDnYPh4UjdYb9O497bMsJ4Mt3Ws1tckQWHX35GIaOGvsOF1KtoVZM3sPL3riY80Au4E9fIRiAySKFv8RT4TIWGyxOJJ2AnLzC/Ygyt5ztC+tzTObbnAnB/w10zQrWavITIskh8fDHCOLyWKPAECsSFQ0+V7X9Ffn5urpUb/ytmD+g0uTTXjxDKcD1W2OYNFFxtNJ3AVAWX/lT+oECQQgnD89Yh8FzpILyWYA/rk7k026pQLzug==; Received: from bach.cs.huji.ac.il ([132.65.80.20] helo=smtpclient.apple) by kabab.cs.huji.ac.il with esmtp id 1sWuRi-000Neg-S2 for hackers@freebsd.org; Thu, 25 Jul 2024 12:07:34 +0300 From: Daniel Braniss Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\)) Subject: kevent and NFS mounted file Message-Id: <8A898922-4D2B-453D-967E-E9F197B66DC0@cs.huji.ac.il> Date: Thu, 25 Jul 2024 12:07:34 +0300 To: freebsd-hackers X-Mailer: Apple Mail (2.3696.120.41.1.8) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.15 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.950]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:378, ipnet:132.64.0.0/15, country:IL]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[danny]; R_DKIM_PERMFAIL(0.00)[cse.huji.ac.il:s=57791128]; RCVD_COUNT_ONE(0.00)[1]; R_SPF_NA(0.00)[no SPF record]; MLMMJ_DEST(0.00)[hackers@freebsd.org]; DKIM_MIXED(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[cs.huji.ac.il:+,cse.huji.ac.il:~] X-Rspamd-Queue-Id: 4WV4lz5v4Pz3xTQ Hi, using kevent on an NFS mounted file did not work in the past, but just for fun (I actually forgot) I tried on an NFS mounted file, and = it works! BUT: only if the server is NetAPP, freebsd still does not work. so what gives? danny From nobody Thu Jul 25 12:33:14 2024 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WV9Kd2YNlz5S08L for ; Thu, 25 Jul 2024 12:33:41 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WV9Kb03ZVz4KsC for ; Thu, 25 Jul 2024 12:33:38 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=eVCuc0a4; dkim=none ("invalid DKIM record") header.d=cse.huji.ac.il header.s=57791128 header.b=bML5eWx7; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=Message-Id:In-Reply-To:To:References:Date:Subject:Mime-Version:Content-Transfer-Encoding:Content-Type:From; bh=XnmPU8YjsDoAExRGvSyGUh6jtz0E7rnFAYvzqZSCr6A=; b=eVCuc0a4m5Vi/btY4rDMif/DJviHLTZldRXxYr96rv6fR3Jz5dEPGJmNTN3D1XT5hXQCbX6iBfjPDyHDvgvAfpketIdAHdJkJzXjOkLW775U1PFv4dotCr0Bp2XGujbT4kgjnPgl63N3FcpxwjnlOsgPWJdY8/z+FljLus/ATz57E9ZOKaY1isjYB46LFHVvNx/6apLrYOXyfcRCg/zrw2qPe/IND30PnKQZ1kQsUiDiw101ZnHxfAidb+s2L5w//GYvwSbkEk86G354g78V0qMcEKwna3k0CHQaSV/qxIi4tFY2t3Rq61NtNdub5kiQNCB1NHVVu00PvTIzYX+DgQ==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cse.huji.ac.il; s=57791128; h=Message-Id:In-Reply-To:To:References:Date:Subject:Mime-Version:Content-Transfer-Encoding:Content-Type:From; bh=XnmPU8YjsDoAExRGvSyGUh6jtz0E7rnFAYvzqZSCr6A=; b=bML5eWx7NS2l9vmK73YWZZmqdWR2HSS2Zyehlcpjn+nqUec6ygW7tfdR5SKYjKnzn1NwV4QwShFRH574IDSCkHHROrQT8aceIhe3P+otfvP/FOOI7p4Vbw4yb00xc6WllTIyw5wmyF3MmcPXX73Dzk6byT4DHaZFbUpP2If0K4ErmE4fvtv4B+um2gxpl39vvmncz5czp+Ws9QpKv8SHFTxedY99mmqYpzl7qrS9OExNVlfSLapCaZHg+PkPt7AiUw/7ZzbRDgj4W7QcwU7N9blv9JSXtf/R6u5Nrou2RV1xSxW1O7qUYSo8FdEeykx6RSuPb/Kg9MTYpdDtCFIk3A==; Received: from imac.bk.cs.huji.ac.il ([132.65.179.42] helo=smtpclient.apple) by kabab.cs.huji.ac.il with esmtp id 1sWxf4-0007UG-BW for hackers@freebsd.org; Thu, 25 Jul 2024 15:33:34 +0300 From: Daniel Braniss Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: kevent and NFS mounted file Date: Thu, 25 Jul 2024 15:33:14 +0300 References: <8A898922-4D2B-453D-967E-E9F197B66DC0@cs.huji.ac.il> To: freebsd-hackers In-Reply-To: <8A898922-4D2B-453D-967E-E9F197B66DC0@cs.huji.ac.il> Message-Id: X-Mailer: Apple Mail (2.3774.600.62) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.70 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[cs.huji.ac.il:+,cse.huji.ac.il:~]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_NA(0.00)[no SPF record]; TO_DN_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_MIXED(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FREEFALL_USER(0.00)[danny]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[hackers@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/15, country:IL]; R_DKIM_PERMFAIL(0.00)[cse.huji.ac.il:s=57791128] X-Rspamd-Queue-Id: 4WV9Kb03ZVz4KsC Sorry, It only works if the update is done on the same host the kevent is = running, No matter where the file is. Sorry for the noise. Danny > On 25 Jul 2024, at 12:07, Daniel Braniss wrote: >=20 > Hi, > using kevent on an NFS mounted file did not work in the past, > but just for fun (I actually forgot) I tried on an NFS mounted file, = and it works! > BUT: only if the server is NetAPP, freebsd still does not work. >=20 > so what gives? >=20 > danny >=20 From nobody Thu Jul 25 18:46:17 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WVKbj0plrz5R3tl for ; Thu, 25 Jul 2024 18:46:25 +0000 (UTC) (envelope-from jake@technologyfriends.net) Received: from ci74p00im-qukt09081901.me.com (ci74p00im-qukt09081901.me.com [17.57.156.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WVKbg6b7gz571k for ; Thu, 25 Jul 2024 18:46:23 +0000 (UTC) (envelope-from jake@technologyfriends.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=technologyfriends.net header.s=sig1 header.b=jwmINP7L; dmarc=none; spf=pass (mx1.freebsd.org: domain of jake@technologyfriends.net designates 17.57.156.8 as permitted sender) smtp.mailfrom=jake@technologyfriends.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=technologyfriends.net; s=sig1; t=1721933182; bh=Q80iHPaARjWulI57uzQ2GC1jq9X4U5ZyPmqcB2++IXo=; h=Message-ID:Date:MIME-Version:From:To:Subject:Content-Type; b=jwmINP7LljR4B3iukMJqmn462Lq2sDuSdRbrnz+O8p6NupK3saYIAel/z3obYtEiU ItqLBlyzxatWa/xe6o3FlB7jW2T1/ng3GyricoQ5bbjQz7BFuAAbT2vK3xb9HHy2mV 1OugCRqh4+WId45DBlBYqCdyEY6WYRPI11vRuAeWOXV8P58/GCJC1Zz4upjDlFOWZh IH1R75N7BPXlaYRS1/Upwv421X3DhVi/jKPBO+xCyp7rcK98WOn5ofYlHFqoCscAdr BKQ/k1dTUdqgx2RBo1oLFhSSapzZ4jnK7JG5cedZAjtjil5ykaNBad1FVZVgphjBeP ghm+ypVWYnpaw== Received: from [10.0.233.209] (ci77p00im-dlb-asmtp-mailmevip.me.com [17.57.156.26]) by ci74p00im-qukt09081901.me.com (Postfix) with ESMTPSA id 2B6855AC048D for ; Thu, 25 Jul 2024 18:46:20 +0000 (UTC) Message-ID: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> Date: Thu, 25 Jul 2024 13:46:17 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US From: Jake Freeland To: freebsd-hackers@freebsd.org Subject: FreeBSD hugepages Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Proofpoint-GUID: k7mM4AnPLDJFixgCVm0UwjdoCrUZLfDq X-Proofpoint-ORIG-GUID: k7mM4AnPLDJFixgCVm0UwjdoCrUZLfDq X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-25_18,2024-07-25_03,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 clxscore=1030 mlxscore=0 mlxlogscore=746 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2308100000 definitions=main-2407250129 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[technologyfriends.net:s=sig1]; R_SPF_ALLOW(-0.20)[+ip4:17.57.156.0/24]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; RWL_MAILSPIKE_GOOD(-0.10)[17.57.156.8:from]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[technologyfriends.net:+]; DMARC_NA(0.00)[technologyfriends.net]; RCVD_TLS_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[jake]; ARC_NA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:714, ipnet:17.57.156.0/24, country:US]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4WVKbg6b7gz571k Hi there, I have been steadily working on bringing Data Plane Development Kit (DPDK) on FreeBSD up to date with the Linux version. The most significant hurdle so far has been supporting concurrent DPDK processes, each with their own contiguous memory regions. These contiguous regions are used by DPDK as a heap for allocating DMA buffers and other miscellaneous resources. Retrieving the underlying memory and mapping these regions is currently different on Linux and FreeBSD: On Linux, hugepages are fetched from the kernel's pre-allocated hugepage pool and are mapped into virtual address space on DPDK initialization. Since the hugepages exist in a pool, multiple processes can reserve their own hugepages and operate concurrently. On FreeBSD, DPDK uses an in-house contigmem kernel module that reserves a large contiguous region of memory on load. During DPDK initialization, the entire region is mapped into virtual address space. This leaves no memory for another independent DPDK process, so only one process can operate at a time. I could modify the DPDK contigmem module to mimic Linux's hugepages, but I thought it would be better to integrate and upstream a hugepage-like interface directly in the FreeBSD kernel source. I am writing this email to see if anyone has any advice on the matter. I did not see any previous attempts at this in Phabriactor or the commit log, but it is possible that I missed it. I have read about transparent superpage promotion, but that seems like a different mechanism altogether. At a quick glance, the implementation seems straightforward: read some loader tunables, allocate persistent hugepages at boot time, and create a pseudo filesystem that supports creating and mapping hugepages. I could be underestimating the magnitude of this task, but that is why I'm asking for thoughts and advice :) For reference, here is Linux's documentation on hugepages: https://docs.kernel.org/admin-guide/mm/hugetlbpage.html Let me know if you have any thoughts, Jake Freeland From nobody Thu Jul 25 19:02:23 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WVKyQ56RPz5R5MX for ; Thu, 25 Jul 2024 19:02:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WVKyP6TKGz59L3 for ; Thu, 25 Jul 2024 19:02:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 46PJ2NJ8014268; Thu, 25 Jul 2024 22:02:26 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 46PJ2NJ8014268 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 46PJ2N5M014267; Thu, 25 Jul 2024 22:02:23 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 25 Jul 2024 22:02:23 +0300 From: Konstantin Belousov To: Jake Freeland Cc: freebsd-hackers@freebsd.org Subject: Re: FreeBSD hugepages Message-ID: References: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.1 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on tom.home X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4WVKyP6TKGz59L3 On Thu, Jul 25, 2024 at 01:46:17PM -0500, Jake Freeland wrote: > Hi there, > > I have been steadily working on bringing Data Plane Development Kit (DPDK) > on FreeBSD up to date with the Linux version. The most significant hurdle so > far has been supporting concurrent DPDK processes, each with their own > contiguous memory regions. > > These contiguous regions are used by DPDK as a heap for allocating DMA > buffers and other miscellaneous resources. Retrieving the underlying memory > and mapping these regions is currently different on Linux and FreeBSD: > > On Linux, hugepages are fetched from the kernel's pre-allocated hugepage > pool and are mapped into virtual address space on DPDK initialization. Since > the hugepages exist in a pool, multiple processes can reserve their own > hugepages and operate concurrently. > > On FreeBSD, DPDK uses an in-house contigmem kernel module that reserves a > large contiguous region of memory on load. During DPDK initialization, the > entire region is mapped into virtual address space. This leaves no memory > for another independent DPDK process, so only one process can operate at a > time. > > I could modify the DPDK contigmem module to mimic Linux's hugepages, but I > thought it would be better to integrate and upstream a hugepage-like > interface directly in the FreeBSD kernel source. I am writing this email to > see if anyone has any advice on the matter. I did not see any previous > attempts at this in Phabriactor or the commit log, but it is possible that I > missed it. I have read about transparent superpage promotion, but that seems > like a different mechanism altogether. > > At a quick glance, the implementation seems straightforward: read some > loader tunables, allocate persistent hugepages at boot time, and create a > pseudo filesystem that supports creating and mapping hugepages. I could be > underestimating the magnitude of this task, but that is why I'm asking for > thoughts and advice :) > > For reference, here is Linux's documentation on hugepages: > https://docs.kernel.org/admin-guide/mm/hugetlbpage.html Are posix shm largepages objects enough (they were developed to support DPDK). Look for shm_create_largepage(3). From nobody Thu Jul 25 19:47:16 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WVLy32QTLz5RB06 for ; Thu, 25 Jul 2024 19:47:23 +0000 (UTC) (envelope-from jake@technologyfriends.net) Received: from ms11p00im-qufo17281501.me.com (ms11p00im-qufo17281501.me.com [17.58.38.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WVLy3036Hz43jV for ; Thu, 25 Jul 2024 19:47:22 +0000 (UTC) (envelope-from jake@technologyfriends.net) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=technologyfriends.net; s=sig1; t=1721936840; bh=rJHVRpZdANp7/DiITccDafsUq/D0AxedBh7oc6RN1BM=; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; b=eIXVhcxZdzl8PLKmG5Ylom6sWQj1Dt7gRQyVlnfI83gyoJz+CH3cHGVjrisuAyTqZ E81VcSSnaZCGI4pniylxu5HVJz0sYOe4g2YLFIzyFOiKrnq+hWEbWVvewymoJ4ErZo lhB+SYqpsRj7IeSPfTfrFDvktv8WphjdTwi/CT+dJJraSvLlnF987g2eYuQi4I+IYk 86Tp2S97sXhECkeowaUuM6BuwnhQL1zzll1iP4xAOi2Fmz6YMDIAc08KJFBWyJG5Vx G3vlCc9YtIsiRn2w+gSjZYZFDvISAlMjTtn6iaHzDR2lZ6N80l2HgJkt8wyPNNK5zW r6silvFkMsO8w== Received: from [10.0.233.209] (ms11p00im-dlb-asmtpmailmevip.me.com [17.57.154.19]) by ms11p00im-qufo17281501.me.com (Postfix) with ESMTPSA id 455D7B61C3D; Thu, 25 Jul 2024 19:47:17 +0000 (UTC) Message-ID: <35da66f9-b913-45ea-90f4-16a2fa072848@technologyfriends.net> Date: Thu, 25 Jul 2024 14:47:16 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: FreeBSD hugepages To: Konstantin Belousov Cc: freebsd-hackers@freebsd.org References: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> Content-Language: en-US From: Jake Freeland In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Proofpoint-ORIG-GUID: LPnpgvxYj-5EycG1_HjjpF9PjsmzQVY- X-Proofpoint-GUID: LPnpgvxYj-5EycG1_HjjpF9PjsmzQVY- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-25_19,2024-07-25_03,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1030 mlxscore=0 spamscore=0 malwarescore=0 mlxlogscore=999 bulkscore=0 adultscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2308100000 definitions=main-2407250135 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:714, ipnet:17.58.32.0/20, country:US] X-Rspamd-Queue-Id: 4WVLy3036Hz43jV On 7/25/24 14:02, Konstantin Belousov wrote: > On Thu, Jul 25, 2024 at 01:46:17PM -0500, Jake Freeland wrote: >> Hi there, >> >> I have been steadily working on bringing Data Plane Development Kit (DPDK) >> on FreeBSD up to date with the Linux version. The most significant hurdle so >> far has been supporting concurrent DPDK processes, each with their own >> contiguous memory regions. >> >> These contiguous regions are used by DPDK as a heap for allocating DMA >> buffers and other miscellaneous resources. Retrieving the underlying memory >> and mapping these regions is currently different on Linux and FreeBSD: >> >> On Linux, hugepages are fetched from the kernel's pre-allocated hugepage >> pool and are mapped into virtual address space on DPDK initialization. Since >> the hugepages exist in a pool, multiple processes can reserve their own >> hugepages and operate concurrently. >> >> On FreeBSD, DPDK uses an in-house contigmem kernel module that reserves a >> large contiguous region of memory on load. During DPDK initialization, the >> entire region is mapped into virtual address space. This leaves no memory >> for another independent DPDK process, so only one process can operate at a >> time. >> >> I could modify the DPDK contigmem module to mimic Linux's hugepages, but I >> thought it would be better to integrate and upstream a hugepage-like >> interface directly in the FreeBSD kernel source. I am writing this email to >> see if anyone has any advice on the matter. I did not see any previous >> attempts at this in Phabriactor or the commit log, but it is possible that I >> missed it. I have read about transparent superpage promotion, but that seems >> like a different mechanism altogether. >> >> At a quick glance, the implementation seems straightforward: read some >> loader tunables, allocate persistent hugepages at boot time, and create a >> pseudo filesystem that supports creating and mapping hugepages. I could be >> underestimating the magnitude of this task, but that is why I'm asking for >> thoughts and advice :) >> >> For reference, here is Linux's documentation on hugepages: >> https://docs.kernel.org/admin-guide/mm/hugetlbpage.html > Are posix shm largepages objects enough (they were developed to support > DPDK). Look for shm_create_largepage(3). Yes, shm_create_largepage(2) looks promising, but I would like the ability to allocate these largepages at boot time when memory fragmentation as at a minimum. Perhaps a couple sysctl tunables could be added onto the vm.largepages node to specify a pagesize and allocate some number of pages at boot? It seems Linux had an interface similar to shm_create_largepage(2) back in v2.5, but they removed it in favor of their hugetlbfs filesystem. It would be nice to stay close to the file-backed Linux interface to maximize code sharing in userspace. It looks like the foundation for hugepages is there, but the interface for allocation and access needs to be extended. Jake Freeland From nobody Thu Jul 25 20:18:16 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WVMdn6CNJz5RDhp for ; Thu, 25 Jul 2024 20:18:21 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WVMdn32vrz46dN for ; Thu, 25 Jul 2024 20:18:21 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x831.google.com with SMTP id d75a77b69052e-44fee8813c3so4303491cf.2 for ; Thu, 25 Jul 2024 13:18:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721938700; x=1722543500; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=F2yMAo42eKCdj+x5rUSQZ/59ssvkBYL/zC6TyIZDFxs=; b=YAwoN4kaNsd9owPkw8hqFr9siOCA/xMqvHFqotfWpxVhiGpgGgM6yUCLmqMR+Z/lBN itHRPQtLTenBOJDSNKs0q1v5T5ySYw/uGxSGB8EK3LFRM9Oq6EOfNsCE+ut/lJOKs/7B G6W0yQP0p55CEOtfOnaj0FHXdVn77E1CyarsPP58WnkBZ+3x8vbgWz0yGA+vkFMSinHh FuP4YjIs+L85Zn03yTsUeqwQxce3eZAKYHJI/NXYabyPQxYDEvwWvKc3OAevEAX1nnko EHkf1ljVMERTwbnZzF/Ya0zavEUtL7ddBHUxdS3rRQF7uNeXaycoPJpWegoO2bLv5p/7 o9XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721938700; x=1722543500; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F2yMAo42eKCdj+x5rUSQZ/59ssvkBYL/zC6TyIZDFxs=; b=Pb2/cbAMxjq7OIYJrsjzWU+l29+n74OJSUJqft/Egz+Bml4MicuhLQvvL1ABM3uAuA MYosUOof/J16YbU9QXOCHecjE/Ica0ZFqDWixKsrKJMLVRpu2Yk02/c1wR3aK3AdUaCR nxCAIIsIDxoedISWVR9ddW+Y0+qYMof0CVmSdPuxxHDtc7jOQuVpEoJtsrvPxOzHhm/v YwdaRRoAZjHkZfd3JKeRvlbZiOQPbi57IQIgo9hi0s8Sqp9oHeVv3FDEdDHEIA8QVMlp SlJlLSjhw+N7NfTA3hmfrGF0laiH2+KnXW30BDL4CB6J9nCjMB0ucZRsK6CffxGxo7ac Yz+Q== X-Forwarded-Encrypted: i=1; AJvYcCUYcUN79BFz/9uE86WwhOBAIGWlFQzX5F+nKz/yWeVeStBBActoKS+L8pjuSPWN7NqR5FAlD0xEOWnZTQcpulm/wLr3pZccZuddw/c= X-Gm-Message-State: AOJu0YyE9tj3Gg5KljKzPlYmHisfYez6TMVlpnbXWtZBFzE0qq53Cud8 zJPfTjvFnzTyp31vStbm9cotXQDuX+lCYN8DIzlqX++JplboeSA7Efbd/QHJ X-Google-Smtp-Source: AGHT+IFocUYph05vYNrqpwTPxe7RiNxm8l1cZxS0EYZRYNJyLCPUOeeEo3H+cBYO0V6vru4szhQGCw== X-Received: by 2002:ac8:59c2:0:b0:446:4c0f:ef03 with SMTP id d75a77b69052e-44fe3283c46mr53121111cf.10.1721938699832; Thu, 25 Jul 2024 13:18:19 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-44fe840c096sm9056561cf.79.2024.07.25.13.18.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jul 2024 13:18:19 -0700 (PDT) Date: Thu, 25 Jul 2024 16:18:16 -0400 From: Mark Johnston To: Jake Freeland Cc: Konstantin Belousov , freebsd-hackers@freebsd.org Subject: Re: FreeBSD hugepages Message-ID: References: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> <35da66f9-b913-45ea-90f4-16a2fa072848@technologyfriends.net> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <35da66f9-b913-45ea-90f4-16a2fa072848@technologyfriends.net> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4WVMdn32vrz46dN On Thu, Jul 25, 2024 at 02:47:16PM -0500, Jake Freeland wrote: > On 7/25/24 14:02, Konstantin Belousov wrote: > > On Thu, Jul 25, 2024 at 01:46:17PM -0500, Jake Freeland wrote: > > > Hi there, > > > > > > I have been steadily working on bringing Data Plane Development Kit (DPDK) > > > on FreeBSD up to date with the Linux version. The most significant hurdle so > > > far has been supporting concurrent DPDK processes, each with their own > > > contiguous memory regions. > > > > > > These contiguous regions are used by DPDK as a heap for allocating DMA > > > buffers and other miscellaneous resources. Retrieving the underlying memory > > > and mapping these regions is currently different on Linux and FreeBSD: > > > > > > On Linux, hugepages are fetched from the kernel's pre-allocated hugepage > > > pool and are mapped into virtual address space on DPDK initialization. Since > > > the hugepages exist in a pool, multiple processes can reserve their own > > > hugepages and operate concurrently. > > > > > > On FreeBSD, DPDK uses an in-house contigmem kernel module that reserves a > > > large contiguous region of memory on load. During DPDK initialization, the > > > entire region is mapped into virtual address space. This leaves no memory > > > for another independent DPDK process, so only one process can operate at a > > > time. > > > > > > I could modify the DPDK contigmem module to mimic Linux's hugepages, but I > > > thought it would be better to integrate and upstream a hugepage-like > > > interface directly in the FreeBSD kernel source. I am writing this email to > > > see if anyone has any advice on the matter. I did not see any previous > > > attempts at this in Phabriactor or the commit log, but it is possible that I > > > missed it. I have read about transparent superpage promotion, but that seems > > > like a different mechanism altogether. > > > > > > At a quick glance, the implementation seems straightforward: read some > > > loader tunables, allocate persistent hugepages at boot time, and create a > > > pseudo filesystem that supports creating and mapping hugepages. I could be > > > underestimating the magnitude of this task, but that is why I'm asking for > > > thoughts and advice :) > > > > > > For reference, here is Linux's documentation on hugepages: > > > https://docs.kernel.org/admin-guide/mm/hugetlbpage.html > > Are posix shm largepages objects enough (they were developed to support > > DPDK). Look for shm_create_largepage(3). > Yes, shm_create_largepage(2) looks promising, but I would like the ability > to allocate these largepages at boot time when memory fragmentation as at a > minimum. Perhaps a couple sysctl tunables could be added onto the > vm.largepages node to specify a pagesize and allocate some number of pages > at boot? We could add an rc script which creates named largepage objects. This can be done using the posixshmcontrol utility. That might not be early enough during boot for some purposes. In that case, we could have a module which creates such objects from within the kernel. This is pretty straightforward to do; I wrote a dumb version of this for a mips-specific project a few years ago, feel free to take code or inspiration from it: https://people.freebsd.org/~markj/tlbdemo.c > It seems Linux had an interface similar to shm_create_largepage(2) back in > v2.5, but they removed it in favor of their hugetlbfs filesystem. It would > be nice to stay close to the file-backed Linux interface to maximize code > sharing in userspace. It looks like the foundation for hugepages is there, > but the interface for allocation and access needs to be extended. POSIX shm objects have most of the properties one would want, I'd expect, save the ability to access them via standard syscalls. What else is missing besides the ability to reserve memory at boot time? From nobody Thu Jul 25 21:11:22 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WVNq31cRPz5RKTh for ; Thu, 25 Jul 2024 21:11:27 +0000 (UTC) (envelope-from jake@technologyfriends.net) Received: from ci74p00im-qukt09090501.me.com (ci74p00im-qukt09090501.me.com [17.57.156.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WVNq26xNfz4G0N for ; Thu, 25 Jul 2024 21:11:26 +0000 (UTC) (envelope-from jake@technologyfriends.net) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=technologyfriends.net; s=sig1; t=1721941885; bh=miwWTOTo4TOzWgG4TNgpUhVhQhtkNzkYMcQBBEeMLz4=; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; b=jcSfAQ8hop1AV80Se9K4M50/r++4spR48NTz4ewVwc6b3gNg+KvUUYtnN4pJj1iGf XUyuwhj3Y4H7f0pGK3P11HmHxzeeYoUU4qs8GCE6XUVabr1iDhwi0zyOBoY7JwR7Vf OYfszJTEAvANaLQttTVUgkzk7ZnI0ijlyf3aiWjqjQLA/28b/+epb7h+SC7CZ3sjfP hsxI2/srfKuXFKR6dawP688XFXNyBrt5iZRDQrI2f6ymFfbv8Quvfg54x0Q4UnvBo9 /qp+BscbEBKlxqKm3SOs5BQzc7/TRkU+kiHfjCV8SSpifJPlSr36LvPME9Hy3gQrv0 Rj5DcVMCdo0iQ== Received: from [10.0.233.209] (ci77p00im-dlb-asmtp-mailmevip.me.com [17.57.156.26]) by ci74p00im-qukt09090501.me.com (Postfix) with ESMTPSA id 4201D46401FD; Thu, 25 Jul 2024 21:11:24 +0000 (UTC) Message-ID: <4d4398e5-81ba-4fbd-9806-649ec70abdb4@technologyfriends.net> Date: Thu, 25 Jul 2024 16:11:22 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: FreeBSD hugepages To: Mark Johnston Cc: Konstantin Belousov , freebsd-hackers@freebsd.org References: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> <35da66f9-b913-45ea-90f4-16a2fa072848@technologyfriends.net> Content-Language: en-US From: Jake Freeland In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Proofpoint-GUID: e_CfyNCBo8I2LLM_8NiIBs3OBGYI2-85 X-Proofpoint-ORIG-GUID: e_CfyNCBo8I2LLM_8NiIBs3OBGYI2-85 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-25_21,2024-07-25_03,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0 suspectscore=0 malwarescore=0 mlxscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 clxscore=1030 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2308100000 definitions=main-2407250145 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:714, ipnet:17.57.156.0/24, country:US] X-Rspamd-Queue-Id: 4WVNq26xNfz4G0N On 7/25/24 15:18, Mark Johnston wrote: > On Thu, Jul 25, 2024 at 02:47:16PM -0500, Jake Freeland wrote: >> On 7/25/24 14:02, Konstantin Belousov wrote: >>> On Thu, Jul 25, 2024 at 01:46:17PM -0500, Jake Freeland wrote: >>>> Hi there, >>>> >>>> I have been steadily working on bringing Data Plane Development Kit (DPDK) >>>> on FreeBSD up to date with the Linux version. The most significant hurdle so >>>> far has been supporting concurrent DPDK processes, each with their own >>>> contiguous memory regions. >>>> >>>> These contiguous regions are used by DPDK as a heap for allocating DMA >>>> buffers and other miscellaneous resources. Retrieving the underlying memory >>>> and mapping these regions is currently different on Linux and FreeBSD: >>>> >>>> On Linux, hugepages are fetched from the kernel's pre-allocated hugepage >>>> pool and are mapped into virtual address space on DPDK initialization. Since >>>> the hugepages exist in a pool, multiple processes can reserve their own >>>> hugepages and operate concurrently. >>>> >>>> On FreeBSD, DPDK uses an in-house contigmem kernel module that reserves a >>>> large contiguous region of memory on load. During DPDK initialization, the >>>> entire region is mapped into virtual address space. This leaves no memory >>>> for another independent DPDK process, so only one process can operate at a >>>> time. >>>> >>>> I could modify the DPDK contigmem module to mimic Linux's hugepages, but I >>>> thought it would be better to integrate and upstream a hugepage-like >>>> interface directly in the FreeBSD kernel source. I am writing this email to >>>> see if anyone has any advice on the matter. I did not see any previous >>>> attempts at this in Phabriactor or the commit log, but it is possible that I >>>> missed it. I have read about transparent superpage promotion, but that seems >>>> like a different mechanism altogether. >>>> >>>> At a quick glance, the implementation seems straightforward: read some >>>> loader tunables, allocate persistent hugepages at boot time, and create a >>>> pseudo filesystem that supports creating and mapping hugepages. I could be >>>> underestimating the magnitude of this task, but that is why I'm asking for >>>> thoughts and advice :) >>>> >>>> For reference, here is Linux's documentation on hugepages: >>>> https://docs.kernel.org/admin-guide/mm/hugetlbpage.html >>> Are posix shm largepages objects enough (they were developed to support >>> DPDK). Look for shm_create_largepage(3). >> Yes, shm_create_largepage(2) looks promising, but I would like the ability >> to allocate these largepages at boot time when memory fragmentation as at a >> minimum. Perhaps a couple sysctl tunables could be added onto the >> vm.largepages node to specify a pagesize and allocate some number of pages >> at boot? > We could add an rc script which creates named largepage objects. This > can be done using the posixshmcontrol utility. That might not be early > enough during boot for some purposes. In that case, we could have a > module which creates such objects from within the kernel. This is > pretty straightforward to do; I wrote a dumb version of this for a > mips-specific project a few years ago, feel free to take code or > inspiration from it: https://people.freebsd.org/~markj/tlbdemo.c Looks simple enough. Thanks for the example code. >> It seems Linux had an interface similar to shm_create_largepage(2) back in >> v2.5, but they removed it in favor of their hugetlbfs filesystem. It would >> be nice to stay close to the file-backed Linux interface to maximize code >> sharing in userspace. It looks like the foundation for hugepages is there, >> but the interface for allocation and access needs to be extended. > POSIX shm objects have most of the properties one would want, I'd > expect, save the ability to access them via standard syscalls. What > else is missing besides the ability to reserve memory at boot time? Most notably, I would like the ability to allocate pages in a specific NUMA domain. Otherwise, in a perfect world, I'd like a unified interface for both Linux and FreeBSD. Linux hugepages are managed using standard system calls; files are mmap(2)'d into virtual address space from hugetlbfs and ftruncate(2)'d. A matching interface would not add an extra kernel entrypoint and even more importantly, it would ease the Linux-to-FreeBSD porting process for programs that use hugepages. Thanks, Jake Freeland From nobody Thu Jul 25 22:40:31 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WVQnw4Dfhz5RSZ2 for ; Thu, 25 Jul 2024 22:40:36 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-oo1-xc2a.google.com (mail-oo1-xc2a.google.com [IPv6:2607:f8b0:4864:20::c2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WVQnv4HnPz4QcB for ; Thu, 25 Jul 2024 22:40:35 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=I77+nYeJ; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::c2a as permitted sender) smtp.mailfrom=markjdb@gmail.com Received: by mail-oo1-xc2a.google.com with SMTP id 006d021491bc7-5ce74defe42so285256eaf.0 for ; Thu, 25 Jul 2024 15:40:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721947234; x=1722552034; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=1fP/NXahIOTHypv7uwPmL2sR9bnLW2uKYfkRalYuv2w=; b=I77+nYeJowNsT+YCHfGtWeGYBcGESsHJ1xkrvZKRyYFzSAZjf4su0FOv9a+g2y8Jd6 HXmw1Oy4dTGnIkMMJ9Jwq6BTUp6ghLNFHWicJqmauhTM51pNoXSELAQpdruRjw+fhPxM FRr/THpmI2xmnzLA0rw3p8SuTqQUKFNNBbjC5IeBOX4bzisPpXM9WRMVDSDUFbyXPvFj 0z1LEj0eu6O+I/3VqSEZ+XkikxsN/F/lYOsmP7xYrrMZD1hEXx4Lwt4D/EWUYf4D55mN MluhlyNABecmLTe/cbV4Q2LIX3hxBai9t8LZ3ZLdF2wc35I9pM7eynAkdHRmwFa56DNn x7aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721947234; x=1722552034; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1fP/NXahIOTHypv7uwPmL2sR9bnLW2uKYfkRalYuv2w=; b=K2w3TWjlgJBmNurtxx342MKrv1oKWP7K4XoS4atVg4xHn41YK4a+zDsubOPQDpVV8C eWHY78A7Zwpip+re/rvk3au2UdzGZFCde7y1/4NUeKpMtnTEd7YVAxO3kDGrljtZE1LR gGLBSyLk8FDMDnHUaXN2p64zLCoprGokdSJ9U7H8Ar/2mkqODA2tGIsIsTFo+LbV8gzs rfQEnO7ONZTYMo5HxcSymEdF0OtSA+VwAcVcJKkVFsm5CNJqcuDtwiNpGERHfe4EL71I JA7NmW8a42hDmXDP0bHE3MrMa3x77csdWIyL3U5vUxzvv89BdlosBe3SUoJvjZAduvYW DdhA== X-Forwarded-Encrypted: i=1; AJvYcCXO8AS3RFFqfh+RkmgwbHC+yYTxt6lfOXDRGurZdzA5mMJCtjQJTXGoV5OaoiBMv6g0T2iU8aFnDWSWAv1tr+D8SpjlSuQs6edQtTA= X-Gm-Message-State: AOJu0Yz4MDsQ3lKE0IngJYRlnGMABkngokc0YB1eRijk/74nUMSK04oC 2z3+bFf1+zbP4OootzKCmlEwHYVLgE2u8QFf2QH5JrSsfUxLGD2N X-Google-Smtp-Source: AGHT+IGMTcU9SmtioO8WdljWo00ZFxMdeIqxgu/W9uVjQXb1XB/Khc855MOgRIF9UB4xvlt3iPYiMw== X-Received: by 2002:a05:6358:52c9:b0:1aa:a19e:f195 with SMTP id e5c5f4694b2df-1acfb894dfemr442080555d.4.1721947234194; Thu, 25 Jul 2024 15:40:34 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb3f8d8269sm11111246d6.20.2024.07.25.15.40.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jul 2024 15:40:33 -0700 (PDT) Date: Thu, 25 Jul 2024 18:40:31 -0400 From: Mark Johnston To: Jake Freeland Cc: Konstantin Belousov , freebsd-hackers@freebsd.org Subject: Re: FreeBSD hugepages Message-ID: References: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> <35da66f9-b913-45ea-90f4-16a2fa072848@technologyfriends.net> <4d4398e5-81ba-4fbd-9806-649ec70abdb4@technologyfriends.net> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.59 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::c2a:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCPT_COUNT_THREE(0.00)[3]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4WVQnv4HnPz4QcB On Thu, Jul 25, 2024 at 06:34:43PM -0400, Mark Johnston wrote: > On Thu, Jul 25, 2024 at 04:11:22PM -0500, Jake Freeland wrote: > > On 7/25/24 15:18, Mark Johnston wrote: > > > On Thu, Jul 25, 2024 at 02:47:16PM -0500, Jake Freeland wrote: > > > > On 7/25/24 14:02, Konstantin Belousov wrote: > > > > > On Thu, Jul 25, 2024 at 01:46:17PM -0500, Jake Freeland wrote: > > > > > > Hi there, > > > > > > > > > > > > I have been steadily working on bringing Data Plane Development Kit (DPDK) > > > > > > on FreeBSD up to date with the Linux version. The most significant hurdle so > > > > > > far has been supporting concurrent DPDK processes, each with their own > > > > > > contiguous memory regions. > > > > > > > > > > > > These contiguous regions are used by DPDK as a heap for allocating DMA > > > > > > buffers and other miscellaneous resources. Retrieving the underlying memory > > > > > > and mapping these regions is currently different on Linux and FreeBSD: > > > > > > > > > > > > On Linux, hugepages are fetched from the kernel's pre-allocated hugepage > > > > > > pool and are mapped into virtual address space on DPDK initialization. Since > > > > > > the hugepages exist in a pool, multiple processes can reserve their own > > > > > > hugepages and operate concurrently. > > > > > > > > > > > > On FreeBSD, DPDK uses an in-house contigmem kernel module that reserves a > > > > > > large contiguous region of memory on load. During DPDK initialization, the > > > > > > entire region is mapped into virtual address space. This leaves no memory > > > > > > for another independent DPDK process, so only one process can operate at a > > > > > > time. > > > > > > > > > > > > I could modify the DPDK contigmem module to mimic Linux's hugepages, but I > > > > > > thought it would be better to integrate and upstream a hugepage-like > > > > > > interface directly in the FreeBSD kernel source. I am writing this email to > > > > > > see if anyone has any advice on the matter. I did not see any previous > > > > > > attempts at this in Phabriactor or the commit log, but it is possible that I > > > > > > missed it. I have read about transparent superpage promotion, but that seems > > > > > > like a different mechanism altogether. > > > > > > > > > > > > At a quick glance, the implementation seems straightforward: read some > > > > > > loader tunables, allocate persistent hugepages at boot time, and create a > > > > > > pseudo filesystem that supports creating and mapping hugepages. I could be > > > > > > underestimating the magnitude of this task, but that is why I'm asking for > > > > > > thoughts and advice :) > > > > > > > > > > > > For reference, here is Linux's documentation on hugepages: > > > > > > https://docs.kernel.org/admin-guide/mm/hugetlbpage.html > > > > > Are posix shm largepages objects enough (they were developed to support > > > > > DPDK). Look for shm_create_largepage(3). > > > > Yes, shm_create_largepage(2) looks promising, but I would like the ability > > > > to allocate these largepages at boot time when memory fragmentation as at a > > > > minimum. Perhaps a couple sysctl tunables could be added onto the > > > > vm.largepages node to specify a pagesize and allocate some number of pages > > > > at boot? > > > We could add an rc script which creates named largepage objects. This > > > can be done using the posixshmcontrol utility. That might not be early > > > enough during boot for some purposes. In that case, we could have a > > > module which creates such objects from within the kernel. This is > > > pretty straightforward to do; I wrote a dumb version of this for a > > > mips-specific project a few years ago, feel free to take code or > > > inspiration from it: https://people.freebsd.org/~markj/tlbdemo.c > > > > Looks simple enough. Thanks for the example code. > > > > > > It seems Linux had an interface similar to shm_create_largepage(2) back in > > > > v2.5, but they removed it in favor of their hugetlbfs filesystem. It would > > > > be nice to stay close to the file-backed Linux interface to maximize code > > > > sharing in userspace. It looks like the foundation for hugepages is there, > > > > but the interface for allocation and access needs to be extended. > > > POSIX shm objects have most of the properties one would want, I'd > > > expect, save the ability to access them via standard syscalls. What > > > else is missing besides the ability to reserve memory at boot time? > > > > Most notably, I would like the ability to allocate pages in a specific NUMA > > domain. > > I thought this was already supported, but it seems not... Thinking a bit more, I'm pretty sure I had just been using something like $ cpuset -n prefer: posixshmcontrol create -l 1G /largepage-1G- so didn't need an explicit NUMA configuration parameter. In C one would use cpuset_setdomain(2) instead, but that's not as convenient. So, imbuing a NUMA domain in struct shm_largepage_conf is still probably a reasonable thing to do. > It should be very easy to implement: extend shm_largepage_conf to > include a NUMA domain parameter, and specify that domain when allocating > pages for the object (in shm_largepage_dotruncate(), the > vm_page_alloc_contig() call should become a > vm_page_alloc_contig_domain() call). > > > Otherwise, in a perfect world, I'd like a unified interface for both > > Linux and FreeBSD. Linux hugepages are managed using standard system calls; > > files are mmap(2)'d into virtual address space from hugetlbfs and > > ftruncate(2)'d. > > largepage shm objects work this way as well. > > > A matching interface would not add an extra kernel > > entrypoint and even more importantly, it would ease the Linux-to-FreeBSD > > porting process for programs that use hugepages. From nobody Thu Jul 25 22:34:43 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WVQrD4lXwz5RSZW for ; Thu, 25 Jul 2024 22:42:36 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-oa1-x31.google.com (mail-oa1-x31.google.com [IPv6:2001:4860:4864:20::31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WVQrC65W0z4RWd for ; Thu, 25 Jul 2024 22:42:35 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x31.google.com with SMTP id 586e51a60fabf-260f033fda3so443578fac.3 for ; Thu, 25 Jul 2024 15:42:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721947353; x=1722552153; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=3y3GDzvlNSyTxAYBOKFfG6Zj9eS8Znc44Hqp7c6rdkg=; b=HBDdsm2qCZxFjWwhigWwZfHyT0WtwH8uOmcgTEZPhrSLfVQH6MdDy9Tkkcfm9diI+A v/utiGT9+HQCcuJx5J6/6v96Gb3/uw+v4LgUx0hV3sDUiufEZRvoqgPm8QKaPFoVczm0 84l3PCYwPnA5GGF8PyVqWaBLHoPA+JH/308IUX1sYoN9RexJc7Q2ewUaFairplLgUQWk 0G5oFJ0Td4Hg3iGleREJLbsyRb62lOS4ny/RovwmCNK7a/0jLI8f0zq6CU1+9rJMI8iT VjdkUA64JVIk1RVBuBrp8IdAoYuSNNNg6afNe22uOBKF4L5nYhc1xlScSCZNZsrpG5iD MQ+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721947353; x=1722552153; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3y3GDzvlNSyTxAYBOKFfG6Zj9eS8Znc44Hqp7c6rdkg=; b=CKQQe/v5iatPIO/cXtmk0zF2aT8d7I/si+xwFPdHoaBtpBVPjwGAWmVYp1CAtJvZ0U DwaA9mhL5bRrWow+/fZFoaSMPv1qIAnL6N0DzlM9ERogpeN2Qj4oKunrkBg/ypV2yWHd KHfbpJqmYbKNRAxbjzD/CV2TDYvAunzIRj3vgGJZYn+W2BgPDjYmKnriLrfjkDcX4S+3 KL2XzCfMttle3hAbkURNxnH/x5w2v3tJaJv58PFcs06n0hTZyr1oi/QQDrXV2K8CwQQn 9yhbBSmV6LR+lDEtk+xDcmEHHSG+5ISpjTGlHqEIFVEKDjNYEU2GfL1Q9kmTBc5nGb5Q 1D1g== X-Forwarded-Encrypted: i=1; AJvYcCXkGkkrABQ8xjRMEsDBPwAvW/GduNng1AlaA4h4L8YrnAcJp8oLGpHUv/ui+js7/uOiwg9Ac3ZzuNy1pH+RcQvftrT6SPZLeSCDKME= X-Gm-Message-State: AOJu0YwOuAXPNaGR4wq8mZH4esMJyQwVMBBKS5F4g8Z3CixOoHQNlAM8 zzPdvmm/M4opsApp5tYsN6/ndTvuDxTFg7CLtIDa5yn9Rbfvm3VLTD59TVsa X-Google-Smtp-Source: AGHT+IGM7kAWPDeN9ZuYPHuyNw0WSlvul5/UuwKV2G1rxpJ+LtTSP+Ox7Qo5ufVOEa5pSJ/NLr7jHA== X-Received: by 2002:a05:6808:3093:b0:3d9:2bab:de1f with SMTP id 5614622812f47-3db10f566famr5821207b6e.25.1721946886735; Thu, 25 Jul 2024 15:34:46 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-44fe81463easm9629191cf.26.2024.07.25.15.34.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jul 2024 15:34:45 -0700 (PDT) Date: Thu, 25 Jul 2024 18:34:43 -0400 From: Mark Johnston To: Jake Freeland Cc: Konstantin Belousov , freebsd-hackers@freebsd.org Subject: Re: FreeBSD hugepages Message-ID: References: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> <35da66f9-b913-45ea-90f4-16a2fa072848@technologyfriends.net> <4d4398e5-81ba-4fbd-9806-649ec70abdb4@technologyfriends.net> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4d4398e5-81ba-4fbd-9806-649ec70abdb4@technologyfriends.net> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US] X-Rspamd-Queue-Id: 4WVQrC65W0z4RWd On Thu, Jul 25, 2024 at 04:11:22PM -0500, Jake Freeland wrote: > On 7/25/24 15:18, Mark Johnston wrote: > > On Thu, Jul 25, 2024 at 02:47:16PM -0500, Jake Freeland wrote: > > > On 7/25/24 14:02, Konstantin Belousov wrote: > > > > On Thu, Jul 25, 2024 at 01:46:17PM -0500, Jake Freeland wrote: > > > > > Hi there, > > > > > > > > > > I have been steadily working on bringing Data Plane Development Kit (DPDK) > > > > > on FreeBSD up to date with the Linux version. The most significant hurdle so > > > > > far has been supporting concurrent DPDK processes, each with their own > > > > > contiguous memory regions. > > > > > > > > > > These contiguous regions are used by DPDK as a heap for allocating DMA > > > > > buffers and other miscellaneous resources. Retrieving the underlying memory > > > > > and mapping these regions is currently different on Linux and FreeBSD: > > > > > > > > > > On Linux, hugepages are fetched from the kernel's pre-allocated hugepage > > > > > pool and are mapped into virtual address space on DPDK initialization. Since > > > > > the hugepages exist in a pool, multiple processes can reserve their own > > > > > hugepages and operate concurrently. > > > > > > > > > > On FreeBSD, DPDK uses an in-house contigmem kernel module that reserves a > > > > > large contiguous region of memory on load. During DPDK initialization, the > > > > > entire region is mapped into virtual address space. This leaves no memory > > > > > for another independent DPDK process, so only one process can operate at a > > > > > time. > > > > > > > > > > I could modify the DPDK contigmem module to mimic Linux's hugepages, but I > > > > > thought it would be better to integrate and upstream a hugepage-like > > > > > interface directly in the FreeBSD kernel source. I am writing this email to > > > > > see if anyone has any advice on the matter. I did not see any previous > > > > > attempts at this in Phabriactor or the commit log, but it is possible that I > > > > > missed it. I have read about transparent superpage promotion, but that seems > > > > > like a different mechanism altogether. > > > > > > > > > > At a quick glance, the implementation seems straightforward: read some > > > > > loader tunables, allocate persistent hugepages at boot time, and create a > > > > > pseudo filesystem that supports creating and mapping hugepages. I could be > > > > > underestimating the magnitude of this task, but that is why I'm asking for > > > > > thoughts and advice :) > > > > > > > > > > For reference, here is Linux's documentation on hugepages: > > > > > https://docs.kernel.org/admin-guide/mm/hugetlbpage.html > > > > Are posix shm largepages objects enough (they were developed to support > > > > DPDK). Look for shm_create_largepage(3). > > > Yes, shm_create_largepage(2) looks promising, but I would like the ability > > > to allocate these largepages at boot time when memory fragmentation as at a > > > minimum. Perhaps a couple sysctl tunables could be added onto the > > > vm.largepages node to specify a pagesize and allocate some number of pages > > > at boot? > > We could add an rc script which creates named largepage objects. This > > can be done using the posixshmcontrol utility. That might not be early > > enough during boot for some purposes. In that case, we could have a > > module which creates such objects from within the kernel. This is > > pretty straightforward to do; I wrote a dumb version of this for a > > mips-specific project a few years ago, feel free to take code or > > inspiration from it: https://people.freebsd.org/~markj/tlbdemo.c > > Looks simple enough. Thanks for the example code. > > > > It seems Linux had an interface similar to shm_create_largepage(2) back in > > > v2.5, but they removed it in favor of their hugetlbfs filesystem. It would > > > be nice to stay close to the file-backed Linux interface to maximize code > > > sharing in userspace. It looks like the foundation for hugepages is there, > > > but the interface for allocation and access needs to be extended. > > POSIX shm objects have most of the properties one would want, I'd > > expect, save the ability to access them via standard syscalls. What > > else is missing besides the ability to reserve memory at boot time? > > Most notably, I would like the ability to allocate pages in a specific NUMA > domain. I thought this was already supported, but it seems not... It should be very easy to implement: extend shm_largepage_conf to include a NUMA domain parameter, and specify that domain when allocating pages for the object (in shm_largepage_dotruncate(), the vm_page_alloc_contig() call should become a vm_page_alloc_contig_domain() call). > Otherwise, in a perfect world, I'd like a unified interface for both > Linux and FreeBSD. Linux hugepages are managed using standard system calls; > files are mmap(2)'d into virtual address space from hugetlbfs and > ftruncate(2)'d. largepage shm objects work this way as well. > A matching interface would not add an extra kernel > entrypoint and even more importantly, it would ease the Linux-to-FreeBSD > porting process for programs that use hugepages. From nobody Thu Jul 25 23:11:00 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WVRT65pnpz5RW3r for ; Thu, 25 Jul 2024 23:11:06 +0000 (UTC) (envelope-from jake@technologyfriends.net) Received: from st43p00im-ztdg10073201.me.com (st43p00im-ztdg10073201.me.com [17.58.63.177]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WVRT649Gxz4X4s for ; Thu, 25 Jul 2024 23:11:06 +0000 (UTC) (envelope-from jake@technologyfriends.net) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=technologyfriends.net; s=sig1; t=1721949065; bh=T3MZGmC91k5hpL4U50c8dRSVYIpdzQiDrawMq5J1R6U=; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; b=MTcGnCANnIZtcP0AHTy/4H56RoKxkkiDH4A0Q3n+YdqZqELnqjfJvnLCPWHRl4Beg hCSzz9wL2rnvShqrKiefd58bp/yxtLRMcIk3PzzUeuBSX0vM0/FbkE/JCp7dYIGxU+ tMNRcCmZZgp9e4AV5ykv42SudzOhWfxJ3xIsIpyfd7i0/nGUzII4wY3bavwLtIVjyi k7WhxqrvgfQXxuYmHDUthEzvxtJZj0JnvjIR4EyY9fwo0vhKVucfKjNScRAksTLrKW Iy2LBj5iYrRD0MBXLG0+eQtJVaBajkzzznuyORIdJ1plqt4B8eypIObJO9oT0G0lDW jk2cHzCKlc6WQ== Received: from [10.0.233.209] (st43p00im-dlb-asmtp-mailmevip.me.com [17.42.251.41]) by st43p00im-ztdg10073201.me.com (Postfix) with ESMTPSA id 42E3D9C047F; Thu, 25 Jul 2024 23:11:02 +0000 (UTC) Message-ID: <912f849b-95ac-4d29-8a86-300999a0a9c4@technologyfriends.net> Date: Thu, 25 Jul 2024 18:11:00 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: FreeBSD hugepages To: Mark Johnston Cc: Konstantin Belousov , freebsd-hackers@freebsd.org References: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> <35da66f9-b913-45ea-90f4-16a2fa072848@technologyfriends.net> <4d4398e5-81ba-4fbd-9806-649ec70abdb4@technologyfriends.net> Content-Language: en-US From: Jake Freeland In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Proofpoint-ORIG-GUID: jmEUEnRQZvL04qP9iu74v2lE2-bSXS5B X-Proofpoint-GUID: jmEUEnRQZvL04qP9iu74v2lE2-bSXS5B X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-25_12,2024-07-25_03,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 bulkscore=0 phishscore=0 suspectscore=0 mlxscore=0 clxscore=1030 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2308100000 definitions=main-2407250090 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:714, ipnet:17.58.63.0/24, country:US] X-Rspamd-Queue-Id: 4WVRT649Gxz4X4s On 7/25/24 17:40, Mark Johnston wrote: > On Thu, Jul 25, 2024 at 06:34:43PM -0400, Mark Johnston wrote: >> On Thu, Jul 25, 2024 at 04:11:22PM -0500, Jake Freeland wrote: >>> On 7/25/24 15:18, Mark Johnston wrote: >>>> On Thu, Jul 25, 2024 at 02:47:16PM -0500, Jake Freeland wrote: >>>>> On 7/25/24 14:02, Konstantin Belousov wrote: >>>>>> On Thu, Jul 25, 2024 at 01:46:17PM -0500, Jake Freeland wrote: >>>>>>> Hi there, >>>>>>> >>>>>>> I have been steadily working on bringing Data Plane Development Kit (DPDK) >>>>>>> on FreeBSD up to date with the Linux version. The most significant hurdle so >>>>>>> far has been supporting concurrent DPDK processes, each with their own >>>>>>> contiguous memory regions. >>>>>>> >>>>>>> These contiguous regions are used by DPDK as a heap for allocating DMA >>>>>>> buffers and other miscellaneous resources. Retrieving the underlying memory >>>>>>> and mapping these regions is currently different on Linux and FreeBSD: >>>>>>> >>>>>>> On Linux, hugepages are fetched from the kernel's pre-allocated hugepage >>>>>>> pool and are mapped into virtual address space on DPDK initialization. Since >>>>>>> the hugepages exist in a pool, multiple processes can reserve their own >>>>>>> hugepages and operate concurrently. >>>>>>> >>>>>>> On FreeBSD, DPDK uses an in-house contigmem kernel module that reserves a >>>>>>> large contiguous region of memory on load. During DPDK initialization, the >>>>>>> entire region is mapped into virtual address space. This leaves no memory >>>>>>> for another independent DPDK process, so only one process can operate at a >>>>>>> time. >>>>>>> >>>>>>> I could modify the DPDK contigmem module to mimic Linux's hugepages, but I >>>>>>> thought it would be better to integrate and upstream a hugepage-like >>>>>>> interface directly in the FreeBSD kernel source. I am writing this email to >>>>>>> see if anyone has any advice on the matter. I did not see any previous >>>>>>> attempts at this in Phabriactor or the commit log, but it is possible that I >>>>>>> missed it. I have read about transparent superpage promotion, but that seems >>>>>>> like a different mechanism altogether. >>>>>>> >>>>>>> At a quick glance, the implementation seems straightforward: read some >>>>>>> loader tunables, allocate persistent hugepages at boot time, and create a >>>>>>> pseudo filesystem that supports creating and mapping hugepages. I could be >>>>>>> underestimating the magnitude of this task, but that is why I'm asking for >>>>>>> thoughts and advice :) >>>>>>> >>>>>>> For reference, here is Linux's documentation on hugepages: >>>>>>> https://docs.kernel.org/admin-guide/mm/hugetlbpage.html >>>>>> Are posix shm largepages objects enough (they were developed to support >>>>>> DPDK). Look for shm_create_largepage(3). >>>>> Yes, shm_create_largepage(2) looks promising, but I would like the ability >>>>> to allocate these largepages at boot time when memory fragmentation as at a >>>>> minimum. Perhaps a couple sysctl tunables could be added onto the >>>>> vm.largepages node to specify a pagesize and allocate some number of pages >>>>> at boot? >>>> We could add an rc script which creates named largepage objects. This >>>> can be done using the posixshmcontrol utility. That might not be early >>>> enough during boot for some purposes. In that case, we could have a >>>> module which creates such objects from within the kernel. This is >>>> pretty straightforward to do; I wrote a dumb version of this for a >>>> mips-specific project a few years ago, feel free to take code or >>>> inspiration from it: https://people.freebsd.org/~markj/tlbdemo.c >>> Looks simple enough. Thanks for the example code. >>> >>>>> It seems Linux had an interface similar to shm_create_largepage(2) back in >>>>> v2.5, but they removed it in favor of their hugetlbfs filesystem. It would >>>>> be nice to stay close to the file-backed Linux interface to maximize code >>>>> sharing in userspace. It looks like the foundation for hugepages is there, >>>>> but the interface for allocation and access needs to be extended. >>>> POSIX shm objects have most of the properties one would want, I'd >>>> expect, save the ability to access them via standard syscalls. What >>>> else is missing besides the ability to reserve memory at boot time? >>> Most notably, I would like the ability to allocate pages in a specific NUMA >>> domain. >> I thought this was already supported, but it seems not... > Thinking a bit more, I'm pretty sure I had just been using something > like > > $ cpuset -n prefer: posixshmcontrol create -l 1G /largepage-1G- > > so didn't need an explicit NUMA configuration parameter. In C one would > use cpuset_setdomain(2) instead, but that's not as convenient. So, > imbuing a NUMA domain in struct shm_largepage_conf is still probably a > reasonable thing to do. I just looked at the code, this seems very manageable. I'll draft up a review. >> It should be very easy to implement: extend shm_largepage_conf to >> include a NUMA domain parameter, and specify that domain when allocating >> pages for the object (in shm_largepage_dotruncate(), the >> vm_page_alloc_contig() call should become a >> vm_page_alloc_contig_domain() call). >> >>> Otherwise, in a perfect world, I'd like a unified interface for both >>> Linux and FreeBSD. Linux hugepages are managed using standard system calls; >>> files are mmap(2)'d into virtual address space from hugetlbfs and >>> ftruncate(2)'d. >> largepage shm objects work this way as well. After reading through the man page, this is quite apparent. Not sure how I failed make that connection. Anyway, this is starting to look easier than I thought it would be. The only difference from a userspace perspective that I can think of right now is how the pages are created (e.g. hugetlbfs open(2) on Linux vs. shm_create_largepage(2) on FreeBSD). Thanks for the guidance Mark and Konstantin. Jake Freeland >>> A matching interface would not add an extra kernel >>> entrypoint and even more importantly, it would ease the Linux-to-FreeBSD >>> porting process for programs that use hugepages. From nobody Thu Jul 25 23:20:42 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WVRhH3KTNz5RWpQ for ; Thu, 25 Jul 2024 23:20:47 +0000 (UTC) (envelope-from jake@technologyfriends.net) Received: from ci74p00im-qukt09082502.me.com (ci74p00im-qukt09082502.me.com [17.57.156.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WVRhG3kKlz4Ybx for ; Thu, 25 Jul 2024 23:20:46 +0000 (UTC) (envelope-from jake@technologyfriends.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=technologyfriends.net header.s=sig1 header.b=fp8WFvDe; dmarc=none; spf=pass (mx1.freebsd.org: domain of jake@technologyfriends.net designates 17.57.156.15 as permitted sender) smtp.mailfrom=jake@technologyfriends.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=technologyfriends.net; s=sig1; t=1721949645; bh=DQzdc9nlSQ884hFM/xd4qNbKhM4/Hg4UCxfzxu6qSxo=; h=Message-ID:Date:MIME-Version:Subject:From:To:Content-Type; b=fp8WFvDe+2MWmlKlceGF/naLJRiTAFvnOZBRD/fBYKMwYFo/MFMU1jNAOduz2AqAv zx+IqlKibIwY1KlnuL5GIoHaUKBFyaUl4PRUdA1CHc8MrJZCU5TvfKvEmK+kfQq/J+ qaiXDUJ95D+wlqKI7HmfYzV16f9zCE5orgYEWVIMfN9zf+KzubYQUKfPy870EhsB+J Qi+kJUj2j4W68mFPa4KH4m+UgeDU94B1siG5drbYcnHBTfhF4XBMr4F7YCbRaiXosu zEOcKQd+p6dcdL+ZeI19lDO+RRRrS+6sEQyK8Y1tE1npKev7A09tQK2fhFTdw24koC Ml7K8L8asxNAg== Received: from [10.0.233.209] (ci77p00im-dlb-asmtp-mailmevip.me.com [17.57.156.26]) by ci74p00im-qukt09082502.me.com (Postfix) with ESMTPSA id 4F4E811C00BC; Thu, 25 Jul 2024 23:20:44 +0000 (UTC) Message-ID: Date: Thu, 25 Jul 2024 18:20:42 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: FreeBSD hugepages From: Jake Freeland To: Mark Johnston Cc: Konstantin Belousov , freebsd-hackers@freebsd.org References: <1ced4290-4a31-4218-8611-63a44c307e87@technologyfriends.net> <35da66f9-b913-45ea-90f4-16a2fa072848@technologyfriends.net> <4d4398e5-81ba-4fbd-9806-649ec70abdb4@technologyfriends.net> <912f849b-95ac-4d29-8a86-300999a0a9c4@technologyfriends.net> Content-Language: en-US In-Reply-To: <912f849b-95ac-4d29-8a86-300999a0a9c4@technologyfriends.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Proofpoint-GUID: oooY2YmLlt9racBzSjLmSvjRwrmgRz5J X-Proofpoint-ORIG-GUID: oooY2YmLlt9racBzSjLmSvjRwrmgRz5J X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-25_25,2024-07-25_03,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 malwarescore=0 clxscore=1030 adultscore=0 bulkscore=0 suspectscore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2308100000 definitions=main-2407250158 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.39 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip4:17.57.156.0/24]; R_DKIM_ALLOW(-0.20)[technologyfriends.net:s=sig1]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[technologyfriends.net:+]; DMARC_NA(0.00)[technologyfriends.net]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; RCPT_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:714, ipnet:17.57.156.0/24, country:US]; FREEFALL_USER(0.00)[jake]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[17.57.156.15:from]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4WVRhG3kKlz4Ybx On 7/25/24 18:11, Jake Freeland wrote: > On 7/25/24 17:40, Mark Johnston wrote: >> On Thu, Jul 25, 2024 at 06:34:43PM -0400, Mark Johnston wrote: >>> On Thu, Jul 25, 2024 at 04:11:22PM -0500, Jake Freeland wrote: >>>> On 7/25/24 15:18, Mark Johnston wrote: >>>>> On Thu, Jul 25, 2024 at 02:47:16PM -0500, Jake Freeland wrote: >>>>>> On 7/25/24 14:02, Konstantin Belousov wrote: >>>>>>> On Thu, Jul 25, 2024 at 01:46:17PM -0500, Jake Freeland wrote: >>>>>>>> Hi there, >>>>>>>> >>>>>>>> I have been steadily working on bringing Data Plane Development >>>>>>>> Kit (DPDK) >>>>>>>> on FreeBSD up to date with the Linux version. The most >>>>>>>> significant hurdle so >>>>>>>> far has been supporting concurrent DPDK processes, each with >>>>>>>> their own >>>>>>>> contiguous memory regions. >>>>>>>> >>>>>>>> These contiguous regions are used by DPDK as a heap for >>>>>>>> allocating DMA >>>>>>>> buffers and other miscellaneous resources. Retrieving the >>>>>>>> underlying memory >>>>>>>> and mapping these regions is currently different on Linux and >>>>>>>> FreeBSD: >>>>>>>> >>>>>>>> On Linux, hugepages are fetched from the kernel's pre-allocated >>>>>>>> hugepage >>>>>>>> pool and are mapped into virtual address space on DPDK >>>>>>>> initialization. Since >>>>>>>> the hugepages exist in a pool, multiple processes can reserve >>>>>>>> their own >>>>>>>> hugepages and operate concurrently. >>>>>>>> >>>>>>>> On FreeBSD, DPDK uses an in-house contigmem kernel module that >>>>>>>> reserves a >>>>>>>> large contiguous region of memory on load. During DPDK >>>>>>>> initialization, the >>>>>>>> entire region is mapped into virtual address space. This leaves >>>>>>>> no memory >>>>>>>> for another independent DPDK process, so only one process can >>>>>>>> operate at a >>>>>>>> time. >>>>>>>> >>>>>>>> I could modify the DPDK contigmem module to mimic Linux's >>>>>>>> hugepages, but I >>>>>>>> thought it would be better to integrate and upstream a >>>>>>>> hugepage-like >>>>>>>> interface directly in the FreeBSD kernel source. I am writing >>>>>>>> this email to >>>>>>>> see if anyone has any advice on the matter. I did not see any >>>>>>>> previous >>>>>>>> attempts at this in Phabriactor or the commit log, but it is >>>>>>>> possible that I >>>>>>>> missed it. I have read about transparent superpage promotion, >>>>>>>> but that seems >>>>>>>> like a different mechanism altogether. >>>>>>>> >>>>>>>> At a quick glance, the implementation seems straightforward: >>>>>>>> read some >>>>>>>> loader tunables, allocate persistent hugepages at boot time, >>>>>>>> and create a >>>>>>>> pseudo filesystem that supports creating and mapping hugepages. >>>>>>>> I could be >>>>>>>> underestimating the magnitude of this task, but that is why I'm >>>>>>>> asking for >>>>>>>> thoughts and advice :) >>>>>>>> >>>>>>>> For reference, here is Linux's documentation on hugepages: >>>>>>>> https://docs.kernel.org/admin-guide/mm/hugetlbpage.html >>>>>>> Are posix shm largepages objects enough (they were developed to >>>>>>> support >>>>>>> DPDK).  Look for shm_create_largepage(3). >>>>>> Yes, shm_create_largepage(2) looks promising, but I would like >>>>>> the ability >>>>>> to allocate these largepages at boot time when memory >>>>>> fragmentation as at a >>>>>> minimum. Perhaps a couple sysctl tunables could be added onto the >>>>>> vm.largepages node to specify a pagesize and allocate some number >>>>>> of pages >>>>>> at boot? >>>>> We could add an rc script which creates named largepage objects.  >>>>> This >>>>> can be done using the posixshmcontrol utility.  That might not be >>>>> early >>>>> enough during boot for some purposes.  In that case, we could have a >>>>> module which creates such objects from within the kernel. This is >>>>> pretty straightforward to do; I wrote a dumb version of this for a >>>>> mips-specific project a few years ago, feel free to take code or >>>>> inspiration from it: https://people.freebsd.org/~markj/tlbdemo.c >>>> Looks simple enough. Thanks for the example code. >>>> >>>>>> It seems Linux had an interface similar to >>>>>> shm_create_largepage(2) back in >>>>>> v2.5, but they removed it in favor of their hugetlbfs filesystem. >>>>>> It would >>>>>> be nice to stay close to the file-backed Linux interface to >>>>>> maximize code >>>>>> sharing in userspace. It looks like the foundation for hugepages >>>>>> is there, >>>>>> but the interface for allocation and access needs to be extended. >>>>> POSIX shm objects have most of the properties one would want, I'd >>>>> expect, save the ability to access them via standard syscalls.  What >>>>> else is missing besides the ability to reserve memory at boot time? >>>> Most notably, I would like the ability to allocate pages in a >>>> specific NUMA >>>> domain. >>> I thought this was already supported, but it seems not... >> Thinking a bit more, I'm pretty sure I had just been using something >> like >> >> $ cpuset -n prefer: posixshmcontrol create -l 1G >> /largepage-1G- >> >> so didn't need an explicit NUMA configuration parameter.  In C one would >> use cpuset_setdomain(2) instead, but that's not as convenient. So, >> imbuing a NUMA domain in struct shm_largepage_conf is still probably a >> reasonable thing to do. > > I just looked at the code, this seems very manageable. I'll draft up a > review. > >>> It should be very easy to implement: extend shm_largepage_conf to >>> include a NUMA domain parameter, and specify that domain when >>> allocating >>> pages for the object (in shm_largepage_dotruncate(), the >>> vm_page_alloc_contig() call should become a >>> vm_page_alloc_contig_domain() call). >>> >>>> Otherwise, in a perfect world, I'd like a unified interface for both >>>> Linux and FreeBSD. Linux hugepages are managed using standard >>>> system calls; >>>> files are mmap(2)'d into virtual address space from hugetlbfs and >>>> ftruncate(2)'d. >>> largepage shm objects work this way as well. > > After reading through the man page, this is quite apparent. Not sure > how I failed make that connection. Anyway, this is starting to look > easier than I thought it would be. The only difference from a > userspace perspective that I can think of right now is how the pages > are created (e.g. hugetlbfs open(2) on Linux vs. > shm_create_largepage(2) on FreeBSD). I suppose I should clarify that hugetlbfs open(2) does not create a hugepage, but rather attaches to one. So it would be analogous to a shm_open(2) instead of shm_create_largepage(2). The hugepages are created at boottime or via sysfs on Linux. My mistake. Jake Freeland > > Thanks for the guidance Mark and Konstantin. > > Jake Freeland >>>> A matching interface would not add an extra kernel >>>> entrypoint and even more importantly, it would ease the >>>> Linux-to-FreeBSD >>>> porting process for programs that use hugepages. > From nobody Sun Jul 28 21:04:03 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WXDWQ5rzlz5Rg91 for ; Sun, 28 Jul 2024 21:04:18 +0000 (UTC) (envelope-from obiwac@gmail.com) Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WXDWQ0sWqz4NTB; Sun, 28 Jul 2024 21:04:18 +0000 (UTC) (envelope-from obiwac@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of obiwac@gmail.com designates 209.85.208.178 as permitted sender) smtp.mailfrom=obiwac@gmail.com Received: by mail-lj1-f178.google.com with SMTP id 38308e7fff4ca-2eeb1ba040aso49385561fa.1; Sun, 28 Jul 2024 14:04:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722200655; x=1722805455; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=7ri29mH4vMe/q+tnSR3HBQp5xvI+J71o8t9aMCqtgdU=; b=X3kyJYa9T9SaNUrGIDahg5QZyzfP5BSmWkajf9VAlTlZWv9mzqM2gTH3eXkSly0JO0 t1r79bTLpBsjsFBedbA64BcSKcwr9wfcMzMGdENiVt8qIgpMOSfl5p7S/bHt42jwF9aM w3wn23bGfhaJjE66b10lvO23bfk3fiXnLekes5aerNMRDpUS/mYBO7m09/aciZS2R84E aAJiEMI/1x1vo43eqI5kyAw4+6SKsrJtejEikB/aFy5P0IXaeRleul8TWXYBQ9QGdAZW AjrPOtOWlLWtNEAeu2GJYueldo+mQiuQP2We4XwiG92y89nUzQ7uEmOAS15cyRd1tZaS SvYw== X-Forwarded-Encrypted: i=1; AJvYcCWIM0y0MebYULir4pykeJhr3t7d4dTx4kmwO5VGQLMZflBtZZr4K/HBlrLahnPiVaF5iVnf3GGVbtMCDCqJYPWykiTaYrRs7asiYmpMNnDGng== X-Gm-Message-State: AOJu0YypcQ1pwD/tzFgXzVMZjCj4ElTE+51Qa/6P69FlGehTjZyJm9U1 4ULN5hJavlnM1HheBzk/MIGTWx9eGrIIRPYd1Q/vy9uNKFijAmCJFVIXQ7jyX7U= X-Google-Smtp-Source: AGHT+IHqfQDXF1d5N7mOEihfULO4bCt3Ul9pM8z8GD3Z1LKB4xqfcv7kT5wbXI1lf5mtKGy1YJPEAg== X-Received: by 2002:a2e:3507:0:b0:2ef:2c0f:284a with SMTP id 38308e7fff4ca-2f12ee69e6fmr36174451fa.44.1722200655018; Sun, 28 Jul 2024 14:04:15 -0700 (PDT) Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com. [209.85.167.53]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-2f03cf0dc4bsm10968581fa.18.2024.07.28.14.04.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 28 Jul 2024 14:04:14 -0700 (PDT) Received: by mail-lf1-f53.google.com with SMTP id 2adb3069b0e04-52f00ad303aso4468148e87.2; Sun, 28 Jul 2024 14:04:14 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCW8wyYNvsOXsA396AtYtOlSW0dECBSSZ4B4OujhiDOijJ8PGHphDGxmjXOn32KUqrYWy+s5MU/82a659W4QkaZ1V4+QuexsHxx+1D7AtOdZCw== X-Received: by 2002:ac2:434d:0:b0:52c:df5f:7b4e with SMTP id 2adb3069b0e04-5309b2b6e49mr3241667e87.38.1722200654550; Sun, 28 Jul 2024 14:04:14 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 From: obiwac Date: Sun, 28 Jul 2024 23:04:03 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Questions about best practices w.r.t. writing a kernel module port which modified LinuxKPI (BATMAN) To: freebsd-hackers , imp@freebsd.org, John Baldwin , "shawn.webb@hardenedbsd.org" Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.07 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.62)[-0.621]; NEURAL_HAM_MEDIUM(-0.55)[-0.552]; FORGED_SENDER(0.30)[obiwac@freebsd.org,obiwac@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_THREE(0.00)[4]; FREEFALL_USER(0.00)[obiwac]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.178:from,209.85.167.53:received]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[obiwac@freebsd.org,obiwac@gmail.com]; MISSING_XM_UA(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.178:from]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4WXDWQ0sWqz4NTB Hi! I worked on porting the batman_adv Linux kernel module to FreeBSD using the LinuxKPI last year as part of a GSoC project and I now have time to work on WiFi support for it (which is necessary for it to actually be useful in practice). Before I do so though, I'd like to create a port for it. I have a few questions about best practices w.r.t. going about this which I was unable to answer by myself by looking around at other ports (specifically drm-kmod, it's the only other port that I know of that distributes a kernel module that depends on the LinuxKPI). Here are the two main questions I have: 1. I have made changes to the LinuxKPI headers and other parts of the kernel in order to accommodate batman_adv. Is it okay for me to upstream those changes, or should I expect users to apply a patchset on their kernel source and to recompile it? If I can upstream them, what should I do about older versions than -CURRENT? Will I just have to wait for those changes to go into the next -STABLE release? And if so, will that mean that any updates that I make to the LinuxKPI headers necessary for newer versions of batman_adv will either have to wait until the next release or be distributed alongside the port? 2. I have made changes to ifconfig to support the creation of BATMAN soft interfaces. Should I upstream those changes and somehow disable them when the kernel module is not loaded, or should I distribute a patched version of ifconfig with my port? Or should I go with a different solution entirely, and write and distribute a tool similar to batctl (which from what I understand was the route taken when distributing BATMAN on most Linux distros before iproute2 added support for managing BATMAN interfaces)? Thank you so much in advance for your answers & help! (Warner, John, I've CC'd you two as you were in the thread on the possibility of upstreaming this to the source tree.)