From owner-freebsd-hackers@freebsd.org Sun Oct 7 04:21:04 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30B2010AFA9B for ; Sun, 7 Oct 2018 04:21:04 +0000 (UTC) (envelope-from cneirabustos@gmail.com) Received: from mail-vs1-xe33.google.com (mail-vs1-xe33.google.com [IPv6:2607:f8b0:4864:20::e33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C025E72595 for ; Sun, 7 Oct 2018 04:21:03 +0000 (UTC) (envelope-from cneirabustos@gmail.com) Received: by mail-vs1-xe33.google.com with SMTP id g201so9626132vsd.12 for ; Sat, 06 Oct 2018 21:21:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=wzaB+psnzqDCJMtHEYWRYAIfEEctjQ51TEjR7yGBb0o=; b=NPfI7olEvaBwwdcBJYoPrOzXJnmZ98I1U9RB0lM/PAn3sygHJtdDjjxHxtXAg/nlq9 LBOZuB4jHdSI3UMwpPRrYuRQbkA8Fc5lhFoJb4RitCClT1EDyYWSw45CqPhq+q5kL14o sSa4NYonqckEPXiLm2wVsTiAyUUIKcr2xyiLAJk5f6Fms/xOn/uWET4aO/Fj6sxejzm2 KZIcAMoXLxE9nzAaZsGFb8xeBK1XYlU1VS8DLGd2XEKbNS09fCodL9Fz1nFVqVboIxvY NFaL5q/BXaDP/6QWT7aY7y2k4xEMqHqpXt+13nqIMsIBdtu2vv7cKKE1xjqtD215msPW JyHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=wzaB+psnzqDCJMtHEYWRYAIfEEctjQ51TEjR7yGBb0o=; b=TbWJYKTTExBW7tndpiNp2MrbhdD7LC1moN/eUUBnNLA5xaysBvBSRynnGD9jz0LTC9 9YYl1of4br/uRa7IELeC8B0/IAzwfurgajQO2cGcpHa3ewrP6pW8OwDMht8+e5U05gSg 0oIu8Y5Gn3kS43jtIDz2o6So8LXzOn+SEgcY9cKm0eQ1pUfDfPiCAx3TYEjTt66BcER9 XlxvEHgpvWEtuKJDTw8jkqtO8YagR0SOXy8VOHbZzOpC6tvrIoVe/q0leEUBdWg33ZoX a1BC/q9M69Xm9TlCcsM6/4aIPotp8ruYTTZDulg2Mun7JFxEiX02ACl6gALigEhLDqZz KCQQ== X-Gm-Message-State: ABuFfohF6T5U7Z0U2+FfmBlXRa1PTqT0UWnVlYTLU4OE1b7v/WRERXzO gIO3VYmPsiyoIafUG1L0s6EFCPVdVgrwaPrzSv3utP460Xw= X-Google-Smtp-Source: ACcGV635mgPtBeGNvOqhEV8/f2C+nbYZB/q4gNSXsieY0l67D7NuPomLRmT9jOEhTNSDm44imk2puDIMmkI8z1vh3T0= X-Received: by 2002:a67:468a:: with SMTP id a10mr6885464vsg.202.1538886062673; Sat, 06 Oct 2018 21:21:02 -0700 (PDT) MIME-Version: 1.0 From: carlos antonio neira bustos Date: Sun, 7 Oct 2018 01:20:50 -0300 Message-ID: Subject: vm statistics per jail To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Oct 2018 04:21:04 -0000 Hello, Hackers, I have added mib member vm.vmtotal to be part of a prison with the intent that each jail could report the correct memory usage using their vm counters. Unfortunately for me, I found that t_free depends on the global vmmeter counters, so I'm trying to add those as well to the prison struct, that will make consumers of those counters to report the correct information inside a jail. The problem so far trying to add vmeter is when I'm trying to determine to which jail a vm_page_t belongs to using vm_pages_t's object cred struct I'm getting a kernel panic due to a page fault. I have modified vm_phys_freecnt_adj(vm_page_t m, int adj) from /usr/src/sys/vm/vm_phys.h to increment the vmeter.v_free_count on the cr_prison that's belongs to vm_page_t. The panic does not produce a dump, so I'm unable to debug the fault. My rc.conf contains dumpdev="AUTO" so I think it should produce a dump or could it be that the panic happens to early? Is it possible to obtain to which cr_prison a vm_page_t belongs to ?. Bests From owner-freebsd-hackers@freebsd.org Sun Oct 7 09:42:01 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 25BB810BAD2A for ; Sun, 7 Oct 2018 09:42:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 99E5A7CF68 for ; Sun, 7 Oct 2018 09:42:00 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w979foEV051816 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 7 Oct 2018 12:41:53 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w979foEV051816 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w979foGN051815; Sun, 7 Oct 2018 12:41:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 7 Oct 2018 12:41:50 +0300 From: Konstantin Belousov To: carlos antonio neira bustos Cc: freebsd-hackers@freebsd.org Subject: Re: vm statistics per jail Message-ID: <20181007094150.GY5335@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Oct 2018 09:42:01 -0000 On Sun, Oct 07, 2018 at 01:20:50AM -0300, carlos antonio neira bustos wrote: > Hello, Hackers, > > I have added mib member vm.vmtotal to be part of a prison with the intent > that each jail could report the correct memory usage using their vm > counters. > Unfortunately for me, I found that t_free depends on the global vmmeter > counters, so I'm trying to add those as well to the prison struct, that > will make consumers of those counters to report the correct information > inside a jail. > The problem so far trying to add vmeter is when I'm trying to determine to > which jail a vm_page_t belongs to using vm_pages_t's object cred struct I'm > getting a kernel panic due to a page fault. > I have modified vm_phys_freecnt_adj(vm_page_t m, int adj) from > /usr/src/sys/vm/vm_phys.h to increment the vmeter.v_free_count on the > cr_prison that's belongs to vm_page_t. > The panic does not produce a dump, so I'm unable to debug the fault. > My rc.conf contains dumpdev="AUTO" so I think it should produce a dump or > could it be that the panic happens to early? > Is it possible to obtain to which cr_prison a vm_page_t belongs to ?. No. vm_page_t does not belong to a jail. From owner-freebsd-hackers@freebsd.org Sun Oct 7 18:58:35 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3A8C10C986E for ; Sun, 7 Oct 2018 18:58:35 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2AD1A8FAAD for ; Sun, 7 Oct 2018 18:58:35 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id E6C8CBAD for ; Sun, 7 Oct 2018 14:58:26 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Sun, 07 Oct 2018 14:58:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.net; h=to :from:subject:message-id:date:mime-version:content-type :content-transfer-encoding; s=fm3; bh=4s5QFGvr0aVuyWlwQxocwgR711 RFVSmKzcz+DJCK8AQ=; b=ACCwNoLemPuyvXlPm03X9ICtVOy6TP7hrQlxEwV8BU 9GRamPvabBpNZHaycfaHe3LK8oKYjVXzBX36kUsOglpQn+knrEMvVqjliexeE87Q LXRI019owR0Hci57OdjQ13EOBLVQL7mD/NOCSUck4AETzDuvc80cW6g2P4r+vRqX KQzQL87zG7IVBfY5Tha5Kdrv1J7xxP53sNA1DvmeCte2b0O+ORHGpfxL2snJ6CHx JCMW4Qb4ovjt1jw4sL5x/quU0eRX+//YtvVPGTf1+0GuQX9ezPmGrI8zHsTdtRlW MAIlMW5mEIKbrC6wX1FjZiZXmGylSBpL5z3OytCRBwgQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=4s5QFG vr0aVuyWlwQxocwgR711RFVSmKzcz+DJCK8AQ=; b=ob4bHQaDuTYQH8YZCgYCXt SPFjsReQqqokWsQlNStAKvi1WmOkzZ9WtI9NaWmo3VVqPokUFZpKTs5YZn7X3HNr WYCTobF01CEHnwW5RjfA91pabQEDcwue3FlkiJyvL0LjvRAzFIIfm+eoFoMKyna9 z+0m3jtvStacggklVVblVIBpljsvMDWjH5I/f6+KoIT9ijbvK4duZJX/fQG5wklI KSYJeaOPw6mPXP8vO6YJXm62j6zp/Jw6noPyZu0Hr1YvmcS9cNeqtab5DGFQeiNT jxn72nmNQnHHnyz5V/gUWceG9LWhEThckDgY29hZrgxWs7F6fAVDsnXxO1DrJSMQ == X-ME-Sender: X-ME-Proxy: Received: from [192.168.1.2] (unknown [62.183.124.173]) by mail.messagingengine.com (Postfix) with ESMTPA id DB9E4102A0 for ; Sun, 7 Oct 2018 14:58:25 -0400 (EDT) To: freebsd-hackers From: Yuri Pankov Subject: vfs.usermount allows users to do arbitrary unmounts Message-ID: Date: Sun, 7 Oct 2018 21:58:21 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Oct 2018 18:58:35 -0000 Hi, Going through https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232021, I noticed that unmount(2) will not fail with EPERM as expected on mounts that don't belong to the user, and which were not mounted by that user, e.g.: # zfs create data/test # ls -ld /data/test drwxr-xr-x 2 root wheel 2 Oct 7 21:55 /data/test $ umount /data/test; echo $? 0 Even if vfs.usermount isn't great from security perspective, is this expected? From owner-freebsd-hackers@freebsd.org Sun Oct 7 19:08:18 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 78D2A10C9C5C for ; Sun, 7 Oct 2018 19:08:18 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F05B6901AB for ; Sun, 7 Oct 2018 19:08:17 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 84034187 for ; Sun, 7 Oct 2018 15:08:16 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Sun, 07 Oct 2018 15:08:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.net; h= subject:from:to:references:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm3; bh=b xRP/f0gU5CvWEmVj0Q0qVzT4EKKXORm61xLCRGCP3Y=; b=RyOB7rl+iRONt5Ez0 bNRwiO0f0cFzVnZAorTw1n1rBfbqRCYmpnDiaYOvKTZIYntX3PoUovnd2yZVn1vt jtHL+JTCcSEX8dUhSLBKkovx/CfA9XrsdhELgGq7SoEETYmrWPqgo2swciZJ4XzP zDwGIS4bFb+edaNmW1A1S9b4FE2zwZhRgfwHiivaWCGF3l28ehVAcXrCnlS/1P0n DfgKRUoBWaxCC/YGpGFq2EMc17GCUasjdpkNuzjLxgZ46ZY/1L40kmhxcNgQi3Ny j+81CMnmSIyhKBe7vmTnFBSyR8Yes8qIVZ3fhtHrH+wIbBa7MJFkfD9RAIQnvS+n LWYbA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=bxRP/f0gU5CvWEmVj0Q0qVzT4EKKXORm61xLCRGCP 3Y=; b=rcdg6uyiPTEE7fUfrBlD3ou1SMV5r0uwBo2Y+jRYu97bOHfwtfZo0R0wx vbhlmwWtZype7PDkI2YSEfRe6bh7scTdt5r7yHxfaUIBegY8G0kezw8v/ECilhwN nbhVfStBAKai29lCdWtLhfLE0a+Hj+9zk5PU7deXjmmHwYwnK7NM6WzvzBH9niz+ nMxN7TVPSOSvZe4hCqxymCFC3HQ1ihpeGfuluLtNOsKlB2uYKZGABNlTKvVztXp7 B6l9GlzYXQmGXxNNRE/n+OcnoMZoh22F5t7HrM9N9kz6+ZQPFWXlGY7VFWKrh8DZ so4DA5TVzZY7GTrKijo6LwhFsRHAg== X-ME-Sender: X-ME-Proxy: Received: from [192.168.1.2] (unknown [62.183.124.173]) by mail.messagingengine.com (Postfix) with ESMTPA id 164AF102DE for ; Sun, 7 Oct 2018 15:08:14 -0400 (EDT) Subject: Re: vfs.usermount allows users to do arbitrary unmounts From: Yuri Pankov To: freebsd-hackers References: Message-ID: <4c5d948d-f129-2787-1f34-8e4fd96d7144@yuripv.net> Date: Sun, 7 Oct 2018 22:08:11 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Oct 2018 19:08:18 -0000 Yuri Pankov wrote: > Hi, > > Going through https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232021, > I noticed that unmount(2) will not fail with EPERM as expected on mounts > that don't belong to the user, and which were not mounted by that user, > e.g.: > > # zfs create data/test > # ls -ld /data/test > drwxr-xr-x  2 root  wheel  2 Oct  7 21:55 /data/test > $ umount /data/test; echo $? > 0 > > Even if vfs.usermount isn't great from security perspective, is this > expected? Nevermind me, looks like I got it all confused with using `zfs allow`. From owner-freebsd-hackers@freebsd.org Sun Oct 7 23:40:41 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D48C10ABD0D for ; Sun, 7 Oct 2018 23:40:41 +0000 (UTC) (envelope-from cneirabustos@gmail.com) Received: from mail-vs1-xe2f.google.com (mail-vs1-xe2f.google.com [IPv6:2607:f8b0:4864:20::e2f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA83F70E33 for ; Sun, 7 Oct 2018 23:40:40 +0000 (UTC) (envelope-from cneirabustos@gmail.com) Received: by mail-vs1-xe2f.google.com with SMTP id m5so3861186vsk.6 for ; Sun, 07 Oct 2018 16:40:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=71FWLoeb6iEcg/LE1XBv8SDyIvXrUM25igNeM2TLCLw=; b=q4V4aKKGllSCaV8l+h17pjRcIwzgU2X+UmWk+7gi7n54VqeKZ3ZHEFHX5O+WsIcPOC 1HyXCetwEy6b5k52AM1oitOSXGqDRBd8sa7FCbWx9XvgMB6JpQ1H0QbD7OiafMNpGtMA 6Oq3LGaohrrpNK82DuZKOg+vlr3x41PizZpiU8twZ0FXGbGeSNvnchUU9gB6cYA2Cufw HRA3RBEBlDpZ/ZDE151tqd2hmQyG+sCXZjuZvBlfK5RTsPKZqHmkAgNWmURNDcLIrRQ5 QOJHlzxHmVtEYOA0O/g521sV9AYbzWu/KvoOkKmGhzVh9bLIFsmBwD/lMcvlzGOWyu/z Svtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=71FWLoeb6iEcg/LE1XBv8SDyIvXrUM25igNeM2TLCLw=; b=bR8fKEvwLywnNRVVC1cNIJx+Y1j5nLfjNwZT1AsVgIfByYlnodr7GYV+PLQOkCnbxD ypGmLx/rGTQc9m0O7YOPH13FXklwrAEKVbkBAAxYTPUHJ3i4O7u6j/4CAXhARvYXGEdC bWmBPw+12DXVGkWMLTOHIinv3xbaEKKHIP7nz0pBCwkfw5n12FNK9lAGAia4DIWbZ/wk LeCFUWQHR9Vb5YaA7MJaF1cRckWHwQQkcyAoyDNbBn6nBwUKaVUzs/lI1Y7iEkB6x8nZ yDfCKQCL8PRtN7Fu9/Np6oAKJwg7YjHWwPpk2ijNK2tlf4yNQJqxaZlJ+n2UiDn5MJHY QcYQ== X-Gm-Message-State: ABuFfojW0Xyv4egMj2UE+0zKhT5PKMJ494L2wUfQ3mQRpSJ8lRilY5iH TGGyEsEjsygRcLTNxsPXVjgHNw9NsN9UWHSz9A4= X-Google-Smtp-Source: ACcGV63mqBeNhYMPkWcsah6Lnb97QMhQ3hZyMub2uJpwKr08GpWVR7d6cqvnRp11wFji4bbl8KhPE0I0bkx8qnYGtsQ= X-Received: by 2002:a67:f458:: with SMTP id r24mr7678413vsn.164.1538955639935; Sun, 07 Oct 2018 16:40:39 -0700 (PDT) MIME-Version: 1.0 References: <20181007094150.GY5335@kib.kiev.ua> In-Reply-To: <20181007094150.GY5335@kib.kiev.ua> From: carlos antonio neira bustos Date: Sun, 7 Oct 2018 20:40:23 -0300 Message-ID: Subject: Re: vm statistics per jail To: kostikbel@gmail.com Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Oct 2018 23:40:41 -0000 Thanks, Konstantin, So that means that for a vm_page_t object the cred member from a vm_object_t in the vm_page_t will never have it cr_prison struct pointing to a jail? Is there a method to associate a vm_page_t to a jail ? as vm_object contains a cred member I thought I could use that. Bests On Sun, Oct 7, 2018 at 6:42 AM Konstantin Belousov wrote: > On Sun, Oct 07, 2018 at 01:20:50AM -0300, carlos antonio neira bustos > wrote: > > Hello, Hackers, > > > > I have added mib member vm.vmtotal to be part of a prison with the > intent > > that each jail could report the correct memory usage using their vm > > counters. > > Unfortunately for me, I found that t_free depends on the global vmmeter > > counters, so I'm trying to add those as well to the prison struct, that > > will make consumers of those counters to report the correct information > > inside a jail. > > The problem so far trying to add vmeter is when I'm trying to determine > to > > which jail a vm_page_t belongs to using vm_pages_t's object cred struct > I'm > > getting a kernel panic due to a page fault. > > I have modified vm_phys_freecnt_adj(vm_page_t m, int adj) from > > /usr/src/sys/vm/vm_phys.h to increment the vmeter.v_free_count on the > > cr_prison that's belongs to vm_page_t. > > The panic does not produce a dump, so I'm unable to debug the fault. > > My rc.conf contains dumpdev="AUTO" so I think it should produce a dump or > > could it be that the panic happens to early? > > Is it possible to obtain to which cr_prison a vm_page_t belongs to ?. > > No. vm_page_t does not belong to a jail. > From owner-freebsd-hackers@freebsd.org Sun Oct 7 21:40:43 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1711B10CDE72; Sun, 7 Oct 2018 21:40:43 +0000 (UTC) (envelope-from freebsd@edvax.de) Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.126.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.kundenserver.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8321E95CBD; Sun, 7 Oct 2018 21:40:42 +0000 (UTC) (envelope-from freebsd@edvax.de) Received: from r56.edvax.de ([92.195.59.48]) by mrelayeu.kundenserver.de (mreue010 [212.227.15.167]) with ESMTPA (Nemesis) id 1MPp0l-1gM6o71R7z-00Mx6g; Sun, 07 Oct 2018 23:40:40 +0200 Received: from r56.edvax.de ([92.195.59.48]) by mrelayeu.kundenserver.de (mreue010 [212.227.15.167]) with ESMTPA (Nemesis) id 1MPp0l-1gM6o71R7z-00Mx6g; Sun, 07 Oct 2018 23:40:40 +0200 Date: Sun, 7 Oct 2018 23:40:40 +0200 From: Polytropon To: FreeBSD Questions Cc: FreeBSD Hackers Subject: FreeBSD boot manager customization for single-disk multiboot (4 x DOS) Message-Id: <20181007234040.028639f8.freebsd@edvax.de> Reply-To: Polytropon Organization: EDVAX X-Mailer: Sylpheed 3.1.1 (GTK+ 2.24.5; i386-portbld-freebsd8.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:vz9xg9+WtbrjajfHP/Hkwh33J2jKP6ygPkEzsmWQtx2ZyizkVKW vD9X6h5tyVq7HzEplWkXN1e+zJ7JDIxxhIb/osRkHKP8nGcG4UImj+szvc/61kuQrCpOUjr 7kaZvkJcVO6KHJmDW4am0nogPmPbgycmMG5T+A3lxRKxeWlbYSJHBQ6SQdXfLfCbF37TgU9 9FUBxTcgPfqbjotkJgBtQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:OAYtdTyZHVw=:tOgPkujMvyjFqH8XbJWLe1 7+uF18M6k2CcS82TiAWUxwWjLddnqByvhOR0KpMprGygR6LEDSg3L2dHfCsZtlzm9IKc3r4N+ aJz3CqpelKVWxODWDbJ/fyOoOHcKISeglIAku8Cv1+Ab0QWEd5hKcttEOQlpgiVtrSDLz1npw FtoLl3Ao8DaBKn0KxIP7nahEqDJv/7gPouJz++TiWa5yI5t9zXHjeYnTDYo+fUP/7Mkkhu9lS lISBhAtI/Jhrc26cbG2YwUXP1mXMmVyGCxL3WURTFnQuDubniLl64bTFSh/eNw3SedQtdDyNt vnUTdmY5a8dBsP2eaD6iovLC+1omqzV+sH8P8/LDmc8LZU28FzT02CmLwd/CzpogichkLSFuV igwQKTJ/H20CbM3eX1hRP4HjyODLW2Qpc3u6ON3v4iylEhZ0pl/cVrbtP/fhboA6rfFdxVP+P wlJH68WfJGUph4SObBFkiUP3SCEtxsvzlYKdRL0LUKKL3tbm1hZXTwZTv8A+LCmFTocUmj7BW 2iFJC4uDDXgtjN2kt4sEfD6JrYhnAxi9gbLboCdTIsdu/+0o+DkqxmbwWCuMFOQrVQrGnTcff wQq+9JR++0sLtgBLcBzj4kKLCqqK1f91k9zvmymdLzmWjmzNgKmvCPb41jc6vU+999K3n4FJS 7fEo3ukFGT/BOZMGAy3eaIUS/+bOMf2HKKjYtqPxmgug5etr6eu6SZmxs59ej0y51vyRzIupQ EhLa4cKfLddAlf60 X-Mailman-Approved-At: Mon, 08 Oct 2018 01:36:11 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Oct 2018 21:40:43 -0000 I'm crossposting to the freebsd-hackers@ list, hope that's okay. Replies from that list please keep me CC'd as I'm not subscribed to that list. Thanks in advance! I have a strange and unique problem to solve and would like to ask for pointers into the right direction (or at least pointers into wrong directions _not_ to take). My IBM Thinkpad 755C died. In fact, the very special PSU exploded. It provided special voltages (10V=, 20V=) on a special plug (4-pin semi-rectangular). Its special disk can't be placed into a comparable laptop as it's not a regular 2.5" unit - it's a little bit taller and therefore won't fit the usual (P)ATA 2.5" disk bays. There is some special software on that disk that I need to use from time to time, and there is no way to use a modern laptop (too fast, missing required connectors and drives, as well as low-level control over those resources). The software I have to use is very special and cannot be replaced by anything else. I can access (P)ATA laptop disks fine with FreeBSD and a USB disk adapter, so reading disk content and copying it to somewhere else is not a problem. I plan to dedicate a Fujitsu-Siemens Litebook LFL-G (500 MHz AMD CPU, 128 MB RAM) as a replacement. It will also obtain the functions formerly provided by a Toshiba T-2130CT laptop whose disk drive, power supply, battery, case etc. died. But the disk is still there. The Litebook will get a hard disk of 10 GB (or less, if I still have one) partitioned as follows: 1st slice: 100 MB MS-DOS 5.0 from the Toshiba T-2130CT 2nd slice: 500 MB MS-DOS 6.22 + NDOS from a different DOS system 3rd slice: 1000 MB "Windows '95" + NDOS hybrid that was on the IBM disk in the Thinkpad 755C 4th slice: remaining space: "Windows '98" as it came with that laptop (original CD and license key present) The intended use of "slice" indicates that there will be 4 "DOS primary partitions" on the disk. Staying with the "drive letters" from DOS, every OS should see the partition it booted from as C:, and it would be great if it didn't touch (or maybe even didn't recognize) any of the other partitions. I think I could even use a 500 MB disk and strip everything down to 50 + 100 + 150 + 200 MB as per the layout shown above! That would also be fully sufficient for the special purposes. What I'm asking for _now_ is a convenient way to select which OS to boot. I would not want to use today's GRUB because I don't plan to use Linux on that particular system. In my opinion, the FreeBSD boot manager will work nicely, but it will list all 4 systems with the name "DOS". From what I learned by reading its source, this is due to auto detection. I don't need that feature, I'd think about altering the code in order to hard-code 4 names, like this: F1 DOS 5 F2 DOS 6 F3 Win 95 F4 Win 98 Default: F1 So if I needed to boot the system that formerly ran on the Thinkpad 755C, I'd press F3 and it would boot. Sure, I understand that there is a good chance that I have to reconfigure drivers and such (as it's a typical thing with "Windows"), but that won't be the big problem. I know everything has to fit 512 bytes (and the code of the boot manager illustrates this requirement very nicely by how the text for the possible boot options is stored and selected). Questions: Would you say this is a possible way to go? Or would you suggest that I refresh my assembly knowledge and write something myself, maybe using the existing FreeBSD boot manager as inspiration and organ donor? ;-) Or maybe there is something that can be installed and initially configured from DOS? Distant past... I know there _were_ multi-boot utilities decades ago, but I cannot remember any names. There is no luxury required, no GUI, no boot graphics - just a simple means to select the OS that should be booted. Oh, or maybe there is a long-forgotten functionality in the newer "Windows" versions ('95 and '98 will be finally installed, but none of then will ever see the Internet, of course), those also once contained some kind of strange boot manager? If everything fails, I could still try to use the FreeBSD boot manager "as is", with the following result predicted: F1 DOS F2 DOS F3 DOS F4 DOS Default: F1 I'm not sure if it will recognize old-fashioned "Windows '95" as "DOS" or "Windows"... Thank you for any inspiration! :-) -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... From owner-freebsd-hackers@freebsd.org Mon Oct 8 11:19:33 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E08210BB61B for ; Mon, 8 Oct 2018 11:19:33 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D107A85908; Mon, 8 Oct 2018 11:19:32 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id w98BJNRO028558 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 8 Oct 2018 13:19:23 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id w98BJIBZ028555; Mon, 8 Oct 2018 13:19:18 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Mon, 8 Oct 2018 13:19:18 +0200 (CEST) From: Wojciech Puchar To: Ian Lepore cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: ntpd strange problem In-Reply-To: <1538855151.14264.54.camel@freebsd.org> Message-ID: References: <20181005061829.GG21091@server.rulingia.com> <1538855151.14264.54.camel@freebsd.org> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Oct 2018 11:19:33 -0000 >> >> what else can i do? > > The original output you posted showed the time as unsynchronized when > ntpd started up. That seems normal to me, it can't be synchronized > until ntpd has been running for a while. In a followup message you > seemed to say that it doesn't stay synchronized, but you haven't > provided much info about that. > > What's in the logs when it become unsynchronized or when you notice the > time is several seconds off? after starting ntpd - it seems to be no different than upsteream time server. After week or so it got out of sync in spite of ntpd still running. Strange but today (4 days after i restarted ntpd) it seems to be in sync # ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== LOCAL(0) .LOCL. 2 l 93h 64 0 0.000 0.000 0.000 *ntp.task.gda.pl 194.146.251.100 2 u 227 256 377 7.127 0.006 0.050 # ntpq -c rv associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, version="ntpd 4.2.8p12-a (1)", processor="amd64", system="FreeBSD/11.2-STABLE", leap=00, stratum=3, precision=-24, rootdelay=13.877, rootdisp=31.084, refid=153.19.250.123, reftime=df65bb67.6caa4c22 Mon, Oct 8 2018 13:17:59.424, clock=df65bb79.e7c2ec64 Mon, Oct 8 2018 13:18:17.905, peer=60901, tc=9, mintc=3, offset=0.065124, frequency=35.388, sys_jitter=0.000000, clk_jitter=0.044, clk_wander=0.001 my ntp.conf rlimit memlock 512 server ntp.task.gda.pl iburst maxpoll 9 restrict ntp.task.gda.pl nomodify nopeer noquery notrap restrict -4 default ignore restrict -6 default ignore restrict 127.0.0.1 nomodify nopeer notrap restrict 10.0.0.0 mask 255.0.0.0 nomodify nopeer notrap server 127.127.1.0 fudge 127.127.1.0 stratum 2 > > What does the output of ntptime show when the time is drifted off? > > How about the output of "ntpq -p" and "ntpq -c rv"? > > What's in your ntp.conf? > > If you set sysctl kern.timecounter.stepwarnings=1, do you see any > warnings about the clock stepping as time drifts out of sync? > > -- Ian > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@freebsd.org Mon Oct 8 11:20:25 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C4A1210BB693 for ; Mon, 8 Oct 2018 11:20:25 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3CF25859AA for ; Mon, 8 Oct 2018 11:20:25 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id w98BKNd6028907 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 8 Oct 2018 13:20:23 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id w98BKIQl028904; Mon, 8 Oct 2018 13:20:18 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Mon, 8 Oct 2018 13:20:18 +0200 (CEST) From: Wojciech Puchar To: Warner Losh cc: FreeBSD Hackers Subject: Re: Looking for a python hacker to help me parse dmesg In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Oct 2018 11:20:25 -0000 while from the start you require to use python? maybe i could help you On Sat, 6 Oct 2018, Warner Losh wrote: > Greetings, > > I'm looking for a python hacker to help parse a series dmesg files and > produce summary results. Things like counts of different versions, summary > of which architectures used, summary of memory on the machines, summary of > drivers used. In addition to the one dimensional results, I'd be interested > in the cross product of these as well. > > Please contact me off list if you can be of help. > > Warner > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@freebsd.org Mon Oct 8 21:31:26 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31B4010CC67C for ; Mon, 8 Oct 2018 21:31:26 +0000 (UTC) (envelope-from rpokala@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E01987E9FD for ; Mon, 8 Oct 2018 21:31:25 +0000 (UTC) (envelope-from rpokala@freebsd.org) Received: from [172.17.133.69] (unknown [12.202.168.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: rpokala) by smtp.freebsd.org (Postfix) with ESMTPSA id 883FA1C72F for ; Mon, 8 Oct 2018 21:31:25 +0000 (UTC) (envelope-from rpokala@freebsd.org) User-Agent: Microsoft-MacOutlook/10.11.0.180909 Date: Mon, 08 Oct 2018 14:31:22 -0700 Subject: coredumps disallowed when creds are changed? From: Ravi Pokala To: "freebsd-hackers@freebsd.org" Message-ID: Thread-Topic: coredumps disallowed when creds are changed? Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Oct 2018 21:31:26 -0000 Greetings hackers. core(5) states: > By default, a process that changes user or group credentials whether real or effective will not create a corefile. This behaviour can be changed to generate a core dump by setting the sysctl(8) variable kern.sugid_coredump to 1. Can someone explain why? Thanks, Ravi (rpokala@) From owner-freebsd-hackers@freebsd.org Mon Oct 8 21:02:22 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5E6610CB4FC; Mon, 8 Oct 2018 21:02:21 +0000 (UTC) (envelope-from jude.obscure@yandex.com) Received: from forward104j.mail.yandex.net (forward104j.mail.yandex.net [IPv6:2a02:6b8:0:801:2::107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5DAF87D272; Mon, 8 Oct 2018 21:02:20 +0000 (UTC) (envelope-from jude.obscure@yandex.com) Received: from mxback12g.mail.yandex.net (mxback12g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:91]) by forward104j.mail.yandex.net (Yandex) with ESMTP id 987A541CD9; Tue, 9 Oct 2018 00:01:18 +0300 (MSK) Received: from smtp1p.mail.yandex.net (smtp1p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:6]) by mxback12g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id hplfwsPQ2D-1IgSrv6x; Tue, 09 Oct 2018 00:01:18 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.com; s=mail; t=1539032478; bh=PgQFt5AWpvRxcev8k01xcf5LUrQHhljsjUoS+kX8kQM=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=NAOIAXazNQyfTXH534ZYYJGx4eczIJtZmJTOTqYRzOwrl59O07hWrs9kaiX7ItUpM 2KIVI7tYmy00Ene6qQ9c/UGbuE0YZUL+6Ht7A4v9JXEETasZOxQnRf/xq3EyocKCVG OOEtH+xboDLn4qb4eQO8wwKR3q5D1KJ61Zd3Fq3I= Received: by smtp1p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id Ucz6wPNY68-1GJqRHS0; Tue, 09 Oct 2018 00:01:17 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.com; s=mail; t=1539032477; bh=PgQFt5AWpvRxcev8k01xcf5LUrQHhljsjUoS+kX8kQM=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=ABFBd2A7TlzJ2M7s0csLsZuh0ujzyrH0Tg7XoDMptNl4LsqnXqtLeyhHZXNl6fzQl AO1supZO5/i5RlF+S6Bx2vI+69Y6kLDbtDk2/ge80BTrJXqEuMxjtS9abYL4GmXNd4 /ZXkd6q7Z3BIUtTvN8JfNTV63L2H0+tddXpMATb8= Authentication-Results: smtp1p.mail.yandex.net; dkim=pass header.i=@yandex.com Subject: Re: FreeBSD boot manager customization for single-disk multiboot (4 x DOS) To: Polytropon , FreeBSD Questions Cc: FreeBSD Hackers References: <20181007234040.028639f8.freebsd@edvax.de> From: Manish Jain Message-ID: <38342ce8-990a-ce05-7ab9-9f7d6017f91b@yandex.com> Date: Tue, 9 Oct 2018 02:30:10 +0530 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <20181007234040.028639f8.freebsd@edvax.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 08 Oct 2018 21:48:19 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Oct 2018 21:02:22 -0000 On 10/08/18 03:10, Polytropon wrote: > I'm crossposting to the freebsd-hackers@ list, hope that's okay. > Replies from that list please keep me CC'd as I'm not subscribed > to that list. Thanks in advance! > > I have a strange and unique problem to solve and would like to > ask for pointers into the right direction (or at least pointers > into wrong directions _not_ to take). > > My IBM Thinkpad 755C died. In fact, the very special PSU exploded. > It provided special voltages (10V=, 20V=) on a special plug (4-pin > semi-rectangular). Its special disk can't be placed into a comparable > laptop as it's not a regular 2.5" unit - it's a little bit taller > and therefore won't fit the usual (P)ATA 2.5" disk bays. There is > some special software on that disk that I need to use from time > to time, and there is no way to use a modern laptop (too fast, > missing required connectors and drives, as well as low-level > control over those resources). The software I have to use is > very special and cannot be replaced by anything else. > > I can access (P)ATA laptop disks fine with FreeBSD and a USB > disk adapter, so reading disk content and copying it to somewhere > else is not a problem. > > I plan to dedicate a Fujitsu-Siemens Litebook LFL-G (500 MHz AMD > CPU, 128 MB RAM) as a replacement. It will also obtain the functions > formerly provided by a Toshiba T-2130CT laptop whose disk drive, > power supply, battery, case etc. died. But the disk is still there. > The Litebook will get a hard disk of 10 GB (or less, if I still > have one) partitioned as follows: > > 1st slice: 100 MB MS-DOS 5.0 from the Toshiba T-2130CT > > 2nd slice: 500 MB MS-DOS 6.22 + NDOS from a different DOS system > > 3rd slice: 1000 MB "Windows '95" + NDOS hybrid that was on the > IBM disk in the Thinkpad 755C > > 4th slice: remaining space: "Windows '98" as it came with that > laptop (original CD and license key present) > > The intended use of "slice" indicates that there will be 4 "DOS > primary partitions" on the disk. Staying with the "drive letters" > from DOS, every OS should see the partition it booted from as C:, > and it would be great if it didn't touch (or maybe even didn't > recognize) any of the other partitions. > > I think I could even use a 500 MB disk and strip everything down > to 50 + 100 + 150 + 200 MB as per the layout shown above! That > would also be fully sufficient for the special purposes. > > What I'm asking for _now_ is a convenient way to select which OS > to boot. I would not want to use today's GRUB because I don't plan > to use Linux on that particular system. In my opinion, the FreeBSD > boot manager will work nicely, but it will list all 4 systems with > the name "DOS". From what I learned by reading its source, this is > due to auto detection. I don't need that feature, I'd think about > altering the code in order to hard-code 4 names, like this: > > F1 DOS 5 > F2 DOS 6 > F3 Win 95 > F4 Win 98 > > Default: F1 > > So if I needed to boot the system that formerly ran on the Thinkpad > 755C, I'd press F3 and it would boot. Sure, I understand that there > is a good chance that I have to reconfigure drivers and such (as > it's a typical thing with "Windows"), but that won't be the big > problem. > > I know everything has to fit 512 bytes (and the code of the boot > manager illustrates this requirement very nicely by how the text > for the possible boot options is stored and selected). > > Questions: > > Would you say this is a possible way to go? Or would you suggest > that I refresh my assembly knowledge and write something myself, > maybe using the existing FreeBSD boot manager as inspiration and > organ donor? ;-) > > Or maybe there is something that can be installed and initially > configured from DOS? > > Distant past... I know there _were_ multi-boot utilities decades > ago, but I cannot remember any names. There is no luxury required, > no GUI, no boot graphics - just a simple means to select the OS > that should be booted. > > Oh, or maybe there is a long-forgotten functionality in the newer > "Windows" versions ('95 and '98 will be finally installed, but > none of then will ever see the Internet, of course), those also > once contained some kind of strange boot manager? > > If everything fails, I could still try to use the FreeBSD boot > manager "as is", with the following result predicted: > > F1 DOS > F2 DOS > F3 DOS > F4 DOS > > Default: F1 > > I'm not sure if it will recognize old-fashioned "Windows '95" > as "DOS" or "Windows"... > > > > Thank you for any inspiration! :-) > > > Poly, I suggest a better way than to use FreeBSD/Linux boot manager. Create 1 primary partition + an EBR with 4 logical drives (with the primary partition at the beginning of the disk). Install 4 x DOS/Win into the logical drives (all using the first primary partition as C:) Finally and if needed, alter the boot.ini in C: as you wish. Does that solve your problem, or compound it ? Regards, Manish Jain From owner-freebsd-hackers@freebsd.org Mon Oct 8 22:33:42 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89A0810CE140 for ; Mon, 8 Oct 2018 22:33:42 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F301182AC1; Mon, 8 Oct 2018 22:33:41 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id w98MXW9C054359 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 9 Oct 2018 00:33:33 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: rpokala@freebsd.org Received: from [10.58.0.4] (dadv@[10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id w98MXV90029831 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 9 Oct 2018 05:33:31 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: coredumps disallowed when creds are changed? To: Ravi Pokala , "freebsd-hackers@freebsd.org" References: From: Eugene Grosbein Message-ID: <2e5b1b34-7bd7-f1b8-4f6a-9c794402d2c4@grosbein.net> Date: Tue, 9 Oct 2018 05:33:31 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.5 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_48_96, LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.2 DATE_IN_FUTURE_48_96 Date: is 48 to 96 hours after Received: date * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Oct 2018 22:33:42 -0000 09.10.2018 4:31, Ravi Pokala wrote: > Greetings hackers. > > core(5) states: > >> By default, a process that changes user or group credentials >> whether real or effective will not create a corefile. >> This behaviour can be changed to generate a core dump by setting the sysctl(8) variable kern.sugid_coredump to 1. > > Can someone explain why? Real/effective user/group id often are changed for a process started by non-privilegied user running set[ug]id binary like csh/chpass/passwd(1) that can read sensitive system data similar to /etc/master.passwd containing password hashes. If such utility reads sensitive data and then crashes due to a bug, its coredump may leak data to unexpected places of file system like /home partition, then go to a dump/backup of file system, get uploaded offsite as part of backup etc. That should not happen by default. From owner-freebsd-hackers@freebsd.org Mon Oct 8 22:48:49 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0097D10CE5DE for ; Mon, 8 Oct 2018 22:48:49 +0000 (UTC) (envelope-from rpokala@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AD7748357C; Mon, 8 Oct 2018 22:48:48 +0000 (UTC) (envelope-from rpokala@freebsd.org) Received: from [172.17.133.69] (unknown [12.202.168.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: rpokala) by smtp.freebsd.org (Postfix) with ESMTPSA id 58CED1CE42; Mon, 8 Oct 2018 22:48:48 +0000 (UTC) (envelope-from rpokala@freebsd.org) User-Agent: Microsoft-MacOutlook/10.11.0.180909 Date: Mon, 08 Oct 2018 15:48:39 -0700 Subject: Re: coredumps disallowed when creds are changed? From: Ravi Pokala To: Eugene Grosbein , "freebsd-hackers@freebsd.org" Message-ID: <505EC621-4F7D-4857-A81A-38AF71909000@panasas.com> Thread-Topic: coredumps disallowed when creds are changed? References: <2e5b1b34-7bd7-f1b8-4f6a-9c794402d2c4@grosbein.net> In-Reply-To: <2e5b1b34-7bd7-f1b8-4f6a-9c794402d2c4@grosbein.net> Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Oct 2018 22:48:49 -0000 -----Original Message----- From: Eugene Grosbein Date: 2018-10-08, Monday at 15:33 To: Ravi Pokala , "freebsd-hackers@freebsd.org" Subject: Re: coredumps disallowed when creds are changed? > 09.10.2018 4:31, Ravi Pokala wrote: > >> Greetings hackers. >> >> core(5) states: >> >>> By default, a process that changes user or group credentials >>> whether real or effective will not create a corefile. >>> This behaviour can be changed to generate a core dump by setting the sysctl(8) variable kern.sugid_coredump to 1. >> >> Can someone explain why? > > Real/effective user/group id often are changed for a process started > by non-privilegied user running set[ug]id binary like csh/chpass/passwd(1) > that can read sensitive system data similar to /etc/master.passwd > containing password hashes. If such utility reads sensitive data > and then crashes due to a bug, its coredump may leak data to unexpected places > of file system like /home partition, then go to a dump/backup of file system, > get uploaded offsite as part of backup etc. That should not happen by default. That makes perfect sense. Thanks Eugene! -Ravi (rpokala@) From owner-freebsd-hackers@freebsd.org Tue Oct 9 05:29:47 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C4D310BFED7 for ; Tue, 9 Oct 2018 05:29:47 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vtr.rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 500F775FFB for ; Tue, 9 Oct 2018 05:29:45 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp59-167-167-3.static.internode.on.net [59.167.167.3]) by vtr.rulingia.com (8.15.2/8.15.2) with ESMTPS id w995TYef043687 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 9 Oct 2018 16:29:40 +1100 (AEDT) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id w995TO0K021070 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 9 Oct 2018 16:29:25 +1100 (AEDT) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id w995TO9Q021069; Tue, 9 Oct 2018 16:29:24 +1100 (AEDT) (envelope-from peter) Date: Tue, 9 Oct 2018 16:29:24 +1100 From: Peter Jeremy To: Wojciech Puchar Cc: freebsd-hackers@freebsd.org Subject: Re: ntpd strange problem Message-ID: <20181009052924.GJ21091@server.rulingia.com> References: <20181005061829.GG21091@server.rulingia.com> <1538855151.14264.54.camel@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="K/NRh952CO+2tg14" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Oct 2018 05:29:47 -0000 --K/NRh952CO+2tg14 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2018-Oct-08 13:19:18 +0200, Wojciech Puchar wrote: >After week or so it got out of sync in spite of ntpd still running. > >Strange but today (4 days after i restarted ntpd) it seems to be in sync A couple of suggestions: * (especially if you have an ADSL link): The NTP protocol assumes that the path between the client and server are have symmetric timing. In my experience, bulk uploads or downloads can cause significant path delay asymmetries, which can confuse ntpd. * You only have a single server and are therefore dependent on the trustworthiness of that server. A few more servers could be useful. You could try adding enabling some monitoring (eg loopstats and peerstats) and see what is happening leading up to ntpd losing sync. --=20 Peter Jeremy --K/NRh952CO+2tg14 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAlu8PLRfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzSkog/+I7lIuVmwdIJWV1T78GObZnJwK/H3XRW6wFSXGnmkM0z5Psxrn+ezakWw 6SdQZYvr092QyO3HsYXacOaKFWR+EzObWZp99U8hlFq+RWig1if8WxpgzCakHPxb TLRBRbPBwu0UTz5zzbMr0hkmAs1dA7P04Qu3OnivRq6QN658ozw4x3iur4dpCqlM 6ynyNEql+I0tjfLEa2VFLa6q90lHQE380JpZinr9w7C3QnFolmA5CNQMSxp/Gl9v TysqdaJR6fvfSOqbt2W6wtTOqdcn9oH3JHD3vVrXcBcXMDO0zojkd0jha56sTHyy PWqLB4XhNQ/R88w0U5jdRR4WAkdgbYPcTCFhWvTTK5pp6SIEg9kZhaq8cQgEG9Em ygwMALWnt2g4fPacLhcx9Ybs0MTYzK/q2+aIzITJQACCo81T0dG2b7JySJihih8a if/dR+7HSWwDwlu3amt9xdml8UhCLyAa5EYb3rMYeO4v7bItI7V+xYTyn2lsL0Ol ilaPio92N3wcVYN5BDVxzjK+1gTckIcdnNMSroizJmWT1OlvmCwF9MW5CpbgifGH Se3QRTj4yN1NgWENjoe6aqffkaU/rYh0s+kRz2lV52oc38WkTwRNRvDH7ik+Hr9k +VefnPXCyJhXzG+6UfFvYONoO2Ah4NfkzQZ7KI0VtrT//qqVQBY= =K3YB -----END PGP SIGNATURE----- --K/NRh952CO+2tg14-- From owner-freebsd-hackers@freebsd.org Tue Oct 9 06:43:05 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D72D410C1D34 for ; Tue, 9 Oct 2018 06:43:05 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 57CFB786AB for ; Tue, 9 Oct 2018 06:43:04 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id w996h0ch042321 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 9 Oct 2018 08:43:00 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id w996gtnK042318; Tue, 9 Oct 2018 08:42:55 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Tue, 9 Oct 2018 08:42:55 +0200 (CEST) From: Wojciech Puchar To: Peter Jeremy cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: ntpd strange problem In-Reply-To: <20181009052924.GJ21091@server.rulingia.com> Message-ID: References: <20181005061829.GG21091@server.rulingia.com> <1538855151.14264.54.camel@freebsd.org> <20181009052924.GJ21091@server.rulingia.com> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Oct 2018 06:43:06 -0000 >> Strange but today (4 days after i restarted ntpd) it seems to be in sync > > A couple of suggestions: > * (especially if you have an ADSL link): The NTP protocol assumes that > the path between the client and server are have symmetric timing. In i have gigabit connectivity with low and stable delays. so it's not this. > my experience, bulk uploads or downloads can cause significant path > delay asymmetries, which can confuse ntpd. > * You only have a single server and are therefore dependent on the > trustworthiness of that server. A few more servers could be useful. this server is quite (actually excellently) stable and i trust it. From owner-freebsd-hackers@freebsd.org Tue Oct 9 06:44:26 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DBEFA10C1DDB; Tue, 9 Oct 2018 06:44:26 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 6002A7878A; Tue, 9 Oct 2018 06:44:26 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id w996iORg042757 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 9 Oct 2018 08:44:24 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id w996iJXU042752; Tue, 9 Oct 2018 08:44:19 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Tue, 9 Oct 2018 08:44:19 +0200 (CEST) From: Wojciech Puchar To: Manish Jain cc: Polytropon , FreeBSD Questions , FreeBSD Hackers Subject: Re: FreeBSD boot manager customization for single-disk multiboot (4 x DOS) In-Reply-To: <38342ce8-990a-ce05-7ab9-9f7d6017f91b@yandex.com> Message-ID: References: <20181007234040.028639f8.freebsd@edvax.de> <38342ce8-990a-ce05-7ab9-9f7d6017f91b@yandex.com> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Oct 2018 06:44:27 -0000 > > I suggest a better way than to use FreeBSD/Linux boot manager. > > Create 1 primary partition + an EBR with 4 logical drives (with the primary > partition at the beginning of the disk). > > Install 4 x DOS/Win into the logical drives (all using the first primary > partition as C:) > > Finally and if needed, alter the boot.ini in C: as you wish. > > Does that solve your problem, or compound it ? what is boot.ini? i don't remember something like this in DOS. From owner-freebsd-hackers@freebsd.org Tue Oct 9 14:06:37 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BDB7910CFE97 for ; Tue, 9 Oct 2018 14:06:37 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2r.ore.mailhop.org (outbound2r.ore.mailhop.org [54.200.129.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B1AC8A898 for ; Tue, 9 Oct 2018 14:06:37 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-RoutePath: aGlwcGll X-MHO-User: 804053b6-cbcc-11e8-a70c-1d534e3451d5 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound2.ore.mailhop.org (Halon) with ESMTPSA id 804053b6-cbcc-11e8-a70c-1d534e3451d5; Tue, 09 Oct 2018 14:06:25 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w99E6Lwn012894; Tue, 9 Oct 2018 08:06:21 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1539093981.72469.30.camel@freebsd.org> Subject: Re: ntpd strange problem From: Ian Lepore To: Wojciech Puchar , Peter Jeremy Cc: freebsd-hackers@freebsd.org Date: Tue, 09 Oct 2018 08:06:21 -0600 In-Reply-To: References: <20181005061829.GG21091@server.rulingia.com> <1538855151.14264.54.camel@freebsd.org> <20181009052924.GJ21091@server.rulingia.com> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Oct 2018 14:06:37 -0000 On Tue, 2018-10-09 at 08:42 +0200, Wojciech Puchar wrote: > > > > > > > > Strange but today (4 days after i restarted ntpd) it seems to be > > > in sync > > A couple of suggestions: > > * (especially if you have an ADSL link): The NTP protocol assumes > > that > > the path between the client and server are have symmetric > > timing.  In > i have gigabit connectivity with low and stable delays. so it's not > this. > > > > > my experience, bulk uploads or downloads can cause significant path > > delay asymmetries, which can confuse ntpd. > > * You only have a single server and are therefore dependent on the > > trustworthiness of that server.  A few more servers could be > > useful. > this server is quite (actually excellently) stable and i trust it. When I asked for various outputs, I meant "when the unsync problem is happening"; those outputs are uninteresting when everything is working right. You should remove the config for the "local" clock from your ntp.conf. Configuring a bogus local clock is NEVER the right thing to do, and it's even worse to configure it at a high stratum number. -- Ian From owner-freebsd-hackers@freebsd.org Wed Oct 10 11:10:54 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55C5A10D004D for ; Wed, 10 Oct 2018 11:10:54 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CBFCC7AAC5; Wed, 10 Oct 2018 11:10:53 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id w9ABAhfk036188 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 10 Oct 2018 13:10:43 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id w9ABAcjg036185; Wed, 10 Oct 2018 13:10:38 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Wed, 10 Oct 2018 13:10:38 +0200 (CEST) From: Wojciech Puchar To: Ian Lepore cc: Wojciech Puchar , Peter Jeremy , freebsd-hackers@freebsd.org Subject: Re: ntpd strange problem In-Reply-To: <1539093981.72469.30.camel@freebsd.org> Message-ID: References: <20181005061829.GG21091@server.rulingia.com> <1538855151.14264.54.camel@freebsd.org> <20181009052924.GJ21091@server.rulingia.com> <1539093981.72469.30.camel@freebsd.org> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Oct 2018 11:10:54 -0000 >>> useful. >> this server is quite (actually excellently) stable and i trust it. > > When I asked for various outputs, I meant "when the unsync problem is > happening"; those outputs are uninteresting when everything is working > right. so i will look back and tell you WHEN it will happen again. > > You should remove the config for the "local" clock from your ntp.conf. > Configuring a bogus local clock is NEVER the right thing to do, and > it's even worse to configure it at a high stratum number. thank you. this is an error. From owner-freebsd-hackers@freebsd.org Wed Oct 10 23:59:55 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B5E110C03A7 for ; Wed, 10 Oct 2018 23:59:55 +0000 (UTC) (envelope-from munro@penski.net) Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B56987741E for ; Wed, 10 Oct 2018 23:59:54 +0000 (UTC) (envelope-from munro@penski.net) Received: by mail-ed1-x542.google.com with SMTP id c26-v6so6572084edt.3 for ; Wed, 10 Oct 2018 16:59:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ip9-org.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to:cc; bh=UEx2aorFrp6z6QLtGLEr4opajjQ7MYj6H9PG+T5s6WQ=; b=CB3gEwI6awNmLSDMW6zPJqwxWOULfH+fIjxtpA3N5W76z4N6NBq/qVWHuzbhV2/J1u e4pOi3XOOvhMdGsSpG4JoKuxLiKgBJ4R0JcESjrpyFTeBlL9fzOANdXTiUkY4EokGpGo uFXklzqbz7a4/lrT5CyeyPE4y23BUdeCKsWglJ9s1OvJsSsIOWYf19UENXMOpcokMlJX 8+q3HXRngfH5G7cp8+yQ2JDo4e1dUqRWKObipDq3ZGrE/iaXzTLuvTdNWvxHGBQUg1YT IR0AQlWry4a8dlkxcYh5R3CYasaMjmpCoKTePBN544EO7Cn9Bvd/UJEzOsy7MNKV8qLJ OUnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=UEx2aorFrp6z6QLtGLEr4opajjQ7MYj6H9PG+T5s6WQ=; b=rnCEf7Iwv9tf1NlyQiBs/q4SGety58ENnc27whCXAahVgcrMoYjMQlUoOQWfrqE5wY 1ipdiQW7Jl5GgCU2MZioWKDeQTcUVDA/5tSmqiISp/rY6YZNhHr5u7k95ynhoOFVoR6x yXQ0eZZFhxMV8k/Ii8OJJdkZdvWdxZMCoxNMYfHo8pohIcQ/zv3k822Jl+DYMzGLQNPM H1qXUkH99Ai72QcFuCaggE6eTR7N5WjjdOtcoCwu2sE0DofXq2YVBrkqH99KE6RWMIKA ms8emF3ZHwMXF5tGd+KE+5ZW1ta2bQ+7D6gbUtKZG/gZCVQaqXiZuUyMPFavRcD3upa5 ylaw== X-Gm-Message-State: ABuFfoh+VWdxL5GRlNCTdHPn/CJgep2dAyHIuGj6vwFK5ttSzQ80L2io C97Ns18PwawVhelgGZphIyj0ZqZku5/RShV3nge6wtRDZCev5w== X-Google-Smtp-Source: ACcGV62nubKXEykU0YvDK6b7qI1Wxc1JaBwbqzthCFENylHSLr4Is2NOF3VGPPIpNELYMKoCvHop5SFLoPYMUNZ3K3o= X-Received: by 2002:a17:906:f254:: with SMTP id gy20-v6mr36262199ejb.2.1539215993199; Wed, 10 Oct 2018 16:59:53 -0700 (PDT) MIME-Version: 1.0 From: Thomas Munro Date: Thu, 11 Oct 2018 12:59:41 +1300 Message-ID: Subject: PostgresSQL vs super pages To: freebsd-hackers@freebsd.org Cc: alc@freebsd.org, mjg@freebsd.org, Konstantin Belousov Content-Type: multipart/mixed; boundary="00000000000082055d0577e8a67d" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Oct 2018 23:59:55 -0000 --00000000000082055d0577e8a67d Content-Type: text/plain; charset="UTF-8" Hello FreeBSD hackers, In version 11 of PostgreSQL (about to escape) we've introduced shared memory-based parallel hash joins. Academic reports and basic intuition tell me that big hash tables should benefit from super pages due to random access. I'm interested in exploring this effect on FreeBSD. How can I encourage super pages in short-lived mappings, given our access pattern? I don't have a good understanding of virtual memory (though I'm trying to learn), but let me explain what we're doing, and what I see so far on FreeBSD. We have two kinds of shared memory, and here I'm interested in the second one: 1. We have a big fixed sized area that lives as long as the database is up, that holds our buffer pool and other shared state. It is inherited by every PostgreSQL process. Its size is set in postgresql.conf with eg shared_buffers = '1GB'. On any long running not-completely-idle system I eventually see that it is using super pages (though I wish procstat -v would tell me how many): 752 0x802e00000 0x80bb9e000 rw- 3730 3730 5 0 --S- df That's cool, and we know from benchmarks and experience on Linux and Windows (where we explicitly request huge/large pages with MAP_HUGETLB and MEM_LARGE_PAGES respectively) that this has beneficial performance effects, so I'm happy that FreeBSD eventually reaches that state too (though I haven't yet grokked exactly how and when that happens or attempted to measure its impact on FreeBSD). No problem here AFAIK. 2. In recent versions of PostgreSQL we got parallel computing fever, and decided we needed more dynamic shared memory for that. So we create chunks of memory with shm_open(), size them with ftruncate() and then map them into our main process + worker processes (yeah, still no threads after 30 years). This is where parallel hash join data goes. To get rid of obvious obstacles to super page promotion, I patched my local copy of PostgreSQL to make sure that we always ask for multiples of 2MB and set MAP_ALIGNED_SUPER, but still no cigar (or not much cigar, anyway). Here's what I see on my super slow laptop running recent HEAD. I'll put repro instructions below in case anyone is interested. I ran a 3-process ~380MB join that takes ~90s. These mappings appeared in my memory: 18288 0x826600000 0x82a621000 rw- 16385 16385 4 0 ---- df 18288 0x82a800000 0x82ac00000 rw- 1024 1024 4 0 ---- df 18288 0x82ac00000 0x82b000000 rw- 1024 1024 4 0 ---- df 18288 0x82b000000 0x82b400000 rw- 1024 1024 4 0 ---- df 18288 0x82b400000 0x82b800000 rw- 1024 1024 4 0 ---- df 18288 0x82b800000 0x82c000000 rw- 2048 2048 4 0 --S- df 18288 0x82c000000 0x82c800000 rw- 2048 2048 4 0 ---- df 18288 0x82c800000 0x82d000000 rw- 2048 2048 4 0 ---- df 18288 0x82d000000 0x82d800000 rw- 2048 2048 4 0 ---- df 18288 0x82d800000 0x82e800000 rw- 4096 4096 4 0 ---- df 18288 0x82e800000 0x82f800000 rw- 4096 4096 4 0 ---- df 18288 0x82f800000 0x830800000 rw- 4096 4096 4 0 ---- df 18288 0x830800000 0x831800000 rw- 4096 4096 4 0 ---- df 18288 0x831800000 0x833800000 rw- 8192 8192 4 0 --S- df 18288 0x833800000 0x835800000 rw- 8192 8192 4 0 ---- df 18288 0x835800000 0x837800000 rw- 8192 8192 4 0 ---- df 18288 0x837800000 0x839800000 rw- 8192 8192 4 0 ---- df 18288 0x839800000 0x83d800000 rw- 16102 16102 4 0 ---- df That's actually the best case I've seen, with two S. Usually there are no cases of S, and sometimes just 1. The big mapping at the top holds the hash table buckets, and I've never seen an S there. The rest of them hold tuples. Looking at the output of sysctl vm.pmap before and after a run, I saw: vm.pmap.ad_emulation_superpage_promotions: 0 vm.pmap.num_superpage_accessed_emulations: 0 vm.pmap.num_accessed_emulations: 0 vm.pmap.num_dirty_emulations: 0 vm.pmap.pdpe.demotions: no change vm.pmap.pde.promotions: +20 vm.pmap.pde.p_failures: +1 vm.pmap.pde.mappings: no change vm.pmap.pde.demotions: +48 vm.pmap.pcid_save_cnt: 21392597 vm.pmap.pti: 1 vm.pmap.invpcid_works: 1 vm.pmap.pcid_enabled: 1 vm.pmap.pg_ps_enabled: 1 vm.pmap.pat_works: 1 With the attached patch, the syscalls look like this in truss in the backend that creates each shm segment: shm_open("/PostgreSQL.1721888107",O_RDWR|O_CREAT|O_EXCL,0600) = 46 (0x2e) ftruncate(46,0x400000) = 0 (0x0) mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_HASSEMAPHORE|MAP_NOSYNC|MAP_ALIGNED_SUPER,46,0x0) = 35081158656 (0x82b000000) close(46) = 0 (0x0) ... and like this in other backends that map them in: shm_open("/PostgreSQL.1214430502",O_RDWR,0600) = 46 (0x2e) fstat(46,{ mode=-rw------- ,inode=20,size=8388608,blksize=4096 }) = 0 (0x0) mmap(0x0,8388608,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_HASSEMAPHORE|MAP_NOSYNC|MAP_ALIGNED_SUPER,46,0x0) = 35110518784 (0x82cc00000) close(46) = 0 (0x0) The access pattern for the memory is as follows: 1. In the "build" phase we first initialise the bucket segment with zeroes (sequential), and then load all the tuples into the other segments (sequential) and insert them into the buckets (random, compare-and-swap). We add more segments as necessary, gradually cranking up the sizes. 2. In the "probe" phase, all access is read only. We probe the buckets (random) and follow pointers to tuples in the other segments (random). Afterwards we unmap them and shm_unlink() them, and the parallel worker processes exit. It's possibly that we'll want to recycle memory segments and worker processes in future, but I thought I'd point out that we don't do that in case it's relevant. I understand that the philosophy is not to provide explicit control over page size. That makes sense, but I'd be grateful for any tips on how to encourage super pages for this use case. Thanks, Thomas Munro ==== How to see this (assuming you have git, gmake, flex, bison, readline, curl, ccache): # grab postgres git clone https://github.com/postgres/postgres.git cd postgres # you might want to apply the attached patch to get aligned segments patch -p1 < super-aligned.patch # build ./configure --prefix=$HOME/install --with-includes=/usr/local/include --with-libs=/usr/local/lib CC="ccache cc" gmake -s -j8 gmake -s install gmake -C contrib/pg_prewarm install # create a db cluster ~/install/bin/initdb -D ~/pgdata echo "shared_buffers = '1GB'" >> ~/pgdata/postgresql.conf # you can either start (and later stop) postgres in the background with pg_ctl: ~/install/bin/pg_ctl start -D ~/pgdata # ... or just run it in the foreground and hit ^C to stop it: # ~/install/bin/postgres -D ~/pgdata # run the psql shell ~/install/bin/psql postgres # inside psql, find your backend's pid # (you can also find the parallel workers with top, but they come and go with each query) select pg_backend_pid(); # create a table and set memory size to avoid more complicated batching behaviour create table t as select generate_series(1, 8000000)::int i; analyze t; set work_mem = '1GB'; # if for some reason you want to change the number of parallel workers, try: # set max_parallel_workers_per_gather = 2; # this is quite handy for removing all disk IO from the picture create extension pg_prewarm; select pg_prewarm('t'::regclass); # run a toy parallel hash join explain analyze select count(*) from t t1 join t t2 using (i); In procstat -v you should see that it spends about half its time "building" which looks like slowly adding new mappings and touching more and more pages, and then about half of its time "probing", where there are no further changes visible in procstat -v. If your results are like mine, only after building will you see any S mappings appear, and then only rarely. --00000000000082055d0577e8a67d Content-Type: application/octet-stream; name="super-aligned.patch" Content-Disposition: attachment; filename="super-aligned.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_jn3t1p8f0 ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3N0b3JhZ2UvaXBjL2RzbV9pbXBsLmMgYi9zcmMvYmFj a2VuZC9zdG9yYWdlL2lwYy9kc21faW1wbC5jCmluZGV4IDcwZjg5OWU3NjUuLjRhZjE2MzgxZTkg MTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3N0b3JhZ2UvaXBjL2RzbV9pbXBsLmMKKysrIGIvc3Jj L2JhY2tlbmQvc3RvcmFnZS9pcGMvZHNtX2ltcGwuYwpAQCAtMzc3LDYgKzM3Nyw5IEBAIGRzbV9p bXBsX3Bvc2l4KGRzbV9vcCBvcCwgZHNtX2hhbmRsZSBoYW5kbGUsIFNpemUgcmVxdWVzdF9zaXpl LAogCiAJLyogTWFwIGl0LiAqLwogCWFkZHJlc3MgPSBtbWFwKE5VTEwsIHJlcXVlc3Rfc2l6ZSwg UFJPVF9SRUFEIHwgUFJPVF9XUklURSwKKyNpZmRlZiBNQVBfQUxJR05FRF9TVVBFUgorCQkJCSAg IE1BUF9BTElHTkVEX1NVUEVSIHwKKyNlbmRpZgogCQkJCSAgIE1BUF9TSEFSRUQgfCBNQVBfSEFT U0VNQVBIT1JFIHwgTUFQX05PU1lOQywgZmQsIDApOwogCWlmIChhZGRyZXNzID09IE1BUF9GQUlM RUQpCiAJewpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvdXRpbHMvbW1nci9kc2EuYyBiL3NyYy9i YWNrZW5kL3V0aWxzL21tZ3IvZHNhLmMKaW5kZXggM2I3ZTJhZDEyNi4uYWU1ZDNiZTI5MCAxMDA2 NDQKLS0tIGEvc3JjL2JhY2tlbmQvdXRpbHMvbW1nci9kc2EuYworKysgYi9zcmMvYmFja2VuZC91 dGlscy9tbWdyL2RzYS5jCkBAIC02Myw5ICs2MywxMCBAQAogICogVGhlIHNpemUgb2YgdGhlIGlu aXRpYWwgRFNNIHNlZ21lbnQgdGhhdCBiYWNrcyBhIGRzYV9hcmVhIGNyZWF0ZWQgYnkKICAqIGRz YV9jcmVhdGUuICBBZnRlciBjcmVhdGluZyBzb21lIG51bWJlciBvZiBzZWdtZW50cyBvZiB0aGlz IHNpemUgd2UnbGwKICAqIGRvdWJsZSB0aGlzIHNpemUsIGFuZCBzbyBvbi4gIExhcmdlciBzZWdt ZW50cyBtYXkgYmUgY3JlYXRlZCBpZiBuZWNlc3NhcnkKLSAqIHRvIHNhdGlzZnkgbGFyZ2UgcmVx dWVzdHMuCisgKiB0byBzYXRpc2Z5IGxhcmdlIHJlcXVlc3RzLiAgMk1CIGlzIGNob3NlbiB0byBt YXRjaCB0aGUgc2l6ZSBvZiBodWdlIHBhZ2VzCisgKiBvbiBjb21tb24gaGFyZHdhcmUgKHRob3Vn aCBub3QgbWFueSBPU2VzIGN1cnJlbnRseSB0cnkgdG8gdXNlIHRoZW0pLgogICovCi0jZGVmaW5l IERTQV9JTklUSUFMX1NFR01FTlRfU0laRSAoKHNpemVfdCkgKDEgKiAxMDI0ICogMTAyNCkpCisj ZGVmaW5lIERTQV9JTklUSUFMX1NFR01FTlRfU0laRSAoKHNpemVfdCkgKDIgKiAxMDI0ICogMTAy NCkpCiAKIC8qCiAgKiBIb3cgbWFueSBzZWdtZW50cyB0byBjcmVhdGUgYmVmb3JlIHdlIGRvdWJs ZSB0aGUgc2VnbWVudCBzaXplLiAgSWYgdGhpcyBpcwpAQCAtMjA5MCw5ICsyMDkxLDggQEAgbWFr ZV9uZXdfc2VnbWVudChkc2FfYXJlYSAqYXJlYSwgc2l6ZV90IHJlcXVlc3RlZF9wYWdlcykKIAkg KiB3ZSB3b3VsZG4ndCBuZWVkIHRvIHVzZSBGcmVlUGFnZU1hbmFnZXIpLgogCSAqCiAJICogV2Ug ZGVjaWRlIG9uIGEgdG90YWwgc2VnbWVudCBzaXplIGZpcnN0LCBzbyB0aGF0IHdlIHByb2R1Y2Ug dGlkeQotCSAqIHBvd2VyLW9mLXR3byBzaXplZCBzZWdtZW50cy4gIFRoaXMgaXMgYSBnb29kIHBy b3BlcnR5IHRvIGhhdmUgaWYgd2UKLQkgKiBtb3ZlIHRvIGh1Z2UgcGFnZXMgaW4gdGhlIGZ1dHVy ZS4gIFRoZW4gd2Ugd29yayBiYWNrIHRvIHRoZSBudW1iZXIgb2YKLQkgKiBwYWdlcyB3ZSBjYW4g Zml0LgorCSAqIHBvd2VyLW9mLXR3byBzaXplZCBzZWdtZW50cyB0byBhbGlnbiB3aXRoIG9wZXJh dGluZyBzeXN0ZW0gcGFnZXMuCisJICogVGhlbiB3ZSB3b3JrIGJhY2sgdG8gdGhlIG51bWJlciBv ZiBwYWdlcyB3ZSBjYW4gZml0LgogCSAqLwogCXRvdGFsX3NpemUgPSBEU0FfSU5JVElBTF9TRUdN RU5UX1NJWkUgKgogCQkoKHNpemVfdCkgMSA8PCAobmV3X2luZGV4IC8gRFNBX05VTV9TRUdNRU5U U19BVF9FQUNIX1NJWkUpKTsK --00000000000082055d0577e8a67d-- From owner-freebsd-hackers@freebsd.org Thu Oct 11 00:20:06 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 319E310C0B38 for ; Thu, 11 Oct 2018 00:20:06 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A601277F09; Thu, 11 Oct 2018 00:20:05 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w9B0Jsj3057794 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 11 Oct 2018 03:19:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w9B0Jsj3057794 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w9B0Js2e057793; Thu, 11 Oct 2018 03:19:54 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 11 Oct 2018 03:19:54 +0300 From: Konstantin Belousov To: Thomas Munro Cc: freebsd-hackers@freebsd.org, alc@freebsd.org, mjg@freebsd.org Subject: Re: PostgresSQL vs super pages Message-ID: <20181011001954.GV5335@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Oct 2018 00:20:06 -0000 On Thu, Oct 11, 2018 at 12:59:41PM +1300, Thomas Munro wrote: > shm_open("/PostgreSQL.1721888107",O_RDWR|O_CREAT|O_EXCL,0600) = 46 (0x2e) > ftruncate(46,0x400000) = 0 (0x0) Try to write zeroes instead of truncating. This should activate the fast path in the fault handler, and if the pages allocated for backing store of the shm object were from reservation, you should get superpage mapping on the first fault without promotion. > mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_HASSEMAPHORE|MAP_NOSYNC|MAP_ALIGNED_SUPER,46,0x0) > = 35081158656 (0x82b000000) > close(46) = 0 (0x0) From owner-freebsd-hackers@freebsd.org Thu Oct 11 01:01:34 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31E5510C1CD3 for ; Thu, 11 Oct 2018 01:01:34 +0000 (UTC) (envelope-from munro@penski.net) Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AB8EE79A82 for ; Thu, 11 Oct 2018 01:01:33 +0000 (UTC) (envelope-from munro@penski.net) Received: by mail-ed1-x542.google.com with SMTP id c1-v6so6650101ede.5 for ; Wed, 10 Oct 2018 18:01:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ip9-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zwp+dFBIiDDN9RarXmJpsfMWx/+z0vOLjUiO2YdChVA=; b=mM0PZ66aRL59K8SQvhZN24dEwzuBJFHdbVQBMIKqCCdEbu+94eZ2t3PW7skssyuF9Y bzgMZqgA/Nih8+cufbUAf/UZtWwp4MQUcH/mz98z7keQSvFLsazNgM6qENWXquYGQogf dFu0+vYNo83W5XYzWctdzk81jnTVU4msjlcf6U4gAfJeSMqpJEObw3CzPXVtO9X2hoO9 px2kS87jrf8/4EM2RsekZmrZozR73lcOmxbexY4RXLafm/XYzL+NlsMQjyaGL2qkgtIj iUeNrVipc0k0wBKSfbGZixtL7+56DdgKCCvnZvP4NiVw+S1usVTqhNSBQ6bsbN5aN2dp K4Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zwp+dFBIiDDN9RarXmJpsfMWx/+z0vOLjUiO2YdChVA=; b=oxcs4+hc3yURJ2wTd1ihjywFDwhjUcghFOmElZy4Vkc1zr8QDWpUPPYCMnURF4Omvr s/wfei39A3TjPgrDI5faP0qmAPLYfQatkH0JLHx5HKZ/d8U0YKRrDKfa73/ka9lU6eoj Gu2R7GWr4/GjJ7oB7oG+nOUZ1UG0NBbG0qwq4BzFMaXXbuPQeRzN+l/gc7PQ6dL5j25e uMRbOXN2lBCXS1quAG2FMzVROkoX7VEF5yjT60ZoJ0Vu/+P5whj2JFntaOIWw7FZi5e2 uJNEeAai4zY3hBPex6FqBhqclHPcP/sMMC6V9xTL4MIx014Zi8HPDqgvPsH8qwG73U1m 9upA== X-Gm-Message-State: ABuFfoio9+1abHWCgsu95uC1x+Pfj1ItFXQ+jo1HnvRSs59K77g5pWW5 evCuCiKN3soCmF+wPFIPc7s/2gn1XiFlkIP5LLFVSw== X-Google-Smtp-Source: ACcGV61N95qq1LL0eMR26aYl+TbUb3H/mtocwfNLV0s0T6cVtdznD95Oudxrl99IDspOPVOkWAQXs7BXKPC+RR0Zo+A= X-Received: by 2002:a50:9806:: with SMTP id g6-v6mr43931901edb.235.1539219692327; Wed, 10 Oct 2018 18:01:32 -0700 (PDT) MIME-Version: 1.0 References: <20181011001954.GV5335@kib.kiev.ua> In-Reply-To: <20181011001954.GV5335@kib.kiev.ua> From: Thomas Munro Date: Thu, 11 Oct 2018 14:01:20 +1300 Message-ID: Subject: Re: PostgresSQL vs super pages To: Konstantin Belousov Cc: freebsd-hackers@freebsd.org, alc@freebsd.org, mjg@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Oct 2018 01:01:34 -0000 On Thu, 11 Oct 2018 at 13:20, Konstantin Belousov wrote: > On Thu, Oct 11, 2018 at 12:59:41PM +1300, Thomas Munro wrote: > > shm_open("/PostgreSQL.1721888107",O_RDWR|O_CREAT|O_EXCL,0600) = 46 (0x2e) > > ftruncate(46,0x400000) = 0 (0x0) > Try to write zeroes instead of truncating. > This should activate the fast path in the fault handler, and if the > pages allocated for backing store of the shm object were from reservation, > you should get superpage mapping on the first fault without promotion. If you just write() to a newly shm_open()'d fd you get a return code of 0 so I assume that doesn't work. If you ftruncate() to the desired size first, then loop writing 8192 bytes of zeroes at a time, it works. But still no super pages. I tried also with a write buffer of 2MB of zeroes, but still no super pages. I tried abandoning shm_open() and instead using a mapped file, and still no super pages. From owner-freebsd-hackers@freebsd.org Thu Oct 11 11:25:34 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D047F10B722F for ; Thu, 11 Oct 2018 11:25:34 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 572689402E for ; Thu, 11 Oct 2018 11:25:34 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mailman.ysv.freebsd.org (Postfix) id 18F2210B722E; Thu, 11 Oct 2018 11:25:34 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 075C710B722C for ; Thu, 11 Oct 2018 11:25:34 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A27489402C for ; Thu, 11 Oct 2018 11:25:33 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mail-qk1-x736.google.com with SMTP id 23-v6so5137803qkh.8 for ; Thu, 11 Oct 2018 04:25:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount-org.20150623.gappssmtp.com; s=20150623; h=from:content-transfer-encoding:mime-version:date:subject:message-id :to; bh=JbYHkqJ3oANzfKwMDKrhOy7dRvEKQy/LIfGlWDbg2NU=; b=elPn4tUTIOdPKfBvrf+eGxz7YqtAGjU6u7FpUel3374SP8rqcdohik6wOPH8qEgxna CSDfB4V5YKul1itJDEgKIbARY/QjDR35dHACwm8ZziT3Rsb7c+zT0F5fPeUhlP5AOb0k nvIEWxan/+GApCKSeuYiuOmpM1HjuF40zkcpVgOuNXEJdU001EUhP3qsNMCDvJGUwvOT aB1kLlgwkVM2UUKYHzVrRu5Nm4TB3GLRJEeLeCeVQ72LxaYcFn+zPoQ2C5tsPto0muiB m4SAn3N65rjG9U2GfmKz/Cvlqwb7TbgkItoRBJImHp04KyLgz8ugHzfsZ+BeCYOzVLCv HEtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version:date :subject:message-id:to; bh=JbYHkqJ3oANzfKwMDKrhOy7dRvEKQy/LIfGlWDbg2NU=; b=gLoU9UFdElkHFF8NrQYrl4h45T+PtIw280/ejSxxNVypVJrCOl7jS9r4EFst+mvZF0 FfuJH1RiMhTAgJ9tI0he8fZpR+JydJM3g+hgRxn+d3HRCaUn6VNzRkA5WU2wOrmwVEE8 fwL1PHcJ3z99Xre9wwuUCW5LkNYHXc1vLF04r9G3H10BPxL5QsEXH6XiqvK7M/ACb8Dl ISIbAyxI0Tx3y/VFu9JFSfvQieftC/k8Cjt29uR5x96esQnqlmzGTxtnX7IpBr6S2/86 1QHARFMIEoxoNEEsEYxiiPZpxe5ZKn25qgnZ5GsdQhEBxz4gw3+hW0JUhNaPANHClQyW IegA== X-Gm-Message-State: ABuFfojBJZ3yT+vQY891hoUKwkaLrbKmf41DeLxwdAPjcnBq9rdtDN2W Admb7prmT29GB72OaQNVbng0AvqNxcA= X-Google-Smtp-Source: ACcGV61lMQ1NIWc0bc84A7TCwjgBdYrl+NyYu9EijJsjpC3eycH1qNdJc5qzbY6u3EDkYFThDU78VA== X-Received: by 2002:a37:7745:: with SMTP id s66-v6mr996131qkc.240.1539257132729; Thu, 11 Oct 2018 04:25:32 -0700 (PDT) Received: from [192.168.1.31] (ool-43522828.dyn.optonline.net. [67.82.40.40]) by smtp.gmail.com with ESMTPSA id y50-v6sm14329559qta.60.2018.10.11.04.25.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 04:25:31 -0700 (PDT) From: Mark Saad Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Date: Thu, 11 Oct 2018 07:25:30 -0400 Subject: Carp questions Message-Id: <12F8CE17-6E79-4AB5-B309-080F270F5F01@longcount.org> To: hackers@freebsd.org X-Mailer: iPhone Mail (16A366) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Oct 2018 11:25:35 -0000 All I am running into two carp questions I can=E2=80=99t easily find an answer= too . Before I go off and try to break stuff I figured i=E2=80=99de ask if a= nyone has tried this and has any pointers . Issue one : adding more user defined bits to the custom carp Mac . So this i= s not all that common but in some cases you can have two pairs of carp serve= rs with overlapping vhids that eventually cause trouble with each other. Ha= s anyone tried this ? I am ok with random-ish mac=E2=80=99s like how crossbo= w does this on illumos .=20 Issue two: one way carp fail over . For example node a and b are carp partne= rs . Node a crashes node b takes over but there is no automatic fail back wh= en node a recovers . I can=E2=80=99t find a way to do this in 11.x . I am st= umped anyone have any built into magic for this ?=20 --- Mark Saad | nonesuch@longcount.org= From owner-freebsd-hackers@freebsd.org Thu Oct 11 11:58:56 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1A35210B7EFD for ; Thu, 11 Oct 2018 11:58:56 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A79A395056 for ; Thu, 11 Oct 2018 11:58:55 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: by mailman.ysv.freebsd.org (Postfix) id 68DF110B7EFB; Thu, 11 Oct 2018 11:58:55 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5781B10B7EFA for ; Thu, 11 Oct 2018 11:58:55 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.fagskolen.gjovik.no", Issuer "Fagskolen i Gj??vik" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C629095055 for ; Thu, 11 Oct 2018 11:58:54 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTPS id w9BBwnB0017880 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 11 Oct 2018 13:58:49 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id w9BBwn9O017877 for ; Thu, 11 Oct 2018 13:58:49 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Thu, 11 Oct 2018 13:58:48 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: hackers@freebsd.org Subject: Re: Carp questions In-Reply-To: <12F8CE17-6E79-4AB5-B309-080F270F5F01@longcount.org> Message-ID: References: <12F8CE17-6E79-4AB5-B309-080F270F5F01@longcount.org> User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.fig.ol.no Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Oct 2018 11:58:56 -0000 On Thu, 11 Oct 2018 07:25-0400, Mark Saad wrote: > I am running into two carp questions I can’t easily find an answer > too . Before I go off and try to break stuff I figured i’de ask if > anyone has tried this and has any pointers . > > Issue one : adding more user defined bits to the custom carp Mac . > So this is not all that common but in some cases you can have two > pairs of carp servers with overlapping vhids that eventually cause > trouble with each other. Has anyone tried this ? I am ok with > random-ish mac’s like how crossbow does this on illumos . Maybe someone else can answer this. > Issue two: one way carp fail over . For example node a and b are > carp partners . Node a crashes node b takes over but there is no > automatic fail back when node a recovers . I can’t find a way to do > this in 11.x . I am stumped anyone have any built into magic for > this ? I run doubly DHCP servers using CARP, and I have net.inet.carp.preempt=1 in /etc/sysctl.conf on the master only. -- Trond. From owner-freebsd-hackers@freebsd.org Thu Oct 11 15:52:22 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E6B210BECBB for ; Thu, 11 Oct 2018 15:52:22 +0000 (UTC) (envelope-from stb@lassitu.de) Received: from gilb.zs64.net (gilb.zs64.net [IPv6:2a00:14b0:4200:32e0::1ea]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gilb.zs64.net", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D9BEE760EA for ; Thu, 11 Oct 2018 15:52:21 +0000 (UTC) (envelope-from stb@lassitu.de) Received: by gilb.zs64.net (Postfix, from stb@lassitu.de) id 68D271CB48D for ; Thu, 11 Oct 2018 15:52:19 +0000 (UTC) From: Stefan Bethke Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Sponsored by: in commit messages Message-Id: <4ACC0E9B-0DC1-4316-BACB-9EF6958ADFBA@lassitu.de> Date: Thu, 11 Oct 2018 17:52:17 +0200 To: freebsd-hackers@FreeBSD.org X-Mailer: Apple Mail (2.3445.5.20) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Oct 2018 15:52:22 -0000 Hi there, I just had another update to a port that I maintain committed. Thanks = for that! However, the commit message contains a Sponsored by line. I'm not = sponsored by that company, and neither is the upstream project, AFAICT. What is the significance of this line? The port update was mainly mechanical, although it included adding a = vuxml entry. At any rate, the submission contained a more specifc entry = for vuxml than what got committed. So I'm wondering what the sponsoring = actually entails. =46rom what I have observed over the years, Sponsored by has often = indicated significant contributions to important subsystems. Is this = impression misguided? Does it simply indicate that the committer working = on that commit on sponsored time, no matter the amount of work? Stefan --=20 Stefan Bethke Fon +49 151 14070811 From owner-freebsd-hackers@freebsd.org Thu Oct 11 15:55:51 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85BB210BEE85 for ; Thu, 11 Oct 2018 15:55:51 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1a.eu.mailhop.org (outbound1a.eu.mailhop.org [52.58.109.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1265E762B7 for ; Thu, 11 Oct 2018 15:55:50 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-RoutePath: aGlwcGll X-MHO-User: 187feee0-cd6e-11e8-af31-edadc92cdc1a X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound1.eu.mailhop.org (Halon) with ESMTPSA id 187feee0-cd6e-11e8-af31-edadc92cdc1a; Thu, 11 Oct 2018 15:55:40 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w9BFtbac017761; Thu, 11 Oct 2018 09:55:37 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1539273337.72469.57.camel@freebsd.org> Subject: Re: Sponsored by: in commit messages From: Ian Lepore To: Stefan Bethke , freebsd-hackers@FreeBSD.org Date: Thu, 11 Oct 2018 09:55:37 -0600 In-Reply-To: <4ACC0E9B-0DC1-4316-BACB-9EF6958ADFBA@lassitu.de> References: <4ACC0E9B-0DC1-4316-BACB-9EF6958ADFBA@lassitu.de> Content-Type: text/plain; charset="ASCII" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Oct 2018 15:55:51 -0000 On Thu, 2018-10-11 at 17:52 +0200, Stefan Bethke wrote: > Hi there, > > I just had another update to a port that I maintain committed. Thanks > for that! > > However, the commit message contains a Sponsored by line. I'm not > sponsored by that company, and neither is the upstream project, > AFAICT. > > What is the significance of this line? > > The port update was mainly mechanical, although it included adding a > vuxml entry. At any rate, the submission contained a more specifc > entry for vuxml than what got committed. So I'm wondering what the > sponsoring actually entails. > > From what I have observed over the years, Sponsored by has often > indicated significant contributions to important subsystems. Is this > impression misguided? Does it simply indicate that the committer > working on that commit on sponsored time, no matter the amount of > work? > > > Stefan > My guess would be that the committer does much of their work for a company or organization and has customized a local commit-message template to include the sponsor line automatically. -- Ian From owner-freebsd-hackers@freebsd.org Sat Oct 13 15:24:35 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B0C410CD658 for ; Sat, 13 Oct 2018 15:24:35 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D79C67F00A for ; Sat, 13 Oct 2018 15:24:34 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id w9DFOR2R069289 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 13 Oct 2018 17:24:27 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id w9DFOMGp069286 for ; Sat, 13 Oct 2018 17:24:22 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Sat, 13 Oct 2018 17:24:21 +0200 (CEST) From: Wojciech Puchar To: freebsd-hackers@FreeBSD.org Subject: recursive GPT partitioning Message-ID: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Oct 2018 15:24:35 -0000 i have 2 SSDs in server which is GPT partitioned. Among others there are lots of partitions for virtual machines. i use glabel to label these partitions Each partition is geli encrypted andthen partitioned MBR or GPT way depending on many things. for geli encrypted partition then subdivided with MBR i can directly access (of course if bhyve is not running on it) it with for example /dev/label/partitionname.elis1 - to access first subpartition of geli encrypted "partitionname" labeled partition. but with GPT subpartitioned partition it doesn't work. i need to use ggatel to access subpartition. can it be changed? From owner-freebsd-hackers@freebsd.org Sat Oct 13 15:40:08 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68A0710CDF24 for ; Sat, 13 Oct 2018 15:40:08 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 0D4BA7F84D for ; Sat, 13 Oct 2018 15:40:07 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:5d37:62c7:7926:ca83]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 5C3D7750 for ; Sat, 13 Oct 2018 18:40:06 +0300 (MSK) Date: Sat, 13 Oct 2018 18:40:06 +0300 From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD Message-ID: <1551957390.20181013184006@serebryakov.spb.ru> To: freebsd-hackers@FreeBSD.org Subject: What are ck_queue.h guarantees? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Oct 2018 15:40:08 -0000 Hello Freebsd-hackers, Concurrency Kit documentation says: ==== ck_queue is a queue.h-compatible implementation of many-reader-single- writer queues. It allows for safe concurrent iteration, peeking and read- side access in the presence of a single concurrent writer without any usage of locks. ==== But in all places at kernel I peeked, CK_XXXX macros are protected by locks. Yes, even read ones. For example, there are a lot of addresses enumeration under locks in networks drivers, like this: if_maddr_rlock(ifp); CK_STAILQ_FOREACH(inm, &ifp->if_multiaddrs, ifma_link) { if (inm->ifma_addr->sa_family != AF_LINK) continue; crc = ether_crc32_le(LLADDR((struct sockaddr_dl *) inm->ifma_addr), ETHER_ADDR_LEN); /* Just want the 6 most significant bits. */ crc >>= 26; /* Set the corresponding bit in the filter. */ hash[crc >> 4] |= 1 << (crc & 0xf); } if_maddr_runlock(ifp); Why is it so? Why do we bother to use CK_XXX API (which adds all needed barriers and uses CASes) if all accesses are protected by locks, anyway? -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-hackers@freebsd.org Sat Oct 13 15:47:30 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C107F10CE363 for ; Sat, 13 Oct 2018 15:47:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 440897FD77 for ; Sat, 13 Oct 2018 15:47:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io1-xd36.google.com with SMTP id l25-v6so11333893ioj.0 for ; Sat, 13 Oct 2018 08:47:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=TidZ04wsHSDyTb8XvVuEM+lnk6LupR7u7hZaIP+EAhI=; b=Wtn4g/hqOI2dE3J2Hku5ktS0AWJfIh1JOp20e5kowojHxnVCkpJWBWOwP5yPLWVzGA 5nizOLRi8pWPg2L+lSu1OlzS9aNE4UWA3e0IDCF52WviYtnI6Cp0yUW1tow8ZxfqVk6d 7hi+Ecoi9whdFK2H7CAwcJlF/ODcsD654GHMRdNMTzkJA9G+x4nwbH6zV+8/MUHtf5uw w7TB8ebcEIYqpc2TQsXjY21rPJP2P1ScMsVoXTKlgilDCduKPs2Q9rYsn2dLoEkRjX5H D0ReYj7IbYESVSnvP8EZxqjXhZ/c82DAX9rsnEJIeIbv1D1uz19QQPnlFcttogFYbyI9 H6BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=TidZ04wsHSDyTb8XvVuEM+lnk6LupR7u7hZaIP+EAhI=; b=EdGQytjEr1wSsRcFe325IorbxbCSa/ewBLLw9I/4VcEaDQA3k3ua4PNujHk6WP/mla eQvt1HTACpA56Y7OcaudiPrHXNFbhj+TMzKiuSSZIsbev/rLuaABgLBueZx+zNA5j+Gx dg8WOI+O/jOp+SOA+O5rHZC5TK2veS9a1aZKomRvy3gTxVeHsCLaqxBQlsg9AKmi/8Be HG2ilfYhl05Hh9gX7ua6EdT9LmAslJ49tVhdJ7T5XhN4371CL6WEj07kO1XP7up8oPHK IvPCY/eGDQJq20hagf+kx4rFCRk4KdQBvQB01iaUKJzA39iydj29RIAkOPymb3ihGshV Px/Q== X-Gm-Message-State: ABuFfog9AFr3HsLnmMgb10hb6cpPCzEEYKypn787Aob/xcznltKn8/eM TIvzQgKSzYN4WfhXopc7mfdzYGMH72CWicsmR7Sf/DtYpOs= X-Google-Smtp-Source: ACcGV61+nrSLbz9iKdWqTxKOzFtqSQXm4csHhVRg8/jHEWHvHokWPHS3hquKttFuzdyNngfnxMoG0M9ABx1gaI7vMNo= X-Received: by 2002:a6b:3902:: with SMTP id g2-v6mr7220108ioa.168.1539445648210; Sat, 13 Oct 2018 08:47:28 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Sat, 13 Oct 2018 09:47:17 -0600 Message-ID: Subject: Re: recursive GPT partitioning To: Wojciech Puchar Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Oct 2018 15:47:30 -0000 On Sat, Oct 13, 2018 at 9:27 AM Wojciech Puchar wrote: > i have 2 SSDs in server which is GPT partitioned. Among others there are > lots of partitions for virtual machines. i use glabel to label these > partitions > > Each partition is geli encrypted andthen partitioned MBR or GPT way > depending on many things. > > for geli encrypted partition then subdivided with MBR i can directly > access (of course if bhyve is not running on it) it with for example > > /dev/label/partitionname.elis1 - to access first subpartition of geli > encrypted "partitionname" labeled partition. > > > but with GPT subpartitioned partition it doesn't work. i need to use > ggatel to access subpartition. > > can it be changed? > static int g_part_gpt_probe(struct g_part_table *table, struct g_consumer *cp) { struct g_provider *pp; u_char *buf; int error, index, pri, res; /* We don't nest, which means that our depth should be 0. */ if (table->gpt_depth != 0) return (ENXIO); is what enforces that. It would be easy enough to add an option here such that you could do: #ifndef GPT_MAX_DEPTH #define GPT_MAX_DEPTH 0 #endif ... if (table->gpt_depth > GPT_MAX_DEPTH) return ENXIO; Warner From owner-freebsd-hackers@freebsd.org Sat Oct 13 23:50:37 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76D6110D80C3 for ; Sat, 13 Oct 2018 23:50:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DC3568E6EE; Sat, 13 Oct 2018 23:50:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w9DNoLxc049887 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 14 Oct 2018 02:50:24 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w9DNoLxc049887 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w9DNoLFx049882; Sun, 14 Oct 2018 02:50:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 14 Oct 2018 02:50:21 +0300 From: Konstantin Belousov To: Thomas Munro Cc: freebsd-hackers@freebsd.org, alc@freebsd.org, mjg@freebsd.org Subject: Re: PostgresSQL vs super pages Message-ID: <20181013235021.GX5335@kib.kiev.ua> References: <20181011001954.GV5335@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Oct 2018 23:50:37 -0000 On Thu, Oct 11, 2018 at 02:01:20PM +1300, Thomas Munro wrote: > On Thu, 11 Oct 2018 at 13:20, Konstantin Belousov wrote: > > On Thu, Oct 11, 2018 at 12:59:41PM +1300, Thomas Munro wrote: > > > shm_open("/PostgreSQL.1721888107",O_RDWR|O_CREAT|O_EXCL,0600) = 46 (0x2e) > > > ftruncate(46,0x400000) = 0 (0x0) > > Try to write zeroes instead of truncating. > > This should activate the fast path in the fault handler, and if the > > pages allocated for backing store of the shm object were from reservation, > > you should get superpage mapping on the first fault without promotion. > > If you just write() to a newly shm_open()'d fd you get a return code > of 0 so I assume that doesn't work. If you ftruncate() to the desired > size first, then loop writing 8192 bytes of zeroes at a time, it > works. But still no super pages. I tried also with a write buffer of > 2MB of zeroes, but still no super pages. I tried abandoning > shm_open() and instead using a mapped file, and still no super pages. I did not quite scientific experiment, but you would need to try to find the differences between what I did and what you observe. Below is the naive test program that directly implements my suggestion, and the output from the procstat -v for it after all things were set up. /* $Id: shm_super.c,v 1.1 2018/10/13 23:49:37 kostik Exp kostik $ */ #include #include #include #include #include #include #include #include #define M(x) ((x) * 1024 * 1024) #define SZ M(4) int main(void) { char buf[128]; void *ptr; off_t cnt; int error, shmfd; shmfd = shm_open(SHM_ANON, O_CREAT | O_RDWR, 0600); if (shmfd == -1) err(1, "shm_open"); error = ftruncate(shmfd, SZ); if (error == -1) err(1, "truncate"); memset(buf, 0, sizeof(buf)); for (cnt = 0; cnt < SZ; cnt += sizeof(buf)) { error = write(shmfd, buf, sizeof(buf)); if (error == -1) err(1, "write"); else if (error != sizeof(buf)) errx(1, "short write %d", (int)error); } ptr = mmap(NULL, SZ, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ALIGNED_SUPER, shmfd, 0); if (ptr == MAP_FAILED) err(1, "mmap"); for (cnt = 0; cnt < SZ; cnt += PAGE_SIZE) *((char *)ptr + cnt) = 0; printf("ptr %p\n", ptr); snprintf(buf, sizeof(buf), "procstat -v %d", getpid()); system(buf); } $ ./shm_super ptr 0x800e00000 PID START END PRT RES PRES REF SHD FLAG TP PATH 98579 0x400000 0x401000 r-x 1 3 1 0 CN-- vn /usr/home/kostik/work/build/bsd/DEV/stuff/tests/shm_super 98579 0x600000 0x601000 rw- 1 1 1 0 ---- df 98579 0x800600000 0x800620000 r-x 32 34 146 72 CN-- vn /libexec/ld-elf.so.1 98579 0x800620000 0x800644000 rw- 24 24 1 0 ---- df 98579 0x80081f000 0x800820000 rw- 1 0 1 0 C--- vn /libexec/ld-elf.so.1 98579 0x800820000 0x800821000 rw- 1 1 1 0 ---- df 98579 0x800821000 0x8009b3000 r-x 402 440 146 72 CN-- vn /lib/libc.so.7 98579 0x8009b3000 0x800bb3000 --- 0 0 0 0 CN-- -- 98579 0x800bb3000 0x800bbf000 rw- 12 0 1 0 C--- vn /lib/libc.so.7 98579 0x800bbf000 0x800bd9000 rw- 5 14 2 0 ---- df 98579 0x800c00000 0x800e00000 rw- 9 14 2 0 ---- df 98579 0x800e00000 0x801200000 rw- 1024 1030 3 0 --S- df 98579 0x801200000 0x801400000 rw- 6 1030 3 0 ---- df 98579 0x7fffdffff000 0x7ffffffdf000 --- 0 0 0 0 ---- -- 98579 0x7ffffffdf000 0x7ffffffff000 rw- 4 4 1 0 ---D df 98579 0x7ffffffff000 0x800000000000 r-x 1 1 81 0 ---- ph