Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 21 May 2021 10:53:51 -0700
From:      Cy Schubert <Cy.Schubert@cschubert.com>
To:        Marcin Wojtas <mw@semihalf.com>
Cc:        Cy Schubert <Cy.Schubert@cschubert.com>, Jessica Clarke <jrtc27@freebsd.org>, shawn.webb@hardenedbsd.org, Marcin Wojtas <mw@freebsd.org>, "src-committers@freebsd.org" <src-committers@freebsd.org>, "dev-commits-src-all@freebsd.org" <dev-commits-src-all@freebsd.org>, "dev-commits-src-main@freebsd.org" <dev-commits-src-main@freebsd.org>, =?UTF-8?Q?Dawid_G=C3=B3recki?= <dgr@semihalf.com>
Subject:   Re: git: af949c590bd8 - main - Disable stack gap for ntpd during  build.
Message-ID:  <202105211753.14LHrpAN004663@slippy.cwsent.com>
In-Reply-To: <CAPv3WKe4O--Jne20ozpMfLe3XvyPZXawUx%2BLgvOF8bsDEVsa7g@mail.gmail.com>
References:  <202105211334.14LDYqoa004343@gitrepo.freebsd.org>  <04F25FD0-7863-4AC1-A257-EF0F1EB90659@freebsd.org>  <CAPv3WKeV1Oz8Gbv0LBFD03J6k3k%2B2XMBEvi28DuMM8LVq8cjrQ@mail.gmail.com>  <02078965-24BE-4F23-92D5-5E8E54A0C3E7@freebsd.org>  <202105211446.14LEk8kZ009266@slippy.cwsent.com> <CAPv3WKe4O--Jne20ozpMfLe3XvyPZXawUx%2BLgvOF8bsDEVsa7g@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
In message <CAPv3WKe4O--Jne20ozpMfLe3XvyPZXawUx+LgvOF8bsDEVsa7g@mail.gmail.c
om>
, Marcin Wojtas writes:
> Hi Cy,
>
> pt., 21 maj 2021 o 16:46 Cy Schubert <Cy.Schubert@cschubert.com> napisał(a):
> >
> > In message <02078965-24BE-4F23-92D5-5E8E54A0C3E7@freebsd.org>, Jessica
> > Clarke w
> > rites:
> > > > On 21 May 2021, at 15:11, Marcin Wojtas <mw@semihalf.com> wrote:
> > > >
> > > > Hi Jess
> > > >
> > > > pt., 21 maj 2021 o 15:39 Jessica Clarke <jrtc27@freebsd.org> napisał(a
> ):
> > > >>
> > > >> On 21 May 2021, at 14:34, Marcin Wojtas <mw@FreeBSD.org> wrote:
> > > >>>
> > > >>> The branch main has been updated by mw:
> > > >>>
> > > >>> URL: https://cgit.FreeBSD.org/src/commit/?id=af949c590bd8a00a5973b587
> 5d7e
> > > 0fa6832ea64a
> > > >>>
> > > >>> commit af949c590bd8a00a5973b5875d7e0fa6832ea64a
> > > >>> Author:     Marcin Wojtas <mw@FreeBSD.org>
> > > >>> AuthorDate: 2021-05-21 09:29:22 +0000
> > > >>> Commit:     Marcin Wojtas <mw@FreeBSD.org>
> > > >>> CommitDate: 2021-05-21 13:33:06 +0000
> > > >>>
> > > >>>   Disable stack gap for ntpd during build.
> > > >>>
> > > >>>   When starting, ntpd calls setrlimit(2) to limit maximum size of its
> > > >>>   stack. The stack limit chosen by ntpd is 200K, so when stack gap
> > > >>>   is enabled, the stack gap is larger than this limit, which results
> > > >>>   in ntpd crashing.
> > > >>
> > > >> Isn’t the bug that the unusable gap counts as usage?
> > > >>
> > > >> Jess
> > > >>
> > > >
> > > > An alternative solution was submitted
> > > > (https://reviews.freebsd.org/D29832), so that to extend the limit for
> > > > ntpd, but eventually it was recommended to simple disable the stack
> > > > gap for it until it's fixed upstream (see the last comment in the
> > > > linked revision).
> > >
> > > That’s my point, there is nothing to “fix” upstream. NTPD uses less
>  tha
> > > n 200K
> > > of stack, thus it is perfectly reasonable for it to set its limit to that
> . Th
> > > e
> > > fact that FreeBSD decides to count an arbitrary, non-deterministic amount
>  of
> > > additional unusable virtual address space towards that limit is not its f
> ault
> > > ,
> > > but a bug in FreeBSD that needs to be fixed as it’s entirely unreasonab
> le f
> > > or
> > > applications to have to account for that.
> >
> > This latest problem is not stack gap. It is PIE.
> >
>
> I have to disagree.

We are talking cross purposes. Your examples later on in your email prove 
my point.

> ntpd does not start because of stack gap, not PIE, even though it may
> seem like PIE causes this. This is due to the fact that stack gap is
> disabled if PIE is disabled. Because of that value of sysctl
> kern.elf64.aslr.stack_gap does not matter when kern.elf64.aslr.pie_enable
> is set to 0. When pie_enabled is set to 1 and stack gap is enabled, then
> ntpd fails to start, but when pie_enabled is set to 1 and stack_gap
> is set to 0, then ntpd starts without any issue. We verified this on
> FreeBSD-CURRENT snapshot from 2021-05-20.

I verified the PIE problem on a -CURRENT as of my comments in the review. 
Enabling stack gap and disabling PIE resolved the issue. The reason for 
stack gap is not a problem is that ntpd disables stack gap at line 441 of 
ntpd.c.

Furthermore enabling stack gap and disabling PIE circumvents the problem. I 
tested this myself and left that note in the review.

Enable stack gap and disable PIE: It works. But look at line 441 of ntpd.c 
to see stack gap disabled before ntpd forks itself.

>
> The fact that this is a stack gap issue can be verified using following
> procedure:
> 1. Install FreeBSD-CURRENT snapshot from 2021-05-20 using default
> configuration.
> 2. On a newly installed system start ntpd. With default configuration
> it should start successfully.
> 3. Set sysctl kern.elf64.aslr.pie_enable=1 and start ntpd. This time ntpd
> should fail. An entry indicating that ntpd was killed because of signal
> 11 should be visible in /var/log/messages.
> 4. Set sysctl kern.elf64.aslr.stack_gap=0 and start ntpd once again. This
> time ntpd should start even though pie_enable is set to 1.
>
> Exact log from the boot it was tested:
> root@freebsd-ntpd-test:~ # sysctl -a | grep aslr
> kern.elf32.aslr.stack_gap: 3
> kern.elf32.aslr.honor_sbrk: 1
> kern.elf32.aslr.pie_enable: 0
> kern.elf32.aslr.enable: 0
> kern.elf64.aslr.stack_gap: 3
> kern.elf64.aslr.honor_sbrk: 1
> kern.elf64.aslr.pie_enable: 0
> kern.elf64.aslr.enable: 0
> vm.aslr_restarts: 0
> root@freebsd-ntpd-test:~ # ntpd
> root@freebsd-ntpd-test:~ # ps aux | grep ntpd
> root  826   0.0  0.2 22060 6960  -  Ss   17:38    0:00.01 ntpd
> root  828   0.0  0.1 12976 2416  0  S+   17:38    0:00.00 grep ntpd
> root@freebsd-ntpd-test:~ # killall ntpd
> root@freebsd-ntpd-test:~ # ps aux | grep ntpd
> root  831   0.0  0.1 12976 2416  0  S+   17:38    0:00.00 grep ntpd
> root@freebsd-ntpd-test:~ # sysctl kern.elf64.aslr.pie_enable=1
> kern.elf64.aslr.pie_enable: 0 -> 1

This causes the problem.

> root@freebsd-ntpd-test:~ # ntpd
> root@freebsd-ntpd-test:~ # ps aux | grep ntpd
> root  836   0.0  0.1 14128 2452  0  S+   17:39    0:00.00 grep ntpd
> root@freebsd-ntpd-test:~ # cat /var/log/messages | tail
> May 21 17:38:25 freebsd-ntpd-test ntpd[826]: ntpd exiting on signal 15
> (Terminated)
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: ntpd 4.2.8p15-a (1): Starting
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: Command line: ntpd
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]:
> ----------------------------------------------------
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: ntp-4 is maintained by
> Network Time Foundation,
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: Inc. (NTF), a non-profit
> 501(c)(3) public-benefit
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: corporation.  Support and
> training for ntp-4 are
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: available at
> https://www.nwtime.org/support
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]:
> ----------------------------------------------------
> May 21 17:39:14 freebsd-ntpd-test kernel: pid 834 (ntpd), jid 0, uid
> 0: exited on signal 11 (core dumped)
> root@freebsd-ntpd-test:~ # sysctl kern.elf64.aslr.stack_gap=0
> kern.elf64.aslr.stack_gap: 3 -> 0
> root@freebsd-ntpd-test:~ # sysctl -a | grep aslr
> kern.elf32.aslr.stack_gap: 3
> kern.elf32.aslr.honor_sbrk: 1
> kern.elf32.aslr.pie_enable: 0
> kern.elf32.aslr.enable: 0
> kern.elf64.aslr.stack_gap: 0
> kern.elf64.aslr.honor_sbrk: 1
> kern.elf64.aslr.pie_enable: 1

This is the problem.

> kern.elf64.aslr.enable: 0
> vm.aslr_restarts: 1
> root@freebsd-ntpd-test:~ # ntpd
> root@freebsd-ntpd-test:~ # ps aux | grep ntpd
> root  845   0.0  0.2 22060 6924  -  Ss   17:40    0:00.01 ntpd
> root  847   0.0  0.1 12976 2440  0  S+   17:40    0:00.00 grep ntpd
> root@freebsd-ntpd-test:~ # cat /var/log/messages | tail
> May 21 17:39:14 freebsd-ntpd-test kernel: pid 834 (ntpd), jid 0, uid
> 0: exited on signal 11 (core dumped)
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: ntpd 4.2.8p15-a (1): Starting
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: Command line: ntpd
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]:
> ----------------------------------------------------
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: ntp-4 is maintained by
> Network Time Foundation,
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: Inc. (NTF), a non-profit
> 501(c)(3) public-benefit
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: corporation.  Support and
> training for ntp-4 are
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: available at
> https://www.nwtime.org/support
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]:
> ----------------------------------------------------
> May 21 17:40:52 freebsd-ntpd-test ntpd[845]: leapsecond file
> ('/var/db/ntpd.leap-seconds.list'): stat failed: No such file or
> directory
> root@freebsd-ntpd-test:~ # killall ntpd
>
> Best regards,
> Marcin

Running on my firewall, which has had this same ASLR configuration for 
about a year.

cwfw# sysctl kern.elf64.aslr
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 1
cwfw# ps auxww | grep ntpd
ntpd    1499   0.0  0.1  22044   5776  -  Ss   09:30    0:00.28 
/usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f 
/var/db/ntp/ntpd.drift -g
root    3032   0.0  0.0  13044   2384  0  S+   10:49    0:00.00 grep ntpd
cwfw# uptime
10:49AM  up  1:20, 1 user, load averages: 1.06, 1.02, 0.97
cwfw# uname -a
FreeBSD cwfw 14.0-CURRENT FreeBSD 14.0-CURRENT #151 
komquats-n246804-af949c590bd8-dirty: Fri May 21 07:09:32 PDT 2021     
root@cwsys:/export/obj/opt/src/git-src/amd64.amd64/sys/PROD2  amd64
cwfw# 

My laptop:

slippy# sysctl kern.elf64.aslr
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 1
slippy# ps auxww | grep ntpd
ntpd  2100   0.0  0.1  22036  8600  -  Ss   09:35     0:00.27 
/usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f 
/var/db/ntp/ntpd.drift -g
root  4632   0.0  0.0  13040  2724  1  S+   10:51     0:00.00 grep ntpd
slippy# uptime
10:51AM  up  1:17, 0 users, load averages: 0.11, 0.16, 0.16
slippy# uname -a
FreeBSD slippy 14.0-CURRENT FreeBSD 14.0-CURRENT #155 
komquats-n246804-af949c590bd8-dirty: Fri May 21 07:07:22 PDT 2021     
root@cwsys:/export/obj/opt/src/git-src/amd64.amd64/sys/BREAK  amd64
slippy# 

One of my poudriere machines:

cwsys# sysctl kern.elf64.aslr
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 1
cwsys# ps auxww | grep ntpd
ntpd  4039   0.0  0.1  22040   7340  -  Ss   09:34     0:00.46 
/usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f 
/var/db/ntp/ntpd.drift -g
root  6385   0.0  0.0  13044   2712  2  S+   10:52     0:00.01 grep ntpd
cwsys# uptime
10:52AM  up  1:19, 2 users, load averages: 0.26, 0.25, 0.24
cwsys# uname -a
FreeBSD cwsys 14.0-CURRENT FreeBSD 14.0-CURRENT #155 
komquats-n246804-af949c590bd8-dirty: Fri May 21 07:07:22 PDT 2021     
root@cwsys:/export/obj/opt/src/git-src/amd64.amd64/sys/BREAK  amd64
cwsys# 

Three examples of stack gap enabled and PIE disabled. When I enable PIE, 
ntpd fails.


-- 
Cheers,
Cy Schubert <Cy.Schubert@cschubert.com>
FreeBSD UNIX:  <cy@FreeBSD.org>   Web:  https://FreeBSD.org
NTP:           <cy@nwtime.org>    Web:  https://nwtime.org

	The need of the many outweighs the greed of the few.





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202105211753.14LHrpAN004663>