Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 22 Aug 2001 00:29:28 -0700
From:      Terry Lambert <tlambert2@mindspring.com>
To:        Mitsuru IWASAKI <iwasaki@jp.FreeBSD.org>
Cc:        peter@wemm.org, arch@FreeBSD.ORG, audit@FreeBSD.ORG, kumabu@t3.rim.or.jp
Subject:   Re: CFR: Timing to enable CR4.PGE bit
Message-ID:  <3B835F58.68534CCE@mindspring.com>
References:  <20010809035801V.iwasaki@jp.FreeBSD.org> <20010817072149.0BCD63811@overcee.netplex.com.au> <20010822020634P.iwasaki@jp.FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Mitsuru IWASAKI wrote:
> > This part is fine.
> 
> OK, I'll commit this one first.

What does setting PGE early do for you?

I use PGE to avoid TLB shootdown on a number of memory regions
shared between user and kernel space (including zero system
call time functions), but setting it early seems wrong.

Specifically, the conceptual idea is to make a VM that looks
exactly like real memory, with the smallest relocation code
chunk possible, so that as much as possible can be done in C
code, and there's as little strangeness as possible (e.g. the
evil that is machdep.c, and the "magic" numbers in pmap.h
that have to match exactly the magic address at which the
kernel gets linked, and have to be offset exactly by the SMP
pages and other "off by one" hidden values).


> > However:
> >
> > > Also I have another thing to be confirmed.  Should we utilize TLB by
> > > enabling PGE bit at very later stage?  I think it would be more
> > > efficient to cache page entries with G flag in multi-user environment,
> > > not in kernel bootstrap.  If we enable PGE bit in locore.s, TLB could
> > > be occupied by entries which is referenced by initialization code
> > > (yes, most of them are executed only once).
> > > # but I could be wrong...

PGE might be useful for shared libraries.  It's set on the
kernel itself, which means that trapping to kernel mode does
not end up costing unnecessary overhead.  It's kind of ugly,
when the 4M page is set on the kernel, which loses the page
table page for the 4k pages (yuck), and it's not nice for
the case where the kernel gets larger than 4M.

From a practical point of view, the hassle of having to set
and unset a bit in CR3 to cause the TLB shootdown to occur
is not really worth setting the PGE bit so early that you do
not have most of the PTE's set up.


> > The G bit does not "lock" the TLB entries in.  All it does is stop
> > unnecessary flushes when %cr3 is changed.  If entries are not used
> > for a short while, they will be recycled when the TLB slot is needed
> > for something else soon enough.  ie: this should not be a problem.

It also stops necessary ones, unless you bounce it off, hit
CR3, and bounce it back on... that's the strange code around
the 4M page enable code.

> My point is that users need higher system performance in multi-user
> environment rather than in kernel bootstrap.  Also PGE bit has effects
> in multi-user environment where %cr3 is changed frequently.
> I think enabling PGE in early stage of kernel bootstrap won't give us
> performance advantages, entries which is used in bootstrap will remain
> in the TLB as Intel's document says;
> ----
> 3.7. TRANSLATION LOOKASIDE BUFFERS (TLBS)
> [snip]
> When the processor loads a page-directory or page-table entry for a
> global page into a TLB, the entry will remain in the TLB indefinitely.
> The only way to deterministically invalidate global page entries is to
> clear the PGE flag and then invalidate the TLBs or to use the INVLPG
> instruction to invalidate individual page-directory or page-table
> entries in the TLBs.
> ----

The INVLPG doesn't work exactly like you think it should, with
PGE on, on more recent processors, unfortunately.


> According to i386/locore.s, it seems that PTEs for kernel text, data,
> bss and symbols have PG_G bit, I worry that it is enough many to fill
> TLB slot out...

The kernel is in a 4M page in most cases, so it's not an
issue in most cases.  It's really very important that you
not have to flush in the case of a kernel entry (interrupt,
system call, etc.), since it _will_ make a protection domain
crossing significantly more expensive.

Also, note that the 4M pages are in a seperate 8 entry conflict
domain, and aren't in the same 16 entry data or 16 entry
instruction TLB's, on every processor where they are supported,
so the kernel is not competing with user space code anyway.

NB: 4M pages only make sense in certain specific limited
situations... using up 4M chunks of KVA space is generally a
bad idea, unless the objects you are using them for are really
4M or larger in size.  This is particularly true on 4G machines,
where you really don't have any sparseness to burn on unused
pages, and can't afford to use the remainder space without the
same mapping you used for the rest of it (e.g. for libc.so, a
copy-on-write page that is also executable, unless you split
the code and data across the page boundary).

-- Terry

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-audit" in the body of the message




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3B835F58.68534CCE>