From owner-freebsd-questions@FreeBSD.ORG Tue Jun 7 05:41:28 2005 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0CF1016A41C for ; Tue, 7 Jun 2005 05:41:28 +0000 (GMT) (envelope-from tedm@toybox.placo.com) Received: from mail.freebsd-corp-net-guide.com (mail.freebsd-corp-net-guide.com [65.75.192.90]) by mx1.FreeBSD.org (Postfix) with ESMTP id 88D3C43D1D for ; Tue, 7 Jun 2005 05:41:27 +0000 (GMT) (envelope-from tedm@toybox.placo.com) Received: from tedwin2k (nat-rtr.freebsd-corp-net-guide.com [65.75.197.130]) by mail.freebsd-corp-net-guide.com (8.11.1/8.11.1) with SMTP id j575gDb44175; Mon, 6 Jun 2005 22:42:13 -0700 (PDT) (envelope-from tedm@toybox.placo.com) From: "Ted Mittelstaedt" To: "Ben Hockenhull" Date: Mon, 6 Jun 2005 22:41:21 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.6604 (9.0.2911.0) In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1478 Importance: Normal Cc: freebsd-questions@freebsd.org Subject: RE: FreeBSD 5.x with > 70k users? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jun 2005 05:41:28 -0000 You need to try it and find out. Since pwd_mkdb is what is failing the db build, and it pays attention to the PW_SCAN_BIG_IDS variable, that is going to be the very first question any of the developers ask. While it is true that someone could setup UID's greater than 65535 with less than 65535 actual password entries (for sheer idiocy if nothing else) it seems obvious that in a normal situation that you would only have UID's greater than 65535 if you actually had more than 65535 entries in the password file - hence the need for the variable. Originally FreeBSD didn't use db files for passwords, that mechanism was added because - you guessed it - the need to support very large password files. Linear scans of password files greater than 3-5K entries were getting noticeably slower. The fact that the db mechanism exists at all, not to mention that variable, most definitely indicates that someone at the time wanted to support very large password files. And I'm talking in the hundreds of thousands of users. The fact that it's broke now probably means someone accidentally used the wrong data type somewhere in the code during one of the many revisions of FreeBSD. Thus I would recommend strongly that as soon as you get the preliminary testing out of the way that you file a PR since this really looks like an introduced bug IF IT BLOWS UP on the tests I outlined below. And that bug might be in a library that pwd_mkdb is using, not in the actual program, so the core definitely needs to be looking at this, not one of us. The FreeBSD password structure has already been fundamentally modified from the original ATT format to support 16 character usernames and md5 passwords, there is no reason that the 65535 limit needs to be present either. If a few apps break so what, bitch to those developers or fix them yourself. I'm assuming this is a mailserver and if so, for <150K users there is no reason that the current FreeBSD system authentication mechanism couldn't handle it with a fast CPU. Sure, if you got 200-300K users then you want to bypass the system authentication and use a SQL server - patches for that for the major open source mail apps like /bin/mail, imapd, popper, procmail and such are floating around the Internet - but in your case the only thing you should need to do for 70K users is to patch the delivery agent and the pop/imap server to use a heirarchial directory structure for the mailboxes - once again, patches for that are also floating around the Internet. What you need to do for testing IMHO is do the following: 1) Generate a test password file of 70K users with a script as such: user1::1:20::0:0:User1:/usr/home/user1:/bin/sh user2::2:20::0:0:User2:/usr/home/user2:/bin/sh user3::3:20::0:0:User3:/usr/home/user3:/bin/sh . . . user79997::79997:20::0:0:User79997:/usr/home/user79997:/bin/sh user79998::79998:20::0:0:User79998:/usr/home/user79998:/bin/sh user79999::79999:20::0:0:User79999:/usr/home/user79999:/bin/sh and feed this to the pwd_mkdb program. The reason why is that with your production password file you have NO assurance that every line in it is correct - it is possible that pwd_mkdb is blowing up because of something wrong in your password file. Remember pwd_mkdb is intended to be called by adduser, vipw and the like, and so it has few provisions for error checking since it assumes that whatever is fed to it is absolutely correctly formatted. 2) If this blows chunks then start truncating it down by halves until you find the exact number of lines that pwd_mkdb will build, and the number that it blows chunks at. The developers will need to know this to help them know what to look for. Good luck! Ted >-----Original Message----- >From: Ben Hockenhull [mailto:benh@jpj.net] >Sent: Monday, June 06, 2005 9:33 PM >To: Ted Mittelstaedt >Subject: RE: FreeBSD 5.x with > 70k users? > > >I didn't, but I'm less concerned about pwd_mkdb complaining about UIDs > >65535 than I am about the less than enlightening db build failure I get >with the entire 71k+ file. > >/etc/master.passwd rebuilds just fine with UIDs > 65535 present (but >considerably fewer than 65535 actual entries), but just >complains about it. > > >Will setting that variable fix the build failure? I got the >sense that it >just suppresses the complaining. > >Ben > >>Did you set the PW_SCAN_BIG_IDS as documented in the man page for >>pwd_mkdb? >> >>Ted >> >>>-----Original Message----- >>>From: owner-freebsd-questions@freebsd.org >>>[mailto:owner-freebsd-questions@freebsd.org]On Behalf Of Ben >Hockenhull >>>Sent: Monday, June 06, 2005 6:36 AM >>>To: freebsd-questions@freebsd.org >>>Subject: FreeBSD 5.x with > 70k users? >>> >>> >>>Hi there, >>> >>>I'm attempting to migrate an old BSDI system to FreeBSD. The >system in >>>question has about 71k users, with UIDs from about 2000 up to >>>about 70000. >>>When I import the master.passwd file (formats are the same) to the new >>>system and try to rebuild the .db files, it fails with a >pwd_mkdb: put: >>>Unknown error: 0 . >>> >>>As best I can tell, if I import a small subset of the users >(about 5k), >>>things work fine. From what I understand, FreeBSD can have >>>massive UIDs, >>>with the caveat that some applications may not like UIDs > 65535. >>> >>>I did some research and found someone reporting an identical >>>problem, but >>>didn't see that he'd found a solution. Any input, pointers, solutions >>>greatly appreciated. >>> >>>Ben >>> >>> >>>_______________________________________________ >>>freebsd-questions@freebsd.org mailing list >>>http://lists.freebsd.org/mailman/listinfo/freebsd-questions >>>To unsubscribe, send any mail to >>>"freebsd-questions-unsubscribe@freebsd.org" >>> > > > >