From owner-freebsd-hackers  Sun Sep 29 19:38:55 1996
Return-Path: owner-hackers
Received: (from root@localhost)
          by freefall.freebsd.org (8.7.5/8.7.3) id TAA12307
          for hackers-outgoing; Sun, 29 Sep 1996 19:38:55 -0700 (PDT)
Received: from alpo.whistle.com ([207.76.204.16])
          by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id TAA12268
          for <hackers@freebsd.org>; Sun, 29 Sep 1996 19:38:49 -0700 (PDT)
Received: from current1.whistle.com (current1.whistle.com [207.76.205.22]) by alpo.whistle.com (8.7.5/8.7.3) with SMTP id TAA12207; Sun, 29 Sep 1996 19:36:36 -0700 (PDT)
Message-ID: <324F31C6.41C67EA6@whistle.com>
Date: Sun, 29 Sep 1996 19:34:46 -0700
From: Julian Elischer <julian@whistle.com>
Organization: Whistle Communications
X-Mailer: Mozilla 3.0b6 (X11; I; FreeBSD 2.2-CURRENT i386)
MIME-Version: 1.0
To: eng@alpo.whistle.com, hackers@freebsd.org
Subject: flock/sendmail stuffup
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@freebsd.org
X-Loop: FreeBSD.org
Precedence: bulk

It's taken a couple of days to track down, but in the
implimentation we're using here of sendmail there appears to be some
strangeness.
(this is as of a few weeks ago)
I understand that this may not be relevant with the sendmail,
new version (I'll check it tomorrow or tonight)
but it appears to point to a weakness.

Sendmail was openning /var/tmp/dead.letter and locking the file
it then re-openned it or whatever, and got a second file
structure for that file which it attempted to lock
and immediatly went into deadlock with itself.

all following sendmails (sendmail -q5m) all lined up obediently waiting
for that file to come free (as it already existed)
I eventually had 85 sendmails all stopped in "lockf"

Unfortunatly it seems that whole file locks are 
feferenced to the struct file, rather than to the process itself.
so children can inherrit the  file-locks, however
it ALSO bypasses the deadlock detection if
it's not a "POSIX" type lock. this means that
it's trivial to deadlock oneself
with flock.

e.g.
#include <sys/file.h>

main()
{
        int fd1, fd2;

        fd1 = open("/tmp/xx",O_RDWR,0666);
        fd2 = open("/tmp/xx",O_RDWR,0666);
        flock(fd1,LOCK_EX);
        flock(fd2,LOCK_EX);
}


of course the quick answer is 
"Don't Do That"
but it shouldn't be possible to deadlock yourself so easily..
and if something like sendmail can be configured in such a way that it
happens then maybe it should be thought out a bit better.


(I'm not sure WHY sendmail was doing this... it olny seemed to do it
if /var/tmp/dead.letter already existed. Once I deleted it and killed
the deadlocked
sendmail, all the rest went to completion,
but if I didn't delete it, then each one would dead-lock, as I killed
the one before it..

ggest a way of keeping track of what processes
have a particular 'file' structure referenced, so that the 
deadlock detection in the POSIX case can be extended to the
'flock' case.

julian