From nobody Wed Jun 5 10:24:34 2024 X-Original-To: dev-commits-ports-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VvNqk2cYhz5MSgT; Wed, 05 Jun 2024 10:24:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VvNqk25fgz4ZNH; Wed, 5 Jun 2024 10:24:34 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1717583074; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=rzeWFk1dLuySqa5QzmQGTxKj/uUlugz3vU7zwNkeRwI=; b=JTF+BqUBlqITo/Nq6TwB1XBRXVL/Qf52fyR3968zkkFiWm+CV6NPoRzRAhl2qDi8v5iVWo wrV9FHChuIrHmlmBYuBKfeSPw1tj3I5hKcjE5BpnWdLS7n6DRZ53DzIno5wL0ymisc80Ub SNvR2CoAsH0S1ps4MuYBUzHZvUdFUmtM3hoMcRmCJjLUFmr+9Xzno+SgCE2xRGOe4WEINj xo7gQZmNzRMpwzM9SZKWXBomTEYwuDae4PPXEWO8ged196eIQr3QupZs3rRxXevcH+I6DP xanClH4Ejzf5l+U730v9dhfI8yATbNRQV9EMRU2NULEK4FZobreEnAQ4KnkwdQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1717583074; a=rsa-sha256; cv=none; b=nCsLCgu+YK6DhEV7DhPhCY2JXRVMBIoqj4Fg0hOi2Q6/ZIh1pFJxn5Q5zb9gwlf7V03Ocg +Sz/S/GdVkwUJwV8VOQsL4OcCV5lA7VtMtDqqsxI1eKApg6a6SY/kznn/RNU51gV5Abg2s VU7Iyii2jSJ6QOONFmKL1xdRA2OEv9zbHDr/oSskZbYDhuqU0D1vJfX38H7T9WmaU1bCkT +nYGEDVL3RID2M0mLt6yHR9ymdQvJmtjBY2jO9kH2IX1CGd6xutWJF6sqxwk2cws5uB8Sq 5Cg85Qy0HqUP68ETdHgCugC2FPjB7uAb6wkaPlfUZADb4KfEg5v2WWtyqxwSHg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1717583074; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=rzeWFk1dLuySqa5QzmQGTxKj/uUlugz3vU7zwNkeRwI=; b=ynvqwzI3xzyJ2TuQqxQL9oYqfWla+CZonFM0sI+hwPEsvC6dmBu7Rd1Y+8RQZsKTMI0dEv hR1Wxkg+JVTge7WvydtkS8J2cRaWIod82RzRDU3oMny6zRH+6SF9YO427vDWPzsCBMi1IU CkFALqlwx4/ZSaYImlIHy83GPdcEZPPciM60Q51PBMVA2t9qfl25322QA1FQkqY9vxqotx UTH7o3tboOveMuKyv/52MHlxl0pOY+0xxBfFpCuoy/2cL7YC1a3OYJLJ0gQXX6Vc4ttcj7 +wkBYtIKTkIxH+Xy4gIsPhJUmBQ14fYlpIIPMvBtrRKSKqQrbTxR4Rb17bSePg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VvNqk1gqdzxyH; Wed, 5 Jun 2024 10:24:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 455AOYSA028361; Wed, 5 Jun 2024 10:24:34 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 455AOYg8028359; Wed, 5 Jun 2024 10:24:34 GMT (envelope-from git) Date: Wed, 5 Jun 2024 10:24:34 GMT Message-Id: <202406051024.455AOYg8028359@gitrepo.freebsd.org> To: ports-committers@FreeBSD.org, dev-commits-ports-all@FreeBSD.org, dev-commits-ports-branches@FreeBSD.org From: Hajimu UMEMOTO Subject: git: af362b0d7efa - 2024Q2 - mail/cyrus-imapd3[468]: vulnerability fix. List-Id: Commit messages for all branches of the ports repository List-Archive: https://lists.freebsd.org/archives/dev-commits-ports-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-ports-all@freebsd.org Sender: owner-dev-commits-ports-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ume X-Git-Repository: ports X-Git-Refname: refs/heads/2024Q2 X-Git-Reftype: branch X-Git-Commit: af362b0d7efa8dd79563f162d4e1fb0b2bdf4680 Auto-Submitted: auto-generated The branch 2024Q2 has been updated by ume: URL: https://cgit.FreeBSD.org/ports/commit/?id=af362b0d7efa8dd79563f162d4e1fb0b2bdf4680 commit af362b0d7efa8dd79563f162d4e1fb0b2bdf4680 Author: Hajimu UMEMOTO AuthorDate: 2024-06-05 10:15:09 +0000 Commit: Hajimu UMEMOTO CommitDate: 2024-06-05 10:24:16 +0000 mail/cyrus-imapd3[468]: vulnerability fix. Security: CVE-2024-34055 (cherry picked from commit 51303a9aa6d52fa38602579485f09d2d73fc39a0) --- mail/cyrus-imapd34/Makefile | 4 +- mail/cyrus-imapd34/files/v34-CVE-2024-34055.patch | 5815 +++++++++++++++++++++ mail/cyrus-imapd36/Makefile | 4 +- mail/cyrus-imapd36/files/v36-CVE-2024-34055.patch | 5348 +++++++++++++++++++ mail/cyrus-imapd38/Makefile | 4 +- mail/cyrus-imapd38/files/v38-CVE-2024-34055.patch | 5402 +++++++++++++++++++ 6 files changed, 16574 insertions(+), 3 deletions(-) diff --git a/mail/cyrus-imapd34/Makefile b/mail/cyrus-imapd34/Makefile index b3c75282e273..876d2ddb9c79 100644 --- a/mail/cyrus-imapd34/Makefile +++ b/mail/cyrus-imapd34/Makefile @@ -1,6 +1,6 @@ PORTNAME= cyrus-imapd PORTVERSION= 3.4.7 -PORTREVISION= 0 +PORTREVISION= 1 CATEGORIES= mail MASTER_SITES= https://github.com/cyrusimap/cyrus-imapd/releases/download/${PORTNAME}-${PORTVERSION}/ PKGNAMESUFFIX= ${CYRUS_IMAPD_VER} @@ -20,6 +20,8 @@ http_PKGNAMESUFFIX= ${CYRUS_IMAPD_VER}-http CYRUS_IMAPD_VER= 34 +EXTRA_PATCHES= ${FILESDIR}/v34-CVE-2024-34055.patch:-p1 + LIB_DEPENDS= libsasl2.so:security/cyrus-sasl2 \ libicuuc.so:devel/icu \ libjansson.so:devel/jansson \ diff --git a/mail/cyrus-imapd34/files/v34-CVE-2024-34055.patch b/mail/cyrus-imapd34/files/v34-CVE-2024-34055.patch new file mode 100644 index 000000000000..c1719ea49b28 --- /dev/null +++ b/mail/cyrus-imapd34/files/v34-CVE-2024-34055.patch @@ -0,0 +1,5815 @@ +From b6682068bf8c754a87f98ee59d2616d48ed756c7 Mon Sep 17 00:00:00 2001 +From: Robert Stepanek +Date: Wed, 3 Jan 2024 09:51:36 +0100 +Subject: [PATCH 01/22] SearchFuzzy.pm: do not use non-standard XSNIPPETS + command + +The XSNIPPETS and XCONVMULTISTANDARD commands in Cyrus got +deprecated, so don't keep our test using it. + +Signed-off-by: Robert Stepanek +--- + cassandane/Cassandane/Cyrus/SearchFuzzy.pm | 344 +++++++++------------ + 1 file changed, 146 insertions(+), 198 deletions(-) + +diff --git a/cassandane/Cassandane/Cyrus/SearchFuzzy.pm b/cassandane/Cassandane/Cyrus/SearchFuzzy.pm +index 1ac00dc49..dd1a369bd 100644 +--- a/cassandane/Cassandane/Cyrus/SearchFuzzy.pm ++++ b/cassandane/Cassandane/Cyrus/SearchFuzzy.pm +@@ -43,6 +43,8 @@ use warnings; + use Cwd qw(abs_path); + use DateTime; + use Data::Dumper; ++use MIME::Base64 qw(encode_base64); ++use Encode qw(decode encode); + + use lib '.'; + use base qw(Cassandane::Cyrus::TestCase); +@@ -50,10 +52,19 @@ use Cassandane::Util::Log; + + sub new + { ++ + my ($class, @args) = @_; + my $config = Cassandane::Config->default()->clone(); +- $config->set(conversations => 'on'); +- return $class->SUPER::new({ config => $config }, @args); ++ $config->set( ++ conversations => 'on', ++ httpallowcompress => 'no', ++ httpmodules => 'jmap', ++ ); ++ return $class->SUPER::new({ ++ config => $config, ++ jmap => 1, ++ services => [ 'imap', 'http' ] ++ }, @args); + } + + sub set_up +@@ -134,6 +145,55 @@ sub create_testmessages + $self->{instance}->run_command({cyrus => 1}, 'squatter'); + } + ++sub get_snippets ++{ ++ # Previous versions of this test module used XSNIPPETS to ++ # assert snippets but this command got removed from Cyrus. ++ # Use JMAP instead. ++ ++ my ($self, $folder, $uids, $filter) = @_; ++ ++ my $imap = $self->{store}->get_client(); ++ my $jmap = $self->{jmap}; ++ ++ $self->assert_not_null($jmap); ++ ++ $imap->select($folder); ++ my $res = $imap->fetch($uids, ['emailid']); ++ my %emailIdToImapUid = map { $res->{$_}{emailid}[0] => $_ } keys %$res; ++ ++ $res = $jmap->CallMethods([ ++ ['SearchSnippet/get', { ++ filter => $filter, ++ emailIds => [ keys %emailIdToImapUid ], ++ }, 'R1'], ++ ]); ++ ++ my @snippets; ++ foreach (@{$res->[0][1]{list}}) { ++ if ($_->{subject}) { ++ push(@snippets, [ ++ 0, ++ $emailIdToImapUid{$_->{emailId}}, ++ 'SUBJECT', ++ $_->{subject}, ++ ]); ++ } ++ if ($_->{preview}) { ++ push(@snippets, [ ++ 0, ++ $emailIdToImapUid{$_->{emailId}}, ++ 'BODY', ++ $_->{preview}, ++ ]); ++ } ++ } ++ ++ return { ++ snippets => [ sort { $a->[1] <=> $b->[1] } @snippets ], ++ }; ++} ++ + sub test_copy_messages + :needs_search_xapian + { +@@ -151,12 +211,13 @@ sub test_copy_messages + } + + sub test_stem_verbs +- :min_version_3_0 :needs_search_xapian ++ :min_version_3_0 :needs_search_xapian :JMAPExtensions + { + my ($self) = @_; + $self->create_testmessages(); + + my $talk = $self->{store}->get_client(); ++ $self->assert_not_null($self->{jmap}); + + xlog $self, "Select INBOX"; + my $r = $talk->select("INBOX") || die; +@@ -175,11 +236,8 @@ sub test_stem_verbs + $r = $talk->search('fuzzy', ['subject', { Quote => "runs" }]) || die; + $self->assert_num_equals(3, scalar @$r); + +- xlog $self, 'XSNIPPETS for FUZZY subject "runs"'; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'subject', { Quote => 'runs' }] +- ) || die; ++ xlog $self, 'Get snippets for FUZZY subject "runs"'; ++ $r = $self->get_snippets('INBOX', $uids, { subject => 'runs' }); + $self->assert_num_equals(3, scalar @{$r->{snippets}}); + } + +@@ -250,12 +308,8 @@ sub test_snippet_wildcard + $talk->select("INBOX") || die; + my $uidvalidity = $talk->get_response_code('uidvalidity'); + +- xlog $self, "XSNIPPETS for $term"; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'text', { Quote => "$term*" }] +- ) || die; +- xlog $self, Dumper($r); ++ xlog $self, "Get snippets for $term"; ++ $r = $self->get_snippets('INBOX', $uids, { 'text' => "$term*" }); + $self->assert_num_equals(2, scalar @{$r->{snippets}}); + } + +@@ -358,13 +412,17 @@ sub test_normalize_snippets + my ($self) = @_; + + # Set up test message with funny characters +- my $body = "foo gären советской diĝir naïve léger"; +- my @terms = split / /, $body; ++use utf8; ++ my @terms = ( "gären", "советской", "diĝir", "naïve", "léger" ); ++no utf8; ++ my $body = encode_base64(encode('UTF-8', join(' ', @terms))); ++ $body =~ s/\r?\n/\r\n/gs; + + xlog $self, "Generate and index test messages."; + my %params = ( + mime_charset => "utf-8", +- body => $body ++ mime_encoding => 'base64', ++ body => $body, + ); + $self->make_message("1", %params) || die; + +@@ -380,24 +438,20 @@ sub test_normalize_snippets + + # Assert that diacritics are matched and returned + foreach my $term (@terms) { +- xlog $self, "XSNIPPETS for FUZZY text \"$term\""; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'text', { Quote => $term }] +- ) || die; +- $self->assert_num_not_equals(index($r->{snippets}[0][3], "$term"), -1); ++ $r = $self->get_snippets('INBOX', $uids, { text => $term }); ++ $self->assert_num_not_equals(index($r->{snippets}[0][3], "$term"), -1); + } + + # Assert that search without diacritics matches + if ($self->{skipdiacrit}) { + my $term = "naive"; +- xlog $self, "XSNIPPETS for FUZZY text \"$term\""; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'text', { Quote => $term }] +- ) || die; +- $self->assert_num_not_equals(index($r->{snippets}[0][3], "naïve"), -1); ++ xlog $self, "Get snippets for FUZZY text \"$term\""; ++ $r = $self->get_snippets('INBOX', $uids, { 'text' => $term }); ++use utf8; ++ $self->assert_num_not_equals(index($r->{snippets}[0][3], "naïve"), -1); ++no utf8; + } ++ + } + + sub test_skipdiacrit +@@ -499,38 +553,23 @@ sub test_snippets_termcover + my $r = $talk->select("INBOX") || die; + my $uidvalidity = $talk->get_response_code('uidvalidity'); + my $uids = $talk->search('1:*', 'NOT', 'DELETED'); +- my $want = "favourite cereal"; ++ my $want = "favourite cereal"; + +- $r = $talk->xsnippets( [ [ 'inbox', $uidvalidity, $uids ] ], +- 'utf-8', [ +- 'fuzzy', 'text', 'favourite', +- 'fuzzy', 'text', 'cereal', +- 'fuzzy', 'text', { Quote => 'bogus gnarly' } +- ] +- ) || die; ++ $r = $self->get_snippets('INBOX', $uids, { ++ operator => 'AND', ++ conditions => [{ ++ text => 'favourite', ++ }, { ++ text => 'cereal', ++ }, { ++ text => '"bogus gnarly"' ++ }], ++ }); + $self->assert_num_not_equals(-1, index($r->{snippets}[0][3], $want)); + +- $r = $talk->xsnippets( [ [ 'inbox', $uidvalidity, $uids ] ], +- 'utf-8', [ +- 'fuzzy', 'text', 'favourite cereal' +- ] +- ) || die; +- $self->assert_num_not_equals(-1, index($r->{snippets}[0][3], $want)); +- +- # Regression - a phrase is treated as a loose term +- $r = $talk->xsnippets( [ [ 'INBOX', $uidvalidity, $uids ] ], +- 'utf-8', [ +- 'fuzzy', 'text', { Quote => 'favourite nope cereal' }, +- 'fuzzy', 'text', { Quote => 'bogus gnarly' } +- ] +- ) || die; +- $self->assert_num_not_equals(-1, index($r->{snippets}[0][3], $want)); +- +- $r = $talk->xsnippets( [ [ 'inbox', $uidvalidity, $uids ] ], +- 'utf-8', [ +- 'fuzzy', 'text', { Quote => 'favourite cereal' } +- ] +- ) || die; ++ $r = $self->get_snippets('INBOX', $uids, { ++ text => 'favourite cereal', ++ }); + $self->assert_num_not_equals(-1, index($r->{snippets}[0][3], $want)); + } + +@@ -542,18 +581,28 @@ sub test_cjk_words + + xlog $self, "Generate and index test messages."; + ++use utf8; + my $body = "明末時已經有香港地方的概念"; ++no utf8; ++ $body = encode_base64(encode('UTF-8', $body)); ++ $body =~ s/\r?\n/\r\n/gs; + my %params = ( + mime_charset => "utf-8", +- body => $body ++ mime_encoding => 'base64', ++ body => $body, + ); + $self->make_message("1", %params) || die; + + # Splits into the words: "み, 円, 月額, 申込 ++use utf8; + $body = "申込み！月額円"; ++no utf8; ++ $body = encode_base64(encode('UTF-8', $body)); ++ $body =~ s/\r?\n/\r\n/gs; + %params = ( + mime_charset => "utf-8", +- body => $body ++ mime_encoding => 'base64', ++ body => $body, + ); + $self->make_message("2", %params) || die; + +@@ -569,50 +618,45 @@ sub test_cjk_words + + my $term; + # Search for a two-character CJK word ++use utf8; + $term = "已經"; +- xlog $self, "XSNIPPETS for FUZZY text \"$term\""; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'text', { Quote => $term }] +- ) || die; +- $self->assert_num_not_equals(index($r->{snippets}[0][3], "$term"), -1); ++no utf8; ++ xlog $self, "Get snippets for FUZZY text \"$term\""; ++ $r = $self->get_snippets('INBOX', $uids, { text => $term }); ++ $self->assert_num_not_equals(index($r->{snippets}[0][3], "$term"), -1); + + # Search for the CJK words 明末 and 時, note that the + # word order is reversed to the original message ++use utf8; + $term = "時明末"; +- xlog $self, "XSNIPPETS for FUZZY text \"$term\""; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'text', { Quote => $term }] +- ) || die; ++no utf8; ++ xlog $self, "Get snippets for FUZZY text \"$term\""; ++ $r = $self->get_snippets('INBOX', $uids, { text => $term }); + $self->assert_num_equals(scalar @{$r->{snippets}}, 1); + + # Search for the partial CJK word 月 ++use utf8; + $term = "月"; +- xlog $self, "XSNIPPETS for FUZZY text \"$term\""; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'text', { Quote => $term }] +- ) || die; ++no utf8; ++ xlog $self, "Get snippets for FUZZY text \"$term\""; ++ $r = $self->get_snippets('INBOX', $uids, { text => $term }); + $self->assert_num_equals(scalar @{$r->{snippets}}, 0); + + # Search for the interleaved, partial CJK word 額申 ++use utf8; + $term = "額申"; +- xlog $self, "XSNIPPETS for FUZZY text \"$term\""; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'text', { Quote => $term }] +- ) || die; ++no utf8; ++ xlog $self, "Get snippets for FUZZY text \"$term\""; ++ $r = $self->get_snippets('INBOX', $uids, { text => $term }); + $self->assert_num_equals(scalar @{$r->{snippets}}, 0); + + # Search for three of four words: "み, 月額, 申込", + # in different order than the original. ++use utf8; + $term = "月額み申込"; +- xlog $self, "XSNIPPETS for FUZZY text \"$term\""; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'text', { Quote => $term }] +- ) || die; ++no utf8; ++ xlog $self, "Get snippets for FUZZY text \"$term\""; ++ $r = $self->get_snippets('INBOX', $uids, { text => $term }); + $self->assert_num_equals(scalar @{$r->{snippets}}, 1); + } + +@@ -805,86 +849,6 @@ sub test_xattachmentname + } + + +-sub test_xapianv2 +- :min_version_3_0 :needs_search_xapian +-{ +- my ($self) = @_; +- +- my $talk = $self->{store}->get_client(); +- +- # This is a smallish regression test to check if we break something +- # obvious by moving Xapian indexing from folder:uid to message guids. +- # +- # Apart from the tests in this module, at least also the following +- # imodules are relevant: Metadata for SORT, Thread for THREAD. +- +- xlog $self, "Generate message"; +- my $r = $self->make_message("I run", body => "Run, Forrest! Run!" ) || die; +- my $uid = $r->{attrs}->{uid}; +- +- xlog $self, "Copy message into INBOX"; +- $talk->copy($uid, "INBOX"); +- +- xlog $self, "Run squatter"; +- $self->{instance}->run_command({cyrus => 1}, 'squatter'); +- +- $r = $talk->xconvmultisort( +- [ qw(reverse arrival) ], +- [ 'conversations', position => [1,10] ], +- 'utf-8', 'fuzzy', 'text', "run", +- ); +- $self->assert_num_equals(2, scalar @{$r->{sort}[0]} - 1); +- $self->assert_num_equals(1, scalar @{$r->{sort}}); +- +- xlog $self, "Create target mailbox"; +- $talk->create("INBOX.target"); +- +- xlog $self, "Copy message into INBOX.target"; +- $talk->copy($uid, "INBOX.target"); +- +- xlog $self, "Run squatter"; +- $self->{instance}->run_command({cyrus => 1}, 'squatter'); +- +- $r = $talk->xconvmultisort( +- [ qw(reverse arrival) ], +- [ 'conversations', position => [1,10] ], +- 'utf-8', 'fuzzy', 'text', "run", +- ); +- $self->assert_num_equals(3, scalar @{$r->{sort}[0]} - 1); +- $self->assert_num_equals(1, scalar @{$r->{sort}}); +- +- xlog $self, "Generate message"; +- $self->make_message("You run", body => "A running joke" ) || die; +- +- xlog $self, "Run squatter"; +- $self->{instance}->run_command({cyrus => 1}, 'squatter'); +- +- $r = $talk->xconvmultisort( +- [ qw(reverse arrival) ], +- [ 'conversations', position => [1,10] ], +- 'utf-8', 'fuzzy', 'text', "run", +- ); +- $self->assert_num_equals(2, scalar @{$r->{sort}}); +- +- xlog $self, "SEARCH FUZZY"; +- $r = $talk->search( +- "charset", "utf-8", "fuzzy", "text", "run", +- ) || die; +- $self->assert_num_equals(3, scalar @$r); +- +- xlog $self, "Select INBOX"; +- $r = $talk->select("INBOX") || die; +- my $uidvalidity = $talk->get_response_code('uidvalidity'); +- my $uids = $talk->search('1:*', 'NOT', 'DELETED'); +- +- xlog $self, "XSNIPPETS"; +- $r = $talk->xsnippets( +- [['INBOX', $uidvalidity, $uids]], 'utf-8', +- ['fuzzy', 'body', 'run'], +- ) || die; +- $self->assert_num_equals(3, scalar @{$r->{snippets}}); +-} +- + sub test_snippets_escapehtml + :min_version_3_0 :needs_search_xapian + { +@@ -914,21 +878,15 @@ sub test_snippets_escapehtml + my $uids = $talk->search('1:*', 'NOT', 'DELETED'); + my %m; + +- $r = $talk->xsnippets( [ [ 'inbox', $uidvalidity, $uids ] ], +- 'utf-8', [ 'fuzzy', 'text', 'test1' ] +- ) || die; +- ++ $r = $self->get_snippets('INBOX', $uids, { 'text' => 'test1' }); + %m = map { lc($_->[2]) => $_->[3] } @{ $r->{snippets} }; +- $self->assert_str_equals("Test1 body with the same tag as snippets", $m{body}); +- $self->assert_str_equals("Test1 subject with an unescaped & in it", $m{subject}); +- +- $r = $talk->xsnippets( [ [ 'inbox', $uidvalidity, $uids ] ], +- 'utf-8', [ 'fuzzy', 'text', 'test2' ] +- ) || die; ++ $self->assert_str_equals("Test1 body with the same tag as snippets", $m{body}); ++ $self->assert_str_equals("Test1 subject with an unescaped & in it", $m{subject}); + ++ $r = $self->get_snippets('INBOX', $uids, { 'text' => 'test2' }); + %m = map { lc($_->[2]) => $_->[3] } @{ $r->{snippets} }; +- $self->assert_str_equals("Test2 body with a <tag/>, although it's plain text", $m{body}); +- $self->assert_str_equals("Test2 subject with a <tag> in it", $m{subject}); ++ $self->assert_str_equals("Test2 body with a <tag/>, although it's plain text", $m{body}); ++ $self->assert_str_equals("Test2 subject with a <tag> in it", $m{subject}); + } + + sub test_search_exactmatch +@@ -963,13 +921,10 @@ sub test_search_exactmatch + $self->assert_num_equals(1, scalar @$uids); + + my %m; +- $r = $talk->xsnippets( [ [ 'inbox', $uidvalidity, $uids ] ], +- 'utf-8', [ 'fuzzy', 'body', $query ] +- ) || die; +- ++ $r = $self->get_snippets('INBOX', $uids, { body => $query }); + %m = map { lc($_->[2]) => $_->[3] } @{ $r->{snippets} }; +- $self->assert(index($m{body}, "some text") != -1); +- $self->assert(index($m{body}, "some long text") == -1); ++ $self->assert(index($m{body}, "some text") != -1); ++ $self->assert(index($m{body}, "some long text") == -1); + } + + sub test_search_subjectsnippet +@@ -1004,10 +959,7 @@ sub test_search_subjectsnippet + $self->assert_num_equals(1, scalar @$uids); + + my %m; +- $r = $talk->xsnippets( [ [ 'inbox', $uidvalidity, $uids ] ], +- 'utf-8', [ 'fuzzy', 'text', $query ] +- ) || die; +- ++ $r = $self->get_snippets('INBOX', $uids, { text => $query }); + %m = map { lc($_->[2]) => $_->[3] } @{ $r->{snippets} }; + $self->assert_matches(qr/^\[plumbing\]/, $m{subject}); + } +@@ -1317,11 +1269,10 @@ sub test_detect_language + $self->assert_deep_equals([1], $uids); + + my $r = $talk->select("INBOX") || die; +- my $uidvalidity = $talk->get_response_code('uidvalidity'); +- $r = $talk->xsnippets( [ [ 'inbox', $uidvalidity, $uids ] ], +- 'utf-8', [ 'fuzzy', 'body', 'atmet' ] +- ) || die; +- $self->assert_num_not_equals(-1, index($r->{snippets}[0][3], ' Höhe atmeten.')); ++ $r = $self->get_snippets('INBOX', $uids, { body => 'atmet' }); ++use utf8; ++ $self->assert_num_not_equals(-1, index($r->{snippets}[0][3], ' Höhe atmeten.')); ++no utf8; + } + + sub test_detect_language_subject +@@ -1377,12 +1328,9 @@ sub test_detect_language_subject + $self->assert_deep_equals([1], $uids); + + my $r = $talk->select("INBOX") || die; +- my $uidvalidity = $talk->get_response_code('uidvalidity'); +- $r = $talk->xsnippets( [ [ 'inbox', $uidvalidity, $uids ] ], +- 'utf-8', [ 'fuzzy', 'subject', 'Landschaft' ] +- ) || die; ++ $r = $self->get_snippets('INBOX', $uids, { subject => 'Landschaft' }); + $self->assert_str_equals( +- 'A subject with the German word Landschaften', ++ 'A subject with the German word Landschaften', + $r->{snippets}[0][3] + ); + } +-- +2.39.2 + + +From 00aafb0fd51aaac1badc3370a250605cff4313b0 Mon Sep 17 00:00:00 2001 +From: Bron Gondwana +Date: Fri, 20 Nov 2020 11:24:58 +1100 +Subject: [PATCH 02/22] imapd: maxsize for appends + +--- + imap/imapd.c | 4 ++++ + lib/imapoptions | 4 ++++ + 2 files changed, 8 insertions(+) + +diff --git a/imap/imapd.c b/imap/imapd.c +index a617ff80c..48055ccce 100644 +--- a/imap/imapd.c ++++ b/imap/imapd.c +@@ -3829,6 +3829,8 @@ static void cmd_append(char *tag, char *name, const char *cur_name) + const char *parseerr = NULL, *url = NULL; + struct appendstage *curstage; + mbentry_t *mbentry = NULL; ++ size_t maxsize = config_getint(IMAPOPT_APPEND_MAXSIZE) * 1024; ++ if (!maxsize) maxsize = UINT32_MAX; + + memset(&appendstate, 0, sizeof(struct appendstate)); + +@@ -4004,12 +4006,14 @@ static void cmd_append(char *tag, char *name, const char *cur_name) + size = 0; + r = append_catenate(curstage->f, cur_name, &size, + &(curstage->binary), &parseerr, &url); ++ if (!r && size > maxsize) r = IMAP_MESSAGE_TOO_LARGE; + if (r) goto done; + } + else { + /* Read size from literal */ + r = getliteralsize(arg.s, c, &size, &(curstage->binary), &parseerr); + if (!r && size == 0) r = IMAP_ZERO_LENGTH_LITERAL; ++ if (!r && size > maxsize) r = IMAP_MESSAGE_TOO_LARGE; + if (r) goto done; + + /* Copy message to stage */ +diff --git a/lib/imapoptions b/lib/imapoptions +index 5cb8ef7b8..786b288fe 100644 +--- a/lib/imapoptions ++++ b/lib/imapoptions +@@ -296,6 +296,10 @@ Blank lines and lines beginning with ``#'' are ignored. + but might be useful in the meantime for supporting old clients that + do not implement the RFC 5464 IMAP METADATA extension. */ + ++{ "append_maxsize", 0, INT, "3.3.2" } ++/* The size in kilobytes of the largest message that can be appended ++ via IMAP. If zero, no limit (i.e UINT32_MAX) */ ++ + { "aps_topic", NULL, STRING, "3.0.0" } + /* Topic for Apple Push Service registration. */ + { "aps_topic_caldav", NULL, STRING, "3.0.0" } +-- +2.39.2 + + +From 02f158782578d4d99e0915c317ffe9d339180cca Mon Sep 17 00:00:00 2001 +From: Bron Gondwana +Date: Fri, 20 Nov 2020 12:54:58 +1100 +Subject: [PATCH 03/22] imapd: push the maxsize down into each parser to avoid + spooling + +--- + imap/imap_proxy.c | 7 ++++++- + imap/imap_proxy.h | 2 +- + imap/imapd.c | 42 ++++++++++++++++++------------------------ + imap/index.c | 7 ++++++- + imap/index.h | 2 +- + 5 files changed, 32 insertions(+), 28 deletions(-) + +diff --git a/imap/imap_proxy.c b/imap/imap_proxy.c +index fb585e680..2dac80455 100644 +--- a/imap/imap_proxy.c ++++ b/imap/imap_proxy.c +@@ -1207,7 +1207,7 @@ void proxy_copy(const char *tag, char *sequence, char *name, int myrights, + /* xxx end of separate proxy-only code */ + + int proxy_catenate_url(struct backend *s, struct imapurl *url, FILE *f, +- unsigned long *size, const char **parseerr) ++ size_t maxsize, unsigned long *size, const char **parseerr) + { + char mytag[128]; + int c, r = 0, found = 0; +@@ -1309,6 +1309,11 @@ int proxy_catenate_url(struct backend *s, struct imapurl *url, FILE *f, + if (c == '}') c = prot_getc(s->in); + if (c == '\r') c = prot_getc(s->in); + if (c != '\n') c = EOF; ++ if (sz > maxsize) { ++ r = IMAP_MESSAGE_TOO_LARGE; ++ eatline(s->in, c); ++ goto next_resp; ++ } + } + else if (c == 'n' || c == 'N') { + c = chomp(s->in, "il"); +diff --git a/imap/imap_proxy.h b/imap/imap_proxy.h +index aa2170960..89cb02002 100644 +--- a/imap/imap_proxy.h ++++ b/imap/imap_proxy.h +@@ -86,7 +86,7 @@ void proxy_copy(const char *tag, char *sequence, char *name, int myrights, + int usinguid, struct backend *s); + + int proxy_catenate_url(struct backend *s, struct imapurl *url, FILE *f, +- unsigned long *size, const char **parseerr); ++ size_t maxsize, unsigned long *size, const char **parseerr); + + int annotate_fetch_proxy(const char *server, const char *mbox_pat, + const strarray_t *entry_pat, +diff --git a/imap/imapd.c b/imap/imapd.c +index 48055ccce..2e55a6285 100644 +--- a/imap/imapd.c ++++ b/imap/imapd.c +@@ -3534,7 +3534,7 @@ static int isokflag(char *s, int *isseen) + } + } + +-static int getliteralsize(const char *p, int c, ++static int getliteralsize(const char *p, int c, size_t maxsize, + unsigned *size, int *binary, const char **parseerr) + + { +@@ -3573,6 +3573,9 @@ static int getliteralsize(const char *p, int c, + return IMAP_PROTOCOL_ERROR; + } + ++ if (num > maxsize) ++ return IMAP_MESSAGE_TOO_LARGE; ++ + if (!isnowait) { + /* Tell client to send the message */ + prot_printf(imapd_out, "+ go ahead\r\n"); +@@ -3584,7 +3587,7 @@ static int getliteralsize(const char *p, int c, + return 0; + } + +-static int catenate_text(FILE *f, unsigned *totalsize, int *binary, ++static int catenate_text(FILE *f, size_t maxsize, unsigned *totalsize, int *binary, + const char **parseerr) + { + int c; +@@ -3597,11 +3600,9 @@ static int catenate_text(FILE *f, unsigned *totalsize, int *binary, + c = getword(imapd_in, &arg); + + /* Read size from literal */ +- r = getliteralsize(arg.s, c, &size, binary, parseerr); ++ r = getliteralsize(arg.s, c, maxsize - *totalsize, &size, binary, parseerr); + if (r) return r; + +- if (*totalsize > UINT_MAX - size) r = IMAP_MESSAGE_TOO_LARGE; +- + /* Catenate message part to stage */ + while (size) { + n = prot_read(imapd_in, buf, size > 4096 ? 4096 : size); +@@ -3629,7 +3630,7 @@ static int catenate_text(FILE *f, unsigned *totalsize, int *binary, + } + + static int catenate_url(const char *s, const char *cur_name, FILE *f, +- unsigned *totalsize, const char **parseerr) ++ size_t maxsize, unsigned *totalsize, const char **parseerr) + { + struct imapurl url; + struct index_state *state; +@@ -3668,11 +3669,8 @@ static int catenate_url(const char *s, const char *cur_name, FILE *f, + proxy_userid, &backend_cached, + &backend_current, &backend_inbox, imapd_in); + if (be) { +- r = proxy_catenate_url(be, &url, f, &size, parseerr); +- if (*totalsize > UINT_MAX - size) +- r = IMAP_MESSAGE_TOO_LARGE; +- else +- *totalsize += size; ++ r = proxy_catenate_url(be, &url, f, maxsize - *totalsize, &size, parseerr); ++ *totalsize += size; + } + else + r = IMAP_SERVER_UNAVAILABLE; +@@ -3727,14 +3725,12 @@ static int catenate_url(const char *s, const char *cur_name, FILE *f, + struct protstream *s = prot_new(fileno(f), 1); + + r = index_urlfetch(state, msgno, 0, url.section, +- url.start_octet, url.octet_count, s, &size); ++ url.start_octet, url.octet_count, s, ++ maxsize - *totalsize, &size); + if (r == IMAP_BADURL) + *parseerr = "No such message part"; + else if (!r) { +- if (*totalsize > UINT_MAX - size) +- r = IMAP_MESSAGE_TOO_LARGE; +- else +- *totalsize += size; ++ *totalsize += size; + } + + prot_flush(s); +@@ -3751,7 +3747,7 @@ static int catenate_url(const char *s, const char *cur_name, FILE *f, + return r; + } + +-static int append_catenate(FILE *f, const char *cur_name, unsigned *totalsize, ++static int append_catenate(FILE *f, const char *cur_name, size_t maxsize, unsigned *totalsize, + int *binary, const char **parseerr, const char **url) + { + int c, r = 0; +@@ -3765,7 +3761,7 @@ static int append_catenate(FILE *f, const char *cur_name, unsigned *totalsize, + } + + if (!strcasecmp(arg.s, "TEXT")) { +- int r1 = catenate_text(f, totalsize, binary, parseerr); ++ int r1 = catenate_text(f, maxsize, totalsize, binary, parseerr); + if (r1) return r1; + + /* if we see a SP, we're trying to catenate more than one part */ +@@ -3781,7 +3777,7 @@ static int append_catenate(FILE *f, const char *cur_name, unsigned *totalsize, + } + + if (!r) { +- r = catenate_url(arg.s, cur_name, f, totalsize, parseerr); ++ r = catenate_url(arg.s, cur_name, f, maxsize, totalsize, parseerr); + if (r) { + *url = arg.s; + return r; +@@ -4004,16 +4000,14 @@ static void cmd_append(char *tag, char *name, const char *cur_name) + + /* Catenate the message part(s) to stage */ + size = 0; +- r = append_catenate(curstage->f, cur_name, &size, ++ r = append_catenate(curstage->f, cur_name, maxsize, &size, + &(curstage->binary), &parseerr, &url); +- if (!r && size > maxsize) r = IMAP_MESSAGE_TOO_LARGE; + if (r) goto done; + } + else { + /* Read size from literal */ +- r = getliteralsize(arg.s, c, &size, &(curstage->binary), &parseerr); ++ r = getliteralsize(arg.s, c, maxsize, &size, &(curstage->binary), &parseerr); + if (!r && size == 0) r = IMAP_ZERO_LENGTH_LITERAL; +- if (!r && size > maxsize) r = IMAP_MESSAGE_TOO_LARGE; + if (r) goto done; + + /* Copy message to stage */ +@@ -14010,7 +14004,7 @@ static void cmd_urlfetch(char *tag) + } else { + r = index_urlfetch(state, msgno, params, url.section, + url.start_octet, url.octet_count, +- imapd_out, NULL); ++ imapd_out, UINT32_MAX, NULL); + } + + err: +diff --git a/imap/index.c b/imap/index.c +index ef537aa55..35ca866aa 100644 +--- a/imap/index.c ++++ b/imap/index.c +@@ -4550,7 +4550,7 @@ static int index_fetchreply(struct index_state *state, uint32_t msgno, + EXPORTED int index_urlfetch(struct index_state *state, uint32_t msgno, + unsigned params, const char *section, + unsigned long start_octet, unsigned long octet_count, +- struct protstream *pout, unsigned long *outsize) ++ struct protstream *pout, size_t maxsize, unsigned long *outsize) + { + /* dumbass eM_Client sends this: + * A4 APPEND "INBOX.Junk Mail" () "14-Jul-2013 17:01:02 +0000" +@@ -4723,6 +4723,11 @@ EXPORTED int index_urlfetch(struct index_state *state, uint32_t msgno, + n = size - start_octet; + } + ++ if (n > maxsize) { ++ r = IMAP_MESSAGE_TOO_LARGE; ++ goto done; ++ } ++ + if (outsize) { + /* Return size (CATENATE) */ + *outsize = n; +diff --git a/imap/index.h b/imap/index.h +index 196607f3f..bf8006d9b 100644 +--- a/imap/index.h ++++ b/imap/index.h +@@ -303,7 +303,7 @@ extern struct seqset *index_vanished(struct index_state *state, + extern int index_urlfetch(struct index_state *state, uint32_t msgno, + unsigned params, const char *section, + unsigned long start_octet, unsigned long octet_count, +- struct protstream *pout, unsigned long *size); ++ struct protstream *pout, size_t maxsize, unsigned long *size); + extern char *index_get_msgid(struct index_state *state, uint32_t msgno); + extern struct nntp_overview *index_overview(struct index_state *state, + uint32_t msgno); +-- +2.39.2 + + +From 133a11ebfd9e3f659da3081d8e7c9f416c8ead3b Mon Sep 17 00:00:00 2001 +From: Bron Gondwana +Date: Tue, 1 Dec 2020 08:11:31 +1100 +Subject: [PATCH 04/22] use maxmessagesize rather than our own config option + +--- + imap/imapd.c | 2 +- + lib/imapoptions | 4 ---- + 2 files changed, 1 insertion(+), 5 deletions(-) + +diff --git a/imap/imapd.c b/imap/imapd.c +index 2e55a6285..d9a9dd776 100644 +--- a/imap/imapd.c ++++ b/imap/imapd.c +@@ -3825,7 +3825,7 @@ static void cmd_append(char *tag, char *name, const char *cur_name) + const char *parseerr = NULL, *url = NULL; + struct appendstage *curstage; + mbentry_t *mbentry = NULL; +- size_t maxsize = config_getint(IMAPOPT_APPEND_MAXSIZE) * 1024; ++ size_t maxsize = config_getint(IMAPOPT_MAXMESSAGESIZE) * 1024; + if (!maxsize) maxsize = UINT32_MAX; + + memset(&appendstate, 0, sizeof(struct appendstate)); +diff --git a/lib/imapoptions b/lib/imapoptions +index 786b288fe..5cb8ef7b8 100644 +--- a/lib/imapoptions ++++ b/lib/imapoptions +@@ -296,10 +296,6 @@ Blank lines and lines beginning with ``#'' are ignored. + but might be useful in the meantime for supporting old clients that + do not implement the RFC 5464 IMAP METADATA extension. */ + +-{ "append_maxsize", 0, INT, "3.3.2" } +-/* The size in kilobytes of the largest message that can be appended +- via IMAP. If zero, no limit (i.e UINT32_MAX) */ +- + { "aps_topic", NULL, STRING, "3.0.0" } + /* Topic for Apple Push Service registration. */ + { "aps_topic_caldav", NULL, STRING, "3.0.0" } +-- +2.39.2 + + +From ddc431769b61eef06550da624c1c99a2fd620dbb Mon Sep 17 00:00:00 2001 +From: ellie timoney +Date: Wed, 27 Mar 2024 11:31:58 +1100 +Subject: [PATCH 05/22] imapd: read maxmsgsize once at startup + +Based on: +40793dfde8c96797d86f80e9f461bea61bca3bc9 imapd.c: Advertise APPENDLIMIT= capability + +but without introducing the APPENDLIMIT= capability +--- + imap/imapd.c | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +diff --git a/imap/imapd.c b/imap/imapd.c +index d9a9dd776..e7cf600c7 100644 +--- a/imap/imapd.c ++++ b/imap/imapd.c +@@ -135,6 +135,7 @@ static int imaps = 0; + static sasl_ssf_t extprops_ssf = 0; + static int nosaslpasswdcheck = 0; + static int apns_enabled = 0; ++static size_t maxsize = 0; + + /* PROXY STUFF */ + /* we want a list of our outgoing connections here and which one we're +@@ -908,6 +909,9 @@ int service_init(int argc, char **argv, char **envp) + + prometheus_increment(CYRUS_IMAP_READY_LISTENERS); + ++ maxsize = config_getint(IMAPOPT_MAXMESSAGESIZE) * 1024; ++ if (!maxsize) maxsize = UINT32_MAX; ++ + return 0; + } + +@@ -3825,8 +3829,6 @@ static void cmd_append(char *tag, char *name, const char *cur_name) + const char *parseerr = NULL, *url = NULL; + struct appendstage *curstage; + mbentry_t *mbentry = NULL; +- size_t maxsize = config_getint(IMAPOPT_MAXMESSAGESIZE) * 1024; +- if (!maxsize) maxsize = UINT32_MAX; + + memset(&appendstate, 0, sizeof(struct appendstate)); + +-- +2.39.2 + + +From a32fe042bc503a36393e7d888b26b6c1759cf6b0 Mon Sep 17 00:00:00 2001 +From: Matthew Horsfall +Date: Wed, 15 Jun 2022 14:57:02 -0400 +Subject: [PATCH 06/22] imap/imapd.c: IMAPOPT_MAXMESSAGESIZE is bytes, not + kilobytes + +I think this was a mistake added in bf28aa3fb6 when replacing +IMAPOPT_APPEND_MAXSIZE. + +Signed-off-by: Matthew Horsfall +--- + imap/imapd.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/imap/imapd.c b/imap/imapd.c +index e7cf600c7..ce8c6f675 100644 *** 15683 LINES SKIPPED ***