From nobody Fri Jan 14 13:17:52 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7D8FD19595D2 for ; Fri, 14 Jan 2022 13:18:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-54.consmr.mail.gq1.yahoo.com (sonic307-54.consmr.mail.gq1.yahoo.com [98.137.64.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jb20w2kpNz4ngp for ; Fri, 14 Jan 2022 13:18:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1642166281; bh=Kog/N5JRBZhbfEIWsKfgkyYbBDfMmWdJ1UXGBD1ggIs=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=HD1hesAp1GKeCE8BruN6jTRkLf/ZZrYhgxn5YYkBd7MTJhpFHnZTjmL1x82R2iqVP/NWKReW5T9IBk1kg3qx6SrlcVygYY2ougrRsZovwxCY74EKgYygShkJOInk5WLjUmz9qGyW6f47ziJoyleU7UiZeZOi4NiOpYqcSMbLv/KdupYe66s+ONo+gzLmMK7kYe0bT2z0kY4mFp9V4BVbwDdE0ZmMs/MElif4qNhAiq1gspgAlS3iOmKsDTMHfip+eDsZHXyJk1PaAbhokpXl0ojlGqM4XIzHtZMh8H+qMQHBHb8LRx5LDgMZbq+bzr7DZ4whOYRherBXVjLIBKvzPw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1642166281; bh=vAgm7gth+Y6JXAiHtKERRxJ9ivCszyUgn/UYrl+06tJ=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Kh8h75TNqi0L8O77eTwO8o857UorcZck3I9goH3SPx1BEswaWX7ydRI6AiYHRb6RpRodwXNRv1mzgHVjRy2NKrfGqyXGQbNjHe7jDJWzaff7hkNNkx+/uxslrT7OiJzGU36hYyduvWC/bA5EH+BLtkbkwGK+rTbTCbPNLIBBJCgdAmeMMBvbLbW2SKKsDyfW+BuINPNm1iHl51jOpnBrks3oo9q3AeIjKQaP0XPyIdBD261hSVnU/5gerYaNFI4DeLi+hJLzoeXp9uGmBuWqJ9dSDRvQwAgH2hdJDFVV0w+kEwOhLhI1+DHkauXl1V1JaXKjJ1b7WsOE5KevbpzRRg== X-YMail-OSG: p4a4y7gVM1kTIhgIwiKdo9u3X1d6eCRyOmYxywpikz0Ywwmf1t3f9ufDprSgUGZ YoQBMqy6lHEcFpkjFwnhKg1yZVlxstGkVZz.CgNHJmTqcqrOHgD9TxSEw5aO_oRJpbhmcRvuwfWz 5.b.ha9ggTAz026GrobqBtGoH6KV7CYvgEvuXVdAXQQZ505pNMetnb6PQgRtZLwjRnN4y4nvERUV okhnGgscEbuGrZQNrHEpc9vMzdqsOo7JWwQkA3GjcOoYyMYz5y4cQnFSOde2LxE_C7Lv6apH4bsd 1NZ9M6RvUteQivaZIRf0sOHyUV2AIfZtMCFX66.aDUnZ1BX.hiIFVEgT.Qm9QoWLGdWBRGdCAOj3 SgM6uFR1psmu9PMTgeViVKkitX.siwi.KHXL5pejTnhz2m_VsbxSq8heLMUyRMWpUNhA500vlJYZ AXeKDN3pve0JQNGsNZvSIKh1Qn4tly5am3qbTkpTlDvwqISgUg2FORWJMKqmazi2LEXCt7dDKLEl UGARwvMhRJAjIITgStW3KXxQXk3iCcisPtEJO1XtdpdirQG_jfEeFu1guS9Rit_y6snv0nbIrzRp 7g5aXuG1sF8e_3xon_eDms0QQaw4ejlljQJ16nSwT6hnrkoq5FvHveps8dZAQsmEp7AbMsCaG4hn gihRu5FptafoEKs_WYv2CoS2Ssf5GFhmKgotsVIxsHCcVvE4YDSjCG.XOYR3gTHMLrP4A6Ku643r I5eRlfc5Q2HtzVu3VQlx1Da4dCx1mm3durvNsKWrC.fzNp3mt6VhBH00xHIOxLeMHer.AEpiNIF5 twsnaDaUrUuZDICEikkvOj9WMpCTfY0zqnv6exF.oYyCeX4WCaMYz41Ay2kFYaC.uOBS63W9.R05 KFFb7KbfGiZ9KowPmtGucbTPkhO9d8cziCrN6TDOMI.8ts2gBG35gpVcMZBi.OrBZXNtJ5c9eScu BE8asb7jv5GqXzL7FRtuhYfmNi.2BwU.9yweQR8YWo5cR1b2L_Zf72afMeZzKDvzR0WeB14Ra8Mg z2bUCN8vtA_IX.BNL2XtkQRpq1CzIMquEsyqwQR0wYBv2KH8l.Dv_UcbwmTdmzwHwbLGT.DMgXrq 1n2Jxglr3IHn.bu4BJg9QfItW.15G0lYCmwF8fPgECpVX3a3UcfoTQbU8wYZ1WCEZW9PtMJDjFhz SjSd4U4Z3uN_cdIKRCSb.BNlQs.KS9WtdQPBC6vMZSzPlxzFxV9YLWaeFjbxPoZXgsWcMrTVBKNK kmsHrZGr7tNS09XD8J6ostiYCPay03kgKbQcLwmsB1OO0dMaZofxce8lVYbqVeRInaeYUe8dI8Qu fGS7ot5bGKhZ2f_GzgCoUSdWluTrXamHca3I5LMf0G2jM1Dsbb6lZ.6mBobZgh4IY6VQAxheTzY3 sU8YBp6z5nY2ggNcAB8qCrAvGDXZmXSkCmnBLFzsHgzNaczjpvVxeg8aj9VDkwMfdK4B4v9AQP5R tFX3U21u4AZdQa8ss4_So_RjDvlXjnDvL62MInmZwl5AVGkGJKxnyHDyX8qgoyOxA5c31wh.pXbb cuZpML.JerdkQK_ZlA0cVUbjWTOyTNDzs6iKQwrAsfJ36ghVz0h5p1rMf0kvOwa8R6M_Sf8lqDlt HqBxB_FzdXGY_1QRKQLY.k6BdrhT8ocBcOJep6SaLZ8x8_LlvmhUctl.97A68UgCb8d4YWf59GrQ BkKRBXtX.oHupoHOQdWcYafoUfoP.p9iyJD_I_1FJeoscYKMk5SPPwnbsi_px8mHnAzZI65kRign H5UAfPEv3Rr.YRvrp9suNJOW3RPRFSzEk6xoPF8dTdeo8b_xW8p_Ujf6usWmBEyROGaSPWOeinYB Wv1ZH.W.TKyVCmg.BmTzD5xKTpD8sKNRZG94YICN1rp3zGdenGbU6JPU0mIiRsy3HPc.YEcOlO.x pmcnAVNTbHHScstMUYQfzPFBxhtTK0WFp5oyFTv6c4mPyjC7tOB_FDAJ.haer2izzIbW18gYPrQz gfqFFefo2ey0L2d1XF9cHmJpDPtU3xU3ywhRrdKuwys7CC1LvtUr3SaDxP1KsyJzCq40dKhtRT2g q.mNinFEzjGlTSWEyZ5RRn.KtLw8VQi5Bqz6OSf.TiWiWIlDtRSxZxq8s5u1Zl9Zaj35gkk_U7TJ vi7.6ad4O0kDFk7FJ11a.q5DlEpIj2CZroWmWo.pVO_lddcGv8.WxfKe0sWK9gnlVjOQlqjFGLtD sHqXWGlOIcVPr50QE8qQZOsS7ub._J5zSiF5ALlbj2_JQ48kLz..2982TzvN.gLOlPY2aoAcrZuE 7 X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Fri, 14 Jan 2022 13:18:01 +0000 Received: by kubenode537.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 18875033a69c27c0934cda8b0b92335c; Fri, 14 Jan 2022 13:17:55 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: An explanation of some "Container overflow" ASAN reports: libc++ does sometimes temporarily overflow the used range of a container Date: Fri, 14 Jan 2022 05:17:52 -0800 References: <22B11810-DB3F-4E81-AB40-ED207CD83EEE@yahoo.com> To: freebsd-current In-Reply-To: <22B11810-DB3F-4E81-AB40-ED207CD83EEE@yahoo.com> Message-Id: <439BD3A5-68D4-40A7-A4CD-77A05A847B18@yahoo.com> X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Jb20w2kpNz4ngp X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=HD1hesAp; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-1.94 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.56)[0.556]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.30:from]; MLMMJ_DEST(0.00)[freebsd-current]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.30:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N > On 2022-Jan-14, at 04:44, Mark Millard wrote: >=20 > Looks like libc++ does the following sort of thing > (from lldb list): >=20 > . . . > 1635=09 > 1636 template > 1637 template > 1638 void > 1639 #ifndef _LIBCPP_CXX03_LANG > (lldb)=20 > 1640 vector<_Tp, _Allocator>::__push_back_slow_path(_Up&& = __x) > 1641 #else > 1642 vector<_Tp, _Allocator>::__push_back_slow_path(_Up& __x) > 1643 #endif > 1644 { > 1645 allocator_type& __a =3D this->__alloc(); > 1646 __split_buffer = __v(__recommend(size() + 1), size(), __a); > 1647 // __v.push_back(_VSTD::forward<_Up>(__x)); > 1648 __alloc_traits::construct(__a, = _VSTD::__to_address(__v.__end_), _VSTD::forward<_Up>(__x)); > 1649 __v.__end_++; > (lldb)=20 > 1650 __swap_out_circular_buffer(__v); > 1651 } > . . . (the bt points to 1650) . . . >=20 > 1648 constructs into __v at __v.__end_ and 1649 then corrects > __v.__end_ to cause the constructed object to no longer be > an example of "Container overflow" but now in the Container. > (At least that is my interpretation.) >=20 > The compiler's code generation may move the detailed > place where __v.__end_++ happens relative to some other > of the activity but the compiler has been told an order > relative to the construction that would lead to writing > memory in the capacity of the container __v but outside > the size of the __v container at the time. >=20 > For reference: >=20 > 970 template > 971 void > 972 vector<_Tp, = _Allocator>::__swap_out_circular_buffer(__split_buffer& __v) > 973 { > 974 =09 > 975 __annotate_delete(); > 976 = _VSTD::__construct_backward_with_exception_guarantees(this->__alloc(), = this->__begin_, this->__end_, __v.__begin_); > 977 _VSTD::swap(this->__begin_, __v.__begin_); > 978 _VSTD::swap(this->__end_, __v.__end_); > 979 _VSTD::swap(this->__end_cap(), __v.__end_cap()); > (lldb)=20 > 980 __v.__first_ =3D __v.__begin_; > 981 __annotate_new(size()); > 982 __invalidate_all_iterators(); > 983 } > . . . (the bt for this points to 976) . . . >=20 >=20 > This suggests to me that using some equivalent of: >=20 > env ASAN_OPTIONS=3Ddetect_container_overflow=3D0 >=20 > my be required fairly generally when libc++ can > be involved. >=20 >=20 > Other notes . . . >=20 > I used ld -v as an example for the above via: >=20 > env ASAN_OPTIONS=3Ddetect_container_overflow=3D0 lldb ld >=20 > and used: >=20 > (lldb) env ASAN_OPTIONS=3D > (lldb) run -v >=20 > in order to have ld itself not have detect_container_overflow > disabled. >=20 > lldb suffers the libc++ Container overflow problems via its > libc++ use and fails to operate without the: >=20 > ASAN_OPTIONS=3Ddetect_container_overflow=3D0 >=20 Hmm. The code in: 772 template 773 _LIBCPP_INLINE_VISIBILITY 774 void __construct_backward_with_exception_guarantees(_Alloc& __a, = _Ptr __begin1, _Ptr __end1, _Ptr& __end2) { 775 static_assert(__is_cpp17_move_insertable<_Alloc>::value, 776 "The specified type does not meet the requirements of = Cpp17MoveInsertable"); 777 typedef allocator_traits<_Alloc> _Traits; 778 while (__end1 !=3D __begin1) { 779 _Traits::construct(__a, _VSTD::__to_address(__end2 - 1), (lldb)=20 780 #ifdef _LIBCPP_NO_EXCEPTIONS 781 _VSTD::move(*--__end1) 782 #else 783 _VSTD::move_if_noexcept(*--__end1) 784 #endif 785 ); 786 --__end2; 787 } 788 } has the same sort of problem going in the other direction. __end2 references the __v.__begin_ mentioned earlier for the context at hand. line 779 constructs just before where __v.__begin_ refers to at the time and then 786 updates __v.__begin_ to make the constructed object no longer be an example of "Container overflow" but now in the Container. This is likely the WRITE activity that is actually reported, although I've not analyzed the machine code at all. This also suggests to me that using some equivalent of: env ASAN_OPTIONS=3Ddetect_container_overflow=3D0 my be required fairly generally when libc++ can be involved. Important . . . I'll note that the construct-then-include order is tied to exception safety. I'm not claiming that the libc++ code is wrong. It just that it and ASAN are currently mismatched so ASAN by default reports libc++ as having an addressing issue for which aborting the program is the handling ASAN does by default. =3D=3D=3D Mark Millard marklmi at yahoo.com