From owner-svn-src-all@freebsd.org Mon Jul 6 08:50:16 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 274BD360361; Mon, 6 Jul 2020 08:50:16 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B0fQw0DsGz3Twh; Mon, 6 Jul 2020 08:50:16 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C6FE51BAAC; Mon, 6 Jul 2020 08:50:15 +0000 (UTC) (envelope-from hselasky@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 0668oFa8098032; Mon, 6 Jul 2020 08:50:15 GMT (envelope-from hselasky@FreeBSD.org) Received: (from hselasky@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 0668oB1o098012; Mon, 6 Jul 2020 08:50:11 GMT (envelope-from hselasky@FreeBSD.org) Message-Id: <202007060850.0668oB1o098012@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: hselasky set sender to hselasky@FreeBSD.org using -f From: Hans Petter Selasky Date: Mon, 6 Jul 2020 08:50:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r362953 - in head/sys: dev/mlx4/mlx4_core dev/mlx4/mlx4_en dev/mlx4/mlx4_ib dev/mlx5/mlx5_core dev/mlx5/mlx5_en dev/mlx5/mlx5_fpga_tools dev/mlx5/mlx5_ib dev/mthca ofed/drivers/infiniba... X-SVN-Group: head X-SVN-Commit-Author: hselasky X-SVN-Commit-Paths: in head/sys: dev/mlx4/mlx4_core dev/mlx4/mlx4_en dev/mlx4/mlx4_ib dev/mlx5/mlx5_core dev/mlx5/mlx5_en dev/mlx5/mlx5_fpga_tools dev/mlx5/mlx5_ib dev/mthca ofed/drivers/infiniband/core ofed/drivers/infi... X-SVN-Commit-Revision: 362953 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2020 08:50:16 -0000 Author: hselasky Date: Mon Jul 6 08:50:11 2020 New Revision: 362953 URL: https://svnweb.freebsd.org/changeset/base/362953 Log: Infiniband clients must be attached and detached in a specific order in ibcore. Currently the linking order of the infiniband, IB, modules decide in which order the clients are attached and detached. For example one IB client may use resources from another IB client. This can lead to a potential deadlock at shutdown. For example if the ipoib is unregistered after the ib_multicast client is detached, then if ipoib is using multicast addresses a deadlock may happen, because ib_multicast will wait for all its resources to be freed before returning from the remove method. Fix this by using module_xxx_order() instead of module_xxx(). Differential Revision: https://reviews.freebsd.org/D23973 MFC after: 1 week Sponsored by: Mellanox Technologies Modified: head/sys/dev/mlx4/mlx4_core/mlx4_main.c head/sys/dev/mlx4/mlx4_en/mlx4_en_main.c head/sys/dev/mlx4/mlx4_ib/mlx4_ib_main.c head/sys/dev/mlx5/mlx5_core/mlx5_main.c head/sys/dev/mlx5/mlx5_en/mlx5_en_main.c head/sys/dev/mlx5/mlx5_fpga_tools/mlx5fpga_tools_main.c head/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c head/sys/dev/mthca/mthca_main.c head/sys/ofed/drivers/infiniband/core/ib_cm.c head/sys/ofed/drivers/infiniband/core/ib_cma.c head/sys/ofed/drivers/infiniband/core/ib_device.c head/sys/ofed/drivers/infiniband/core/ib_iwcm.c head/sys/ofed/drivers/infiniband/core/ib_ucm.c head/sys/ofed/drivers/infiniband/core/ib_ucma.c head/sys/ofed/drivers/infiniband/core/ib_user_mad.c head/sys/ofed/drivers/infiniband/core/ib_uverbs_main.c head/sys/ofed/drivers/infiniband/ulp/ipoib/ipoib_main.c head/sys/ofed/drivers/infiniband/util/madeye.c Modified: head/sys/dev/mlx4/mlx4_core/mlx4_main.c ============================================================================== --- head/sys/dev/mlx4/mlx4_core/mlx4_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/dev/mlx4/mlx4_core/mlx4_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -4218,8 +4218,8 @@ static void __exit mlx4_cleanup(void) destroy_workqueue(mlx4_wq); } -module_init(mlx4_init); -module_exit(mlx4_cleanup); +module_init_order(mlx4_init, SI_ORDER_FIRST); +module_exit_order(mlx4_cleanup, SI_ORDER_FIRST); static int mlx4_evhand(module_t mod, int event, void *arg) Modified: head/sys/dev/mlx4/mlx4_en/mlx4_en_main.c ============================================================================== --- head/sys/dev/mlx4/mlx4_en/mlx4_en_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/dev/mlx4/mlx4_en/mlx4_en_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -304,8 +304,8 @@ static void __exit mlx4_en_cleanup(void) mlx4_unregister_interface(&mlx4_en_interface); } -module_init(mlx4_en_init); -module_exit(mlx4_en_cleanup); +module_init_order(mlx4_en_init, SI_ORDER_SIXTH); +module_exit_order(mlx4_en_cleanup, SI_ORDER_SIXTH); static int mlx4en_evhand(module_t mod, int event, void *arg) Modified: head/sys/dev/mlx4/mlx4_ib/mlx4_ib_main.c ============================================================================== --- head/sys/dev/mlx4/mlx4_ib/mlx4_ib_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/dev/mlx4/mlx4_ib/mlx4_ib_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -3319,8 +3319,8 @@ static void __exit mlx4_ib_cleanup(void) destroy_workqueue(wq); } -module_init_order(mlx4_ib_init, SI_ORDER_THIRD); -module_exit(mlx4_ib_cleanup); +module_init_order(mlx4_ib_init, SI_ORDER_SEVENTH); +module_exit_order(mlx4_ib_cleanup, SI_ORDER_SEVENTH); static int mlx4ib_evhand(module_t mod, int event, void *arg) Modified: head/sys/dev/mlx5/mlx5_core/mlx5_main.c ============================================================================== --- head/sys/dev/mlx5/mlx5_core/mlx5_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/dev/mlx5/mlx5_core/mlx5_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -2026,5 +2026,5 @@ static void __exit cleanup(void) pci_unregister_driver(&mlx5_core_driver); } -module_init(init); -module_exit(cleanup); +module_init_order(init, SI_ORDER_FIRST); +module_exit_order(cleanup, SI_ORDER_FIRST); Modified: head/sys/dev/mlx5/mlx5_en/mlx5_en_main.c ============================================================================== --- head/sys/dev/mlx5/mlx5_en/mlx5_en_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/dev/mlx5/mlx5_en/mlx5_en_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -4723,8 +4723,8 @@ mlx5e_show_version(void __unused *arg) } SYSINIT(mlx5e_show_version, SI_SUB_DRIVERS, SI_ORDER_ANY, mlx5e_show_version, NULL); -module_init_order(mlx5e_init, SI_ORDER_THIRD); -module_exit_order(mlx5e_cleanup, SI_ORDER_THIRD); +module_init_order(mlx5e_init, SI_ORDER_SIXTH); +module_exit_order(mlx5e_cleanup, SI_ORDER_SIXTH); #if (__FreeBSD_version >= 1100000) MODULE_DEPEND(mlx5en, linuxkpi, 1, 1, 1); Modified: head/sys/dev/mlx5/mlx5_fpga_tools/mlx5fpga_tools_main.c ============================================================================== --- head/sys/dev/mlx5/mlx5_fpga_tools/mlx5fpga_tools_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/dev/mlx5/mlx5_fpga_tools/mlx5fpga_tools_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -139,5 +139,5 @@ static void __exit mlx5_fpga_tools_exit(void) mlx5_fpga_tools_char_deinit(); } -module_init(mlx5_fpga_tools_init); -module_exit(mlx5_fpga_tools_exit); +module_init_order(mlx5_fpga_tools_init, SI_ORDER_SECOND); +module_exit_order(mlx5_fpga_tools_exit, SI_ORDER_SECOND); Modified: head/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c ============================================================================== --- head/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -3399,5 +3399,5 @@ mlx5_ib_show_version(void __unused *arg) } SYSINIT(mlx5_ib_show_version, SI_SUB_DRIVERS, SI_ORDER_ANY, mlx5_ib_show_version, NULL); -module_init_order(mlx5_ib_init, SI_ORDER_THIRD); -module_exit_order(mlx5_ib_cleanup, SI_ORDER_THIRD); +module_init_order(mlx5_ib_init, SI_ORDER_SEVENTH); +module_exit_order(mlx5_ib_cleanup, SI_ORDER_SEVENTH); Modified: head/sys/dev/mthca/mthca_main.c ============================================================================== --- head/sys/dev/mthca/mthca_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/dev/mthca/mthca_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -1271,8 +1271,8 @@ static void __exit mthca_cleanup(void) mthca_catas_cleanup(); } -module_init_order(mthca_init, SI_ORDER_THIRD); -module_exit(mthca_cleanup); +module_init_order(mthca_init, SI_ORDER_SEVENTH); +module_exit_order(mthca_cleanup, SI_ORDER_SEVENTH); MODULE_VERSION(mthca, 1); MODULE_DEPEND(mthca, ibcore, 1, 1, 1); Modified: head/sys/ofed/drivers/infiniband/core/ib_cm.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_cm.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/core/ib_cm.c Mon Jul 6 08:50:11 2020 (r362953) @@ -4180,5 +4180,5 @@ static void __exit ib_cm_cleanup(void) } module_init_order(ib_cm_init, SI_ORDER_SECOND); -module_exit_order(ib_cm_cleanup, SI_ORDER_FIRST); +module_exit_order(ib_cm_cleanup, SI_ORDER_SECOND); Modified: head/sys/ofed/drivers/infiniband/core/ib_cma.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_cma.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/core/ib_cma.c Mon Jul 6 08:50:11 2020 (r362953) @@ -4619,5 +4619,5 @@ static void __exit cma_cleanup(void) destroy_workqueue(cma_wq); } -module_init(cma_init); -module_exit(cma_cleanup); +module_init_order(cma_init, SI_ORDER_FOURTH); +module_exit_order(cma_cleanup, SI_ORDER_FOURTH); Modified: head/sys/ofed/drivers/infiniband/core/ib_device.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_device.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/core/ib_device.c Mon Jul 6 08:50:11 2020 (r362953) @@ -1060,8 +1060,25 @@ static void __exit ib_core_cleanup(void) destroy_workqueue(ib_wq); } -module_init(ib_core_init); -module_exit(ib_core_cleanup); +/* + * Typical loading and unloading order values and their use: + * + * SI_ORDER_FIRST (default for module_init): + * Core modules (PCI, infiniband) + * SI_ORDER_SECOND (default for module_exit): + * Infiniband core modules (CM) + * SI_ORDER_THIRD: + * SI_ORDER_FOURTH: + * Infiniband core modules (CMA) + * SI_ORDER_FIFTH: + * Infiniband user-space modules (UCM,UCMA,UMAD,UVERBS,IPOIB) + * SI_ORDER_SIXTH: + * Network HW driver modules + * SI_ORDER_SEVENTH: + * Infiniband HW driver modules + */ +module_init_order(ib_core_init, SI_ORDER_FIRST); +module_exit_order(ib_core_cleanup, SI_ORDER_FIRST); MODULE_VERSION(ibcore, 1); MODULE_DEPEND(ibcore, linuxkpi, 1, 1, 1); Modified: head/sys/ofed/drivers/infiniband/core/ib_iwcm.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_iwcm.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/core/ib_iwcm.c Mon Jul 6 08:50:11 2020 (r362953) @@ -1052,5 +1052,5 @@ static void __exit iw_cm_cleanup(void) destroy_workqueue(iwcm_wq); } -module_init(iw_cm_init); -module_exit(iw_cm_cleanup); +module_init_order(iw_cm_init, SI_ORDER_FIRST); +module_exit_order(iw_cm_cleanup, SI_ORDER_FIRST); Modified: head/sys/ofed/drivers/infiniband/core/ib_ucm.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_ucm.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/core/ib_ucm.c Mon Jul 6 08:50:11 2020 (r362953) @@ -1371,5 +1371,5 @@ static void __exit ib_ucm_cleanup(void) idr_destroy(&ctx_id_table); } -module_init_order(ib_ucm_init, SI_ORDER_THIRD); -module_exit(ib_ucm_cleanup); +module_init_order(ib_ucm_init, SI_ORDER_FIFTH); +module_exit_order(ib_ucm_cleanup, SI_ORDER_FIFTH); Modified: head/sys/ofed/drivers/infiniband/core/ib_ucma.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_ucma.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/core/ib_ucma.c Mon Jul 6 08:50:11 2020 (r362953) @@ -1782,5 +1782,5 @@ static void __exit ucma_cleanup(void) idr_destroy(&multicast_idr); } -module_init(ucma_init); -module_exit(ucma_cleanup); +module_init_order(ucma_init, SI_ORDER_FIFTH); +module_exit_order(ucma_cleanup, SI_ORDER_FIFTH); Modified: head/sys/ofed/drivers/infiniband/core/ib_user_mad.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_user_mad.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/core/ib_user_mad.c Mon Jul 6 08:50:11 2020 (r362953) @@ -1405,5 +1405,5 @@ static void __exit ib_umad_cleanup(void) unregister_chrdev_region(overflow_maj, IB_UMAD_MAX_PORTS * 2); } -module_init_order(ib_umad_init, SI_ORDER_THIRD); -module_exit(ib_umad_cleanup); +module_init_order(ib_umad_init, SI_ORDER_FIFTH); +module_exit_order(ib_umad_cleanup, SI_ORDER_FIFTH); Modified: head/sys/ofed/drivers/infiniband/core/ib_uverbs_main.c ============================================================================== --- head/sys/ofed/drivers/infiniband/core/ib_uverbs_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/core/ib_uverbs_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -1431,5 +1431,5 @@ static void __exit ib_uverbs_cleanup(void) idr_destroy(&ib_uverbs_srq_idr); } -module_init_order(ib_uverbs_init, SI_ORDER_THIRD); -module_exit(ib_uverbs_cleanup); +module_init_order(ib_uverbs_init, SI_ORDER_FIFTH); +module_exit_order(ib_uverbs_cleanup, SI_ORDER_FIFTH); Modified: head/sys/ofed/drivers/infiniband/ulp/ipoib/ipoib_main.c ============================================================================== --- head/sys/ofed/drivers/infiniband/ulp/ipoib/ipoib_main.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/ulp/ipoib/ipoib_main.c Mon Jul 6 08:50:11 2020 (r362953) @@ -1753,8 +1753,8 @@ ipoib_resolvemulti(struct ifnet *ifp, struct sockaddr } } -module_init(ipoib_init_module); -module_exit(ipoib_cleanup_module); +module_init_order(ipoib_init_module, SI_ORDER_FIFTH); +module_exit_order(ipoib_cleanup_module, SI_ORDER_FIFTH); static int ipoib_evhand(module_t mod, int event, void *arg) Modified: head/sys/ofed/drivers/infiniband/util/madeye.c ============================================================================== --- head/sys/ofed/drivers/infiniband/util/madeye.c Mon Jul 6 08:36:14 2020 (r362952) +++ head/sys/ofed/drivers/infiniband/util/madeye.c Mon Jul 6 08:50:11 2020 (r362953) @@ -594,5 +594,5 @@ static void __exit ib_madeye_cleanup(void) ib_unregister_client(&madeye_client); } -module_init(ib_madeye_init); -module_exit(ib_madeye_cleanup); +module_init_order(ib_madeye_init, SI_ORDER_THIRD); +module_exit_order(ib_madeye_cleanup, SI_ORDER_THIRD);