From owner-svn-ports-all@freebsd.org Thu Feb 4 08:48:41 2016 Return-Path: Delivered-To: svn-ports-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3FAAA9B495; Thu, 4 Feb 2016 08:48:41 +0000 (UTC) (envelope-from koobs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 900741979; Thu, 4 Feb 2016 08:48:41 +0000 (UTC) (envelope-from koobs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u148megi000815; Thu, 4 Feb 2016 08:48:40 GMT (envelope-from koobs@FreeBSD.org) Received: (from koobs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u148meUa000812; Thu, 4 Feb 2016 08:48:40 GMT (envelope-from koobs@FreeBSD.org) Message-Id: <201602040848.u148meUa000812@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: koobs set sender to koobs@FreeBSD.org using -f From: Kubilay Kocak Date: Thu, 4 Feb 2016 08:48:40 +0000 (UTC) To: ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: svn commit: r408017 - in head/ports-mgmt/portscout: . files X-SVN-Group: ports-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-ports-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the ports tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2016 08:48:41 -0000 Author: koobs Date: Thu Feb 4 08:48:40 2016 New Revision: 408017 URL: https://svnweb.freebsd.org/changeset/ports/408017 Log: ports-mgmt/portscout: Loop through all PyPI files While processing Issue 206746 [1] for a security update to security/py-rsa (For versions < 3.3), it was noticed that Portscout had not identified the the newer version, released on 2016-01-13. Investigation revealed that the PyPI SiteHandler in Portscout only processed the first url/filename returned by PyPI, which in many cases is not a tar.gz, the default EXTRACT_SUFFIX for source distribution (sdist) files: [py-rsa] VersionCheck() [py-rsa] Checking site: https://pypi.python.org/packages/source/r/rsa/ Does site handler exist ... Yes (Portscout::SiteHandler::PyPI) GET https://pypi.python.org/pypi/rsa/json (Portscout::SiteHandler::PyPI) GET success: 200 Filename: rsa-3.3-py2.py3-none-any.whl FindNewest: Checking rsa-3.3-py2.py3-none-any.whl ... against port DISTFILES. FindNewest: Checking DISTFILE ... rsa-3.1.4.tar.gz (ver: 3.1.4, sufx: .tar.gz) [py-rsa] Done This change backports a commit [1] made to Portroach which adds a loop to enumerate all URLs/filenames in the PyPI JSON response, not just the first. [1] https://github.com/jasperla/portroach/commit/e93b8331f6e5f850bbb5faca866efcbf73de756c PR: 206746 [1] Obtained from: https://github.com/jasperla/portroach Modified: head/ports-mgmt/portscout/Makefile head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm Modified: head/ports-mgmt/portscout/Makefile ============================================================================== --- head/ports-mgmt/portscout/Makefile Thu Feb 4 08:35:33 2016 (r408016) +++ head/ports-mgmt/portscout/Makefile Thu Feb 4 08:48:40 2016 (r408017) @@ -3,7 +3,7 @@ PORTNAME= portscout PORTVERSION= 0.8.1 -PORTREVISION= 6 +PORTREVISION= 7 CATEGORIES= ports-mgmt MASTER_SITES= http://mirror.inerd.com/FreeBSD/distfiles/${PORTNAME}/ \ http://www.atarininja.org/~wxs/distfiles/ \ @@ -39,7 +39,7 @@ PORTDOCS= UPDATING portscout-portconfig. HTTPS_RUN_DEPENDS= p5-LWP-Protocol-https>=0:${PORTSDIR}/www/p5-LWP-Protocol-https -SQLITE_USE= SQLITE=3 +SQLITE_USES= sqlite SQLITE_RUN_DEPENDS= p5-DBD-SQLite>=0:${PORTSDIR}/databases/p5-DBD-SQLite PGSQL_USES= pgsql @@ -60,7 +60,7 @@ post-extract: @${CP} ${FILESDIR}/files-Portscout-SiteHandler-GitHub.pm ${WRKSRC}/Portscout/SiteHandler/GitHub.pm @${CP} ${FILESDIR}/files-Portscout-SiteHandler-PyPI.pm ${WRKSRC}/Portscout/SiteHandler/PyPI.pm -post-patch: +pre-install: .if ${PORT_OPTIONS:MSQLITE} @${REINPLACE_CMD} 's/^\([^#]*DBI:Pg.*\)$$/#\1/g' ${WRKSRC}/portscout.conf @${REINPLACE_CMD} 's/^#\(.*DBI:SQLite.*\)$$/\1/g' ${WRKSRC}/portscout.conf Modified: head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm ============================================================================== --- head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm Thu Feb 4 08:35:33 2016 (r408016) +++ head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm Thu Feb 4 08:48:40 2016 (r408017) @@ -109,14 +109,13 @@ sub GetFiles $ua->agent(USER_AGENT); $resp = $ua->request(HTTP::Request->new(GET => $query)); if ($resp->is_success) { - my ($json, $info, $version); + my ($json, $urls); - $json = decode_json($resp->decoded_content); - $info = $json->{info}; - $version = $info->{version}; - next unless $version; - - push(@$files, $json->{releases}{$version}[0]{filename}); + $json = decode_json($resp->decoded_content); + $urls = $json->{urls}; + foreach my $url (@$urls) { + push(@$files, $url->{filename}); + } } else { _debug("GET failed: " . $resp->code); return 0; Modified: head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm ============================================================================== --- head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm Thu Feb 4 08:35:33 2016 (r408016) +++ head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm Thu Feb 4 08:48:40 2016 (r408017) @@ -1,11 +1,16 @@ ---- Portscout/SiteHandler/PyPI.pm.orig 2015-10-25 05:00:48 UTC +--- Portscout/SiteHandler/PyPI.pm.orig 2016-02-04 08:23:53 UTC +++ Portscout/SiteHandler/PyPI.pm -@@ -115,7 +115,7 @@ sub GetFiles - $info = $json->{info}; - $version = $info->{version}; - next unless $version; -- -+ _debug("GET success: " . $resp->code . " Filename: " . $json->{releases}{$version}[0]{filename}); - push(@$files, $json->{releases}{$version}[0]{filename}); +@@ -109,11 +109,13 @@ sub GetFiles + $ua->agent(USER_AGENT); + $resp = $ua->request(HTTP::Request->new(GET => $query)); + if ($resp->is_success) { ++ _debug("GET success: " . $resp->code); + my ($json, $urls); + + $json = decode_json($resp->decoded_content); + $urls = $json->{urls}; + foreach my $url (@$urls) { ++ _debug("PyPi File: " . $url->{filename}); + push(@$files, $url->{filename}); + } } else { - _debug("GET failed: " . $resp->code);