Date: Sun, 8 Nov 2015 07:37:41 +0000 (UTC) From: Kubilay Kocak <koobs@FreeBSD.org> To: ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: svn commit: r401037 - in head/ports-mgmt/portscout: . files Message-ID: <201511080737.tA87bfB4067213@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: koobs Date: Sun Nov 8 07:37:41 2015 New Revision: 401037 URL: https://svnweb.freebsd.org/changeset/ports/401037 Log: ports-mgmt/portscout: Add GitHub and PyPI site handlers & MORE! Over the past several months portscout.freebsd.org appears to have been more frequently NOT finding updates, particularly for ports that use CHEESESHOP (PyPI) as their MASTER_SITES. Portscout has also never worked for ports using GitHub for distribution files due to the following: a) Portscout, prior to 'guessing', requests a randomly named file from the Site and expects a 4xx (404) in response. If it doesn't receive a 4xx response, it increments a 'lie counter' and does not check the site again in the next run. b) The GitHUB handlers (SUBDIR/MASTER_SITES) in bsd.sites.mk construct a URL that ends in a a dummy query paramater (for the filename), so that fetch saves the correct filename to DISTDIR. This means for any DISTFILE name provided, a 200 OK response is returned These two factors unfortunately leave us in a position where there is no good way to workaround this in the ports framework, including overriding DISTFILES, DISTNAME, FETCH_ARGS, or the SUBDIR URL itself for various reasons (not matching distinfo, file conflicts in DISTDIR, etc) Fortunately, the portroach project (OpenBSD's fork of portscout) contains a site handler for GitHub and PyPI (among others) already [1]. These site handlers use API endpoints at GitHub and PyPI that respond JSON respectively, providing a faster and more accurate way to determine the latest version of a package, without having to go through the 'guessing' process. This commit: - Adds GitHub and PyPI site handlers, and modifies or extends them to accept/match our MASTER_SITES URL's. - Adds authenticated API request support and two settings for the GitHub site handler - Add p5-JSON to RUN_DEPENDS (needed by new site handlers) - Add HTTPS option for supporting https:// MASTER_SITES. Currently portscout does not check (fails) https:// MASTER_SITES [2] - Take MAINTAINER'ship - Adds badly needed logging/debugging messages to key parts of the process retaining the conditional logic that ties the verbosity to "quiet" or "debug" portscout.conf settings. - Renables the SQLITE3 option (previously commented out) and renames it to SQLITE (the standard, as per bsd.options.desk.mk) - Creates a DATABASE option group allowing either/or SQLITE or POSTGRESQL to be selected - Switches option conditionals where possible to options helpers - Backport a fix for maintainer matching/mapping [3] - Adds LICENSE (BSD2CLAUSE) - Updates and sorts pkg-plist This change was tested again ports maintained by me, and resulted in 'new versions' being found and reported for 42 of my ports (of 123). [1] https://github.com/jasperla/portroach/tree/master/Portroach/SiteHandler [2] Reported by: truckman [3] https://github.com/jasperla/portroach/commit/2f6ee134ddc178f74688b37b986b66ecb0481782 PR: 203996 Approved by: maintainer (timeout, 15 days) Differential Revision: https://reviews.freebsd.org/D3996 Added: head/ports-mgmt/portscout/files/ head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-GitHub.pm (contents, props changed) head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm (contents, props changed) head/ports-mgmt/portscout/files/patch-Portscout_DataSrc_Ports.pm (contents, props changed) head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler.pm (contents, props changed) head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_GitHub.pm (contents, props changed) head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm (contents, props changed) head/ports-mgmt/portscout/files/patch-portscout.conf (contents, props changed) head/ports-mgmt/portscout/files/patch-portscout.pl (contents, props changed) Modified: head/ports-mgmt/portscout/Makefile head/ports-mgmt/portscout/pkg-plist Modified: head/ports-mgmt/portscout/Makefile ============================================================================== --- head/ports-mgmt/portscout/Makefile Sun Nov 8 07:24:25 2015 (r401036) +++ head/ports-mgmt/portscout/Makefile Sun Nov 8 07:37:41 2015 (r401037) @@ -3,44 +3,52 @@ PORTNAME= portscout PORTVERSION= 0.8.1 -PORTREVISION= 3 +PORTREVISION= 4 CATEGORIES= ports-mgmt -MASTER_SITES= http://mirror.inerd.com/FreeBSD/distfiles/${PORTNAME}/ \ - http://www.atarininja.org/~wxs/distfiles/ \ +MASTER_SITES= http://mirror.inerd.com/FreeBSD/distfiles/${PORTNAME}/ \ + http://www.atarininja.org/~wxs/distfiles/ \ http://www.inerd.com/software/${PORTNAME}/ -MAINTAINER= shaun@FreeBSD.org +MAINTAINER= koobs@FreeBSD.org COMMENT= Tool to scan for new versions of FreeBSD ports -OPTIONS_DEFINE= SQLITE3 DOCS -SQLITE3_DESC= Use SQLite backend instead of PostgreSQL +LICENSE= BSD2CLAUSE +RUN_DEPENDS= p5-DBI>=0:${PORTSDIR}/databases/p5-DBI \ + p5-Proc-Queue>=0:${PORTSDIR}/devel/p5-Proc-Queue \ + p5-Net>=0:${PORTSDIR}/net/p5-Net \ + p5-URI>=0:${PORTSDIR}/net/p5-URI \ + p5-XML-XPath>=0:${PORTSDIR}/textproc/p5-XML-XPath \ + p5-MIME-Lite>=0:${PORTSDIR}/mail/p5-MIME-Lite \ + p5-libwww>=0:${PORTSDIR}/www/p5-libwww \ + p5-JSON>0:${PORTSDIR}/converters/p5-JSON + +OPTIONS_DEFINE= DOCS HTTPS +OPTIONS_DEFAULT= HTTPS PGSQL + +OPTIONS_MULTI= DATABASE +OPTIONS_MULTI_DATABASE= SQLITE PGSQL + +DATABASE_DESC= Database Backends USES= perl5 shebangfix + NO_BUILD= yes SHEBANG_FILES= ${WRKSRC}/portscout.pl PORTDOCS= UPDATING portscout-portconfig.txt xml-datasrc-example.xml -RUN_DEPENDS= p5-DBI>=0:${PORTSDIR}/databases/p5-DBI \ - p5-Proc-Queue>=0:${PORTSDIR}/devel/p5-Proc-Queue \ - p5-Net>=0:${PORTSDIR}/net/p5-Net \ - p5-URI>=0:${PORTSDIR}/net/p5-URI \ - p5-XML-XPath>=0:${PORTSDIR}/textproc/p5-XML-XPath \ - p5-MIME-Lite>=0:${PORTSDIR}/mail/p5-MIME-Lite \ - p5-libwww>=0:${PORTSDIR}/www/p5-libwww +HTTPS_RUN_DEPENDS= p5-LWP-Protocol-https>=0:${PORTSDIR}/www/p5-LWP-Protocol-https -.include <bsd.port.options.mk> +SQLITE_USE= SQLITE=3 +SQLITE_RUN_DEPENDS= p5-DBD-SQLite>=0:${PORTSDIR}/databases/p5-DBD-SQLite -.if ${PORT_OPTIONS:MSQLITE3} -#USE_SQLITE= 3 -#RUN_DEPENDS+= p5-DBD-SQLite>=0:${PORTSDIR}/databases/p5-DBD-SQLite -.elif !defined(WITHOUT_PGSQL) -#USES+= pgsql -#RUN_DEPENDS+= p5-DBD-Pg>=0:${PORTSDIR}/databases/p5-DBD-Pg -.endif +PGSQL_USES= pgsql +PGSQL_RUN_DEPENDS= p5-DBD-Pg>=0:${PORTSDIR}/databases/p5-DBD-Pg + +.include <bsd.port.options.mk> pre-everything:: -.if ${PORT_OPTIONS:MSQLITE3} +.if ${PORT_OPTIONS:MSQLITE} @${ECHO_MSG} "+-------------------------------------------------------------+" @${ECHO_MSG} "| Warning! although SQLite is supported, portscout will only |" @${ECHO_MSG} "| operate in non-forking mode with this database backend. It |" @@ -48,8 +56,12 @@ pre-everything:: @${ECHO_MSG} "+-------------------------------------------------------------+" .endif +post-extract: + @${CP} ${FILESDIR}/files-Portscout-SiteHandler-GitHub.pm ${WRKSRC}/Portscout/SiteHandler/GitHub.pm + @${CP} ${FILESDIR}/files-Portscout-SiteHandler-PyPI.pm ${WRKSRC}/Portscout/SiteHandler/PyPI.pm + post-patch: -.if ${PORT_OPTIONS:MSQLITE3} +.if ${PORT_OPTIONS:MSQLITE} @${REINPLACE_CMD} 's/^\([^#]*DBI:Pg.*\)$$/#\1/g' ${WRKSRC}/portscout.conf @${REINPLACE_CMD} 's/^#\(.*DBI:SQLite.*\)$$/\1/g' ${WRKSRC}/portscout.conf .endif @@ -96,7 +108,7 @@ do-install: test: ${PERL} ${WRKSRC}/t/00-use.t ${PERL} ${WRKSRC}/t/01-vercompare.t -.if ${PORT_OPTIONS:MSQLITE3} +.if ${PORT_OPTIONS:MSQLITE} ${PERL} ${WRKSRC}/t/10-sqlite.t #.else # ${PERL} ${WRKSRC}/10-postgresql.t Added: head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-GitHub.pm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-GitHub.pm Sun Nov 8 07:37:41 2015 (r401037) @@ -0,0 +1,175 @@ +#------------------------------------------------------------------------------ +# Copyright (C) 2014, Jasper Lievisse Adriaanse <jasper@openbsd.org> +# +# Permission to use, copy, modify, and distribute this software for any +# purpose with or without fee is hereby granted, provided that the above +# copyright notice and this permission notice appear in all copies. +# +# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +# +#------------------------------------------------------------------------------ + +package Portscout::SiteHandler::GitHub; + +use JSON qw(decode_json); +use LWP::UserAgent; + +use Portscout::Const; +use Portscout::Config; + +use strict; + +require 5.006; + + +#------------------------------------------------------------------------------ +# Globals +#------------------------------------------------------------------------------ + +push @Portscout::SiteHandler::sitehandlers, __PACKAGE__; + +our %settings; + + +#------------------------------------------------------------------------------ +# Func: new() +# Desc: Constructor. +# +# Args: n/a +# +# Retn: $self +#------------------------------------------------------------------------------ + +sub new +{ + my $self = {}; + my $class = shift; + + $self->{name} = 'GitHub'; + + bless ($self, $class); + return $self; +} + + +#------------------------------------------------------------------------------ +# Func: CanHandle() +# Desc: Ask if this handler (package) can handle the given site. +# +# Args: $url - URL of site. +# +# Retn: $res - true/false. +#------------------------------------------------------------------------------ + +sub CanHandle +{ + my $self = shift; + + my ($url) = @_; + + return ($url =~ /^https?:\/\/([^\/.]+\.)?github\.com\/(.*?)\/tar.gz/); +} + + +#------------------------------------------------------------------------------ +# Func: GetFiles() +# Desc: Extract a list of files from the given URL. In the case of GitHub, +# we are actually pulling the files from the project's Atom feed and +# extract the release url, containing the tag it was based on. +# +# Args: $url - URL we would normally fetch from. +# \%port - Port hash fetched from database. +# \@files - Array to put files into. +# +# Retn: $success - False if file list could not be constructed; else, true. +#------------------------------------------------------------------------------ + +sub GetFiles +{ + my $self = shift; + + my ($url, $port, $files) = @_; + my $projname; + + if ($url =~ /https:\/\/github\.com\/(.*?)\/archive\//) { + $projname = $1; + } elsif ($url =~ /https:\/\/github.com\/downloads\/(.*)\//) { + $projname = $1; + } + + if ($projname) { + my ($query, $ua, $response, $items, $json); + + # First check if there's a latest releases endpoint + $query = 'https://api.github.com/repos/' . $projname . '/releases/latest'; + + _debug("GET $query"); + $ua = LWP::UserAgent->new; + $ua->agent(USER_AGENT); + $ua->timeout($settings{http_timeout}); + + $response = $ua->request(HTTP::Request->new(GET => $query)); + + if (!$response->is_success || $response->status_line !~ /^2/) { + _debug('GET failed: ' . $response->status_line); + # Project didn't do any releases, so let's try tags instead. + $query = 'https://api.github.com/repos/' . $projname . '/tags'; + _debug("GET $query"); + $ua = LWP::UserAgent->new; + $ua->agent(USER_AGENT); + $ua->timeout($settings{http_timeout}); + + $response = $ua->request(HTTP::Request->new(GET => $query)); + + if (!$response->is_success || $response->status_line !~ /^2/) { + _debug('GET failed: ' . $response->status_line); + return 0; + } + + $json = decode_json($response->decoded_content); + foreach my $tag (@$json) { + my $tag_url = $tag->{tarball_url}; + push(@$files, $tag_url); + } + + _debug('Found ' . scalar @$files . ' files'); + return 1; + } + + $json = decode_json($response->decoded_content); + push(@$files, $json->{tarball_url}); + + _debug('Found ' . scalar @$files . ' files'); + } else { + return 0; + } + + return 1; +} + + +#------------------------------------------------------------------------------ +# Func: _debug() +# Desc: Print a debug message. +# +# Args: $msg - Message. +# +# Retn: n/a +#------------------------------------------------------------------------------ + +sub _debug +{ + my ($msg) = @_; + + $msg = '' if (!$msg); + + print STDERR "(" . __PACKAGE__ . ") $msg\n" if ($settings{debug}); +} + +1; Added: head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm Sun Nov 8 07:37:41 2015 (r401037) @@ -0,0 +1,147 @@ +#------------------------------------------------------------------------------ +# Copyright (C) 2015, Jasper Lievisse Adriaanse <jasper@openbsd.org> +# +# Permission to use, copy, modify, and distribute this software for any +# purpose with or without fee is hereby granted, provided that the above +# copyright notice and this permission notice appear in all copies. +# +# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +# +#------------------------------------------------------------------------------ + +package Portscout::SiteHandler::PyPI; + +use JSON qw(decode_json); +use LWP::UserAgent; + +use Portscout::Const; +use Portscout::Config; + +use strict; + +require 5.006; + + +#------------------------------------------------------------------------------ +# Globals +#------------------------------------------------------------------------------ + +push @Portscout::SiteHandler::sitehandlers, __PACKAGE__; + +our %settings; + + +#------------------------------------------------------------------------------ +# Func: new() +# Desc: Constructor. +# +# Args: n/a +# +# Retn: $self +#------------------------------------------------------------------------------ + +sub new +{ + my $self = {}; + my $class = shift; + + $self->{name} = 'PyPI'; + + bless ($self, $class); + return $self; +} + + +#------------------------------------------------------------------------------ +# Func: CanHandle() +# Desc: Ask if this handler (package) can handle the given site. +# +# Args: $url - URL of site. +# +# Retn: $res - true/false. +#------------------------------------------------------------------------------ + +sub CanHandle +{ + my $self = shift; + + my ($url) = @_; + + return ($url =~ /https?:\/\/pypi\.python\.org\//); +} + + +#------------------------------------------------------------------------------ +# Func: GetFiles() +# Desc: Extract a list of files from the given URL. Simply query the API. +# +# Args: $url - URL we would normally fetch from. +# \%port - Port hash fetched from database. +# \@files - Array to put files into. +# +# Retn: $success - False if file list could not be constructed; else, true. +#------------------------------------------------------------------------------ + +sub GetFiles +{ + my $self = shift; + + my ($url, $port, $files) = @_; + + my ($pypi, $package, $resp, $query, $ua); + $pypi = 'https://pypi.python.org/pypi/'; + + # Strip all the digits at the end to keep the stem of the module. + if ($port->{distname} =~ /(.*?)-(\d+)/) { + $package = $1; + } + + $query = $pypi . $package . '/json'; + + _debug("GET $query"); + $ua = LWP::UserAgent->new; + $ua->agent(USER_AGENT); + $resp = $ua->request(HTTP::Request->new(GET => $query)); + if ($resp->is_success) { + my ($json, $info, $version); + + $json = decode_json($resp->decoded_content); + $info = $json->{info}; + $version = $info->{version}; + next unless $version; + + push(@$files, $json->{releases}{$version}[0]{filename}); + } else { + _debug("GET failed: " . $resp->code); + return 0; + } + + return 1; +} + + +#------------------------------------------------------------------------------ +# Func: _debug() +# Desc: Print a debug message. +# +# Args: $msg - Message. +# +# Retn: n/a +#------------------------------------------------------------------------------ + +sub _debug +{ + my ($msg) = @_; + + $msg = '' if (!$msg); + + print STDERR "(" . __PACKAGE__ . ") $msg\n" if ($settings{debug}); +} + +1; Added: head/ports-mgmt/portscout/files/patch-Portscout_DataSrc_Ports.pm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/ports-mgmt/portscout/files/patch-Portscout_DataSrc_Ports.pm Sun Nov 8 07:37:41 2015 (r401037) @@ -0,0 +1,81 @@ +--- Portscout/DataSrc/Ports.pm.orig 2011-04-09 17:19:03 UTC ++++ Portscout/DataSrc/Ports.pm +@@ -201,7 +201,7 @@ sub BuildDB + + my $lastbuild = getstat('buildtime', TYPE_INT); + +- print "Looking for updated ports...\n\n" ++ print "Incremental build: Looking for updated ports...\n\n" + if ($incremental); + + $got_ports = 0; +@@ -243,12 +243,15 @@ sub BuildDB + my (@fields, $maintainer, $port); + + @fields = split /\|/; +- $maintainer = $fields[5]; ++ $maintainer = lc($fields[5]); + $port = $fields[1]; + $port =~ s/^(?:.*\/)?([^\/]+)\/([^\/]+)$/$1\/$2/; + +- $portsmaintok{$port} = $maintainer +- if ($maintainers{$maintainer}); ++ if ($maintainers{$maintainer}) { ++ $portsmaintok{$port} = $maintainer; ++ print "Maintainer match: $maintainer $port \n" ++ unless ($settings{quiet}); ++ } + } + + close $if; +@@ -264,7 +267,7 @@ sub BuildDB + + opendir my $catdir, $settings{ports_dir}."/$cat"; + +- print "Scanning $cat...\n" ++ print "Scanning $cat ...\n" + unless ($settings{quiet}); + + while (my $name = readdir $catdir) { +@@ -276,9 +279,9 @@ sub BuildDB + # port directory's mtime; skip if not updated. + if ($incremental) { + my ($updated); +- + opendir my $portdir, $settings{ports_dir}."/$cat/$name"; +- ++ print "Scanning $cat/$name ... " ++ unless ($settings{quiet}); + while (my $subfile = readdir $portdir) { + my ($subfile_path, $fi); + +@@ -289,12 +292,18 @@ sub BuildDB + or die "Couldn't stat $subfile_path: $!"; + + if ($fi->mtime > $lastbuild) { ++ print "$subfile (mtime: $fi->mtime) modified updated since last build: $lastbuild \n" ++ if ($settings{debug}); + $updated = 1; + last; + } + } + +- next if (!$updated); ++ if (!$updated) { ++ print "Not modified since last build: $lastbuild \n" ++ if ($settings{debug}); ++ next; ++ } + } + + # Check this port is wanted by user +@@ -307,7 +316,8 @@ sub BuildDB + && $settings{indexfile_enable}) { + next if (!$portsmaintok{"$cat/$name"}); + } +- ++ print "Matched: $cat/$name\n" ++ unless ($settings{quiet}); + push @ports, "$cat/$name"; + } + } Added: head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler.pm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler.pm Sun Nov 8 07:37:41 2015 (r401037) @@ -0,0 +1,11 @@ +--- Portscout/SiteHandler.pm.orig 2010-04-29 01:07:51 UTC ++++ Portscout/SiteHandler.pm +@@ -31,6 +31,8 @@ package Portscout::SiteHandler; + use XML::XPath; + use XML::XPath::XMLParser; + ++use Portscout::SiteHandler::GitHub; ++use Portscout::SiteHandler::PyPI; + use Portscout::SiteHandler::SourceForge; + + use strict; Added: head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_GitHub.pm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_GitHub.pm Sun Nov 8 07:37:41 2015 (r401037) @@ -0,0 +1,26 @@ +--- Portscout/SiteHandler/GitHub.pm.orig 2015-10-25 05:00:48 UTC ++++ Portscout/SiteHandler/GitHub.pm +@@ -97,7 +97,9 @@ sub GetFiles + my ($url, $port, $files) = @_; + my $projname; + +- if ($url =~ /https:\/\/github\.com\/(.*?)\/archive\//) { ++ if ($url =~ /https?:\/\/codeload\.github\.com\/(.*?)\/tar.gz\//) { ++ $projname = $1; ++ } elsif ($url =~ /https:\/\/github\.com\/(.*?)\/archive\//) { + $projname = $1; + } elsif ($url =~ /https:\/\/github.com\/downloads\/(.*)\//) { + $projname = $1; +@@ -108,7 +110,11 @@ sub GetFiles + + # First check if there's a latest releases endpoint + $query = 'https://api.github.com/repos/' . $projname . '/releases/latest'; +- ++ # Add GitHub Client ID & Secret if they are set in settings ++ # https://developer.github.com/v3/#authentication ++ if ($settings{github_client_id} && $settings{github_client_id}) { ++ $query = $query . "?client_id=$settings{github_client_id}&client_secret=$settings{github_client_secret}"; ++ } + _debug("GET $query"); + $ua = LWP::UserAgent->new; + $ua->agent(USER_AGENT); Added: head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm Sun Nov 8 07:37:41 2015 (r401037) @@ -0,0 +1,11 @@ +--- Portscout/SiteHandler/PyPI.pm.orig 2015-10-25 05:00:48 UTC ++++ Portscout/SiteHandler/PyPI.pm +@@ -115,7 +115,7 @@ sub GetFiles + $info = $json->{info}; + $version = $info->{version}; + next unless $version; +- ++ _debug("GET success: " . $resp->code . " Filename: " . $json->{releases}{$version}[0]{filename}); + push(@$files, $json->{releases}{$version}[0]{filename}); + } else { + _debug("GET failed: " . $resp->code); Added: head/ports-mgmt/portscout/files/patch-portscout.conf ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/ports-mgmt/portscout/files/patch-portscout.conf Sun Nov 8 07:37:41 2015 (r401037) @@ -0,0 +1,14 @@ +--- portscout.conf.orig 2015-10-25 05:00:59 UTC ++++ portscout.conf +@@ -172,5 +172,11 @@ db port = # Port + #db connstr = DBI:Pg:dbname=%(db_name);host=%(db_host);port=%(db_port) + db connstr = DBI:SQLite:dbname=/var/db/portscout.db + ++# GitHub site handler settings ++# GitHub rate limits requests to its API to a very low number for unauthenticated ++# requests, and 5000 per hour for authenticated requests. ++ ++# github_client_id = # GitHub Client ID ++# github_client_secret = # GitHub Client Secret + + # ex: ts=4 sw=4 Added: head/ports-mgmt/portscout/files/patch-portscout.pl ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/ports-mgmt/portscout/files/patch-portscout.pl Sun Nov 8 07:37:41 2015 (r401037) @@ -0,0 +1,55 @@ +--- portscout.pl.orig 2015-10-25 05:00:48 UTC ++++ portscout.pl +@@ -463,7 +463,7 @@ sub VersionCheck + + $i++; + +- info($k, 'Checking site: ' . strchop($site, 60)); ++ info($k, 'Checking site: ' . strchop($site, 200)); + + # Look to see if the URL contains the distfile version. + # This will affect our checks and guesses later on. +@@ -493,15 +493,23 @@ sub VersionCheck + } + + # Check for special handler for this site first ++ print "Does site handler exist ... " ++ unless ($settings{quiet}); + if (my $sh = Portscout::SiteHandler->FindHandler($site)) + { +- info($k, $site, 'Using dedicated site handler for site.'); ++ print "Yes \n" ++ unless ($settings{quiet}); + + if (!$sh->GetFiles($site, $port, \@files)) { + info($k, $site, 'SiteHandler::GetFiles() failed for ' . $site); + next; + } + } ++ elsif (!$sh) ++ { ++ print "No \n" ++ unless ($settings{quiet}); ++ } + elsif ($site->scheme eq 'ftp') + { + my $ftp; +@@ -713,7 +721,8 @@ sub VersionCheck + + # Got a response which wasn't HTTP 4xx -> bail out + if ($response->is_success && $response->status_line !~ /^4/) { +- info($k, $site, 'Not doing any guessing; site is lieing to us.'); ++ print "URL: $url\n"; ++ info($k, $site, "Skip guessing: Response not 4xx to a file that shouldnt exist (". $response->status_line .")"); + $sths->{sitedata_initliecount}->execute($sitedata->{host}) + unless($settings{precious_data}); + next; +@@ -797,7 +806,7 @@ sub VersionCheck + $new_found = 1; + last; + } else { +- info($k, $site, "Guess failed $port->{ver} -> $guess_v"); ++ info($k, $site, "Guess failed $port->{ver} -> $guess_v (". $response->status_line .")"); + } + + last if ($new_found); Modified: head/ports-mgmt/portscout/pkg-plist ============================================================================== --- head/ports-mgmt/portscout/pkg-plist Sun Nov 8 07:24:25 2015 (r401036) +++ head/ports-mgmt/portscout/pkg-plist Sun Nov 8 07:37:41 2015 (r401037) @@ -14,19 +14,21 @@ man/man1/portscout.1.gz %%DATADIR%%/sql/sqlite_init.sql %%DATADIR%%/sql/sqlite_destroy.sql %%DATADIR%%/sql/sqlite_upgrade_0.8_to_0.8.1.sql -%%SITE_PERL%%/Portscout/SQL/SQLite.pm -%%SITE_PERL%%/Portscout/SQL/Pg.pm -%%SITE_PERL%%/Portscout/SiteHandler/SourceForge.pm +%%SITE_PERL%%/Portscout.pm +%%SITE_PERL%%/Portscout/API.pm +%%SITE_PERL%%/Portscout/Config.pm +%%SITE_PERL%%/Portscout/Const.pm +%%SITE_PERL%%/Portscout/DataSrc.pm %%SITE_PERL%%/Portscout/DataSrc/Ports.pm %%SITE_PERL%%/Portscout/DataSrc/XML.pm -%%SITE_PERL%%/Portscout/SiteHandler.pm -%%SITE_PERL%%/Portscout/SQL.pm %%SITE_PERL%%/Portscout/Make.pm -%%SITE_PERL%%/Portscout/Const.pm -%%SITE_PERL%%/Portscout/DataSrc.pm +%%SITE_PERL%%/Portscout/SQL.pm +%%SITE_PERL%%/Portscout/SQL/Pg.pm +%%SITE_PERL%%/Portscout/SQL/SQLite.pm +%%SITE_PERL%%/Portscout/SiteHandler.pm +%%SITE_PERL%%/Portscout/SiteHandler/GitHub.pm +%%SITE_PERL%%/Portscout/SiteHandler/PyPI.pm +%%SITE_PERL%%/Portscout/SiteHandler/SourceForge.pm %%SITE_PERL%%/Portscout/Template.pm %%SITE_PERL%%/Portscout/Util.pm -%%SITE_PERL%%/Portscout/API.pm -%%SITE_PERL%%/Portscout/Config.pm -%%SITE_PERL%%/Portscout.pm %%SITE_PERL%%/portscout.pod
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201511080737.tA87bfB4067213>