diff --git a/bin/clientstats.pl b/bin/clientstats.pl index b94fa16..71923c3 100755 --- a/bin/clientstats.pl +++ b/bin/clientstats.pl @@ -275,7 +275,7 @@ clientstats - create reports on client usage =head1 SYNOPSIS -B [B<-Vhcs> B<--comments>] [B<-m> I[:I] | I] [B<-n> I] [B<-r> I] [B<-l> I] [B<-u> I] [B<-g> I] [B<-o> I] [B<-f> I] [B<--filetemplate> I] [B<--db> I] [B<--conffile> I] +B [B<-Vhcsv> B<--comments>] [B<-m> I[:I] | I] [B<-n> I] [B<-r> I] [B<-l> I] [B<-u> I] [B<-g> I] [B<-o> I] [B<-f> I] [B<--filetemplate> I] [B<--db> I] [B<--conffile> I] =head1 REQUIREMENTS @@ -323,18 +323,19 @@ details and exceptions, please see below. The results will be formatted as a kind of table; you can change the output format to a simple list or just a list of names and number of -postings with the B<--format> option. Captions will be added by means of -the B<--caption> option; all comments (and captions) can be supressed by -using B<--nocomments>. +postings with the B<--format> option. Captions will be added by means +of the B<--caption> option; all comments (and captions) can be +supressed by using B<--nocomments>. -Last but not least you can redirect all output to a number of files, e.g. -one for each month, by submitting the B<--filetemplate> option, see below. +Last but not least you can redirect all output to a number of files, +e.g. one for each month, by submitting the B<--filetemplate> option, +see below. =head2 Configuration B will read its configuration from F -which should be present in etc/ via Config::Auto or from a configuration file -submitted by the B<--conffile> option. +which should be present in etc/ via Config::Auto or from a configuration +file submitted by the B<--conffile> option. See doc/INSTALL for an overview of possible configuration options. @@ -346,32 +347,47 @@ You can override some configuration options via the B<--db> option. =item B<-V>, B<--version> -Print out version and copyright information and exit. +Display version and copyright information and exit. =item B<-h>, B<--help> -Print this man page and exit. +Display this man page and exit. =item B<-m>, B<--month> I Set processing period to a single month in YYYY-MM format or to a time period between two month in YYYY-MM:YYYY-MM format (two month, separated by a colon). By using the keyword I instead, you can set no -processing period to process the whole database. +processing period to process the whole database. Defaults to last month. =item B<-n>, B<--names> I Limit processing to a certain set of client names. I -can be a single name (eternal-september.org), a group of names -(*.inka.de) or a list of either of these, separated by colons, for -example +can be a single name (Thunderbird), a group of names (Ice*) or a list +of either of these, separated by colons, for example - eternal-september.org:solani.org:*.inka.de + Forte Agent:Thunderbird:Ice* + +Spaces or special characters like "*" need to be quoted from the shell, +like + + -n 'Forte Agent:Thunderbird:Ice*' + +There is no way to limit processing to a specific version, but you can +alway grep through the output. =item B<-s>, B<--sums|--nosums> (sum per month) Include "virtual" clients named "ALL" for every month in output, -containing the sum of all detected clients for that month. +containing the sum of all detected clients for that month. False +by default. + +=item B<-v>, B<--versions|--noversions> (client versions) + +Include a list of all observed versions of each client in output. +Version information will be displayed with indents ('-') below each +client, sorted in the same way (by postings or alphanumeric). False +by default. =item B<-r>, B<--report> I @@ -379,7 +395,7 @@ Choose the report type: I or I By default, B will report the number of postings for each client in each month. But it can also report the total sum of postings -per client for all months. +per client for all months. Sums of B<--versions> can be included. For report type I, the B option has no meaning and will be silently ignored (see below). @@ -395,10 +411,10 @@ Set the upper boundary. By default, all clients with more postings per month than the upper boundary and/or less postings per month than the lower boundary will be excluded from further processing. For the default report that -means each month only /clients with a number of postings between -the boundaries will be displayed. For the sums report, /clients -with a number of postings exceeding the boundaries in all (!) months -will not be considered. +means each month only clients with a number of postings between the +boundaries will be displayed. For the sums report, clients with a +number of postings exceeding the boundaries in all (!) months will +not be considered. =item B<-g>, B<--group-by> I @@ -406,38 +422,38 @@ By default, all results are grouped by month, sorted chronologically in ascending order, like this: # ----- 2012-01: - arcor-online.net : 9379 - individual.net : 19525 - news.albasani.net: 9063 + 40tude_Dialog: 5873 + Forte Agent : 7735 + Thunderbird : 20925 # ----- 2012-02: - arcor-online.net : 8606 - individual.net : 16768 - news.albasani.net: 7879 + 40tude_Dialog: 4142 + Forte Agent : 5895 + Thunderbird : 19091 The results can be grouped by client instead via B<--group-by> I: - ----- individual.net - 2012-01: 19525 - 2012-02: 16768 - ----- arcor-online.net - 2012-01: 9379 - 2012-02: 8606 - ----- news.albasani.net - 2012-01: 9063 - 2012-02: 7879 + # ----- 40tude_Dialog: + 2012-01: 5873 + 2012-02: 4142 + # ----- Forte Agent: + 2012-01: 7735 + 2012-02: 5895 + # ----- Thunderbird: + 2012-01: 20925 + 2012-02: 19091 By appending I<-desc> to the group-by option parameter, you can reverse the sort order - e.g. B<--group-by> I will give: # ----- 2012-02: - arcor-online.net : 8606 - individual.net : 16768 - news.albasani.net: 7879 + 40tude_Dialog: 4142 + Forte Agent : 5895 + Thunderbird : 19091 # ----- 2012-01: - arcor-online.net : 9379 - individual.net : 19525 - news.albasani.net: 9063 + 40tude_Dialog: 5873 + Forte Agent : 7735 + Thunderbird : 20925 Sums reports (see above) will always be grouped by months; this option will therefore be ignored. @@ -449,41 +465,57 @@ the report will be sorted by name (or month) in ascending alphabetical order by default. You can change the sort order to descending or sort by number of postings instead. -=item B<-f>, B<--format> I - -Select the output format, I being the default: +By default, output is sorted alphabetically: # ----- 2012-01: - arcor-online.net : 9379 - individual.net : 19525 + 40tude_Dialog: 5873 + Forte Agent : 7735 + Thunderbird : 20925 + +Using B<--order-by> I, it will be sorted from most +to least postings: + + # ----- 2012-01: + Thunderbird : 20925 + Forte Agent : 7735 + 40tude_Dialog: 5873 + +=item B<-f>, B<--format> I + +Select the output format, I (a kind of table) being the default: + + # ----- 2012-01: + 40tude_Dialog: 5873 + Forte Agent : 7735 # ----- 2012-02: - arcor-online.net : 8606 - individual.net : 16768 + 40tude_Dialog: 4142 + Forte Agent : 5895 -I format looks like this: +I format looks like this (each client preceded by month): - 2012-01 arcor-online.net 9379 - 2012-01 individual.net 19525 - 2012-02 arcor-online.net 8606 - 2012-02 individual.net 16768 + 2012-01 40tude_Dialog 5873 + 2012-01 Forte Agent 7735 + 2012-02 40tude_Dialog 4142 + 2012-02 Forte Agent 5895 And I format looks like this: # 2012-01: - arcor-online.net 9379 - individual.net 19525 + 40tude_Dialog 5873 + Forte Agent 7735 # 2012-02: - arcor-online.net 8606 - individual.net 16768 + 40tude_Dialog 4142 + Forte Agent 5895 -You can remove the comments by using B<--nocomments>, see below. +You can remove the comments (lines after '#') by using B<--nocomments>, +see below. =item B<-c>, B<--captions|--nocaptions> Add captions to output, like this: ----- Report for 2012-01 to 2012-02 (number of postings for each month) - ----- Names: individual.net + ----- Names: Thunderbird ----- Threshold: 8000 => x (counting only month fulfilling this condition) ----- Grouped by Month (ascending), sorted by number of postings descending @@ -491,11 +523,11 @@ False by default. =item B<--comments|--nocomments> -Add comments (group headers) to I and I output. True by default -as logn as B<--filetemplate> is not set. +Add comments (group headers) to I and I output. True by +default as long as B<--filetemplate> is not set. -Use I<--nocomments> to suppress anything except client names or months and -numbers of postings. +Use I<--nocomments> to suppress anything except client names or months +and numbers of postings. =item B<--filetemplate> I @@ -515,7 +547,7 @@ Override I or I from F. =item B<--conffile> I -Load configuration from I instead of F. +Read configuration from I instead of F. =back @@ -525,26 +557,26 @@ See L. =head1 EXAMPLES -Show number of postings per group for lasth month in I format: +Show number of postings per client for lasth month in I format: clientstats -Show that report for January of 2010 and *.inka plus individual.net: +Show that report for January of 2010 and Thunderbird plus Ice*: - clientstats --month 2010-01 --names *.inka:individual.net: + clientstats --month 2010-01 --names 'Thunderbird:Ice*' -Only show clients with 30 postings or less last month, ordered -by number of postings, descending, in I format: +Only show clients with at least 30 postings last month and the versions +of those clients, ordered each by number of postings, descending, +in I format: - clientstats --upper 30 --order-by postings-desc + clientstats --lower 30 --versions --order-by postings-desc -List number of postings per host for each month of 2010 and redirect +List number of postings per client for each month of 2010 and redirect output to one file for each month, named hosts-2010-01 and so on, in machine-readable form (without formatting): clientstats -m 2010-01:2010-12 -f dump --filetemplate hosts - =head1 FILES =over 4 diff --git a/bin/dbcreate.pl b/bin/dbcreate.pl index e3ad8c6..6bec42d 100755 --- a/bin/dbcreate.pl +++ b/bin/dbcreate.pl @@ -168,7 +168,7 @@ INSTALL my $Upgrade =''; if ($OptUpdate) { - $Upgrade = < 0.02 - # &DoMySQL('...;'); - # print "v0.02: Database upgrades ...\n"; - # &PrintInstructions('0.02',<<" INSTRUCTIONS"); - # INSTRUCTIONS - }; - }; + # TBD # Display general upgrade instructions print $Upgrade; }; @@ -307,11 +298,11 @@ See L for an overview of possible configuration options. =item B<-V>, B<--version> -Print out version and copyright information and exit. +Display version and copyright information and exit. =item B<-h>, B<--help> -Print this man page and exit. +Display this man page and exit. =item B<-u>, B<--update> I @@ -319,7 +310,7 @@ Don't do a fresh install, but update from I. =item B<--conffile> I -Load configuration from I instead of F. +Read configuration from I instead of F. =back diff --git a/bin/feedlog.pl b/bin/feedlog.pl index 6359693..9b1fd96 100755 --- a/bin/feedlog.pl +++ b/bin/feedlog.pl @@ -167,8 +167,9 @@ time. All reporting is done to I via I facility. If B fails to initiate a database connection at startup, it will log to -I with I priority and go in an endless loop, as -terminating would only result in a rapid respawn. +I with I priority and go in an endless loop, trying again +to connect every 5 seconds, as terminating would only result in a rapid +respawn. =head2 Configuration @@ -184,15 +185,15 @@ See L for an overview of possible configuration options. =item B<-V>, B<--version> -Print out version and copyright information and exit. +Display version and copyright information and exit. =item B<-h>, B<--help> -Print this man page and exit. +Display this man page and exit. =item B<-d>, B<--debug> -Output debugging information to STDERR while parsing STDIN. You'll +Print debugging information to STDERR while parsing STDIN. You'll find that information most probably in your B F file. =item B<-q>, B<--quiet> @@ -201,7 +202,7 @@ Suppress logging to syslog. =item B<--conffile> I -Load configuration from I instead of F. +Read configuration from I instead of F. =back diff --git a/bin/gatherstats.pl b/bin/gatherstats.pl index 1b9292c..aee6f28 100755 --- a/bin/gatherstats.pl +++ b/bin/gatherstats.pl @@ -803,12 +803,12 @@ override that default through the B<--clientsdb> option. =head2 Configuration B will read its configuration from F -which should be present in etc/ via Config::Auto or from a configuration file -submitted by the B<--conffile> option. +which should be present in etc/ via Config::Auto or from a configuration +file submitted by the B<--conffile> option. See L for an overview of possible configuration options. -You can override configuration options via the B<--hierarchy>, +You can override configuration options by using the B<--hierarchy>, B<--rawdb>, B<--groupsdb>, B<--clientsdb> and B<--hostsdb> options, respectively. @@ -818,15 +818,15 @@ respectively. =item B<-V>, B<--version> -Print out version and copyright information and exit. +Display version and copyright information and exit. =item B<-h>, B<--help> -Print this man page and exit. +Display this man page and exit. =item B<-d>, B<--debug> -Output debugging information to STDOUT while processing (number of +Print debugging information to STDOUT while processing (number of postings per group). =item B<-t>, B<--test> @@ -838,15 +838,17 @@ conjunction with B<--test> ... everything else seems a bit pointless. Set processing period to a single month in YYYY-MM format or to a time period between two month in YYYY-MM:YYYY-MM format (two month, separated -by a colon). +by a colon). Defaults to last month. =item B<-s>, B<--stats> I -Set processing type to one of I, I or I. Defaults -to all. +Set processing type to one of I, I, I or I. +Defaults to I. =item B<-c>, B<--checkgroups> I +Relevant only for newsgroup stats (I). + Check each group against a list of valid newsgroups read from a file, one group on each line and ignoring everything after the first whitespace (so you can use a file in checkgroups format or (part of) @@ -889,7 +891,7 @@ Override I from F. =item B<--conffile> I -Load configuration from I instead of F. +Read configuration from I instead of F. =back diff --git a/bin/groupstats.pl b/bin/groupstats.pl index 0c193a8..4da8d41 100755 --- a/bin/groupstats.pl +++ b/bin/groupstats.pl @@ -283,7 +283,7 @@ See L. =head1 DESCRIPTION -This script create reports on newsgroup usage (number of postings per +This script creates reports on newsgroup usage (number of postings per group per month) taken from result tables created by B. @@ -291,16 +291,16 @@ B. =head3 Time period and newsgroups -The time period to act on defaults to last month; you can assign another -time period or a single month (or drop all time constraints) via the -B<--month> option (see below). +The time period to act on defaults to last month; you can assign +another time period or a single month (or drop all time constraints) +via the B<--month> option (see below). B will process all newsgroups by default; you can limit -processing to only some newsgroups by supplying a list of those groups via -B<--newsgroups> option (see below). You can include hierarchy levels in -the output by adding the B<--sums> switch (see below). Optionally -newsgroups not present in a checkgroups file can be excluded from output, -sse B<--checkgroups> below. +processing to only some newsgroups by supplying a list of those groups +via B<--newsgroups> option (see below). You can include hierarchy +levels in the output by adding the B<--sums> switch (see below). +Optionally newsgroups not present in a checkgroups file can be excluded +from output, sse B<--checkgroups> below. =head3 Report type @@ -321,26 +321,27 @@ below. =head3 Sorting and formatting the output By default, all results are grouped by month; you can group results by -newsgroup instead via the B<--groupy-by> option. Within those groups, the -list of newsgroups (or months) is sorted alphabetically (or -chronologically, respectively) ascending. You can change that order (and -sort by number of postings) with the B<--order-by> option. For details and -exceptions, please see below. +newsgroup instead via the B<--groupy-by> option. Within those groups, +the list of newsgroups (or months) is sorted alphabetically (or +chronologically, respectively) ascending. You can change that order +(and sort by number of postings) with the B<--order-by> option. For +details and exceptions, please see below. The results will be formatted as a kind of table; you can change the -output format to a simple list or just a list of newsgroups and number of -postings with the B<--format> option. Captions will be added by means of -the B<--caption> option; all comments (and captions) can be supressed by -using B<--nocomments>. +output format to a simple list or just a list of newsgroups and number +of postings with the B<--format> option. Captions will be added by means +of the B<--caption> option; all comments (and captions) can be supressed +by using B<--nocomments>. -Last but not least you can redirect all output to a number of files, e.g. -one for each month, by submitting the B<--filetemplate> option, see below. +Last but not least you can redirect all output to a number of files, +e.g. one for each month, by submitting the B<--filetemplate> option, +see below. =head2 Configuration B will read its configuration from F -which should be present in etc/ via Config::Auto or from a configuration file -submitted by the B<--conffile> option. +which should be present in etc/ via Config::Auto or from a configuration +file submitted by the B<--conffile> option. See doc/INSTALL for an overview of possible configuration options. @@ -352,18 +353,18 @@ You can override some configuration options via the B<--groupsdb> option. =item B<-V>, B<--version> -Print out version and copyright information and exit. +Display version and copyright information and exit. =item B<-h>, B<--help> -Print this man page and exit. +Display this man page and exit. =item B<-m>, B<--month> I Set processing period to a single month in YYYY-MM format or to a time period between two month in YYYY-MM:YYYY-MM format (two month, separated by a colon). By using the keyword I instead, you can set no -processing period to process the whole database. +processing period to process the whole database. Defaults to last month. =item B<-n>, B<--newsgroups> I @@ -388,17 +389,20 @@ See the B man page for details. This option does not work together with the B<--checkgroups> option as all "virtual" groups will not be present in the checkgroups file. +False by default. + =item B<--checkgroups> I -Restrict output to those newgroups present in a file in checkgroups format -(one newgroup name per line; everything after the first whitespace on each -line is ignored). All other newsgroups will be removed from output. +Restrict output to those newgroups present in a file in checkgroups +format (one newgroup name per line; everything after the first +whitespace on each line is ignored). All other newsgroups will be +removed from output. -Contrary to B, I is not a template, but refers to -a single file in checkgroups format. +Contrary to B, I is not a template, but refers +to a single file in checkgroups format. -The B<--sums> option will not work together with this option as "virtual" -groups will not be present in the checkgroups file. +The B<--sums> option will not work together with this option as +"virtual" groups will not be present in the checkgroups file. =item B<-r>, B<--report> I @@ -406,8 +410,8 @@ Choose the report type: I, I or I By default, B will report the number of postings for each newsgroup in each month. But it can also report the average number of -postings per group for all months or the total sum of postings per group -for all months. +postings per group for all months or the total sum of postings per +group for all months. For report types I and I, the B option has no meaning and will be silently ignored (see below). @@ -426,12 +430,13 @@ Set the boundary type to one of I, I, I or I. By default, all newsgroups with more postings per month than the upper -boundary and/or less postings per month than the lower boundary will be -excluded from further processing. For the default report that means each -month only newsgroups with a number of postings between the boundaries -will be displayed. For the other report types, newsgroups with a number of -postings exceeding the boundaries in all (!) months will not be -considered. +boundary and/or less postings per month than the lower boundary will +be +excluded from further processing. For the default report that means +each month only newsgroups with a number of postings between the +boundaries will be displayed. For the other report types, newsgroups +with a number of postings exceeding the boundaries in all (!) months +will not be considered. For example, lets take a list of newsgroups like this: @@ -461,22 +466,23 @@ month. If you want to list all newsgroups with more than 25 postings I, you'll have to set the boundary type to I, see below. A boundary type of I will show only those newsgroups - at all - -that satisfy the boundaries in each and every single month. With the above -list of newsgroups and +that satisfy the boundaries in each and every single month. With the +above list of newsgroups and C, you'll get this result: ----- All months: de.comp.datenbanken.ms-access 293 -de.comp.datenbanken.mysql has not been considered because it had less than -25 postings in 2012-02 (only). +de.comp.datenbanken.mysql has not been considered because it had less +than 25 postings in 2012-02 (only). -You can use that to get a list of newsgroups that have more (or less) then -x postings in every month during the whole reporting period. +You can use that to get a list of newsgroups that have more (or less) +then x postings in every month during the whole reporting period. -A boundary type of I will show only those newsgroups - at all -that -satisfy the boundaries on average. With the above list of newsgroups and +A boundary type of I will show only those newsgroups - at +all - that satisfy the boundaries on average. With the above list of +newsgroups and C, you'll get this result: @@ -491,8 +497,8 @@ The average number of postings in the three groups is: de.comp.datenbanken.mysql 48.33 Last but not least, a boundary type of I will show only those -newsgroups - at all - that satisfy the boundaries with the total sum of -all postings during the reporting period. With the above list of +newsgroups - at all - that satisfy the boundaries with the total sum +of all postings during the reporting period. With the above list of newsgroups and C, you'll finally get this result: @@ -505,8 +511,8 @@ you'll finally get this result: =item B<-g>, B<--group-by> I -By default, all results are grouped by month, sorted chronologically in -ascending order, like this: +By default, all results are grouped by month, sorted chronologically +in ascending order, like this: ----- 2012-01: de.comp.datenbanken.ms-access 84 @@ -525,8 +531,8 @@ B<--group-by> I: 2012-01 88 2012-02 21 -By appending I<-desc> to the group-by option parameter, you can reverse -the sort order - e.g. B<--group-by> I will give: +By appending I<-desc> to the group-by option parameter, you can +reverse the sort order - e.g. B<--group-by> I will give: ----- 2012-02: de.comp.datenbanken.ms-access 126 @@ -541,9 +547,9 @@ this option will therefore be ignored. =item B<-o>, B<--order-by> I Within each group (a single month or single newsgroup, see above), the -report will be sorted by newsgroup names in ascending alphabetical order -by default. You can change the sort order to descending or sort by number -of postings instead. +report will be sorted by newsgroup names in ascending alphabetical +order by default. You can change the sort order to descending or sort +by number of postings instead. =item B<-f>, B<--format> I @@ -587,19 +593,19 @@ False by default. =item B<--comments|--nocomments> -Add comments (group headers) to I and I output. True by default -as logn as B<--filetemplate> is not set. +Add comments (group headers) to I and I output. True by +default as long as B<--filetemplate> is not set. -Use I<--nocomments> to suppress anything except newsgroup names/months and -numbers of postings. +Use I<--nocomments> to suppress anything except newsgroup names/months +and numbers of postings. =item B<--filetemplate> I -Save output to file(s) instead of dumping it to STDOUT. B will -create one file for each month (or each newsgroup, accordant to the -setting of B<--group-by>, see above), with filenames composed by adding -year and month (or newsgroup names) to the I, for -example with B<--filetemplate> I: +Save output to file(s) instead of dumping it to STDOUT. B +will create one file for each month (or each newsgroup, according to +the setting of B<--group-by>, see above), with filenames composed by +adding year and month (or newsgroup names) to the I, +for example with B<--filetemplate> I: stats-2012-01 stats-2012-02 @@ -611,7 +617,7 @@ Override I from F. =item B<--conffile> I -Load configuration from I instead of F. +Read configuration from I instead of F. =back @@ -635,9 +641,9 @@ by number of postings, descending, in I format: groupstats --upper 30 --order-by postings-desc -Show the total of all postings for the year of 2010 for all groups that -had 30 postings or less in every single month in that year, ordered by -number of postings in descending order: +Show the total of all postings for the year of 2010 for all groups +that had 30 postings or less in every single month in that year, +ordered by number of postings in descending order: groupstats -m 2010-01:2010-12 -u 30 -b level -r sums -o postings-desc @@ -651,7 +657,6 @@ machine-readable form (without formatting): groupstats -m 2010-01:2010-12 -f dump --filetemplate stats - =head1 FILES =over 4 diff --git a/bin/hoststats.pl b/bin/hoststats.pl index 9e949ff..7c38d0d 100755 --- a/bin/hoststats.pl +++ b/bin/hoststats.pl @@ -197,7 +197,7 @@ See L. =head1 DESCRIPTION -This script create reports on newsgroup usage (number of postings from +This script creates reports on newsgroup usage (number of postings from each host) taken from result tables created by B. =head2 Features and options @@ -236,18 +236,19 @@ please see below. The results will be formatted as a kind of table; you can change the output format to a simple list or just a list of names and number of -postings with the B<--format> option. Captions will be added by means of -the B<--caption> option; all comments (and captions) can be supressed by -using B<--nocomments>. +postings with the B<--format> option. Captions will be added by means +of the B<--caption> option; all comments (and captions) can be +supressed by using B<--nocomments>. -Last but not least you can redirect all output to a number of files, e.g. -one for each month, by submitting the B<--filetemplate> option, see below. +Last but not least you can redirect all output to a number of files, +e.g. one for each month, by submitting the B<--filetemplate> option, +see below. =head2 Configuration B will read its configuration from F -which should be present in etc/ via Config::Auto or from a configuration file -submitted by the B<--conffile> option. +which should be present in etc/ via Config::Auto or from a configuration +file submitted by the B<--conffile> option. See doc/INSTALL for an overview of possible configuration options. @@ -259,18 +260,18 @@ You can override some configuration options via the B<--db> option. =item B<-V>, B<--version> -Print out version and copyright information and exit. +Display version and copyright information and exit. =item B<-h>, B<--help> -Print this man page and exit. +Display this man page and exit. =item B<-m>, B<--month> I Set processing period to a single month in YYYY-MM format or to a time period between two month in YYYY-MM:YYYY-MM format (two month, separated by a colon). By using the keyword I instead, you can set no -processing period to process the whole database. +processing period to process the whole database. Defaults to last month. =item B<-n>, B<--names> I @@ -284,7 +285,8 @@ example =item B<-s>, B<--sums|--nosums> (sum per month) Include a "virtual" host named "ALL" for every month in output, -containing the sum of all detected hosts for that month. +containing the sum of all detected hosts for that month. False +by default. =item B<-r>, B<--report> I @@ -315,8 +317,8 @@ considered. =item B<-g>, B<--group-by> I -By default, all results are grouped by month, sorted chronologically in -ascending order, like this: +By default, all results are grouped by month, sorted chronologically +in ascending order, like this: # ----- 2012-01: arcor-online.net : 9379 @@ -339,8 +341,8 @@ The results can be grouped by host instead via B<--group-by> I: 2012-01: 9063 2012-02: 7879 -By appending I<-desc> to the group-by option parameter, you can reverse -the sort order - e.g. B<--group-by> I will give: +By appending I<-desc> to the group-by option parameter, you can +reverse the sort order - e.g. B<--group-by> I will give: # ----- 2012-02: arcor-online.net : 8606 @@ -427,7 +429,7 @@ Override I from F. =item B<--conffile> I -Load configuration from I instead of F. +Read configuration from I instead of F. =back @@ -437,7 +439,7 @@ See L. =head1 EXAMPLES -Show number of postings per group for lasth month in I format: +Show number of postings per host for lasth month in I format: hoststats diff --git a/bin/postingstats.pl b/bin/postingstats.pl index e517a45..0b362c3 100755 --- a/bin/postingstats.pl +++ b/bin/postingstats.pl @@ -268,7 +268,7 @@ postingstats - format and post reports =head1 SYNOPSIS -B B<-t> I [B<-Vh> [B<-m> I] +B [B<-Vh>] [B<-t> I] [B<-m> I] =head1 REQUIREMENTS @@ -285,11 +285,13 @@ message that can be posted to Usenet. B will create a table with entries numbered from most to least and percentages calculated from the sum total of all values. -It depends on a sorted list on STDIN in I format with I. +It depends on a sorted list on STDIN in I format with I; +I from B are optional. B needs a B<--type> and a B<--month> to create a caption and select matching lead-ins and lead-outs. B<--type> is also needed -to catch the correct sum total from input. +to catch the correct sum total from input which differs between I +on one hand and I or I on the other hand. It will default to posting statistics (number of postings per group) and last month. @@ -308,7 +310,7 @@ C<----- configuration -----> section. =item C<$TLH> Top level hierarchy the report was created for. Used for display and -sum total. +sum total (only for I). =item C<%Heading> @@ -327,13 +329,13 @@ Output will be truncated otherwise. =item C<%LeadIn> Hash with keys for I, I and I. -Used to create the headers for our posting. Can contain other text +Used to create the headers for the postings. Can contain other text that will be shown before C<%Heading>. =item C<%LeadOut> Hash with keys for I, I and I. -Will be shown at the end of our posting. +Will be shown at the end of the posting. =back @@ -343,11 +345,11 @@ Will be shown at the end of our posting. =item B<-V>, B<--version> -Print out version and copyright information and exit. +Display version and copyright information and exit. =item B<-h>, B<--help> -Print this man page and exit. +Display this man page and exit. =item B<-t>, B<--type> I @@ -356,7 +358,7 @@ statistics accordingly. =item B<-m>, B<--month> I -Set month for display. +Set month (for display only). =back @@ -372,7 +374,7 @@ Create a posting from a posting statistics report for last month: Create a posting from a posting statistics report for 2012-01: - groupstats.pl --nocomments --sums --format dump | postingstats.pl -t groups -m 2012-01 + groupstats.pl --nocomments --sums --format dump -m 2012-01 | postingstats.pl -t groups -m 2012-01 Create a posting from a host statistics report for last month: diff --git a/doc/ChangeLog b/doc/ChangeLog index 857b23e..d327650 100644 --- a/doc/ChangeLog +++ b/doc/ChangeLog @@ -16,6 +16,7 @@ NewsStats 0.4.0 (unreleased) * Add ClientStats to dopostingstats. * Let dopostingstats default to last month. * Set executable bit for new scripts. + * Update documentation. NewsStats 0.3.0 (2025-05-18) * Extract GroupStats (in gatherstats) to subroutine. diff --git a/doc/INSTALL b/doc/INSTALL index 9a37207..3588c21 100644 --- a/doc/INSTALL +++ b/doc/INSTALL @@ -1,11 +1,12 @@ NewsStats (c) 2010-2013, 2025 Thomas Hochstein -NewsStats is a software package used to gather statistical information -from a live Usenet feed and for its subsequent examination. +NewsStats is a software package that can be used to collect +statistical information from a live Usenet feed and then analyze it +to create statistical reports. -This script package is free software; you can redistribute it and/or -modify it under the terms of the GNU Public License as published by -the Free Software Foundation. +This package is free software; you can redistribute it and/or modify +it under the terms of the GNU Public License as published by the Free +Software Foundation. --------------------------------------------------------------------- @@ -16,9 +17,10 @@ INSTALLATION INSTRUCTIONS * Download the current version of NewsStats from . - * Untar it into a directory of your choice: + * Untar it into a directory of your choice, i.e. /srv/newsstats: - # tar -xzf newsstats-nn.tar.gz + $ cd /srv + $ tar -xzf newsstats-n.n.n.tar.gz Scripts in this path - at least feedlog.pl - should be executable by the news user. @@ -28,8 +30,8 @@ INSTALLATION INSTRUCTIONS * Copy the sample configuration file newsstats.conf.sample to newsstats.conf and modify it for your purposes: - # cp etc/newsstats.conf.sample etc/newsstats.conf - # vim etc/newsstats.conf + $ cp etc/newsstats.conf.sample etc/newsstats.conf + $ vim etc/newsstats.conf a) Mandatory configuration options @@ -60,6 +62,9 @@ INSTALLATION INSTRUCTIONS * DBTableHosts = hosts_de Table holding data on postings per server. + * DBTableClnts = clients_de + Table holding data on postings per client. + b) Optional configuration options * TLH = de.alt,news.admin @@ -68,19 +73,21 @@ INSTALLATION INSTRUCTIONS 3) Database (mysql) setup - * Setup your database server with a username, password and - database matching the NewsStats configuration (see 2 a). + * Setup your database server with an username, a password and + (optionally) a database matching the NewsStats configuration + (see 2 a). * Start the database creation script: - # bin/dbcreate.pl + $ bin/dbcreate.pl - It will setup the necessary database tables and display some - information on the next steps. + It will create the database (if not already present), create the + necessary database tables and display some information on the + next steps. 4) Feed (INN) setup - You have to setup an INN feed to feedlog.pl. + You have to set up an INN feed to feedlog.pl. * Edit your 'newsfeeds' file and insert something like @@ -90,39 +97,39 @@ INSTALLATION INSTRUCTIONS :Tc,WmtfbsPNH,Ac:/path/to/feedlog.pl * You should only feed that hierarchy (those hierarchies ...) to - feedlog.pl you'll want to cover with your statistical - examination. It may be a good idea to setup different feeds (to - different databases ...) for different hierarchies. + feedlog.pl that you want to cover with your statistical analysis. + It may be a good idea to setup different feeds (to different + databases ...) for different hierarchies. * Please double check that your path to feedlog.pl is correct and feedlog.pl can be executed by the news user * Check your 'newsfeeds' syntax: - # ctlinnd checkfile + $ ctlinnd checkfile * Reload 'newsfeeds': - # ctlinnd reload newsfeeds 'Adding newsstats! feed' + $ ctlinnd reload newsfeeds 'Adding newsstats! feed' * Watch your 'news.notice' and 'errlog' files: - # tail -f /var/log/news/news.notice + $ tail -f /var/log/news/news.notice ... - # tail -f /var/log/news/errlog + $ tail -f /var/log/news/errlog Everything should be going smoothly now. * If INN is spewing error messages to 'errlog' or reporting continous respawns of feedlog.pl to 'news.notice', stop your feed: - # ctlinnd drop 'newsstats!' + $ ctlinnd drop 'newsstats!' and investigate. 'errlog' may be helpful here. * You can restart the feed with - # ctlinnd begin 'newsstats!' + $ ctlinnd begin 'newsstats!' later. diff --git a/doc/README b/doc/README index f6b1f44..de7e56c 100644 --- a/doc/README +++ b/doc/README @@ -1,21 +1,21 @@ NewsStats (c) 2010-2013, 2025 Thomas Hochstein NewsStats is a software package for gathering statistical data live -from a Usenet feed and subsequent examination. +from a Usenet feed and subsequent analysis. -This script package is free software; you can redistribute it and/or -modify it under the terms of the GNU Public License as published by -the Free Software Foundation. +This package is free software; you can redistribute it and/or modify +it under the terms of the GNU Public License as published by the Free +Software Foundation. --------------------------------------------------------------------- What's that? - There's a multitude of tools for the statistical examination of - newsgroups: number of postings per month or per person, longest - threads, and so on (see - [German language] for an incomplete list). Most of them use a per- - newsgroup approach while NewsStats is hierarchy oriented. + There's a multitude of tools to create statistics about newsgroup + usage: number of postings per month or per person, longest threads, + and so on (see [German language] + for an incomplete list). Most of them use a per-newsgroup approach + while NewsStats is hierarchy oriented. NewsStats will accumulate data from a live INN feed, allowing you to process the saved information later on. @@ -40,7 +40,9 @@ Prerequisites * Perl 5.8.x with standard modules - Cwd + - Encode - File::Basename + - Getopt::Long - Sys::Syslog * Perl modules from CPAN @@ -50,7 +52,7 @@ Prerequisites * mysql 5.0.x - * working installation of INN + * a working installation of INN Installation instructions @@ -67,15 +69,52 @@ Getting Started table. See the feedlog.pl man page for more information. You can process that data via 'gatherstats.pl'; currently the - tabulation of postings per group and injection server per month is - supported. Tabulation of clients (newsreaders) is planned. See - the gatherstats.pl man page for more information. + tabulation of postings per group, injection server and posting + agent (newsreader) per month is supported. See the gatherstats.pl + man page for more information. + + Example: + + bin/gatherstats.pl + + will parse raw data from the last month and save the results in + tables for postings per group, server and client, respectively. Report generation is handled by specialised scripts for each - report type. Currently reports on the number of postings per group - and month and injection server and month are supported; you can - use 'groupstats.pl' and 'hoststats.pl' for that. See the - groupstats.pl and hoststats.pl man pages for more information. + report type: 'groupstats.pl' for postings per group + (s), 'hoststats.pl' for postings per injection server + (s) and 'clientstats.pl' for postings per posting agent. See the + groupstats.pl, hoststats.pl and clientstats.pl man pages for more + information. + + Example: + + bin/groupstats.pl -o postings-desc + bin/hoststats.pl -o postings-desc + bin/clientstats.pl -o postings-desc -v + + will show reports for postings per group, per injection server and + per client (with detailed client versions) for the last month, + using the result tables filled by gatherstats. + + To post those reports to Usenet, change postingstats.pl according + to your needs (sender, newsgroups and other headers, translation + of table headers and text templates) and display a test posting + by piping report data into postingstats.pl: + + bin/groupstats.pl --nocomments -s -f dump | bin/postingstats.pl + + If the result is to your liking, add a pipe to a inews + implementation. + + Example: + + bin/groupstats.pl --nocomments -s -f dump | bin/postingstats.pl | contrib/tinews.pl -X + +More information + + See the man pages for 'gatherstats' and the report generating + scripts. Reporting Bugs @@ -87,7 +126,7 @@ Reporting Bugs Development - This program is maintained using the Git version control system at + This package is maintained using the Git version control system at . Related projects diff --git a/doc/TODO b/doc/TODO index a376c53..9f5e0a1 100644 --- a/doc/TODO +++ b/doc/TODO @@ -1,12 +1,10 @@ NewsStats To-Do List ==================== -This is a list of planned bug fixes, improvements and enhancements for +This is a list of possible bug fixes, improvements and enhancements for NewsStats. * General - - Improve Documentation - The documentation is rather sparse and could use some improvement. - Add a test suite There is currently no kind of test suite or regression tests. Something like that is badly needed. @@ -27,8 +25,6 @@ NewsStats. for late creation and deletion), optionally including the previously mentioned information; and you should be able to get the history of any group. - - Add other reports - NewsStats should include some other kinds of reports (stats on used clients) - Add tools for database management NewsStats should offer tools e.g. to inject postings into the 'raw' database, or to split databases. @@ -53,23 +49,11 @@ NewsStats. Some other tests - working database connection, valid database and table names - would be nice. - + install/install.pl - - Read current version from a file dropped and updated by installer - - Add / enhance / test error handling - - General tests and optimisations - - + feedlog.pl - - Add / enhance / test error handling - - General tests and optimisations - + gatherstats.pl - Use hierarchy information (see GroupInfo above) - - Add gathering of other stats (clients, ...) - - better modularisation (code reuse for other reports!) - Add / enhance / test error handling - - General tests and optimisations - + groupstats.pl - - better modularisation (code reuse for other reports!) + + groupstats.pl, hoststats.pl, clientstats.pl + - better modularisation (code reuse) - Add / enhance / test error handling - General tests and optimisations