thh/newsstats
1
0
Fork 0
Code Issues 3 Pull requests Packages Projects Releases Activity

Update documentation.

Browse source 
- Fix clientstats doc (copied from hoststats).
- Add some more examples ro README.

Signed-off-by: Thomas Hochstein <thh@thh.name>
This commit is contained in:
Thomas Hochstein 2025-06-01 16:39:25 +02:00
parent 0b87e81b08
commit c4360c074f
10 changed files with 328 additions and 263 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

178
bin/clientstats.pl
View file

@ -275,7 +275,7 @@ clientstats - create reports on client usage
=head1 SYNOPSIS
B<clientstats> [B<-Vhcs> B<--comments>] [B<-m> I<YYYY-MM>[:I<YYYY-MM>] | I<all>] [B<-n> I<client(s)>] [B<-r> I<report type>] [B<-l> I<lower boundary>] [B<-u> I<upper boundary>] [B<-g> I<group by>] [B<-o> I<order by>] [B<-f> I<output format>] [B<--filetemplate> I<filename template>] [B<--db> I<database table>] [B<--conffile> I<filename>]
B<clientstats> [B<-Vhcsv> B<--comments>] [B<-m> I<YYYY-MM>[:I<YYYY-MM>] | I<all>] [B<-n> I<client(s)>] [B<-r> I<report type>] [B<-l> I<lower boundary>] [B<-u> I<upper boundary>] [B<-g> I<group by>] [B<-o> I<order by>] [B<-f> I<output format>] [B<--filetemplate> I<filename template>] [B<--db> I<database table>] [B<--conffile> I<filename>]
=head1 REQUIREMENTS
@ -323,18 +323,19 @@ details and exceptions, please see below.
The results will be formatted as a kind of table; you can change the
output format to a simple list or just a list of names and number of
postings with the B<--format> option. Captions will be added by means of
the B<--caption> option; all comments (and captions) can be supressed by
using B<--nocomments>.
postings with the B<--format> option. Captions will be added by means
of the B<--caption> option; all comments (and captions) can be
supressed by using B<--nocomments>.
Last but not least you can redirect all output to a number of files, e.g.
one for each month, by submitting the B<--filetemplate> option, see below.
Last but not least you can redirect all output to a number of files,
e.g. one for each month, by submitting the B<--filetemplate> option,
see below.
=head2 Configuration
B<clientstats> will read its configuration from F<newsstats.conf>
which should be present in etc/ via Config::Auto or from a configuration file
submitted by the B<--conffile> option.
which should be present in etc/ via Config::Auto or from a configuration
file submitted by the B<--conffile> option.
See doc/INSTALL for an overview of possible configuration options.
@ -346,32 +347,47 @@ You can override some configuration options via the B<--db> option.
=item B<-V>, B<--version>
Print out version and copyright information and exit.
Display version and copyright information and exit.
=item B<-h>, B<--help>
Print this man page and exit.
Display this man page and exit.
=item B<-m>, B<--month> I<YYYY-MM[:YYYY-MM]|all>
Set processing period to a single month in YYYY-MM format or to a time
period between two month in YYYY-MM:YYYY-MM format (two month, separated
by a colon). By using the keyword I<all> instead, you can set no
processing period to process the whole database.
processing period to process the whole database. Defaults to last month.
=item B<-n>, B<--names> I<name(s)>
Limit processing to a certain set of client names. I<names(s)>
can be a single name (eternal-september.org), a group of names
(*.inka.de) or a list of either of these, separated by colons, for
example
can be a single name (Thunderbird), a group of names (Ice*) or a list
of either of these, separated by colons, for example
eternal-september.org:solani.org:*.inka.de
Forte Agent:Thunderbird:Ice*
Spaces or special characters like "*" need to be quoted from the shell,
like
-n 'Forte Agent:Thunderbird:Ice*'
There is no way to limit processing to a specific version, but you can
alway grep through the output.
=item B<-s>, B<--sums|--nosums> (sum per month)
Include "virtual" clients named "ALL" for every month in output,
containing the sum of all detected clients for that month.
containing the sum of all detected clients for that month. False
by default.
=item B<-v>, B<--versions|--noversions> (client versions)
Include a list of all observed versions of each client in output.
Version information will be displayed with indents ('-') below each
client, sorted in the same way (by postings or alphanumeric). False
by default.
=item B<-r>, B<--report> I<default|sums>
@ -379,7 +395,7 @@ Choose the report type: I<default> or I<sums>
By default, B<clientstats> will report the number of postings for each
client in each month. But it can also report the total sum of postings
per client for all months.
per client for all months. Sums of B<--versions> can be included.
For report type I<sums>, the B<group-by> option has no meaning and
will be silently ignored (see below).
@ -395,10 +411,10 @@ Set the upper boundary.
By default, all clients with more postings per month than the
upper boundary and/or less postings per month than the lower boundary
will be excluded from further processing. For the default report that
means each month only /clients with a number of postings between
the boundaries will be displayed. For the sums report, /clients
with a number of postings exceeding the boundaries in all (!) months
will not be considered.
means each month only clients with a number of postings between the
boundaries will be displayed. For the sums report, clients with a
number of postings exceeding the boundaries in all (!) months will
not be considered.
=item B<-g>, B<--group-by> I<month[-desc]|name[-desc]>
@ -406,38 +422,38 @@ By default, all results are grouped by month, sorted chronologically in
ascending order, like this:
# ----- 2012-01:
arcor-online.net : 9379
individual.net : 19525
news.albasani.net: 9063
40tude_Dialog: 5873
Forte Agent : 7735
Thunderbird : 20925
# ----- 2012-02:
arcor-online.net : 8606
individual.net : 16768
news.albasani.net: 7879
40tude_Dialog: 4142
Forte Agent : 5895
Thunderbird : 19091
The results can be grouped by client instead via
B<--group-by> I<name>:
----- individual.net
2012-01: 19525
2012-02: 16768
----- arcor-online.net
2012-01: 9379
2012-02: 8606
----- news.albasani.net
2012-01: 9063
2012-02: 7879
# ----- 40tude_Dialog:
2012-01: 5873
2012-02: 4142
# ----- Forte Agent:
2012-01: 7735
2012-02: 5895
# ----- Thunderbird:
2012-01: 20925
2012-02: 19091
By appending I<-desc> to the group-by option parameter, you can reverse
the sort order - e.g. B<--group-by> I<month-desc> will give:
# ----- 2012-02:
arcor-online.net : 8606
individual.net : 16768
news.albasani.net: 7879
40tude_Dialog: 4142
Forte Agent : 5895
Thunderbird : 19091
# ----- 2012-01:
arcor-online.net : 9379
individual.net : 19525
news.albasani.net: 9063
40tude_Dialog: 5873
Forte Agent : 7735
Thunderbird : 20925
Sums reports (see above) will always be grouped by months; this option
will therefore be ignored.
@ -449,41 +465,57 @@ the report will be sorted by name (or month) in ascending alphabetical
order by default. You can change the sort order to descending or sort
by number of postings instead.
=item B<-f>, B<--format> I<pretty|list|dump>
Select the output format, I<pretty> being the default:
By default, output is sorted alphabetically:
# ----- 2012-01:
arcor-online.net : 9379
individual.net : 19525
40tude_Dialog: 5873
Forte Agent : 7735
Thunderbird : 20925
Using B<--order-by> I<postings[-desc]>, it will be sorted from most
to least postings:
# ----- 2012-01:
Thunderbird : 20925
Forte Agent : 7735
40tude_Dialog: 5873
=item B<-f>, B<--format> I<pretty|list|dump>
Select the output format, I<pretty> (a kind of table) being the default:
# ----- 2012-01:
40tude_Dialog: 5873
Forte Agent : 7735
# ----- 2012-02:
arcor-online.net : 8606
individual.net : 16768
40tude_Dialog: 4142
Forte Agent : 5895
I<list> format looks like this:
I<list> format looks like this (each client preceded by month):
2012-01 arcor-online.net 9379
2012-01 individual.net 19525
2012-02 arcor-online.net 8606
2012-02 individual.net 16768
2012-01 40tude_Dialog 5873
2012-01 Forte Agent 7735
2012-02 40tude_Dialog 4142
2012-02 Forte Agent 5895
And I<dump> format looks like this:
# 2012-01:
arcor-online.net 9379
individual.net 19525
40tude_Dialog 5873
Forte Agent 7735
# 2012-02:
arcor-online.net 8606
individual.net 16768
40tude_Dialog 4142
Forte Agent 5895
You can remove the comments by using B<--nocomments>, see below.
You can remove the comments (lines after '#') by using B<--nocomments>,
see below.
=item B<-c>, B<--captions|--nocaptions>
Add captions to output, like this:
----- Report for 2012-01 to 2012-02 (number of postings for each month)
----- Names: individual.net
----- Names: Thunderbird
----- Threshold: 8000 => x (counting only month fulfilling this condition)
----- Grouped by Month (ascending), sorted by number of postings descending
@ -491,11 +523,11 @@ False by default.
=item B<--comments|--nocomments>
Add comments (group headers) to I<dump> and I<pretty> output. True by default
as logn as B<--filetemplate> is not set.
Add comments (group headers) to I<dump> and I<pretty> output. True by
default as long as B<--filetemplate> is not set.
Use I<--nocomments> to suppress anything except client names or months and
numbers of postings.
Use I<--nocomments> to suppress anything except client names or months
and numbers of postings.
=item B<--filetemplate> I<filename template>
@ -515,7 +547,7 @@ Override I<DBTableClnts> or I<DBTableClnts> from F<newsstats.conf>.
=item B<--conffile> I<filename>
Load configuration from I<filename> instead of F<newsstats.conf>.
Read configuration from I<filename> instead of F<newsstats.conf>.
=back
@ -525,26 +557,26 @@ See L<doc/INSTALL>.
=head1 EXAMPLES
Show number of postings per group for lasth month in I<pretty> format:
Show number of postings per client for lasth month in I<pretty> format:
clientstats
Show that report for January of 2010 and *.inka plus individual.net:
Show that report for January of 2010 and Thunderbird plus Ice*:
clientstats --month 2010-01 --names *.inka:individual.net:
clientstats --month 2010-01 --names 'Thunderbird:Ice*'
Only show clients with 30 postings or less last month, ordered
by number of postings, descending, in I<pretty> format:
Only show clients with at least 30 postings last month and the versions
of those clients, ordered each by number of postings, descending,
in I<pretty> format:
clientstats --upper 30 --order-by postings-desc
clientstats --lower 30 --versions --order-by postings-desc
List number of postings per host for each month of 2010 and redirect
List number of postings per client for each month of 2010 and redirect
output to one file for each month, named hosts-2010-01 and so on, in
machine-readable form (without formatting):
clientstats -m 2010-01:2010-12 -f dump --filetemplate hosts
=head1 FILES
=over 4

19
bin/dbcreate.pl
View file

@ -168,7 +168,7 @@ INSTALL
my $Upgrade ='';
if ($OptUpdate) {
$Upgrade = <<UPGRADE;
$Upgrade = <<UPGRADE;
----------
Your installation was upgraded from $OptUpdate to $VERSION.
@ -218,16 +218,7 @@ if (!$OptUpdate) {
} else {
##### upgrade mode
print "----------\nStarting upgrade process.\n";
my $PackageVersion = '0.03';
if ($OptUpdate < $PackageVersion) {
if ($OptUpdate < 0.02) {
# 0.01 -> 0.02
# &DoMySQL('...;');
# print "v0.02: Database upgrades ...\n";
# &PrintInstructions('0.02',<<" INSTRUCTIONS");
# INSTRUCTIONS
};
};
# TBD
# Display general upgrade instructions
print $Upgrade;
};
@ -307,11 +298,11 @@ See L<doc/INSTALL> for an overview of possible configuration options.
=item B<-V>, B<--version>
Print out version and copyright information and exit.
Display version and copyright information and exit.
=item B<-h>, B<--help>
Print this man page and exit.
Display this man page and exit.
=item B<-u>, B<--update> I<version>
@ -319,7 +310,7 @@ Don't do a fresh install, but update from I<version>.
=item B<--conffile> I<filename>
Load configuration from I<filename> instead of F<newsstats.conf>.
Read configuration from I<filename> instead of F<newsstats.conf>.
=back

13
bin/feedlog.pl
View file

@ -167,8 +167,9 @@ time.
All reporting is done to I<syslog> via I<news> facility. If B<feedlog>
fails to initiate a database connection at startup, it will log to
I<syslog> with I<CRIT> priority and go in an endless loop, as
terminating would only result in a rapid respawn.
I<syslog> with I<CRIT> priority and go in an endless loop, trying again
to connect every 5 seconds, as terminating would only result in a rapid
respawn.
=head2 Configuration
@ -184,15 +185,15 @@ See L<doc/INSTALL> for an overview of possible configuration options.
=item B<-V>, B<--version>
Print out version and copyright information and exit.
Display version and copyright information and exit.
=item B<-h>, B<--help>
Print this man page and exit.
Display this man page and exit.
=item B<-d>, B<--debug>
Output debugging information to STDERR while parsing STDIN. You'll
Print debugging information to STDERR while parsing STDIN. You'll
find that information most probably in your B<INN> F<errlog> file.
=item B<-q>, B<--quiet>
@ -201,7 +202,7 @@ Suppress logging to syslog.
=item B<--conffile> I<filename>
Load configuration from I<filename> instead of F<newsstats.conf>.
Read configuration from I<filename> instead of F<newsstats.conf>.
=back

22
bin/gatherstats.pl
View file

@ -803,12 +803,12 @@ override that default through the B<--clientsdb> option.
=head2 Configuration
B<gatherstats> will read its configuration from F<newsstats.conf>
which should be present in etc/ via Config::Auto or from a configuration file
submitted by the B<--conffile> option.
which should be present in etc/ via Config::Auto or from a configuration
file submitted by the B<--conffile> option.
See L<doc/INSTALL> for an overview of possible configuration options.
You can override configuration options via the B<--hierarchy>,
You can override configuration options by using the B<--hierarchy>,
B<--rawdb>, B<--groupsdb>, B<--clientsdb> and B<--hostsdb> options,
respectively.
@ -818,15 +818,15 @@ respectively.
=item B<-V>, B<--version>
Print out version and copyright information and exit.
Display version and copyright information and exit.
=item B<-h>, B<--help>
Print this man page and exit.
Display this man page and exit.
=item B<-d>, B<--debug>
Output debugging information to STDOUT while processing (number of
Print debugging information to STDOUT while processing (number of
postings per group).
=item B<-t>, B<--test>
@ -838,15 +838,17 @@ conjunction with B<--test> ... everything else seems a bit pointless.
Set processing period to a single month in YYYY-MM format or to a time
period between two month in YYYY-MM:YYYY-MM format (two month, separated
by a colon).
by a colon). Defaults to last month.
=item B<-s>, B<--stats> I<type>
Set processing type to one of I<all>, I<groups> or I<hosts>. Defaults
to all.
Set processing type to one of I<all>, I<groups>, I<hosts> or I<clients>.
Defaults to I<all>.
=item B<-c>, B<--checkgroups> I<filename template>
Relevant only for newsgroup stats (I<groups>).
Check each group against a list of valid newsgroups read from a file,
one group on each line and ignoring everything after the first
whitespace (so you can use a file in checkgroups format or (part of)
@ -889,7 +891,7 @@ Override I<DBTableClnts> from F<newsstats.conf>.
=item B<--conffile> I<filename>
Load configuration from I<filename> instead of F<newsstats.conf>.
Read configuration from I<filename> instead of F<newsstats.conf>.
=back

147
bin/groupstats.pl
View file

@ -283,7 +283,7 @@ See L<doc/README>.
=head1 DESCRIPTION
This script create reports on newsgroup usage (number of postings per
This script creates reports on newsgroup usage (number of postings per
group per month) taken from result tables created by
B<gatherstats.pl>.
@ -291,16 +291,16 @@ B<gatherstats.pl>.
=head3 Time period and newsgroups
The time period to act on defaults to last month; you can assign another
time period or a single month (or drop all time constraints) via the
B<--month> option (see below).
The time period to act on defaults to last month; you can assign
another time period or a single month (or drop all time constraints)
via the B<--month> option (see below).
B<groupstats> will process all newsgroups by default; you can limit
processing to only some newsgroups by supplying a list of those groups via
B<--newsgroups> option (see below). You can include hierarchy levels in
the output by adding the B<--sums> switch (see below). Optionally
newsgroups not present in a checkgroups file can be excluded from output,
sse B<--checkgroups> below.
processing to only some newsgroups by supplying a list of those groups
via B<--newsgroups> option (see below). You can include hierarchy
levels in the output by adding the B<--sums> switch (see below).
Optionally newsgroups not present in a checkgroups file can be excluded
from output, sse B<--checkgroups> below.
=head3 Report type
@ -321,26 +321,27 @@ below.
=head3 Sorting and formatting the output
By default, all results are grouped by month; you can group results by
newsgroup instead via the B<--groupy-by> option. Within those groups, the
list of newsgroups (or months) is sorted alphabetically (or
chronologically, respectively) ascending. You can change that order (and
sort by number of postings) with the B<--order-by> option. For details and
exceptions, please see below.
newsgroup instead via the B<--groupy-by> option. Within those groups,
the list of newsgroups (or months) is sorted alphabetically (or
chronologically, respectively) ascending. You can change that order
(and sort by number of postings) with the B<--order-by> option. For
details and exceptions, please see below.
The results will be formatted as a kind of table; you can change the
output format to a simple list or just a list of newsgroups and number of
postings with the B<--format> option. Captions will be added by means of
the B<--caption> option; all comments (and captions) can be supressed by
using B<--nocomments>.
output format to a simple list or just a list of newsgroups and number
of postings with the B<--format> option. Captions will be added by means
of the B<--caption> option; all comments (and captions) can be supressed
by using B<--nocomments>.
Last but not least you can redirect all output to a number of files, e.g.
one for each month, by submitting the B<--filetemplate> option, see below.
Last but not least you can redirect all output to a number of files,
e.g. one for each month, by submitting the B<--filetemplate> option,
see below.
=head2 Configuration
B<groupstats> will read its configuration from F<newsstats.conf>
which should be present in etc/ via Config::Auto or from a configuration file
submitted by the B<--conffile> option.
which should be present in etc/ via Config::Auto or from a configuration
file submitted by the B<--conffile> option.
See doc/INSTALL for an overview of possible configuration options.
@ -352,18 +353,18 @@ You can override some configuration options via the B<--groupsdb> option.
=item B<-V>, B<--version>
Print out version and copyright information and exit.
Display version and copyright information and exit.
=item B<-h>, B<--help>
Print this man page and exit.
Display this man page and exit.
=item B<-m>, B<--month> I<YYYY-MM[:YYYY-MM]|all>
Set processing period to a single month in YYYY-MM format or to a time
period between two month in YYYY-MM:YYYY-MM format (two month, separated
by a colon). By using the keyword I<all> instead, you can set no
processing period to process the whole database.
processing period to process the whole database. Defaults to last month.
=item B<-n>, B<--newsgroups> I<newsgroup(s)>
@ -388,17 +389,20 @@ See the B<gatherstats> man page for details.
This option does not work together with the B<--checkgroups> option as
all "virtual" groups will not be present in the checkgroups file.
False by default.
=item B<--checkgroups> I<filename>
Restrict output to those newgroups present in a file in checkgroups format
(one newgroup name per line; everything after the first whitespace on each
line is ignored). All other newsgroups will be removed from output.
Restrict output to those newgroups present in a file in checkgroups
format (one newgroup name per line; everything after the first
whitespace on each line is ignored). All other newsgroups will be
removed from output.
Contrary to B<gatherstats>, I<filename> is not a template, but refers to
a single file in checkgroups format.
Contrary to B<gatherstats>, I<filename> is not a template, but refers
to a single file in checkgroups format.
The B<--sums> option will not work together with this option as "virtual"
groups will not be present in the checkgroups file.
The B<--sums> option will not work together with this option as
"virtual" groups will not be present in the checkgroups file.
=item B<-r>, B<--report> I<default|average|sums>
@ -406,8 +410,8 @@ Choose the report type: I<default>, I<average> or I<sums>
By default, B<groupstats> will report the number of postings for each
newsgroup in each month. But it can also report the average number of
postings per group for all months or the total sum of postings per group
for all months.
postings per group for all months or the total sum of postings per
group for all months.
For report types I<average> and I<sums>, the B<group-by> option has no
meaning and will be silently ignored (see below).
@ -426,12 +430,13 @@ Set the boundary type to one of I<default>, I<level>, I<average> or
I<sums>.
By default, all newsgroups with more postings per month than the upper
boundary and/or less postings per month than the lower boundary will be
excluded from further processing. For the default report that means each
month only newsgroups with a number of postings between the boundaries
will be displayed. For the other report types, newsgroups with a number of
postings exceeding the boundaries in all (!) months will not be
considered.
boundary and/or less postings per month than the lower boundary will
be
excluded from further processing. For the default report that means
each month only newsgroups with a number of postings between the
boundaries will be displayed. For the other report types, newsgroups
with a number of postings exceeding the boundaries in all (!) months
will not be considered.
For example, lets take a list of newsgroups like this:
@ -461,22 +466,23 @@ month. If you want to list all newsgroups with more than 25 postings
I<in total>, you'll have to set the boundary type to I<sum>, see below.
A boundary type of I<level> will show only those newsgroups - at all -
that satisfy the boundaries in each and every single month. With the above
list of newsgroups and
that satisfy the boundaries in each and every single month. With the
above list of newsgroups and
C<groupstats --month 2012-01:2012-03 --lower 25 --boundary level --report sums>,
you'll get this result:
----- All months:
de.comp.datenbanken.ms-access 293
de.comp.datenbanken.mysql has not been considered because it had less than
25 postings in 2012-02 (only).
de.comp.datenbanken.mysql has not been considered because it had less
than 25 postings in 2012-02 (only).
You can use that to get a list of newsgroups that have more (or less) then
x postings in every month during the whole reporting period.
You can use that to get a list of newsgroups that have more (or less)
then x postings in every month during the whole reporting period.
A boundary type of I<average> will show only those newsgroups - at all -that
satisfy the boundaries on average. With the above list of newsgroups and
A boundary type of I<average> will show only those newsgroups - at
all - that satisfy the boundaries on average. With the above list of
newsgroups and
C<groupstats --month 2012-01:2012-03 --lower 25 --boundary avg --report sums>,
you'll get this result:
@ -491,8 +497,8 @@ The average number of postings in the three groups is:
de.comp.datenbanken.mysql 48.33
Last but not least, a boundary type of I<sums> will show only those
newsgroups - at all - that satisfy the boundaries with the total sum of
all postings during the reporting period. With the above list of
newsgroups - at all - that satisfy the boundaries with the total sum
of all postings during the reporting period. With the above list of
newsgroups and
C<groupstats --month 2012-01:2012-03 --lower 25 --boundary sum --report sums>,
you'll finally get this result:
@ -505,8 +511,8 @@ you'll finally get this result:
=item B<-g>, B<--group-by> I<month[-desc]|newsgroups[-desc]>
By default, all results are grouped by month, sorted chronologically in
ascending order, like this:
By default, all results are grouped by month, sorted chronologically
in ascending order, like this:
----- 2012-01:
de.comp.datenbanken.ms-access 84
@ -525,8 +531,8 @@ B<--group-by> I<newsgroup>:
2012-01 88
2012-02 21
By appending I<-desc> to the group-by option parameter, you can reverse
the sort order - e.g. B<--group-by> I<month-desc> will give:
By appending I<-desc> to the group-by option parameter, you can
reverse the sort order - e.g. B<--group-by> I<month-desc> will give:
----- 2012-02:
de.comp.datenbanken.ms-access 126
@ -541,9 +547,9 @@ this option will therefore be ignored.
=item B<-o>, B<--order-by> I<default[-desc]|postings[-desc]>
Within each group (a single month or single newsgroup, see above), the
report will be sorted by newsgroup names in ascending alphabetical order
by default. You can change the sort order to descending or sort by number
of postings instead.
report will be sorted by newsgroup names in ascending alphabetical
order by default. You can change the sort order to descending or sort
by number of postings instead.
=item B<-f>, B<--format> I<pretty|list|dump>
@ -587,19 +593,19 @@ False by default.
=item B<--comments|--nocomments>
Add comments (group headers) to I<dump> and I<pretty> output. True by default
as logn as B<--filetemplate> is not set.
Add comments (group headers) to I<dump> and I<pretty> output. True by
default as long as B<--filetemplate> is not set.
Use I<--nocomments> to suppress anything except newsgroup names/months and
numbers of postings.
Use I<--nocomments> to suppress anything except newsgroup names/months
and numbers of postings.
=item B<--filetemplate> I<filename template>
Save output to file(s) instead of dumping it to STDOUT. B<groupstats> will
create one file for each month (or each newsgroup, accordant to the
setting of B<--group-by>, see above), with filenames composed by adding
year and month (or newsgroup names) to the I<filename template>, for
example with B<--filetemplate> I<stats>:
Save output to file(s) instead of dumping it to STDOUT. B<groupstats>
will create one file for each month (or each newsgroup, according to
the setting of B<--group-by>, see above), with filenames composed by
adding year and month (or newsgroup names) to the I<filename template>,
for example with B<--filetemplate> I<stats>:
stats-2012-01
stats-2012-02
@ -611,7 +617,7 @@ Override I<DBTableGrps> from F<newsstats.conf>.
=item B<--conffile> I<filename>
Load configuration from I<filename> instead of F<newsstats.conf>.
Read configuration from I<filename> instead of F<newsstats.conf>.
=back
@ -635,9 +641,9 @@ by number of postings, descending, in I<pretty> format:
groupstats --upper 30 --order-by postings-desc
Show the total of all postings for the year of 2010 for all groups that
had 30 postings or less in every single month in that year, ordered by
number of postings in descending order:
Show the total of all postings for the year of 2010 for all groups
that had 30 postings or less in every single month in that year,
ordered by number of postings in descending order:
groupstats -m 2010-01:2010-12 -u 30 -b level -r sums -o postings-desc
@ -651,7 +657,6 @@ machine-readable form (without formatting):
groupstats -m 2010-01:2010-12 -f dump --filetemplate stats
=head1 FILES
=over 4

38
bin/hoststats.pl
View file

@ -197,7 +197,7 @@ See L<doc/README>.
=head1 DESCRIPTION
This script create reports on newsgroup usage (number of postings from
This script creates reports on newsgroup usage (number of postings from
each host) taken from result tables created by B<gatherstats.pl>.
=head2 Features and options
@ -236,18 +236,19 @@ please see below.
The results will be formatted as a kind of table; you can change the
output format to a simple list or just a list of names and number of
postings with the B<--format> option. Captions will be added by means of
the B<--caption> option; all comments (and captions) can be supressed by
using B<--nocomments>.
postings with the B<--format> option. Captions will be added by means
of the B<--caption> option; all comments (and captions) can be
supressed by using B<--nocomments>.
Last but not least you can redirect all output to a number of files, e.g.
one for each month, by submitting the B<--filetemplate> option, see below.
Last but not least you can redirect all output to a number of files,
e.g. one for each month, by submitting the B<--filetemplate> option,
see below.
=head2 Configuration
B<hoststats> will read its configuration from F<newsstats.conf>
which should be present in etc/ via Config::Auto or from a configuration file
submitted by the B<--conffile> option.
which should be present in etc/ via Config::Auto or from a configuration
file submitted by the B<--conffile> option.
See doc/INSTALL for an overview of possible configuration options.
@ -259,18 +260,18 @@ You can override some configuration options via the B<--db> option.
=item B<-V>, B<--version>
Print out version and copyright information and exit.
Display version and copyright information and exit.
=item B<-h>, B<--help>
Print this man page and exit.
Display this man page and exit.
=item B<-m>, B<--month> I<YYYY-MM[:YYYY-MM]|all>
Set processing period to a single month in YYYY-MM format or to a time
period between two month in YYYY-MM:YYYY-MM format (two month, separated
by a colon). By using the keyword I<all> instead, you can set no
processing period to process the whole database.
processing period to process the whole database. Defaults to last month.
=item B<-n>, B<--names> I<name(s)>
@ -284,7 +285,8 @@ example
=item B<-s>, B<--sums|--nosums> (sum per month)
Include a "virtual" host named "ALL" for every month in output,
containing the sum of all detected hosts for that month.
containing the sum of all detected hosts for that month. False
by default.
=item B<-r>, B<--report> I<default|sums>
@ -315,8 +317,8 @@ considered.
=item B<-g>, B<--group-by> I<month[-desc]|name[-desc]>
By default, all results are grouped by month, sorted chronologically in
ascending order, like this:
By default, all results are grouped by month, sorted chronologically
in ascending order, like this:
# ----- 2012-01:
arcor-online.net : 9379
@ -339,8 +341,8 @@ The results can be grouped by host instead via B<--group-by> I<name>:
2012-01: 9063
2012-02: 7879
By appending I<-desc> to the group-by option parameter, you can reverse
the sort order - e.g. B<--group-by> I<month-desc> will give:
By appending I<-desc> to the group-by option parameter, you can
reverse the sort order - e.g. B<--group-by> I<month-desc> will give:
# ----- 2012-02:
arcor-online.net : 8606
@ -427,7 +429,7 @@ Override I<DBTableHosts> from F<newsstats.conf>.
=item B<--conffile> I<filename>
Load configuration from I<filename> instead of F<newsstats.conf>.
Read configuration from I<filename> instead of F<newsstats.conf>.
=back
@ -437,7 +439,7 @@ See L<doc/INSTALL>.
=head1 EXAMPLES
Show number of postings per group for lasth month in I<pretty> format:
Show number of postings per host for lasth month in I<pretty> format:
hoststats

22
bin/postingstats.pl
View file

@ -268,7 +268,7 @@ postingstats - format and post reports
=head1 SYNOPSIS
B<postingstats> B<-t> I<groups|hosts|clients> [B<-Vh> [B<-m> I<YYYY-MM>]
B<postingstats> [B<-Vh>] [B<-t> I<groups|hosts|clients>] [B<-m> I<YYYY-MM>]
=head1 REQUIREMENTS
@ -285,11 +285,13 @@ message that can be posted to Usenet.
B<postingstats> will create a table with entries numbered from most
to least and percentages calculated from the sum total of all values.
It depends on a sorted list on STDIN in I<dump> format with I<sums>.
It depends on a sorted list on STDIN in I<dump> format with I<sums>;
I<versions> from B<clientstas.pl> are optional.
B<postingstats> needs a B<--type> and a B<--month> to create a caption
and select matching lead-ins and lead-outs. B<--type> is also needed
to catch the correct sum total from input.
to catch the correct sum total from input which differs between I<groups>
on one hand and I<hosts> or I<clients> on the other hand.
It will default to posting statistics (number of postings per group)
and last month.
@ -308,7 +310,7 @@ C<----- configuration -----> section.
=item C<$TLH>
Top level hierarchy the report was created for. Used for display and
sum total.
sum total (only for I<groups>).
=item C<%Heading>
@ -327,13 +329,13 @@ Output will be truncated otherwise.
=item C<%LeadIn>
Hash with keys for I<GroupStats>, I<HostStats> and I<ClientStats>.
Used to create the headers for our posting. Can contain other text
Used to create the headers for the postings. Can contain other text
that will be shown before C<%Heading>.
=item C<%LeadOut>
Hash with keys for I<GroupStats>, I<HostStats> and I<ClientStats>.
Will be shown at the end of our posting.
Will be shown at the end of the posting.
=back
@ -343,11 +345,11 @@ Will be shown at the end of our posting.
=item B<-V>, B<--version>
Print out version and copyright information and exit.
Display version and copyright information and exit.
=item B<-h>, B<--help>
Print this man page and exit.
Display this man page and exit.
=item B<-t>, B<--type> I<groups|hosts|clients>
@ -356,7 +358,7 @@ statistics accordingly.
=item B<-m>, B<--month> I<YYYY-MM>
Set month for display.
Set month (for display only).
=back
@ -372,7 +374,7 @@ Create a posting from a posting statistics report for last month:
Create a posting from a posting statistics report for 2012-01:
groupstats.pl --nocomments --sums --format dump | postingstats.pl -t groups -m 2012-01
groupstats.pl --nocomments --sums --format dump -m 2012-01 | postingstats.pl -t groups -m 2012-01
Create a posting from a host statistics report for last month:

55
doc/INSTALL
View file

@ -1,11 +1,12 @@
NewsStats (c) 2010-2013, 2025 Thomas Hochstein <thh@thh.name>
NewsStats is a software package used to gather statistical information
from a live Usenet feed and for its subsequent examination.
NewsStats is a software package that can be used to collect
statistical information from a live Usenet feed and then analyze it
to create statistical reports.
This script package is free software; you can redistribute it and/or
modify it under the terms of the GNU Public License as published by
the Free Software Foundation.
This package is free software; you can redistribute it and/or modify
it under the terms of the GNU Public License as published by the Free
Software Foundation.
---------------------------------------------------------------------
@ -16,9 +17,10 @@ INSTALLATION INSTRUCTIONS
* Download the current version of NewsStats from
<https://th-h.de/net/software/newsstats/>.
* Untar it into a directory of your choice:
* Untar it into a directory of your choice, i.e. /srv/newsstats:
# tar -xzf newsstats-nn.tar.gz
$ cd /srv
$ tar -xzf newsstats-n.n.n.tar.gz
Scripts in this path - at least feedlog.pl - should be executable by the
news user.
@ -28,8 +30,8 @@ INSTALLATION INSTRUCTIONS
* Copy the sample configuration file newsstats.conf.sample to
newsstats.conf and modify it for your purposes:
# cp etc/newsstats.conf.sample etc/newsstats.conf
# vim etc/newsstats.conf
$ cp etc/newsstats.conf.sample etc/newsstats.conf
$ vim etc/newsstats.conf
a) Mandatory configuration options
@ -60,6 +62,9 @@ INSTALLATION INSTRUCTIONS
* DBTableHosts = hosts_de
Table holding data on postings per server.
* DBTableClnts = clients_de
Table holding data on postings per client.
b) Optional configuration options
* TLH = de.alt,news.admin
@ -68,19 +73,21 @@ INSTALLATION INSTRUCTIONS
3) Database (mysql) setup
* Setup your database server with a username, password and
database matching the NewsStats configuration (see 2 a).
* Setup your database server with an username, a password and
(optionally) a database matching the NewsStats configuration
(see 2 a).
* Start the database creation script:
# bin/dbcreate.pl
$ bin/dbcreate.pl
It will setup the necessary database tables and display some
information on the next steps.
It will create the database (if not already present), create the
necessary database tables and display some information on the
next steps.
4) Feed (INN) setup
You have to setup an INN feed to feedlog.pl.
You have to set up an INN feed to feedlog.pl.
* Edit your 'newsfeeds' file and insert something like
@ -90,39 +97,39 @@ INSTALLATION INSTRUCTIONS
:Tc,WmtfbsPNH,Ac:/path/to/feedlog.pl
* You should only feed that hierarchy (those hierarchies ...) to
feedlog.pl you'll want to cover with your statistical
examination. It may be a good idea to setup different feeds (to
different databases ...) for different hierarchies.
feedlog.pl that you want to cover with your statistical analysis.
It may be a good idea to setup different feeds (to different
databases ...) for different hierarchies.
* Please double check that your path to feedlog.pl is correct and
feedlog.pl can be executed by the news user
* Check your 'newsfeeds' syntax:
# ctlinnd checkfile
$ ctlinnd checkfile
* Reload 'newsfeeds':
# ctlinnd reload newsfeeds 'Adding newsstats! feed'
$ ctlinnd reload newsfeeds 'Adding newsstats! feed'
* Watch your 'news.notice' and 'errlog' files:
# tail -f /var/log/news/news.notice
$ tail -f /var/log/news/news.notice
...
# tail -f /var/log/news/errlog
$ tail -f /var/log/news/errlog
Everything should be going smoothly now.
* If INN is spewing error messages to 'errlog' or reporting
continous respawns of feedlog.pl to 'news.notice', stop your feed:
# ctlinnd drop 'newsstats!'
$ ctlinnd drop 'newsstats!'
and investigate. 'errlog' may be helpful here.
* You can restart the feed with
# ctlinnd begin 'newsstats!'
$ ctlinnd begin 'newsstats!'
later.

75
doc/README
View file

@ -1,21 +1,21 @@
NewsStats (c) 2010-2013, 2025 Thomas Hochstein <thh@thh.name>
NewsStats is a software package for gathering statistical data live
from a Usenet feed and subsequent examination.
from a Usenet feed and subsequent analysis.
This script package is free software; you can redistribute it and/or
modify it under the terms of the GNU Public License as published by
the Free Software Foundation.
This package is free software; you can redistribute it and/or modify
it under the terms of the GNU Public License as published by the Free
Software Foundation.
---------------------------------------------------------------------
What's that?
There's a multitude of tools for the statistical examination of
newsgroups: number of postings per month or per person, longest
threads, and so on (see <https://th-h.de/net/usenet/stats/>
[German language] for an incomplete list). Most of them use a per-
newsgroup approach while NewsStats is hierarchy oriented.
There's a multitude of tools to create statistics about newsgroup
usage: number of postings per month or per person, longest threads,
and so on (see <https://th-h.de/net/usenet/stats/> [German language]
for an incomplete list). Most of them use a per-newsgroup approach
while NewsStats is hierarchy oriented.
NewsStats will accumulate data from a live INN feed, allowing you
to process the saved information later on.
@ -40,7 +40,9 @@ Prerequisites
* Perl 5.8.x with standard modules
- Cwd
- Encode
- File::Basename
- Getopt::Long
- Sys::Syslog
* Perl modules from CPAN
@ -50,7 +52,7 @@ Prerequisites
* mysql 5.0.x
* working installation of INN
* a working installation of INN
Installation instructions
@ -67,15 +69,52 @@ Getting Started
table. See the feedlog.pl man page for more information.
You can process that data via 'gatherstats.pl'; currently the
tabulation of postings per group and injection server per month is
supported. Tabulation of clients (newsreaders) is planned. See
the gatherstats.pl man page for more information.
tabulation of postings per group, injection server and posting
agent (newsreader) per month is supported. See the gatherstats.pl
man page for more information.
Example:
bin/gatherstats.pl
will parse raw data from the last month and save the results in
tables for postings per group, server and client, respectively.
Report generation is handled by specialised scripts for each
report type. Currently reports on the number of postings per group
and month and injection server and month are supported; you can
use 'groupstats.pl' and 'hoststats.pl' for that. See the
groupstats.pl and hoststats.pl man pages for more information.
report type: 'groupstats.pl' for postings per group
(s), 'hoststats.pl' for postings per injection server
(s) and 'clientstats.pl' for postings per posting agent. See the
groupstats.pl, hoststats.pl and clientstats.pl man pages for more
information.
Example:
bin/groupstats.pl -o postings-desc
bin/hoststats.pl -o postings-desc
bin/clientstats.pl -o postings-desc -v
will show reports for postings per group, per injection server and
per client (with detailed client versions) for the last month,
using the result tables filled by gatherstats.
To post those reports to Usenet, change postingstats.pl according
to your needs (sender, newsgroups and other headers, translation
of table headers and text templates) and display a test posting
by piping report data into postingstats.pl:
bin/groupstats.pl --nocomments -s -f dump | bin/postingstats.pl
If the result is to your liking, add a pipe to a inews
implementation.
Example:
bin/groupstats.pl --nocomments -s -f dump | bin/postingstats.pl | contrib/tinews.pl -X
More information
See the man pages for 'gatherstats' and the report generating
scripts.
Reporting Bugs
@ -87,7 +126,7 @@ Reporting Bugs
Development
This program is maintained using the Git version control system at
This package is maintained using the Git version control system at
<https://code.virtcomm.de/thh/newsstats/>.
Related projects

22
doc/TODO
View file

@ -1,12 +1,10 @@
NewsStats To-Do List
====================
This is a list of planned bug fixes, improvements and enhancements for
This is a list of possible bug fixes, improvements and enhancements for
NewsStats.
* General
- Improve Documentation
The documentation is rather sparse and could use some improvement.
- Add a test suite
There is currently no kind of test suite or regression tests. Something like
that is badly needed.
@ -27,8 +25,6 @@ NewsStats.
for late creation and deletion), optionally including the previously
mentioned information; and you should be able to get the history of any
group.
- Add other reports
NewsStats should include some other kinds of reports (stats on used clients)
- Add tools for database management
NewsStats should offer tools e.g. to inject postings into the 'raw' database,
or to split databases.
@ -53,23 +49,11 @@ NewsStats.
Some other tests - working database connection, valid database and table
names - would be nice.
+ install/install.pl
- Read current version from a file dropped and updated by installer
- Add / enhance / test error handling
- General tests and optimisations
+ feedlog.pl
- Add / enhance / test error handling
- General tests and optimisations
+ gatherstats.pl
- Use hierarchy information (see GroupInfo above)
- Add gathering of other stats (clients, ...)
- better modularisation (code reuse for other reports!)
- Add / enhance / test error handling
- General tests and optimisations
+ groupstats.pl
- better modularisation (code reuse for other reports!)
+ groupstats.pl, hoststats.pl, clientstats.pl
- better modularisation (code reuse)
- Add / enhance / test error handling
- General tests and optimisations