Commit graph

53 commits

Author SHA1 Message Date
Thomas Hochstein cce2fd0b7f Merge branch 'gatherstats' into next
* gatherstats:
  Get empty 'virtual' hierarchies working.
2013-09-02 06:46:40 +02:00
Thomas Hochstein 1703b8e3b4 Get empty 'virtual' hierarchies working.
Commit b5125b1099
was broken.

We didn't add empty .ALL hierarchies as needed;
we added empty (non-existant) hierarchies without
appended '.ALL', and didn't add the original
empty group we started with.

(What's more, gatherstats didn't even start any
more due to missing ex- and import of
&ParseHierarchies from NewsStats.pm.)

Fixes #52 (and some more breakage).

Signed-off-by: Thomas Hochstein <thh@inter.net>
2013-09-01 17:52:21 +02:00
Thomas Hochstein 1c3adb8de2 Merge branch 'feedlog' into next 2013-08-11 22:00:05 +02:00
Thomas Hochstein 98563c619e Refactor database initialisation in feedlog.pl.
* Move database initialisation to a separate function.

* (Re-)try to connect every five seconds
  (instead of going into an endless loop) and
  log successful (re-)connections.

* Log postings that are dropped due to database failures
  to syslog (Message-ID) for recovery.

* If the connection to the database is lost, try to
  recover it (every five seconds) and try again to
  write the pending data.

* Input will be buffered automatically by INN until
  feedlog is able to process it (see man 5 newsfeeds).

Fixes #30, #31.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2013-08-11 21:59:31 +02:00
Thomas Hochstein c3973e7d0d Merge branch 'gatherstats' into next 2013-08-11 21:56:37 +02:00
Thomas Hochstein a915469e0c Merge branch 'rewrite' into next 2013-08-11 21:56:18 +02:00
Thomas Hochstein b5125b1099 Add empty 'virtual' .ALL hierarchies as needed.
When using a --checkgroups file while tabulating,
valid but empty groups will be added with a posting
count of zero as needed. If all groups in a
sub-hierarchy are empty, the virtual '.ALL' group
for that sub-hierarchy was not created, though.

If local.test.dummy and local.test.binary were
both empty, both groups were added with a posting
count of '0', but local.test.ALL was not.

Now we loop through all hierarchy elements using
ParseHierarchies and add empty .ALL hierarchies as
needed.

Fixes #49.

Also fixing a typo in some comment. :-)

Signed-off-by: Thomas Hochstein <thh@inter.net>
2013-08-11 09:45:00 +02:00
Thomas Hochstein 93c8eae2ed Change interpretation of --checkgroups to template
In most hierarchies, the list of valid newsgroups will
change over time, so you'll have to use another
checkgroups file for each month. gatherstats will now
understand the value of --checkgroups to be a template
and amend it with each month it is processing.

Documentation changed accordingly.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:52:24 +02:00
Thomas Hochstein 7662b1065e Be more fault-tolerant when reading checkgroups.
* Accept lines starting with whitespace.

* Drop empty "groups", i.e. lines containing only
  whitespace.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:52:13 +02:00
Thomas Hochstein 0dc13b3980 Remove call to &Bleat where not appropriate.
Some warn()ings are used for debugging purposes.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:52:02 +02:00
Thomas Hochstein 314e31aadf Allow more characters in TLH definitions.
TLH may now also contain literal dots '.',
allowing for using second or third level
hierarchies as "TLH". To faciliate that,
'+' and '-' will be allowed, too.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:51:51 +02:00
Thomas Hochstein 7773fb6d8f Match TLHs correctly, not only partially.
The TLH was checked to match the beginning
of the newsgroup name, not the whole TLH part.
So the TLH "de" would match not only "de.test",
but also "denver.test", which was not the
desired outcome.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:51:40 +02:00
Thomas Hochstein 43a0fc7769 Fix parsing of more than one TLH in config.
The code introduced in 17ffbebad5
did not check the correct variable for being an array.

Improve an unrelated comment, too.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:51:29 +02:00
Thomas Hochstein 1fa9479946 Adapt install.pl to new coding style.
* Switch to Getopt::Long, change coding style;
  limit line length.

* Replace 'die' and 'warn' by calls to &Bleat().

* Completely changed options due to new
  GetOpt::Long processing.

* Adapt to changes in NewsStats.pm

* Redo documentation.

* Update TODO.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:44:40 +02:00
Thomas Hochstein f05ec4aeaa Adapt feedlog.pl to new coding style.
* Switch to Getopt::Long, change coding style;
  limit line length.

* Completely changed options due to new
  GetOpt::Long processing.

* Adapt to changes in NewsStats.pm

* Redo documentation.

* Update TODO.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:44:20 +02:00
Thomas Hochstein 880c3eb227 Adapt gatherstats.pl to new coding style.
* Switch to Getopt::Long, change coding style;
      limit line length.

* Replace 'die' and 'warn' by calls to &Bleat().

* Completely changed options due to new
  GetOpt::Long processing.
  - merged -m/-p into --month

* Adapt to changes in NewsStats.pm

* Redo documentation.

* Update TODO.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:43:59 +02:00
Thomas Hochstein edd250f265 Complete rewrite of groupstats.pl and NewsStats.pm.
* Switch to Getopt::Long, change coding style;
  limit line length.

* Replace 'die' and 'warn' by calls to &Bleat().

* Completely redo options and processing:
  - merge -m/-p/-a into --month
  - replace -i/-q/-d with - much more powerful -
    --group-by/--order-by
  - replace -t/-l with - much more powerful -
    --lower/--upper/--boundary
  - remove -b and replace it with --report

  Fixes #33.

* Add new report types, boundaries and sorting options:
  - report types 'average' and 'sums'
  - boundaries 'average' and 'sums'
  - upper and/or lower boundary
  - sort output independently

  Issue #35.
  Fixes #34, #38.

* Add possibility to cross-check newsgroups against
  checkgroups file.

* Complete rewrite of groupstats.pl internal logic:
  - modularize construction fo SQL queries
  - remove unnecessary special cases
  - refactor code into NewsStats.pm functions as much
    as possible

  Issue #37.
  Fixes #36.

* Rework output formats, fix padding problem by making use
  of modularized SQL queries.

  Fixes #15, #32.

* Add some more consistency checks.

  Issue #12.

* Redo documentation.

* Update TODO list.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-10-13 00:43:39 +02:00
Thomas Hochstein 3c7a7dfef5 Merge branch 'tools' into pu
* tools:
  Add tools directory, move untracked stuff top /tmp
2012-05-07 17:53:52 +02:00
Thomas Hochstein fe46be2168 Add tools directory, move untracked stuff top /tmp
Signed-off-by: Thomas Hochstein <thh@inter.net>
2012-05-04 07:41:02 +02:00
Thomas Hochstein a02fdc165f Documentation: Add ChangeLog
Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 21:54:56 +01:00
Thomas Hochstein 06402a2526 Update To-Do list.
Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 21:54:43 +01:00
Thomas Hochstein 147194c190 Documentation: Add TODO.
Add a comprehensive to-do list.
Add a reference to README accordingly.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 21:54:31 +01:00
Thomas Hochstein 17ffbebad5 gatherstats.pl: Allow for more than one TLH in config.
New syntax for TLH: tlh1:tlh2:tlh3
Check for illegal characters and reformat to RegExp.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 21:45:35 +01:00
Thomas Hochstein d3b6810d3d Fix some typos.
groupsstats.pl: Fix typo in lead-in comment.
groupsstats.pl/NewsStats.pm: Fix typo concerning
GetMaxLength - it isn't 'GetMaxLenght' ...

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 20:28:31 +01:00
Thomas Hochstein 71f0178b2a gatherstats.pl: Change database update mechanism.
DELETE old records and INSERT (INTO) the new ones
instead of just REPLACEing them, which will not
remove wrong entries (i.e. non-existing newsgroups).

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 20:13:33 +01:00
Thomas Hochstein 89db2f904d gatherstats.pl: Move TLH check to NewsStats.pm.
Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 20:13:25 +01:00
Thomas Hochstein ad60979271 gatherstats.pl: Add '-l' option.
Add '-l' option to check newsgroup names against a list of valid
newsgroups read from a file (each group on one line, ignoring
everything after the first whitespace and so accepting files in
checkgroups format as well as (parts of) an INN active file).
New ReadGroupList().
Change ListNewsgroups() accordingly.
Amend documentation accodingly.

See #17.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 20:13:16 +01:00
Thomas Hochstein 54d04e84a5 groupstats.pl: Precedence of -m / -p / -a
Unset -m if either -p or -a is set,
unset -m if -p is set.

Print caption (-c) for -a, too.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 20:13:08 +01:00
Thomas Hochstein 628a183c15 groupstats.pl: Change default output format to 'pretty'.
Change documentation accordingly.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 20:12:59 +01:00
Thomas Hochstein 78389b28e9 groupstats.pl: Add '-f' option.
Add '-f' switch to save monthly reports to a file with
a filename made up by adding year and month to template
submitted via '-f filename': filename-YYYY-MM
Modify OutputData() accordingly.
Ignore '-c' if '-f' is set.
Allow & force '-o dump' if '-f' is set even if a time
period is defined.
Change documentation accodingly.

Fixes #27.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 20:12:50 +01:00
Thomas Hochstein f2ddfd8a92 groupstats.pl: Add '-a' option.
Add '-a' switch to make groupstats.pl consider its whole
database without any time limits ignoring -m and -p.
Change documentation accodingly.

First implementation. Handling needs improvement!

Fixes #14.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-11-01 14:02:47 +01:00
Thomas Hochstein b2e0fd2444 groupstats.pl: Improve formatting of output.
Rewrite captions.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-10-31 23:18:53 +01:00
Thomas Hochstein e742bcf550 groupstats.pl: Improve automatic adaption of output type.
If time period is more than on month, switch from dump
to dumpgroup, if possible (just one newsgroup submitted
via -n), and switch to pretty only if that fails.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-10-31 23:15:18 +01:00
Thomas Hochstein b802bc3d29 groupstats.pl: Redo level (-l) query, gain speedup.
Do the work in the application, using two separate SQL
queries, instead of trying to optimize the single
query in vain ...

Thanks to Juliane. :)

Fixes #16.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-10-31 21:43:18 +01:00
Thomas Hochstein 404c1acdc8 groupstats.pl: Improve comments.
Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-10-31 21:39:20 +01:00
Thomas Hochstein 6b95accbb4 groupstats.pl: Refactor query handling.
Don't misuse variables:
- Push other binding vars to (new) @Params instead of
  filling up @GroupList with things it wasn't meant for.
- Replace generic @QueryPart by @QueryGroupList and
  @QueryThreshold, then build $WhereClause from that.
- Don't use $OrderClause for setting level (more or
  less than x), use (new) $Level instead.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-10-31 21:34:04 +01:00
Thomas Hochstein 59d0c5ef86 install.pl: Change handling of upgrades.
Drop predefined actions in favor of arbitrary statements.
Minor code fixup.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-19 16:53:07 +02:00
Thomas Hochstein 53dcbea63e README: Fix spelling, small changes.
Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-19 13:34:27 +02:00
Thomas Hochstein 313610f63c gatherstats.pl: Add documentation of 'disregard' to POD.
Database entries in DBTableRaw marked 'disregard' will
be ignored by gatherstats. Document that.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-19 13:32:12 +02:00
Thomas Hochstein 76ead5f97b feedlog.pl: Fix annoying logging error.
warn() output to errlog was not activated by '-d' debug
switch, but on by default and just deactivated by '-d'.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-19 13:27:28 +02:00
Thomas Hochstein 4bbd46d336 NewsStats.pm: Fix $VERSION and $PackageVersion.
Set to 0.01 from 0.1

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-19 00:21:42 +02:00
Thomas Hochstein 9c141b0ca7 install.pl: Fix installation instructions.
Example newsfeeds entry was missing a backslash
(had to be quoted).

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-19 00:21:12 +02:00
Thomas Hochstein d8695b1c04 groupstats.pl: More input validation.
Set -o to 'pretty' for time periods.
Set -b to 10 if < 1.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-18 21:59:13 +02:00
Thomas Hochstein f6d15ca78e Add upgrade path.
NewsStats.pm: Add $PackageVersion.

install.pl: Modularise program flow to subroutines.
            Add framework for database upgrade and
            upgrade instructions.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-18 20:54:02 +02:00
Thomas Hochstein 2832c235b2 Introduce end-of-line normalization and add .gitattributes.
Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-18 20:53:31 +02:00
Thomas Hochstein 3430c89868 Add comments and POD.
Add comments NewsStats.pm
Add POD to install.pl
Add POD to feedlog.pl
Add POD and comments to gatherstats.pl
Add POD and comments to groupstats.pl, beautify code

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-18 20:47:32 +02:00
Thomas Hochstein 610b5ef492 Add documentation.
Add doc/
Add doc/README
Add doc/INSTALL

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-18 20:45:41 +02:00
Thomas Hochstein 13c5a175ef groupstats.pl: Add handling of incompatible command line options.
-t/-l/-b are incompatible. Print warning, let -l take precedence
over -t and -b over both.

-q/-d are incompatible with -l and -b. Print warning, ignore
them.

For -l, only -o 'pretty' makes sense. Force that.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-18 20:45:33 +02:00
Thomas Hochstein 2af6c2902f install.pl: Fix chdir.
Don't use path from current working directory, use path
relative to $0 instead.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-18 20:45:26 +02:00
Thomas Hochstein b221278d97 NewsStats.pm: Warn when -p takes precedence over -m.
Signed-off-by: Thomas Hochstein <thh@inter.net>
2010-09-18 20:45:18 +02:00