Just assume UTF-8 for the time being.
Fixes database errors with illegal characters
when writing parsed data.
Signed-off-by: Thomas Hochstein <thh@inter.net>
Our raw data doesn't have the qualitiy one should
expect. There are empty header lines only containing
whitespace (leading to wrong joining of apparent
continuation lines); header lines that contain garbage
without ':' so split is failing; empty 'newsgroups'
fields; unsupported encondings in MIME encoded words
... and so on.
Add fixes for the aforementioned problems.
Signed-off-by: Thomas Hochstein <thh@inter.net>
From:, Sender: etc. may contain more than one
entity in a comma separated list, i.e. a From:
line like
"From: Me <me@example.com>, You <you@example.com>"
is perfectly valid.
Handle multiple entities when splitting those
headers and save all names and all adresses
as (new) comma separated lists in the
corresponding database fields.
Signed-off-by: Thomas Hochstein <thh@inter.net>
Incoming data is written to DBTableRaw without
much interpretation. To allow for more and
better analysis that raw data should be parsed
daily and copied to another database table
with separate fields for most header lines.
All other scripts could use that pre-parsed
data.
* Add database schema to install.pl
* Add DBTableParse to newsstats.conf.sample
and as mandatory to NewsStats.pm
* Add parsedb.pl
TODO:
- Documentation is only rudimentary.
- From:, Sender:, Reply-To: and Subject:
are not yet parsed.
- gatherstats.pl does not yet use DbTableParse.
Signed-off-by: Thomas Hochstein <thh@inter.net>
'Virtual' .ALL groups will never be present in
a checkgroups file, and we can't use them anyway
as they would contain postings from groups that
are filtered out by --checkgroups.
Add a warning, put a note in the documentation.
Signed-off-by: Thomas Hochstein <thh@inter.net>
If --group-by is not set, output will be grouped
by month by default (as long as --boundary is
not set to 'level' or 'average', where grouping
by newsgroup is default).
Now we default to 'newsgroup' if just one newsgroup
is requested by --newsgroups, but more than one
month by --month.
Both defaults can be overridden.
But forced --group-by=month for --report type
'average' or 'sum' in front so defaults are
not checked.
Signed-off-by: Thomas Hochstein <thh@inter.net>
* Move all scripts to /bin
* Move configuration to /etc
* Move NewsStats.pm to /lib
* Add new path to NewsStats.pm to all scripts
* Set $HomePath to top level directory
* Move setting of config file name to ReadConf()
Signed-off-by: Thomas Hochstein <thh@inter.net>
