Commit graph

6 commits

Author SHA1 Message Date
Thomas Hochstein 6deb7dbaa4 Add MID to error message to make it more useful.
Signed-off-by: Thomas Hochstein <thh@inter.net>
2021-05-29 09:47:17 +02:00
Thomas Hochstein 48c8d4bb8e Add some input validation.
Our raw data doesn't have the qualitiy one should
expect. There are empty header lines only containing
whitespace (leading to wrong joining of apparent
continuation lines); header lines that contain garbage
without ':' so split is failing; empty 'newsgroups'
fields; unsupported encondings in MIME encoded words
... and so on.

Add fixes for the aforementioned problems.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2013-09-08 17:53:25 +02:00
Thomas Hochstein 13e006104b Add documentation to parsedb.pl.
Signed-off-by: Thomas Hochstein <thh@inter.net>
2013-09-08 17:53:23 +02:00
Thomas Hochstein aef5467bfe Handle more than one entitiy in From: etc.
From:, Sender: etc. may contain more than one
entity in a comma separated list, i.e. a From:
line like
"From: Me <me@example.com>, You <you@example.com>"
is perfectly valid.

Handle multiple entities when splitting those
headers and save all names and all adresses
as (new) comma separated lists in the
corresponding database fields.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2013-09-08 17:53:21 +02:00
Thomas Hochstein 9630376c31 Add decoding and parsing of From: etc.
Decode From:, Sender:, Reply-To:, Subject:;
parse From:, Sender:, Reply-To:.

Add Mail::Address to prerequisites.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2013-09-08 17:53:17 +02:00
Thomas Hochstein 6d72dad2c0 Create a database table with parsed raw data.
Incoming data is written to DBTableRaw without
much interpretation. To allow for more and
better analysis that raw data should be parsed
daily and copied to another database table
with separate fields for most header lines.
All other scripts could use that pre-parsed
data.

* Add database schema to install.pl
* Add DBTableParse to newsstats.conf.sample
  and as mandatory to NewsStats.pm
* Add parsedb.pl

TODO:
- Documentation is only rudimentary.
- From:, Sender:, Reply-To: and Subject:
  are not yet parsed.
- gatherstats.pl does not yet use DbTableParse.

Signed-off-by: Thomas Hochstein <thh@inter.net>
2013-09-08 17:27:50 +02:00