Add option to add common headers to all projects.

Signed-off-by: Thomas Hochstein <thh@thh.name>

ChangeLog

@@ -1,31 +1,42 @@
+yapfaq 1.1.0 (unreleased)
+  * --test: Set Supersedes and don't modify Message-ID if -o is set.
+  * --test: Force -o if -n is not set.
+  * Add conversion script for old status files.
+  * Accept a posting-frequency of "never", too.
+  * Check for illegal headers and stop posting, if found.
+  * Disabled projects must be posted, if forced.
+  * Warn of unencoded 8bit characters in header or body if -d is set.
+  * Update POD.
+  * Add an option for a common headers file for all projects.
+
 yapfaq 1.0.1 (2025-01-24)
-* Add %t placeholder for Message-ID (feature parity with 0.9).
-* Remove debugging code.
+  * Add %t placeholder for Message-ID (feature parity with 0.9).
+  * Remove debugging code.
 
 yapfaq 1.0.0 (2025-01-23)
-* Complete rewrite.
-* Add POD.
-* Fix file handling (UTF8 mode).
-* Show next posting date if posting is not due.
-* Add --simulation mode.
-* Update examples in POD.
-* Use local time.
+  * Complete rewrite.
+  * Add POD.
+  * Fix file handling (UTF8 mode).
+  * Show next posting date if posting is not due.
+  * Add --simulation mode.
+  * Update examples in POD.
+  * Use local time.
 
 yapfaq 0.10 (unreleased)
-* Add: Charset definition.
+  * Add: Charset definition.
     Fixes #29.
-* Mark yapfaq.pl executable.
-* Change mail address.
+  * Mark yapfaq.pl executable.
+  * Change mail address.
 
 yapfaq 0.9.1 (2010-11-01)
-* Fix: Test mode: Add X-Supersedes only if Supersedes would be set.
+  * Fix: Test mode: Add X-Supersedes only if Supersedes would be set.
     Fixes #28.
     Thomas Hochstein <thh@inter.net> Sun Oct 31 18:42:52 2010 +0100
 
 yapfaq 0.9 (2010-09-11)
-* Drop use of Fcntl (not needed).
+  * Drop use of Fcntl (not needed).
     Thomas Hochstein <thh@inter.net> Tue Jun 15 22:30:11 2010 +0200
-* Changed packaging.
+  * Changed packaging.
     - Drop .yapfaqrc and yapfaq.cfg in favor of .yapfaqrc.sample
       and yapfaq.cfg.sample; rename test.txt to sample.txt.
     - Change contents of these files to reflect the new names.
@@ -33,9 +44,9 @@ yapfaq 0.9 (2010-09-11)
     - Add "INSTALLATION" chapter to documentation.
     Fixes #7.
     Thomas Hochstein <thh@inter.net> Sat May 15 19:16:40 2010 +0200
-* Change default Message-ID format.
+  * Change default Message-ID format.
     Thomas Hochstein <thh@inter.net> Sat May 15 19:04:24 2010 +0200
-* Change: Modify headers for test posts.
+  * Change: Modify headers for test posts.
     - Change MID so you can do multiple tests.
     - Replace Supersedes with X-Supersedes so you do not delete
       postings while testing.
@@ -43,39 +54,39 @@ yapfaq 0.9 (2010-09-11)
     - Change documentation accordingly.
     Fixes #6.
     Thomas Hochstein <thh@inter.net> Sat May 15 17:22:20 2010 +0200
-* Change: Drop %LM from subject if Last-Modified is not found.
+  * Change: Drop %LM from subject if Last-Modified is not found.
     Thomas Hochstein <thh@inter.net> Sat May 15 16:36:52 2010 +0200
 
 yapfaq 0.8.2 (2010-05-15)
-* Fix: Accept leading/trailing whitespace for Last-modified pseudo header.
+  * Fix: Accept leading/trailing whitespace for Last-modified pseudo header.
     Fixes #5.
     Thomas Hochstein <thh@inter.net> Sat May 15 16:32:58 2010 +0200
 
 yapfap 0.8.1 (2010-05-14)
-* Fix broken implementation of "Program" in .yapfaqrc.
+  * Fix broken implementation of "Program" in .yapfaqrc.
     Fixes #4.
     Thomas Hochstein <thh@inter.net>  2010-05-14 21:58:15
 
 yapfaq 0.8 (2010-05-13)
-* Documentation: Add Git repository and bug tracker.
+  * Documentation: Add Git repository and bug tracker.
     Thomas Hochstein <thh@inter.net> Thu May 13 19:21:05 2010 +0200  
-* Making use of Getopt::Std's --help and --version.
+  * Making use of Getopt::Std's --help and --version.
     Fixes #3.
     Thomas Hochstein <thh@inter.net> Thu May 13 19:33:25 2010 +0200    
-* New: Add "Program" to .yapfaqrc.
+  * New: Add "Program" to .yapfaqrc.
     Fixes #2.
     Thomas Hochstein <thh@inter.net> Thu May 13 19:31:49 2010 +0200    
-* Change: Drop PGP support.
+  * Change: Drop PGP support.
     You may use tinews.pl from
     ftp://ftp.tin.org/tin/tools/tinews.pl instead.
     Fixes #1.
     Thomas Hochstein <thh@inter.net> Thu May 13 19:24:44 2010 +0200    
-* Fix: Consistency check for MID-Format fixed (regexp).
+  * Fix: Consistency check for MID-Format fixed (regexp).
     Thomas Hochstein <thh@inter.net> Wed Apr 14 23:17:16 2010 +0200
-* New: MID-Format may now contain %t for a Unix timestamp.
+  * New: MID-Format may now contain %t for a Unix timestamp.
     %t will be replaced by the number of seconds since the epoch.
     Thomas Hochstein <thh@inter.net> Wed Apr 14 23:18:04 2010 +0200    
-* Documentation changes
+  * Documentation changes
     - Change sample yapfaq.cfg (mark optional settings).
       Optional settings are mostly commented out.
       Thomas Hochstein <thh@inter.net> Wed Apr 14 09:10:05 2010 +0200
@@ -84,9 +95,9 @@ yapfaq 0.8 (2010-05-13)
       Thomas Hochstein <thh@inter.net> Wed Apr 14 10:02:48 2010 +0200
     - Add comments pointing to .yapfaqrc to source.
       Thomas Hochstein <thh@inter.net> Wed Apr 14 10:10:28 2010 +0200
-* readconfig(): Add file name to verbose output.
+  * readconfig(): Add file name to verbose output.
     Thomas Hochstein <thh@inter.net> Wed Apr 14 09:38:23 2010 +0200    
-* Change: Reset default for NNTPServer to "unset".
+  * Change: Reset default for NNTPServer to "unset".
     Since yapfaq fill fall back to Perl's build-in defaults,
     that should be no problem; furthermore user may now
     override this setting via .yapfaqrc without changing
@@ -94,25 +105,25 @@ yapfaq 0.8 (2010-05-13)
     Thomas Hochstein <thh@inter.net> Wed Apr 14 09:11:45 2010 +0200
 
 yapfaq 0.7 (2010-04-14)
-* Change: readconfig(): Make mid-format optional.
+  * Change: readconfig(): Make mid-format optional.
     Set defaults for expires and mid-format when they're invalid
     (defaults were already set in postfaq() if undefined).
     Change documentation accordingly; make it more clear if
     parameters are optional or mandatory.
     Thomas Hochstein <thh@inter.net> Tue Apr 13 23:59:43 2010 +0200
-* Fix: Save status information only after successful posting.
+  * Fix: Save status information only after successful posting.
     - New Function: updatestaus
        Move status information save to updatestatus.
     - post() now returns exit code.
     - postfaq() will update status information only
       when post() was successful.
     Thomas Hochstein <thh@inter.net> Sat Apr 10 23:19:44 2010 +0200
-* New: Add option '-s': pipe article to script.
+  * New: Add option '-s': pipe article to script.
     Use an external program to post - or otherwise handle -
     the article.
     Amend documentation.
     Thomas Hochstein <thh@inter.net> Sat Apr 10 02:14:59 2010 +0200
-* New: runtime configuration
+  * New: runtime configuration
     - Moved configuration to a hash (%Config).
       Thomas Hochstein <thh@inter.net> Wed Apr 7 22:09:15 2010 +0200    
     - Add rcfile (.yapfaqrc) for configuration.
@@ -127,41 +138,41 @@ yapfaq 0.7 (2010-04-14)
       Add the according sections to the POD documentation.
       Fix some wording.
       Thomas Hochstein <thh@inter.net> Sat Apr 10 02:17:00 2010 +0200
-* Small changes.
+  * Small changes.
     -t CONSOLE: Change delimiter.
     No leading \n is necessary.
     Add some more comments.
     Thomas Hochstein <thh@inter.net> Sat Apr 10 01:43:19 2010 +0200
-* Add option '-V': print version and copyright information.
+  * Add option '-V': print version and copyright information.
     Thomas Hochstein <thh@inter.net> Thu Apr 8 07:36:11 2010 +0200
-* Change: -h: Replace version/usage information with man page.
+  * Change: -h: Replace version/usage information with man page.
     Feed script to perldoc when called with -h.
     Thomas Hochstein <thh@inter.net> Thu Apr 8 06:21:05 2010 +0200
-* Change: Authenticate only if $NNTPUser is set.
+  * Change: Authenticate only if $NNTPUser is set.
     Thomas Hochstein <thh@inter.net> Sat Apr 10 00:49:24 2010 +0200
-* Add check for MID-Format and fallback to FQDN.
+  * Add check for MID-Format and fallback to FQDN.
     Uses hostfqdn from Net::Domain.
     Thomas Hochstein <thh@inter.net> Thu Apr 8 08:33:01 2010 +0200
-* Add checks for mandatory content in configuration file.
+  * Add checks for mandatory content in configuration file.
     Enhance and optimize existing checks.
     Thomas Hochstein <thh@inter.net> Thu Apr 8 08:30:21 2010 +0200
-* Code optimisation (verbose output).
+  * Code optimisation (verbose output).
     Thomas Hochstein <thh@inter.net> Thu Apr 8 08:00:04 2010 +0200
 
 yapfaq 0.6.2 (2010-02-26)
-* Fix default for Expires.
+  * Fix default for Expires.
     Bug introduced in v0.6.1.
     Thomas Hochstein <thh@inter.net> Fri Feb 26 09:29:01 2010 +0100
 
 yapfaq 0.6.1 (2010-02-26)
-* Fix: Test mode must not update status information.
+  * Fix: Test mode must not update status information.
     Also fix runtime warning concerning expires.
     Thomas Hochstein <thh@inter.net> Fri Feb 26 08:28:06 2010 +0100
   
 yapfaq 0.6 (2010-02-25)
-* Add documentation in POD format.
+  * Add documentation in POD format.
     Thomas Hochstein <thh@inter.net> Thu Feb 25 17:00:07 2010 +0100
-* Add commandline options.
+  * Add commandline options.
     - Using Getopt::Std.
     - Implement option '-h':
       Display version and usage information.
@@ -181,7 +192,7 @@ yapfaq 0.6 (2010-02-25)
       'test mode', post to (an)other newsgroup(s) given on the command
       line or to STDOUT ('console').
     Thomas Hochstein <thh@inter.net> Thu Feb 25 19:22:15 2010 +0100
-* Add variable expiry.
+  * Add variable expiry.
     - New Function: calcdelta
       Move date calculation for new posting date to
       calcdelta.
@@ -189,21 +200,21 @@ yapfaq 0.6 (2010-02-25)
       Parse 'Expires'.
       Use calcdelta to calculate expiry.
     Thomas Hochstein <thh@inter.net> Thu Feb 25 12:55:04 2010 +0100
-* Cleanup on yapfaq.cfg
+  * Cleanup on yapfaq.cfg
     Reformat, translate to English language, add descriptions.
     Thomas Hochstein <thh@inter.net> Thu Feb 25 16:16:49 2010 +0100
-* Change handling of warnings/errors.
+  * Change handling of warnings/errors.
     Don't output line number if .cfg file can't be opened.
     Inform user when writing to ERROR.dat.
     Add script name and Warning/Error to warn() and die() output.
     Thomas Hochstein <thh@inter.net> Thu Feb 25 09:23:14 2010 +0100
-* Fix: Accept case-insensitive Last-modified pseudo header.
+  * Fix: Accept case-insensitive Last-modified pseudo header.
     Thomas Hochstein <thh@inter.net> Sun Feb 21 18:39:05 2010 +0100
-* Change defaults
+  * Change defaults
     Don't use PGP by default.
     Default $NNTPServer to 'localhost'
     Thomas Hochstein <thh@inter.net> Thu Feb 25 15:15:57 2010 +0100
-* Update header/introduction, bump version/copyright information.
+  * Update header/introduction, bump version/copyright information.
     Fix typo/language in header/introduction.
     Add new author / copyright information.
     Bump version to 0.6-unreleased.

bin/yapfaq.pl

@@ -19,7 +19,7 @@
 # It can be redistributed and/or modified under the same terms under 
 # which Perl itself is published.
 
-my $VERSION = "1.0.1";
+my $VERSION = "1.1.0-pre";
 (my $NAME = $0) =~ s#^.*/##;
 
 use utf8;
@@ -45,6 +45,8 @@ $Config{'nntp-pass'}    = '';      # password for AUTHINFO
 $Config{'force-auth'}   = 0;       # set to 1 to force authentication
 $Config{'starttls'}     = 0;       # set to 1 to use STARTTLS if possible
 
+$Config{'xtraheaders'} = '';      # path to file with extra headers
+
 $Config{'verbose'}      = 0;       # set to 1 to get status messages
 $Config{'debug'}        = 0;       # set to 1 to get some debug output,
                                    # set to 2 for NNTP debug output
@@ -100,6 +102,7 @@ GetOptions ('p|project=s'   => \$OptProject,
             'nntp-pass=s'   => \$Config{'nntp-pass'},
             'starttls!'     => \$Config{'starttls'},
             'force-auth!'   => \$Config{'force-auth'},
+            'xtraheaders=s' => \$Config{'xtraheaders'},
             'v|verbose!'    => \$Config{'verbose'},
             'd|debug!'      => \$Config{'debug'},
             'c|config'      => \&ShowConf,
@@ -112,6 +115,9 @@ if ($OptSimulation) {
   $Config{'verbose'} = 1;
 }
 
+# -t implies -o if -n is not set
+$OptOutput = 1 if $OptTest && !$OptNewsgroup;
+
 ### create list of @Projects from $Config{'datadir'} unless -p is set
 my @Projects;
 if (!$OptProject) {
@@ -374,6 +380,11 @@ sub BuildPosting {
     warn "W: '$BodyFile' not found.\n";
     return '';
   }
+  my $XtraHeaderFile = $Config{'xtraheaders'} if $Config{'xtraheaders'};
+  if ($Config{'xtraheaders'} && not -r $XtraHeaderFile) {
+    warn "W: '$XtraHeaderFile' not found.\n";
+    return '';
+  }
 
   # today (TD)
   my $TD = DateTime->now->set_time_zone('local');
@@ -401,6 +412,10 @@ sub BuildPosting {
   print "- Reading headers ($Project.hdr) and body ($Project.txt).\n" if $Config{'debug'};
   my @Headers = path($HeaderFile)->lines;
   my @Body    = path($BodyFile)->lines;
+  if ($Config{'xtraheaders'}) {
+    print "- Reading extra headers ($XtraHeaderFile).\n" if $Config{'debug'};
+    push @Headers, path($XtraHeaderFile)->lines ;
+  }
   my %Header  = &ParseHeaders(@Headers);
 
   # check for mandatory headers
@@ -409,6 +424,17 @@ sub BuildPosting {
     return '';
   }
 
+  # check for illegal headers
+  my $FoundIllegalHeader = 0;
+  foreach (qw/Date User-Agent X-Newsreader X-Mailer Injection-Date
+    Injection-Info NNTP-Posting-Date NNTP-Posting-Host X-Trace/) {
+    if ($Header{lc($_)})  {
+      warn "W: $_ header may not be set in '$HeaderFile'.\n";
+      $FoundIllegalHeader = 1;
+    }
+  }
+  return '' if $FoundIllegalHeader;
+
   # add Date:
   push @Headers, 'Date: ' . $TD->strftime('%a, %d %b %Y %H:%M:%S %z') . "\n";
   # add missing Message-ID:
@@ -416,6 +442,16 @@ sub BuildPosting {
   # add User-Agent
   push @Headers, "User-Agent: $NAME/$VERSION\n";
 
+  # check for unencoded 8bit characters in header or body in --debug mode
+  # taken from tinews.pl
+  if ($Config{'debug'}) {
+    print "- Raw 8-bit data in headers.\n" if (grep {/[\x80-\xff]/} @Headers);
+     # check for MIME headers and warn for 8bit characters in body if missing
+    if (!defined($Header{'mime-version'}) || !defined($Header{'content-type'})) {
+      print "- 8bit data in body without MIME-headers.\n" if (grep {/[\x80-\xff]/} @Body);
+    }
+  }
+
   # parse pseudo headers from body
   my ($InRealBody,$LastModified,$PostingFrequency);
   foreach (@Body) {
@@ -475,7 +511,7 @@ sub BuildPosting {
       $_ =~ s/\%p/$$/g;
       $_ =~ s/\%t/$TimeStamp/g;
       # add random part in test mode
-      if ($OptTest) {
+      if ($OptTest && !$OptOutput) {
         my $random = sprintf("%08X", rand(0xFFFFFFFF));
         $_ =~ s/</<test-$random-/;
       }
@@ -491,7 +527,7 @@ sub BuildPosting {
     }
     # add Supersedes: if set
     if (/^Supersedes: /) {
-      if ($LastMID && !$OptTest) {
+      if ($LastMID && (!$OptTest or $OptOutput)) {
         $_= "Supersedes: $LastMID\n";
       } else {
         $_ = '';
@@ -512,8 +548,8 @@ sub BuildPosting {
     }
   }
 
-  # not due if Posting-Freqency is "none"
-  if ($PostingFrequency =~ /none/) {
+  # not due if Posting-Freqency is "none" or never
+  if ($PostingFrequency =~ /none|never/ && !$OptForce) {
     print "... is disabled.\n" if $Config{'verbose'} or $Config{'debug'};
     return '';
   }
@@ -724,6 +760,14 @@ Use a TLS encrypted connection (via STARTTLS) if available.
 You can override this option on the command line by using
 B<--starttls> or B<--nostarttls> accordingly.
 
+=item B<xtraheaders> = I<path>
+
+Path to a file with common headers for all project files. Those
+headers will be appended to each project.
+
+You can override this option on the command line by using
+B<--xtraheaders> = I<path>.
+
 =back
 
 =head2 Project files
@@ -737,7 +781,14 @@ files need to be in B<datadir>.
 Needs to have at least I<From:>, I<Subject:> and I<Newsgroups:> and
 can contain all other headers that the posting should have. Headers
 must conform to RFC 5536 and RFC 5322 and use MIME encoded words for
-8bit characters. B<yapfaq> won't convert headers.
+8bit characters. B<yapfaq> won't convert headers, but will warn of
+unencoded 8bit characters in B<--debug> mode. Longer headers should
+be folded; B<yapfaq> won't fold headers.
+
+The headers file must not contain any of the following headers:
+I<Date:>, I<User-Agent:>, I<X-Newsreader:>, I<X-Mailer:>,
+I<Injection-Date:>, I<Injection-Info:>, I<NNTP-Posting-Date:>,
+I<NNTP-Posting-Host:> or I<X-Trace:>.
 
 I<Subject:> may contain a I<%LM> placeholder that will be replaced
 with the I<Last-modified:> pseudo-header from the text file
@@ -799,7 +850,8 @@ I<Last-modified:> and I<Posting-frequency> will be evaluated by
 B<yapfaq>.
 
 If your content contains 8bit characters, you'll need suitable MIME
-headers in your headers file.
+headers in your headers file.  B<yapfaq> will warn of unencoded 8bit
+characters with missung MIME headers in B<--debug> mode.
 
 B<Example text file with pseudo-headers>
 
@@ -843,7 +895,7 @@ unique I<Message-ID:> (and no I<Supersedes:> header).
 
 Don't post via NNTP, but print to STDOUT.
 
-Combine with B<--test> to avoid updating project status.
+Use B<--test> instead to avoid updating project status.
 
 Intended for testing purposes or to pipe in another program like
 I<inews> or I<tinews.pl>. If you want to pipe the output to another
@@ -861,20 +913,18 @@ Display this man page and exit.
 =item B<-s>, B<--simulation>
 
 Simulation mode. Don't post, just show which projects would be due.
-Implies B<--test> and B<--verbose>.
+Implies B<--test> (without B<--output>) and B<--verbose>.
 
 Can be combined with B<--project> to show if just one project is due.
 
 =item B<-t>, B<--test>
 
 Test mode. Don't update project status (time and Message-ID of last
-posting), dont' add a I<Supersedes:> header and modify the
-I<Message-ID:> with a random part.
+posting); if project is posted to Usenet, dont' add a I<Supersedes:>
+header and modify the I<Message-ID:> with a random part.
 
-The text(s) will still be posted if due or forced by B<--force>.
-
-Combine with B<--output> to redirect output to STDOUT or with
-B<--newsgroup> to override the I<Newsgroups:> header.
+Implies B<--output> (to redirect output to STDOUT) as long as
+B<--newsgroup> (to override the I<Newsgroups:> header) is not set.
 
 =item B<-V>, B<--version>
 
@@ -911,16 +961,16 @@ that are not:
 Do a test run of your I<example> text and and print it on STDOUT
 (whether ist is due or not):
 
-    yapfaq.pl -t -f -o -p example
-    (or yapfaq.pl -tfop example)
+    yapfaq.pl -t -f -p example
+    (or yapfaq.pl -tfp example)
 
 The same, with debugging output (add "-d"):
 
-    yapfaq.pl -tfdop example
+    yapfaq.pl -tfdp example
 
 Force a test post of your I<example> text to I<alt.test>, even if
-the text is not due to be posted (same as before, just replace "-o"
-by "-n alt-test"):
+the text is not due to be posted (same as before, just add
+"-n alt-test"):
 
     yapfaq.pl -t -f -p example -n alt.test
 

contrib/convert.pl

@@ -0,0 +1,16 @@
+#! /usr/bin/perl -w
+#
+# Convert pre-1.0 yapfaq status file to new format
+# 
+# Usage: convert.pl < old.txt.cfg > new.cfg
+
+while (<>) {
+  if (/Lastpost/) {
+    $_ =~ /Lastpost:\s(\d\d?\.\d\d?\.\d\d\d\d)/;
+    print "Last-Posted: $1\n";
+  }
+  if (/LastMID/) {
+    $_ =~ /LastMID:\s(<[^>]+>)/;
+    print "Last-Message-ID: $1\n";
+  }
+}