How can I debug a triplicate post (free newserver or free newsreader)?

[Danny D]

Every once in a while, a single nntp post ends up posted in triplicate:

Q: How can I debug, on Linux, if the problem is caused by
(a) my freeware newsreader ? or
(b) my public newserver ?

a. Newsreader = Pan 0.135
b. Newserver = aioe.org:119
c. OS = CentOS 6

From: Danny D <***@example.com>
Newsgroups: alt.home.repair
Subject: Re: Have you ever used this funky style 2" PVC sliding pipe
connection
Date: Tue, 18 Jun 2013 14:40:26 +0000 (UTC)
Organization: Aioe.org NNTP Server
Lines: 26
Message-ID: <kpprgq$f1c$***@speranza.aioe.org>

From: Danny D <***@example.com>
Newsgroups: alt.home.repair
Subject: Re: Have you ever used this funky style 2" PVC sliding pipe
connection
Date: Tue, 18 Jun 2013 14:40:57 +0000 (UTC)
Organization: Aioe.org NNTP Server
Lines: 26
Message-ID: <kpprhp$f2t$***@speranza.aioe.org>

From: Danny D <***@example.com>
Newsgroups: alt.home.repair
Subject: Re: Have you ever used this funky style 2" PVC sliding pipe
connection
Date: Tue, 18 Jun 2013 14:42:31 +0000 (UTC)
Organization: Aioe.org NNTP Server
Lines: 26
Message-ID: <kpprkn$fcl$***@speranza.aioe.org>

[Whiskers]

["Followup-To:" header set to alt.free.newsservers.]

Post by Danny D
Q: How can I debug, on Linux, if the problem is caused by
(a) my freeware newsreader ? or
(b) my public newserver ?

You could try a packet sniffer (eg Wireshark) to analyse the traffic
between your system and the news-server. But first, look at all the
headers you've given us:

[...]

Post by Danny D
Date: Tue, 18 Jun 2013 14:40:26 +0000 (UTC)

[...]

Post by Danny D
Date: Tue, 18 Jun 2013 14:40:57 +0000 (UTC)

[...]

Post by Danny D
Date: Tue, 18 Jun 2013 14:42:31 +0000 (UTC)

[...]

Also from the triplicated post here:

Date: Tue, 18 Jun 2013 15:18:07 +0000 (UTC)
Message-ID: <kpptnf$lci$***@speranza.aioe.org>

Date: Tue, 18 Jun 2013 15:19:09 +0000 (UTC)
Message-ID: <kpptpd$lh2$***@speranza.aioe.org>

Date: Tue, 18 Jun 2013 15:18:38 +0000 (UTC)
Message-ID: <kpptoe$leq$***@speranza.aioe.org>

and from your post via Albasani:

Date: Tue, 18 Jun 2013 19:21:23 +0000 (UTC)
Message-ID: <kpqbvi$j1i$***@news.albasani.net>

From these, notice that:

- within each trio, there are three different Message IDs.

- comparing the MIDs in the posts via AIOE and Albasani, it looks as though
your own newsreader is allocating the part of the MID to the left of the
@ sign. The part to the right may also be allocated by your newsreader,
derived from the name or address of the news-server used.

- the timing of the articles in a trio differs by a few seconds, or in one
case by more than a minute.

As both the Date and Message-ID headers are probably created by your own
newsreader, and should be generated when you "send" the message (ie click
on the 'Send' button or whatever), it looks as though that "send" command
is being repeated. The obvious way for that to happen is for the user (ie
you) to do it, perhaps by not waiting long enough for your newsreader and
news-server to finish talking to each other, and clicking again, or by
inadvertently holding the mouse button down long enough for three clicks to
be registered by the software before it has finished handling the first
click.

If there is a keyboard command for sending an article, instead of a GUI
point and click, try using that instead for posts you send via AIOE; there
is less chance of inadvertently duplicating (or triplicating) such
commands.

AIOE is a "special case" among news-servers. Its resources are probably
very limited, and it has a custom-made filter system on the 'posting' side
instead of a conventional access control, so it may well respond much less
quickly than other servers.

[Richard Kettlewell]

Post by Whiskers
- comparing the MIDs in the posts via AIOE and Albasani, it looks as though
your own newsreader is allocating the part of the MID to the left of the
@ sign. The part to the right may also be allocated by your newsreader,
derived from the name or address of the news-server used.

That model isn’t consistent with the way the protocol works. Either the
client chooses the whole message ID or the server does (if the client
doesn’t). The construction in this case is consistent with the latter.

It looks like Pan can generate message IDs if configured to do so but
they start “pan$” (see GNKSA::generate_message_id()), so evidently that
option isn’t set in this case.

Setting it might solve the OP’s problem, depending on exactly when it
operates (i.e. whether it picks one ID and sticks with it or stupidly
generates a new one for each retry).

[Danny D.]

Post by Richard Kettlewell
It looks like Pan can generate message IDs if configured to do so but
they start “pan$” (see GNKSA::generate_message_id()), so evidently that
option isn’t set in this case.

I don't have message IDs set in Pan, so, I always thought the newsserver
was setting the message ID:
Loading Image...

[Richard Kettlewell]

Post by Danny D.

Post by Richard Kettlewell
It looks like Pan can generate message IDs if configured to do so but
they start “pan$” (see GNKSA::generate_message_id()), so evidently that
option isn’t set in this case.

I don't have message IDs set in Pan, so, I always thought the newsserver
http://www5.picturepush.com/photo/a/13357288/img/13357288.png

It’s called add-message-id-header-when-posting in the source code, I
don’t know how that’s exposed in the UI.

[Danny D.]

Post by Richard Kettlewell

Post by Danny D.
I don't have message IDs set in Pan, so, I always thought the newsserver
http://www5.picturepush.com/photo/a/13357288/img/13357288.png

It’s called add-message-id-header-when-posting in the source code, I
don’t know how that’s exposed in the UI.

In ~/.pan2/preferences.xml is this line:
<flag name='add-message-id-header-when-posting' value='false'/>

[Richard Kettlewell]

Post by Danny D.

Post by Richard Kettlewell

Post by Danny D.
I don't have message IDs set in Pan, so, I always thought the
http://www5.picturepush.com/photo/a/13357288/img/13357288.png

It’s called add-message-id-header-when-posting in the source code, I
don’t know how that’s exposed in the UI.

<flag name='add-message-id-header-when-posting' value='false'/>

Looks promising - I would try stopping Pan, editing the ‘false’ to
‘true’ and then restarting Pan. Take a backup copy first so you can
restore it easily if it doesn’t work.

NB I’ve never used Pan - I’m just going on a superficial look at the
code here.

[Danny D.]

Post by Richard Kettlewell
Looks promising - I would try stopping Pan, editing the ‘false’ to
‘true’ and then restarting Pan. Take a backup copy first so you can
restore it easily if it doesn’t work.

OK. I will try that. At the moment, I'm getting aioe server connectivity
problems.

This happens periodically (only with aioe and news4all.se and not any of
the other dozen or so free news servers); so I will have to wait for the
aioe connectivity issues to abate.

Note: I think Paolo disconnects an IP address that has too many errors,
as he had mentioned that to me prior in an email a while ago.

[Richard Kettlewell]

Post by Danny D.

Post by Richard Kettlewell
Looks promising - I would try stopping Pan, editing the ‘false’ to
‘true’ and then restarting Pan. Take a backup copy first so you can
restore it easily if it doesn’t work.

OK. I will try that. At the moment, I'm getting aioe server connectivity
problems.

That’s not entirely surprising; my guess at what’s going on is that the
TCP connection is broken after the article has been sent to the news
server, but before a positive response has been transmitted back to Pan.
(Certain kinds of connectivity problem could readily lead to this
happening.) The outcome of this would be that Pan mistakenly believes
the message hasn’t been posted and retries (or so it seems), leading to
duplicates.

The point of choosing the message ID on the client instead of the server
is that the server won’t accept two messages with the same message ID,
repeat attempts to post with the same ID won’t lead to actual
duplication.

[Danny D.]

Post by Richard Kettlewell
The outcome of this would be that Pan mistakenly believes
the message hasn’t been posted and retries (or so it seems),
leading to duplicates.

This is very interesting, and sounds likely.

The fact there are never more than 3, may indicate that Pan
tries at least 3 times and then gives up (just a guess).

Here's a screenshot of the log file during an attempt:
Loading Image...

I've pretty much ruled out the fact that I might have "bounced"
the send button because, well, for one, there is no "send" button;
but, more importantly, the "File->Send Article" menu that I use
to send makes the whole send gui go away as soon as you click it.

Also, Control+return doesn't seem to make any difference; but it's
hard to test because I'm using another server to send this as
aioe doesn't like me since this morning. :(

[Whiskers]

Post by Richard Kettlewell

Post by Whiskers
- comparing the MIDs in the posts via AIOE and Albasani, it looks as though
your own newsreader is allocating the part of the MID to the left of the
@ sign. The part to the right may also be allocated by your newsreader,
derived from the name or address of the news-server used.

That model isn’t consistent with the way the protocol works. Either the
client chooses the whole message ID or the server does (if the client
doesn’t). The construction in this case is consistent with the latter.

It's also consistent with the former; the newsreader has to know the
address of the news-server (or the IP number) as well as the arbitrary
"name" for it chosen by the user.

The 'unique' parts of each of the OP's MIDs seem to have been created in
the same way, which suggests the same software is involved, regardless of
whether the articles are posted to Albasani or AIOE. So it seems
reasonable to suppose that the user's own system is creating them - whether
Pan or some other tool is doing it.

Post by Richard Kettlewell
It looks like Pan can generate message IDs if configured to do so but
they start “pan$” (see GNKSA::generate_message_id()), so evidently that
option isn’t set in this case.
Setting it might solve the OP’s problem, depending on exactly when it
operates (i.e. whether it picks one ID and sticks with it or stupidly
generates a new one for each retry).

If there's a setting the user can change, then changing it might well
reveal more clues.

[Richard Kettlewell]

Post by Whiskers

Post by Richard Kettlewell

Post by Whiskers
- comparing the MIDs in the posts via AIOE and Albasani, it looks as though
your own newsreader is allocating the part of the MID to the left of the
@ sign. The part to the right may also be allocated by your newsreader,
derived from the name or address of the news-server used.

That model isn’t consistent with the way the protocol works. Either the
client chooses the whole message ID or the server does (if the client
doesn’t). The construction in this case is consistent with the latter.

It's also consistent with the former; the newsreader has to know the
address of the news-server (or the IP number) as well as the arbitrary
"name" for it chosen by the user.

Sorry, it looks like I misread your remarks.

Post by Whiskers
The 'unique' parts of each of the OP's MIDs seem to have been created
in the same way, which suggests the same software is involved,
regardless of whether the articles are posted to Albasani or AIOE. So
it seems reasonable to suppose that the user's own system is creating
them - whether Pan or some other tool is doing it.

Albasani and AIOE both run INN, and strings in question match the syntax
INN uses for message IDs. They don’t match the syntax Pan uses. There
is no mystery about where those IDs were created.

[Danny D.]

Post by Richard Kettlewell
Albasani and AIOE both run INN, and strings in question match the syntax
INN uses for message IDs. They don’t match the syntax Pan uses. There
is no mystery about where those IDs were created.

OK. I see the XXX didn't take (probably because Pan wasn't generating
the Message ID?).

Anyway, I just changed the ~/.pan2/preferences.xml from:
<flag name='add-message-id-header-when-posting' value='false'/>
to:
<flag name='add-message-id-header-when-posting' value='true'/>

I realized I needed to kill and restart pan so this is what I did:
a) I killed pan
b) I modified the preferences.xml
c) I restarted pan and will send this message

[Danny D.]

It looks like Pan generated this message ID based on the
time and date.

I've added a domain of "fakedomain.com" to see what the
message ID is for this message.

[Danny D.]

Now the message ID seems wholly controlled by pan.

When Aioe lets me back in (usually that takes a day after a series
of errors), I'll test it out using aioe.org:119.

[Whiskers]

Post by Danny D.
Now the message ID seems wholly controlled by pan.
When Aioe lets me back in (usually that takes a day after a series
of errors), I'll test it out using aioe.org:119.

Now that you control your MIDs, you can include a personal "Fully Qualified
Domain Name" (FQDN) to the right of the @ sign. Albasani can provide you
with one at no charge; see the FAQ at their web site.

Once your MIDs are 'unique to you', you can use them with your newsreaders
'filter' or 'score' system to make replies to your own posts easier to
find.

[Danny D.]

Post by Whiskers
Now that you control your MIDs, you can include a personal "Fully Qualified
with one at no charge; see the FAQ at their web site.
Once your MIDs are 'unique to you', you can use them with your newsreaders
'filter' or 'score' system to make replies to your own posts easier to
find.

Interesting concept. Up until now, I had thought Message IDs were, somehow,
unique. But they seem not to be, based on what you wrote. So, they're just
useful for tracking within a given time period, I guess.

Very interesting.

[Richard Kettlewell]

Post by Danny D.

Post by Whiskers
Now that you control your MIDs, you can include a personal "Fully
can provide you with one at no charge; see the FAQ at their web site.
Once your MIDs are 'unique to you', you can use them with your
newsreaders 'filter' or 'score' system to make replies to your own
posts easier to find.

Interesting concept. Up until now, I had thought Message IDs were, somehow,
unique. But they seem not to be, based on what you wrote. So, they're just
useful for tracking within a given time period, I guess.

The requirement is that message IDs are globally unique. Any given news
server will only accept one message with a given message ID.

That doesn’t require that the domain half be unique as such - it’s
possible to make clashes vanishingly unlikely just by picking the LHS
uniformly at random from a sufficiently large set.

A unique domain name does help with uniqueness, however, if the left
hand side isn’t very random. In Pan’s case it isn’t “very random” -
it’s the date down to the nearest second. Hence my suggestion in an
early post.

It also doesn’t require that you “own” the domain in any meaningful
sense; and even if you did “own” it, you couldn’t stop someone else
using it for message IDs anyway.

Finally it doesn’t (technically speaking) require that nobody else owns
the domain, if they’re not actually using it for Usenet message-IDs.
But it does seem rude to use someone else’s domain without asking.

[Danny D.]

Post by Richard Kettlewell
The requirement is that message IDs are globally unique. Any given news
server will only accept one message with a given message ID.

I think everyone is suggesting this be the solution, and I don't disagree.

One question: What if I end up using a MessageID that was used, say,
a year or two ago?

Would the news server still reject that message?

[Richard Kettlewell]

Post by Danny D.

Post by Richard Kettlewell
The requirement is that message IDs are globally unique. Any given news
server will only accept one message with a given message ID.

I think everyone is suggesting this be the solution, and I don't disagree.
One question: What if I end up using a MessageID that was used, say,
a year or two ago?
Would the news server still reject that message?

It depends. A common configuration is to forget both articles and
message IDs after (say) a few weeks. But there are news swervers which
retain messages for years (and compete for business based on retention,
among other factors) or indefinitely (e.g. Google).

[Danny D.]

Post by Richard Kettlewell
A common configuration is to forget both articles and
message IDs after (say) a few weeks. But there are news swervers which
retain messages for years

OK. I guess the answer is that it "could" drop your message if the
MessageID is the same as one from as many as a decade prior.

Wow.

I had never thought about message IDs before - and now I'm having
to consider having a machine create unique ones that have to be
unique for all messages within the past decade.

That must be why you guys were so attentive to the proper domain
status half.

[Danny D.]

Post by Richard Kettlewell
A unique domain name does help with uniqueness, however, if the left
hand side isn’t very random. In Pan’s case it isn’t “very random” -
it’s the date down to the nearest second. Hence my suggestion in an
early post.

This is decidedly the case that pan's isn't at all very unique.
So I _do_ understand why the concentration on a better right side.

[Danny D.]

Post by Richard Kettlewell
But it does seem rude to use someone else’s domain without asking.

This seems to be the key point, as you said, even owning the domain
wouldn't stop others from using it in and of itself.

[Jasen Betts]

Post by Danny D.
It looks like Pan generated this message ID based on the
time and date.
I've added a domain of "fakedomain.com" to see what the
message ID is for this message.

fakedoamin.com is owned by information.com

you can get a real domain name from free from dyndsns.com or no-ip.biz etc.

[Richard Kettlewell]

Post by Jasen Betts

Post by Danny D.
It looks like Pan generated this message ID based on the
time and date.
I've added a domain of "fakedomain.com" to see what the
message ID is for this message.

fakedoamin.com is owned by information.com
you can get a real domain name from free from dyndsns.com or no-ip.biz etc.

Registering a domain name just for message IDs seems a bit excessive.
You could use a random string plus .invalid, e.g. 8q82vMuiSABms7xP.invalid.

[Danny D.]

Post by Richard Kettlewell
You could use a random string plus .invalid, e.g. 8q82vMuiSABms7xP.invalid.

How does this look for proper nntp etiquette?
Message-ID: <***@domain.is.invalid>

[Danny D.]

Post by Jasen Betts
fakedoamin.com is owned by information.com

Ooops. I didn't check prior. Thanks. I'll change it.

[Danny D.]

Post by Whiskers
If there's a setting the user can change, then changing it might well
reveal more clues.

Here is one MID setting that I changed for this email:
Loading Image...

I made the pan-included Domain name XXX as shown in the screenshot above.

In the next email, I will change the pan preferences.xml MID setting.

[Whiskers]

Post by Danny D.

Post by Whiskers
If there's a setting the user can change, then changing it might well
reveal more clues.

http://www2.picturepush.com/photo/a/13360135/img/13360135.png
I made the pan-included Domain name XXX as shown in the screenshot above.
In the next email, I will change the pan preferences.xml MID setting.

<cough> <cough> this is usenet; your messages are "articles" or "posts",
but not "emails" ;))

[Danny D.]

Post by Whiskers
<cough> <cough> this is usenet; your messages are "articles" or "posts",
but not "emails" )

Oops. Mea culpa. I appreciate the clarification.

(I often post to alt.usage.english for similar purposes - so I much appreciate
the admonishment.)