UIDL settings & logic - Delayed deletion & duplicate

[SES21]

I offer this, my very 1st post, in honor of dbeusee for his very fast work earlier this month, muerte for his endless loyalty to the community & anujseth for the awesome new site stuff - I'm way behind on payback!

First, by way of introduction, although this is my 1st post, I've been a lurker here off & on for well over a year. I've never felt a need to (register or) post here before because you guys were always a step ahead or more knowledgeable or whatever than me. Every time I've come here, you've always been aware of any problems or new developments & a fix was always in the works so I didn't have anything to offer. I finally saw one place I may be able to help...

I'm hoping this post can serve as a replacement, albeit with any editing deemed appropriate by the mods, for the sticky post that has disappeared about UIDL logic & why it works like it does. Keep in mind I didn't design it so please don't shoot the messenger. The way it's implemented actually offers you some flexibility that you wouldn't have otherwise so be careful what you ask for if you want them to change it! My suggestion is that it be better documented so it doesn't raise so many questions & people understand how to use it then leave it alone until someone comes up with better ideas.

That said, here's my (intentionally) non-techy description of how it works.

When you enable UIDL in YPOPs, you allow your POP client (OE, OL, Mozilla, whatever) to talk in a sort of shorthand with YPOPs using the UIDL message IDs. YPOPs provides the message IDs to the client with an associated sequence number (yes, 1,2,3, etc) in order from your mail. Your POP client maintains its own list of IDs to compare to the list it gets from YPOPs. From there, the POP client then uses the sequence number to tell YPOPs what to do with a particular message like RETR 1 (retrieve msg #1) or DELE 8 (delete msg #8).

Here's where people usually don't get it because the settings are counter-intuitive so please pay close attention:

YPOPs provides a different list of message IDs based on whether you set it to download all messages or only unread messages. If you set YPOPs to only download unread mail, the list of message IDs that's provided to your POP client is only for the unread messages & does NOT include the message IDs that are marked read. When your POP client compares its list to the YPOPs list, it thinks those are the only messages in your mailbox (your POP client doesn't know that the list is only for unread messages!) & it doesn't have an associated sequence number for the old messages to tell YPOPs to delete them, if appropriate. That's why you have to set YPOPs to d/l all messages if you want delayed deletion to work - so YPOPs will provide a msg ID & sequence # for *every* message in your mail.

Now on to why you get duplicates in hopes that this will help the devs in their troubleshooting on this problem (FWIW, I've seen fewer instances of dups since the 0.7.x releases & I believe it's because of the updated cUrl libraries or whatever they're called).

If there is any mismatch in the UIDL list that your POP client has & what YPOPs provides, you will get duplicates. This can be happen for several different reasons:
1. OE, OL, Netscape, whatever didn't get closed properly & has scrambled its list (especially if you have more than one instance running on your system - typically by accident & unknown to you) - in this case, you'll usually get duplicates twice - once during the initial scramble & again when you reload your client, it regains its original list & requests the messages that you downloaded in between.
2. Yahoo has done something on their end to change the msg IDs
3. You can cause this yourself if you delete a message in Yahoo & then move it back to your Inbox - it gets a new msg ID assigned by Yahoo
4. YPOPs scrambles its list somehow & doesn't give your POP client the same message IDs it had previously.

[edit] 1/8/06 - After much time in the forum, I've found a 5th & even more common reason for duplicate emails in typical use! If you use delayed deletion & "apply limit to message list" in YP, you will get duplicates as messages move above & then back below your max email setting. For example, you have max at 50, get more than 50 messages in your Inbox, then delete 2 of the newer ones. You will end up with duplicates of what had become messages 51 & 52 when they drop back to 49 & 50 again. The best fix is to turn off "apply limit." The bad news is that, if you're already over your max limit, you will get dups right after you change that setting. Sorry! [/edit]

You can control what happens to those messages on the Yahoo end if you're careful about which duplicate copy you delete but that's another discussion for another post!

I hope I've been able to shed some light on this oft-confused subject...

Hats off to the folks that make it all happen & thank you for the great work!

SES

[anujseth]

Thanks for the detailed explanation. I'm making this sticky.

[dbeusee]

This is a good summary - thanks!

Here's the technical discussion, which I just converted from sticky to normal (since we only need one sticky article on this):

http://ypops.sourceforge.net/index.php? ... pic&t=1737

And my last short reply, which sums up how to get it to work:

http://ypops.sourceforge.net/index.php? ... =7465#7465

-Don

[mickrussom]

I'm having very hard problems even using this guide to stop massive duplicates with build 0.9.5.1.

Has anyone else had issues with 0.9.5.1 and duplicates?

[wariskampar]

Hello, i think i encounter this problem. i will explain it in my own way since i'm nooby. i deleted an email from email client but the next time i re-open my email client, the same email will be downloaded again. i think it is because there is still a copy of that email in Yahoo account. i think the probable solution to set YP to sync email deletion between email client and yahoo account. is it possible and what is the setting.

Another problem is, YP seem to download everthing in Yahoo account including Junk email. How to prevent this.

Btw, I'm using WIndows XP SP2 and Windows Live Mail. YP 0.9.5.1

Hope to hear from someone soon