Thursday, May 26, 2011

Fedora 15 and offlineimap users beware

This is just a word of warning: if you have upgraded to F15 and you use offlineimap, observe very carefully what it does and ensure you have a backup of your email.

This post has updates. See at the end of the post

I noticed yesterday that I couldn't search for some email that I knew existed. On closer inspection I found a huge chunk of my archive inbox to be missing. Several months worth of emails, they stopped in March and then continued with the emails from yesterday that I had copied in from my Inbox.

The same happened to my fedora-devel and xorg-devel folders, possibly others. Re-running offlineimap to sync these pulled down hundreds of emails, so I let it run overnight. This morning I discovered that sometimes offlineimap simply states something like "Deleted 1018 messages" and they're gone again. This is cumulative, it seems to delete only newer ones (I was at September 2010, but after one more run I'm now back to July 2010 being the newest messages in in my xorg-devel folder).

The good news is that the emails are still on the server. Also, so far only mailboxes with huge numbers of emails (>10000) seem to be affected, smaller mailboxes appear to work fine.

I have not filed a bug yet because I don't know what the actual issue is. Could be offlineimap, could be something else. But in the meantime, beware.
Update May 30: Filed as Bug 708898

Update May 30
I downgraded to offlineimap 6.2.0 which is in F14 (offlineimap-6.2.0-2.fc14.noarch) and managed to eventually resync my mailboxes. Then I went back to F15's offlineimap-6.3.3-1.fc15.noarch. Of course it didn't trigger on my test mailbox, but it did then do the same thing again on my xorg-devel inbox.

Syncing lists-xorg: IMAP -> Maildir
Deleting 8181 messages (33540, 335
in Maildir[lists-xorg]

F15's offlineimap-6.2.0-3.fc15.noarch does not show this behaviour and seems to resync the locally deleted emails. Of course that'll take a while, but I'll try to get logs this time (my first couple of tries to get logs from offlineimap crashed it).


Anonymous said...

I trust my mail system much more since I quit deleting emails in my mail client, and started just rsyncing the maildir from my server. offlineimap always seemed risky (and slow in comparison).

Anonymous said...

Do you get a similar effect with isync too?

Christophe said...

There has recently been this thread on the offlineimap mailing list, I don't know if this is the same issue or not

Dodji said...

I am not sure, but it might be related to this ?

Unknown said...

I use isync... Much faster and not in python :)

Anonymous said...

Yeah, happens all the time to me; it always comes back on the next sync though. So far. Thank god.

Anonymous said...

This is a data loss bug!

Why don't you post a bug so that the developers at least know about it?

They will ask for more information if they need it.

Peter Hutterer said...

I haven't filed a bug yet because I haven't narrowed it down yet. As pointed out in the article, I suspect it is offlineimap but I'm not sure. I've updated the whole system, who knows what could trigger this behavior.

Christophe said...

I forgot to ask what version of offlineimap you're seeing this with? 6.2.0 or 6.3.3 (which is in updates-testing)

Anonymous said...

Hi there, offlineimap hacker here. Unfortunately without a debug log of what has been happening, this is extremly tough to reproduce. (I expect you would say the same about graphics issues with :-)). Unfortunately, loss of emails is much more annoying that a temporarily corrupted screen... Sigh.

If you can log it with a -d imap,maildir -l debug.log and send it to us (or to one dev privately due to privacy issues), we could try to find out what went wrong.

Spaetian "spaetz" Spaeth

P.S. With respect to speed compared to isync, I would say that the python speed is the least of our worries :).

Peter Hutterer said...

I noticed on the offlineimap ML that you found the bug & logs already, but for completeness sake: has the required logs

Peter Hutterer said...

urgh. too early for me. Spaetian, not Sebastian. sorry