The Answer Gang

By Jim Dennis, Jason Creighton, Chris G, Karl-Heinz, and... (meet the Gang) ... the Editors of Linux Gazette... and You!

---

Urk... Mutt just did something *ST00PID*

From Benjamin A. Okopnik

Wow. I've got to say that I'm just stunned by the moronic thing that Mutt just did. It's probably the stupidest thing I've ever seen from any Linux app - it rivals 0utlook and IE for complete slack-jawed idiocy.

Once in a while, I get false positives in my spambox. Today, I got one from somebody posting to TAG (I don't recall the name - somebody who had sent it to the wrong address and then bounced the DSN + original mail to TAG), so I saved it to my main mailbox by hitting 'v' (view), selecting the "pre-Spamassassin message", hitting 's' (save), and choosing "/var/mail/ben". When I opened the message, I decided to repeat the operation (i.e., get rid of the "wrapper" message) - so I again hit 'v', selected the original message, and hit 's'. Mutt then popped up a message that said something like "file exists - are you sure?" - and since I had done the same operation dozens of times before, I hit 'y' for 'yes'... at which point, my mailbox got wiped. Zeroed. Nothing left of the 20 or so messages I was going to answer, not even the message that I had theoretically saved. (Mike, your Python article was part of that - so if you could resend, I'd appreciate it.)

I'm in a bit of shock here, and rather pissed off. In all the years I've used Mutt, I never realized that this essentially random bomb was hidden in it - and that it would be triggered off by a message that seemed to make sense in the context.

Dammit. Double dammit, since I use my mbox as a sort of a backup "to do" list - I leave emails that call for some kind of action in it until I've completed that action. Grrrrr.

[Kapil] Commiserations for your loss.

Thanks. As best as I can recall, there was nothing really critical or earth-shakingly important in there, but important enough for the loss to create a high annoyance factor. Semi-amusingly, two of the messages got "saved" by my despamming mechanism: they had been sent to me by a reader who dressed them in spam-like clothing (all-HTML content, funky mail hosts, etc.), and when I forwarded them to TAG - they were in regard to Heather's query in the Mailbag - they got spam-slammed again. So, between a little info message that Kat sent me, Mike's article, and the two not-spams, I've got four messages back.

[Kapil] You could try ext2 recover mechanisms. They might work.

I hadn't thought of that at the time, and given that the same file had new mail in it just a few seconds later, and a very large email in it yesterday evening, I'm pretty sure that there's nothing left.

[Kapil] At one time I had a similar ToDo list at the top of my mbox and my mailer (vm/emacs) of the time did something similar to what mutt just did to you. I was able to recover my ToDo list (though not the more recent stuff in my mbox).

Partly as a result of the above catastrophe, I moved away from vm/emacs but (I think) more importantly, moved away from the mbox format. I am currently an advocate of the MH or maildir formats for personal folders. One mail---one file. Almost no screw-ups by a mail user agent can screw up all my mail again.

[Heather] Unless it screws up the directory. Also mdir index mechanisms can get mangled; though it seems to take more work, it's much wonkier when it does.

I've thought about that in the past. My hindbrain had made some disquieting noises about not being able to search the archive quite as effectively - which does not appear to stand up to rational analysis when considered soberly - and so I'd left it alone.

Hmm, perhaps this is becoming a Gang-relevant question. Folks, what do you think of the pros and the cons of MH vs. Mbox? The net provides much in a way of "yea" and "nay" answers, with only esoterica for support (speed of opening 2,000 messages - wow, very important criterion to me...), and I'd like to hear if anyone has had other positive or negative experiences with either format.

[Rick] (Note: I've studied the pros and cons of Maildir a bit; MH much less so. To a first approximation, I'll assume they're similar.)

1. People with their mailboxes on Nightmare File System need to migrate to Maildir or MH format with all reasonable speed, because of the greatly increased chance of lossage.

Despite Sun's love affair with NFS and their subsequent attempts to smear it on sandwiches, mix it into house paint as a mold retardant, and use it for greasing subway trains, I avoid it like the plague.

[Rick] 2. Otherwise, the advantages of Maildir/MH format strike me as somewhat but certainly not overwhelmingly compelling. I vaguely recall that the mutt MUA has an (optional) indexing feature that reduces the performance hit of Maildir. People who've migrated to that seem happy with it.

I still keep absolutely everything in mbox files, anyway, because I'm lazy and set in my ways, because I've not yet been bitten by a glitch the way you were, and because something about huge trees of little files just doesn't seem right.

We seem to share a similar set of prejudices, Rick. "Lazy, set in my ways, trees of little files vaguely wrong" - yep, that's me to a tee. So far, I haven't heard any compelling arguments for switching - I was hoping that somebody had one...

[Kapil] Having already called myself an advocate for MH/maildir let me point out one disadvantage of MH/maildir on a multi-user system where you do not have control over disk quotas. Both MH/maildir could cause you to run over file (not space) quota. This could also be a problem over NFS (too many NFS file handles ... ).

Y'know, I really enjoy this kind of thing. When I ask this kind of questions, people's answers tend to trigger off the "oh, yeah... I remember reading/hearing/seeing that!" 8 times out of 10. It calls up a strong echo of Brunner's "Shockwave Rider": "We should not be crippled by the knowledge that no one of us can know what all of us together know."

Thanks for the reminder, Kapil!

[Kapil] I have not noticed speed issues. I know "mutt" is a four-letter word in Ben's book right now, but it can employ "header caching" so that MH/maildir folders can be scanned quickly. This also mitigates the NFS problem somewhat. Other mailers may do the same.

Mutt did this one stupid thing in all the time that I've used it - under a defined set of circumstances. I now know enough to avoid those exact circumstances; the only thing that's left is a question of "do I trust Mutt not to present me with other equally boneheaded non-choices?" Well... there are no guarantees, but I believe that Mutt was written with the best of intentions (as well as being subject to the Open Source debugging mechanism), so, yeah, I'm willing to trust it. Conditionally.

[Kapil] Between MH/maildir, the former should be avoided if there is some possibility that the folder could be accessed by two programs at the same time.

To Ben's question I would add the following compatability related question. "Do most mail-related utilities handle MH/maildir nowadays?"

[Nod] I'm very much a CLI user by preference. Looking at the list of Maildir clients, it seems that most of them are GUIs - which mitigates against my adopting it. Although there certainly are CLI clients, they - other than Mutt - are not very common. Given that I log into a variety of systems to read my mail, often over low-bandwidth links, this definitely reduces my options.

[Rick] (Again, I don't know much about MH-format support.)

I have information on various Linux MDAs (mail delivery agents) and LDAs (local delivery agents), here, including which mail-store formats they support, where known:

"MDAs" on https://linuxmafia.com/kb/Mail

I have similar information on 123 MUAs (mail user agents = mail clients) known to be available for Linux, here:

"MUAs" on https://linuxmafia.com/kb/Mail

[Sluggo] Actually, speed of opening mailboxes is important to me. I switched from mbox to MH format years ago because it's "safer" and more "ideologically correct" (notwithstanding Rick's comment to the contrary; I just prefer not stuffing multiple things into one file with a program-specific "separator".) But then I switched back so I could get the "You have new mail" messages from zsh.

I think that this has been one of my dimly-sensed, not-fully-formed objections to Maildir/MH - a feeling that it's not quite as well supported/debugged as mbox. I don't have much of a problem with the idea of a defined separator in the file; the only time I've seen it screw up was when my mail server went bonkers and delivered me a box of messages where the 1st message was headless (the clue came when I looked at one of the old emails in the box - the original content had been extended by something totally unrelated!)

[Sluggo] That seems to work only with file mailboxes, not directory mailboxes. I was concerned about losing mail, then thought, "How often has mbox ever trashed my messages anyway?" I can't think of a single instance where new messages stomped on existing messages. Mutt does do sometimes display an mbox message as two messages, split arbitrarily in the middle, with empty headers for the second (and thus a date like Jan 1980), but that was never critical. NB. I don't use NFS, especially not for mail spool directories.

It sounds to me like an upstream mail server that fails to use the "From hack". The last time I recall that happening was in an email from a listserv server - about ten years ago.

[Sluggo] But with my current computer I keep my mail on my ISP's mailserver and use IMAP. I'm less concerned about ISP snooping or the mailserver going down than about my own computer going down after it has downloaded mail, or my Internet connection randomly freezing for several hours when I was away from home and had to look up a phone number in an email. But I found mutt's IMAP user interface sucks ass: you can't set an IMAP address as your primary inbox, apparently, meaning you have to type this verbose syntax to access it each time you start mutt. I looked at other mail clients. Kmail kept showing a long-outdated configuration I couldn't override so that alarmed me. I settled on Thunderbird, although it sometimes gets its index pointers out of alignment and won't let me access a new message, so I have to restart it. Someday I'll look over the other clients available. I definitely want a client that can read mail in place in standard formats, rather than one that wants to slurp it up into its own format (and deny access to other mail programs). I think both Thunderbird and Kmail want to slurp up local mailboxes.

[Kapil] This has to be an old version of "mutt". The newer one seems to allow

	set spoolfile = imaps://luser@ghost/INBOX

Of course, "mutt" is not particularly good with IMAP. So my current config is based on "offlineimap" which copies all the mail from the server into a local maildir. It also syncs whatever changes I make to the folder back onto the server; in this it improves on "fetchmail" which is one-way.

Of course, this setup means one uses more bandwidth than is strictly necessary just to read those mails that one is interested in. I have so far not suffered any glitches in this setup (but I'm waiting ...).

[Heather] My positive experiences with mdir are about arrival speed on a heavily loaded server. Also, among the IMAP implementations, Courier was about 10x or 11x as fast as wu-imap, which was fragile, and Cyrus was only about 3x. Courier backends with maildirs, though I'm not sure that's where all of its benefit comes from.

The notes and specs rant about mdir being safer but a lock is a lock. They're probably right since I have seen mbox files get bad hiccups from intermingling messages when locks fail. On the other hand individual mails in mdir use up whatever storage unit is going around. Maybe on reiser (where tails that are tiny are often crammed into one stroage unit) this is less of a pain.

[Kapil]

> I know that this is a bit like asking you to lock the stable door
> after the horse has bolted it, so commiserations once again.
                             ^^

[ puzzled ] If the horse has already bolted it - very smart horse, that - why would I bother locking it? Or is this a multiple-level security implementation?

[Kapil] Thanks to one V. Balaji (Tall-balaji) for this excellent improvement of a classical proverb. I have used this version ever since I learnt it from him.

[Heather] Not only that but after pulling this prank, the horse ran away really fast

---

This page edited and maintained by the Editors of Linux Gazette
HTML script maintained by Heather Stern of Starshine Technical Services, https://www.starshine.org/

---