...making Linux just a little more fun!
Rick Moen [rick at linuxmafia.com]
Thu, 20 Jul 2006 18:41:22 -0700
Ben recently called my attention to incoming mail being rejected at my SMTP server because the system SPF daemon could not be reached. More recently, about an hour ago, a flurry of spam snuck through the system because the SpamAssassin daemon had died. And Exim4 (the SMTP daemon) has occasionally entered a fault mode where a queued-up message is processed repeatedly.
All of these epiphenomena turn out to have the same underlying cause: Baby needs new shoes.
Just as a frog put in a pot of cold water and brought slowly to a boil will not jump out and save himself, mail servers reach the brink of failure by... um... degrees. In this case, what's been happening is that my server's meagre 256MB of RAM is being badly overstressed.
The repeat-sending syndrome was an early clue: The queue-handling Exim4 instance would send the queued mail out, and then call a different routine to process final steps including deletion that just happened to require more RAM, which was not available so the Exim4 instance died before cleanup could occur.
At the same time, I've sung the praises of GNU screen so effectively that several other shell users (Hi, Karsten! Hi, Ben! Hi, Thomas![1]) have joined me in leaving it running 24x7 running (sometimes) large jobs -- more RAM gone. And last, the volume of incoming spam continues to increase and find new tricks, while my primary SMTP-layer defence of Exim4 rulesets isn't being updated because the host is a lame-duck installation intended to be retired.
Right behind the primary defence, intended to backstop spam that gets past Exim4's rules, is -- guess what? SpamAssassin, a rather large, slow, and RAM-grabbing Perl script, running in daemon mode and spawning more instances as needed. The busier and slower it gets, the more RAM-grabbing instances.
So, we've been getting weird MTA repeated-mailing behaviour, and mysteriously dying processes: The kernel OOM-killer gets set loose, and starts shooting large and/or busy processes in the head. If I'm lucky, it's one of my mutt instances running under screen and reading one of my carelessly overlarge mbox files. Or screen itself. If I'm unlucky, it's spfd. Or SpamAssassin. Or Exim4. Or Apache httpd. Or BIND9. All of those have happened.
The pattern was obvious, but I've been reluctant to see it, in part because it's a pain to deal with.
The only fix is to debug my problems with the intended replacement box (needs a kernel with ability to load the root FS from software RAID1), configure its daemons for the needed services, sync up all data files, and do a flag day. The new box is better in every way, including its 1.5 GB of system RAM.
I need to do that. You guys really can't. I just need to find the time.
In the meantime, I'm using screen less. (Karsten, if you can help in that area, too, that would be appreciated.)
[1] Latter two names being a guess -- and it's not like there's anything wrong with using screen. It's great stuff.
--
Cheers, Your eyes are weary from staring at the CRT. You feel
Rick Moen sleepy. Notice how restful it is to watch the cursor
rick at linuxmafia.com blink. Close your eyes. The opinions stated above are
yours. You cannot imagine why you ever felt otherwise.
Rick Moen [rick at linuxmafia.com]
Fri, 21 Jul 2006 11:37:44 -0700
Redirecting back to "tag", just in case we have some diagnostic opinions from there.
Quoting Benjamin A. Okopnik (ben at linuxgazette.net):
> That's how I figured you were using it, Rick. If you wanted money, you'd > have said so - I have this feeling that you got over being all shy and > retiring a while ago.The newer machine in question's still, to my way of thinking, pretty nice: Single-proc PIII/800 or so, VA Linux Systems 2230 2U rackmount chassis, Intel L440GX+ "Lancewood" motherboard, 1.5TB RAM, 2 x 73GB RAID1 pair (Linux software RAID) for the important filesystems, 16GB? boot drive,I just thought that you might still be in the > planning/buying stages.
That's pretty snazzy -- for 2001. ;->
All the filesystems are built, and it's loaded with Debian "etch" 4.0 and a rough cut of all the necessary software, not yet fully configured. Data files haven't yet been copied over (IIRC).
The last time I worked on it, I'd fetched a new Debian-packaged binary kernel 2.6.x and blithely removed the previous, believed-bootable installed 2.6.x kernel. And then rebooted, found that I'd just shot myself in the foot, lost patience / ran out of time, and quit for the day. I've not yet gotten back to it, and meantime other things have been keeping me away.
You know, it's also possible the old box has developed a bad spot of RAM, or something like that. Look at this kernel "oops" from /var/log/messages, which is typical of process blowouts, lately:
Jul 21 11:20:38 linuxmafia kernel: <1>Unable to handle kernel NULL pointer dereference at virtual address 00000004 Jul 21 11:20:38 linuxmafia kernel: printing eip: Jul 21 11:20:38 linuxmafia kernel: c0153ca5 Jul 21 11:20:38 linuxmafia kernel: Oops: 0000 Jul 21 11:20:38 linuxmafia kernel: CPU: 0 Jul 21 11:20:38 linuxmafia kernel: EIP: 0010:[prune_icache+53/464] Not tainted Jul 21 11:20:38 linuxmafia kernel: EFLAGS: 00210213 Jul 21 11:20:38 linuxmafia kernel: eax: 74756564 ebx: 00000000 ecx: 00000006 edx: 00000004 Jul 21 11:20:38 linuxmafia kernel: esi: fffffff8 edi: 00000000 ebp: 00000383 esp: c4e0dddc Jul 21 11:20:38 linuxmafia kernel: ds: 0018 es: 0018 ss: 0018 Jul 21 11:20:38 linuxmafia kernel: Process exim4 (pid: 32387, stackpage=c4e0d000) Jul 21 11:20:38 linuxmafia kernel: Stack: cb7b3e20 00000000 c4e0dde4 c4e0dde400 000009 c1046310 c025ecd8 00001951 Jul 21 11:20:38 linuxmafia kernel: c0153e64 00000383 c01353eb 00000006 000001d2 ffffffff 000001d2 00000009 Jul 21 11:20:38 linuxmafia kernel: 0000001e 000001d2 c025ecd8 c025ecd8 c01357bd c4e0de50 000001d2 0000003c Jul 21 11:20:38 linuxmafia kernel: Call Trace: [shrink_icache_memory+36/64] [shrink_cache+379/944] [shrink_caches+61/96] [try_to_free_pages_zone+98/256] [locate_hd_struct+56/160] Jul 21 11:20:38 linuxmafia kernel: [balance_classzone+66/480] [__alloc_pages+376/640] [do_anonymous_page+92/256] [handle_mm_fault+119/256] [do_page_fault+456/1337] [e100:__insmod_e100_O/lib/modules/2.4.27-2-686/kernel/drivers/net+-687130/96] Jul 21 11:20:38 linuxmafia kernel: [process_timeout+0/80] [bh_action+34/64] [tasklet_hi_action+70/112] [do_IRQ+154/160] [do_page_fault+0/1337] [error_code+52/60] Jul 21 11:20:38 linuxmafia kernel: Jul 21 11:20:38 linuxmafia kernel: Code: 8b 5b 04 8b 86 08 01 00 00 a8 38 0f 84 1c 01 00 00 81 fb a8Off hand, I'm uncertain of the root cause.
Martin Hooper [martinjh at blueyonder.co.uk]
Fri, 21 Jul 2006 19:50:43 +0100
On 21/07/2006 Rick Moen wrote:
> The newer machine in question's still, to my way of thinking, pretty > nice: > Single-proc PIII/800 or so, VA Linux Systems 2230 2U rackmount > chassis, > Intel L440GX+ "Lancewood" motherboard, 1.5TB RAM, 2 x 73GB RAID1 pair > (Linux software RAID) for the important filesystems, 16GB? boot drive
Rick do you mean 1.5Gb memory?? ;) Just thinking that terabytes of memory is going a bit OTT... I also thought that P3's usually only go up something like 2Gb memory, not sure off hand though.
Rick Moen [rick at linuxmafia.com]
Fri, 21 Jul 2006 18:13:27 -0700
Quoting Martin Hooper (martinjh at blueyonder.co.uk):
> On 21/07/2006 Rick Moen wrote: > > The newer machine in question's still, to my way of thinking, pretty > > nice: Single-proc PIII/800 or so, VA Linux Systems 2230 2U rackmount > > chassis, Intel L440GX+ "Lancewood" motherboard, 1.5TB RAM, 2 x 73GB > > RAID1 pair (Linux software RAID) for the important filesystems, > > 16GB? boot drive > > Rick do you mean 1.5Gb memory?? ;)
D'oh! Yes, I only wish I had 1.5 TB of RAM. I'm willing to accept that even in PC-100. Just leave it in a brown bag on my doorstep, please, and nobody need get hurt. ;->
Yes, the old box is back online. Absent-minded members of my household (myself certainly included) had closed up all the doors leading into the garage that is the temporary home of our servers. Today, my town had record high temperatures of 35 degrees (95, if using last millennium's Fahrenheit scale) -- which meant it was probably closer to 40 inside the sealed garage. And the machine simply was unhappy, that way.
There will be a fresh backup, soonish -- and I'll devote serious attention to the long-delayed hardware migration, -and- to creating the planned server-shelf space in the foundation crawlspace, under my house. For now, there's also an electric fan blowing additional air at the server.
For the record, anyway, if you see segfaults and kernel oopses, it may indicate a runaway heat problem. I didn't know that, before.
Benjamin A. Okopnik [ben at linuxgazette.net]
Sat, 22 Jul 2006 22:18:37 -0400
On Fri, Jul 21, 2006 at 06:13:27PM -0700, Rick Moen wrote:
> > For the record, anyway, if you see segfaults and kernel oopses, it may > indicate a runaway heat problem. I didn't know that, before.
One of the laptops that I tested before buying the HP that I have as my backup machine ran ridiculously hot - I actually got to see the kernel spit out a "thermal shutdown" message and halt (I didn't realize it had such a goodie in it until it did that.) During the short time that I ran it - and I actually tried two different machines of the same make and model - a number of the sessions terminated either in a thermal cutout or a segfault.
In general, when I see a segfault that wasn't caused by a known factor (e.g., a just-compiled, highly experimental kernel), I immediately suspect either a) bad hardware or b) overtemp conditions. I suppose you could make the case that b) really resolves to a) - I've always considered memory to be analog hardware, anyway... It sorta works within parameters when the moon in in the right phase, but tends to wander outside of them whenever anything (like the price of pork bellies on the commodities market, or the percentage of carbon dioxide on Mars) changes.
* Ben Okopnik * Editor-in-Chief, Linux Gazette * https://linuxgazette.net *