Down again.

Eric pinged me yesterday afternoon to say that Ourmedia was down. I logged into the webserver using secure shell and it was crawling. So slooooooooooow. There were 92 httpd processes according to

I stopped the server using

and the system became responsive again at the shell level. I restarted with

The database seemed ok, but there were over 400,000 sessions and watchdog records and so I trimmed these down by deleting older records (for user 0 in the case of sessions). The sessions deletes can take a loooooooong time if you try to save some of the recent ones. Over 18 minutes! They system is still responsive and the web server is back up, but the pages are slow while the database query is running.

Need to learn the best ways to deal with these situations when they arise. Also need to return to cron issues so as to ensure these tables do not get so big.

In some ways i am actually happy that we stayed up so long without any administration (I was traveling for several days and this was the longest we have let the system run completely unattended). Not bad, considering. Ideally, Ourmedia needs loving care every single day to grow, and stay, big and strong.

Here are the main things:

• We need maintenance scripts running on regular basis (see cron post below).
• There is still a lot of optimization that can take place.
• Implement caching and more use of static content where appropriate.

Just checked. Things seem to be running nicely. Seems quiet for now. It's Saturday morning, California time. 50 processes. 10 registered users and about 7000 guests (hourly). That's seems pretty normal usage of late. I'm thinking we saw our first busy Friday. It's time to take our capacity to another level.

Had not planned to be a sysadmin. Learning the hard way.