2020-11-04/05 Server Issues [Resolved]

[Jason]

Issue: Pages have been timing out

Analysis: Services and the server itself have not been restarted for a long time.

Action Taken: Services restarted. Full server reboot performed.

Outlook: Still looking choppy. Situation being monitored.

 

[Jason]

Issue: Pages have been timing out

Analysis: The majority of web traffic is coming from bots aggressively spidering the site.

Action Taken: Blocks have been put in place to block bots hogging server resources.

Outlook: Page timeouts still evident. Situation being monitored.

 

[Jason]

Issue: Site down (database unavailable)

Analysis: Database crashed

Action Taken: Database repaired, dumped, reimported, and rechecked. This caused 1 hour of downtime.

Outlook: Page timeouts still evident. Situation being monitored.

 

[Jason]

Issue: Pages still timing out

Analysis: Tracing of database queries showed that searches were creating traffic jams for the database

Action Taken: Search has been disabled for the next 24 hours while we monitor the situation.

In the meantime please use Google for your searches using the search format:

site:diyaudio.com beer

Outlook: At the time of writing this post, server behaviour is back to normal, with average response time matching historical averages of 85-120ms.

We will wait 24 hours and monitor the situation before making a decision on how to restore search without it impacting the regular use of the site.

 

[Jason]

Issue: Pages timing out

Analysis: Combination of vBulletin and plugins with MyISAM table type causing the issue

Action Taken: Table for thread upgraded from MyISAM to InnoDB, search re-enabled, search disabled for non-logged in members

Outlook: Excellent

 

[Jason]

Issue: Infrequent 1 minute delays after making a post

Analysis: Post table causing locks

Action Taken: Posts table upgraded from MyISAM (table level locks) to InnoDB (row level locks)

Outlook: Situation being monitored, bad news is not expected

 

[Jason]

Unexpected downtime just now of ~10 minutes, unrelated to these issues but part of ongoing optimisations, now resolved.

 

[Jason]

Downtime just now of ~5 minutes while the user table was upgraded to InnoDB, to prevent another class of locks occurring. This was not announced in advance due to the short duration of the downtime. Apologies for the inconvenience.