40 hrs of Downtime !
Yeah trak.in was down close to 40 hrs ! its unbelievable but true, Let me be very candid and tell you what happened !
On 22nd March evening, I got many complaints that blog was quite slow. After, logging onto the server I came to know that CPU was showing 100% continuously and any request sent to server was taking ages to get processed. After talking to our support people we decided to restart the server as we could not understand where exactly this issue was. PHP and MySql (normally the culprits) were not showing any usage. I raised the request with our Data Center for reboot –
Here is what came back to me !
This came as a shock to me – My HDD had failed. That was a big jolt. The Data center guys immediately replaced the HDD and did the O/S install, which incidentally took close to 4 hours. It was around 1’am in the morning. The blog was down and there was absolutely nothing we could do about it.
The only thing going on my mind was back-up. From where do I get the back-up? my full daily back-ups of the blog were on my HDD itself, which now was un-readable. I just could not think straight !
On my blog, I also had a Amazon S3 backup plug-in, which unfortunately I had deactivated as of 8th March as it was giving some issues. So, if not anything I atleast had Database backup which was around 2 weeks back.
But even with that, I would loose all the images and rest of stuff !
Here is another twist, when I took this new server 4 months back, I had also taken a paid full backup solution which entitled complete Data backup of server fortnightly. So combination of S3 & this backup could bail me out to a certain extent.
Our support guys in U.S started deploying the server carrying out hardening procedures, installing cpanel and other requisite softwares, which again took about 5 –6 hours – by next day 11 am, the server was set-up. As luck would have it, full backup was corrupted and the size of it was whooping 15 GB compressed.
Support guys spent next 18 hours to rescue some partial data from this back-up.
After about 36 sleepless hours, we were finally able to see something. Yesterday around 6pm, trak.in started functioning, the latest post on it was dated Feb 22nd. We restored our March 8th DB Backup and got another 18 days of posts.
There are close to about 48 posts and about 300 comments which were published since March 8th. We are rescuing posts manually, but on comments front we may not be able to get them !
Thanks to many of our readers who mailed, messaged to us durng this time.
Although, we are still having issues, they should get resolved in next day or two. If you come across any, please do not hesitate to get in touch with us.
Hopefully, never never again !