Trak.in is a popular Indian Business, Technology, Mobile & Startup blog featuring trending News, views and analytical take on Technology, Business, Finance, Telecom, Mobile, startups & Social Media Space

40 hrs of Downtime !

4

Yeah trak.in was down close to 40 hrs ! its unbelievable but true, Let me be very candid and tell you what happened !

On 22nd March evening, I got many complaints that blog was quite slow. After, logging onto the server I came to know that CPU was showing 100% continuously and any request sent to server was taking ages to get processed. After talking to our support people we decided to restart the server as we could not understand where exactly this issue was. PHP and MySql (normally the culprits) were not showing any usage. I raised the request with our Data Center for reboot –

Here is what came back to me !

Support-Request1

This came as a shock to me – My HDD had failed. That was a big jolt. The Data center guys immediately replaced the HDD and did the O/S install, which incidentally took close to 4 hours. It was around 1’am in the morning. The blog was down and there was absolutely nothing we could do about it.

The only thing going on my mind was back-up. From where do I get the back-up? my full daily back-ups of the blog were on my HDD itself, which now was un-readable. I just could not think straight !

On my blog, I also had a Amazon S3 backup plug-in, which unfortunately I had deactivated as of 8th March as it was giving some issues. So, if not anything I atleast had Database backup which was around 2 weeks back.

But even with that, I would loose all the images and rest of stuff !

Here is another twist, when I took this new server 4 months back, I had also taken a paid full backup solution which entitled complete Data backup of server fortnightly. So combination of S3 & this backup could bail me out to a certain extent.

Our support guys in U.S started deploying the server carrying out hardening procedures, installing cpanel and other requisite softwares, which again took about 5 –6 hours – by next day 11 am, the server was set-up. As luck would have it, full backup was corrupted and the size of it was whooping 15 GB compressed.

Support guys spent next 18 hours to rescue some partial data from this back-up.

After about 36 sleepless hours, we were finally able to see something. Yesterday around 6pm, trak.in started functioning, the latest post on it was dated Feb 22nd. We restored our March 8th DB Backup and got another 18 days of posts.

There are close to about 48 posts and about 300 comments which were published since March 8th. We are rescuing posts manually, but on comments front we may not be able to get them !

Thanks to many of our readers who mailed, messaged to us durng this time.

Although, we are still having issues, they should get resolved in next day or two. If you come across any, please do not hesitate to get in touch with us.

Hopefully, never never again !

  1. Pradeep says

    Hi Arun,
    I can understand the pain and know there was nothing much that could have been done.
    Hope you get back-up data and it run smoothly. Trak.in rocks!!

    All the best
    Thanks
    Pradeep

  2. Srini says

    I think it is time to move to the cloud. Have you explored having an instance in Amazon or Rackspace or gogrid cloud?

  3. Madhav Shivpuri says

    Hi Arun,

    Sorry to hear the troubles you encountered… its a wake up call to all of us webmasters. I wonder if there is a way to request hosting companies for daily off site backup and what would the costs of the same be? I need to check up on this.

    All in all, thanks for your efforts to get Trak.in back on track. I need my daily fix of blogs from you guys otherwise my fingers are fidgety with nothing to comment on :-)

  4. Abhilash Kushwaha says

    I am sorry to hear your hard-disk failed. Best of luck in restoring the data. Its in precisely these times you wish you had off-site backup. Its good that you had Amazon S3 for data up to 3/8.

Leave A Reply

Your email address will not be published.