We apologise for the down-time experienced over the last 2 days. Unfortunately this was due to sustained malicious cyber attacks directed at the cloud computing network at the hosting company. There was nothing we could do about it and our tech team could not access the website. This is not supposed to happen with could computing and we have a 99.99% up-time guarantee with the hosting company. Unfortunately, it appears this has been a well co-ordinated attack affecting thousands of websites.
Sequence of developments below if you are interested. (US TIME)
08:41
We are currently looking into delays we are experiencing with the storage system for our Redundant and Cloud products. Rest assured the data is all safe, just the links between the storage and the Cloud servers are being saturated for a reason we are still investigating.
11:05
At this time all hosts on the redundant nodes are being gracefully shut down to diagnose issues with the storage system. We will continue to provide updated on this issue.
11:26
Due to the number of virtual machines we need to power down, unfortunately, this process may take up to two hours to complete. These virtual machines must be powered down before we can diagnose the problems with the storage unit.
12:08
All virtual machines are powered down. We are diagnosing the problems with the storage unit at this time.
12:29
Over the next 30 minutes we are going to begin to gradually bring the virtual machines on-line while monitoring performance.
02:21
Our engineers are currently still working on the NFS settings and as soon as this is completed, the VM's will start being booted. We'll continue to provide updates.
04:50
We designed our cloud system to have redundant hardware systems, and unfortunately this is a software bug that is affecting both storage arrays so both of them were having severe slow write/read speeds. We are working on a fix now and hope to have it back shortly. We will update you as this progresses.
Today
11:52
A few redundant servers have faced an issue again. Our deputy C.T.O. is checking the servers at the moment. We will update this thread as soon as possible.
12:22
The servers are currently coming back up. We sincerely apologize for the downtime. If you have any questions regarding the downtime, please contact support and we'll be happy to assist.
04:29
We have identified the cause of these issues and our Deputy CTO is working actively to fix it completely. Some servers may face service disruption due to this. We will get back to you soon.
04:55
The redundant servers which has been affected by this issue are being restarted now one by one. We will continue monitoring the servers for any further issues. We are very sorry for the inconvenience.
|
