Dear Simplificare Clients,
We wanted to be open and transparent with our customers to understand the timeline of this morning’s service outage.
On June 13 our provider identified that they would be performing maintenance on all servers in their Montreal data centre on June 22. This was shared on social media and added to our network status page.
The maintenance window was expected to last from 10am to 8pm June 22, with downtime expected between 3pm and 6pm, while our provider moved physical hardware from one rack to another. This maintenance window was later modified and extended to 10pm, with downtime now expected between 6pm and 10pm.
Servers pointy and kirk went offline during the maintenance window at 6:45pm as expected however at 10pm they did not come back online. We reached out to our hosting provider at about 10:15pm for a status update. We did not receive a response until 1:05am on June 23. Stating that the schedule maintenance was on-going but had no ETA for completion.
On June 23 at 7:11am pointy server came back online but there were signs of data loss (any changes or mail received on or after June 18th appeared to have been lost). We reached out for a new update from our hosting provider to determine what the restoration plan was for the missing data, and for kirk to come back online. We were informed that the system the servers were formerly hosted on had a data corruption issue and that they were restoring clients from their most recent backups. We were asked to be patient during the downtime as they continued to work on restoring services.
On June 23 at 11:47am kirk server came back online, and similar to pointy, had data loss back to the 18th of June.
At 12:53pm both servers pointy and kirk went offline for approximately 7 minutes while our provider restored the snapshot of the server just prior to it going offline at 6:45pm the evening prior.
At 2:20pm our provider confirmed the maintenance was complete and systems fully restored.
The Good News
Everything is back online and operating.
The Bad News
Services were offline much longer than they were supposed to be. Any data processed by the server (email, web modifications, etc.) between 7:11am and 12:53pm, on either pointy or kirk, will have been lost.
What is Simplificare going to do moving forward?
At the outset, when systems were initially restored, it looked like there was potentially 5 days of data missing, emails, web modifications, etc. This was a major concern. Simplificare already runs weekly backups of all of its accounts, however this event has proven that this is simply not good enough! We are going to start work immediately to implement daily incremental backups of all client hosting account data & email. We intend to provide this service at no additional cost to our customers.
So that’s that! If you have any questions or concerns, please do not hesitate to contact us by opening a support ticket.
Best regards,
Simplificare Hosting and Solutions
https://www.simplificare.net