7 Dec 2012

Salesforce disruption: NA2 and next day, CS1

Posted by iwgcr

SalesForceOn December 4, Salesforce team observed a service disruption from 17:19 UTC to 18:01 UTC. During this time, customers were not able to access the NA2 instance. Again, it was a database fault,  by a corrupt index.

Salesforce team explain:

The service disruption resulted from a large number of requests attempting to access this index. When the requests attempted access, the requests were unable to be processed, and the service became overwhelmed. We suspended service to the database tier and restarted the application tier to restore service to NA2, with the exceptions noted below.
In addition to Sandbox copy and weekly export functionality, we suspended all inbound email requests and blocked WebToCase/WebToLeadrequests for a subset of customers, as a measure to mitigate the risk of performance degradation.

The following day, on December 5, an issue by a hardware failure affecting the CS1 instance that cause a service disruption. Customers on the CS1 instance will experience an inability to access the application from 18:53 UTC to 22:56 UTC

Date

Service

Duration

Critical Data Lost

2012-12-04 Salesforce 1 hour no
2012-12-05 Salesforce 4 hours no