Elevated API Errors
Incident Report for TaskRabbit
Resolved
The ElasticSearch cluster eventually recovered after we deleted a shard that was likely damaged when we filled disk space. We then ingested the dumped data. All systems are back to normal.
Posted Aug 12, 2018 - 19:36 PDT
Update
We are continuing to monitor cluster recovery. This will take several hours. Event data is being dumped so we can run those through later.
Posted Aug 12, 2018 - 14:24 PDT
Monitoring
We were unable to write metrics events to our ElasticSearch cluster. We have upped the disc space and are letting the cluster recover before turning metrics workers back on and reprocessing the events.
Posted Aug 12, 2018 - 12:58 PDT
Investigating
Our automated systems have detected a problem. Our engineers have been notified and we are looking into it.
Posted Aug 12, 2018 - 10:51 PDT