ShotGrid January 17, 2024 Service Disruption

Impacted Services

On January 17, 2024, ShotGrid underwent a service disruption from 16:40 to 20:20 UTC, affecting user login capabilities and displaying a System error page.

Root Cause

An invalid credential change in ShotGrid's daily release disrupted new session creation. Despite immediate alerting of the issue and prompt release rollback action, the specific issue wasn't covered by the rollback procedure and had to be manually fixed.

ShotGrid Actions

ShotGrid has completed a post-incident analysis of the event and identified actions to be taken. These include the following:

We have reviewed our rollback procedure to take into account this specific gap in release/rollback coverage.
We will review our automated and manual pre-release validation procedures to prevent this situation from occurring again.

Thank you for your patience and understanding.

Posted Jan 22, 2024 - 17:27 UTC

Resolved

This incident has been resolved.

If the issue persists, please go to the main page of your ShotGrid website and try to sign in again.

Posted Jan 17, 2024 - 19:28 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jan 17, 2024 - 18:59 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jan 17, 2024 - 18:05 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Jan 17, 2024 - 17:30 UTC

Investigating

We are observing a high number of failed requests to the ShotGrid service which may impact site availability for some clients. Email notifications may also be delayed. This issue is under investigation.

Posted Jan 17, 2024 - 17:18 UTC

This incident affected: Flow Production Tracking and Notification Service.