Certified - certified.one partially unusable due to Railway TLS – Incident details

certified.one partially unusable due to Railway TLS

Resolved
Major outage
Started about 2 months agoLasted about 7 hours

Affected

Certified Infra (production)

Major outage from 8:48 AM to 10:24 AM, Operational from 10:24 AM to 4:03 PM

certified.one (ePDS)

Major outage from 8:48 AM to 10:24 AM, Operational from 10:24 AM to 4:03 PM

Updates
  • Resolved
    UTC
    Resolved

    The new configuration appears stable. However we are working on adding further resilience to the architecture.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    UTC
    Identified

    Railway TLS termination for certified.one has failed in the same way as yesterday. This was already reported upstream to Railway yesterday and we are attempting to work around the problem. This will affect any systems which do not have a working TLS certificate cached.