HFA Authentication Failure (SSPI/Kerberos) - (July 20, 2026)

Duration: Approx. 2 hours (10:35am - 11:55am MDT) July 20, 2026
Affected Services: HFA/Tenant Portal
Status: Resolved

Executive Summary

On July 20, 2026, ProLink Solutions identified that a subset of client APIs supporting the Tenant Portal were not functioning correctly. As investigation began, additional APIs began to fail, indicating that the issue was cascading. At 10:35 AM MDT, the issue cascaded further to impact the HFA application itself, at which point an outage was posted to the status page. Clients in the Mountain and Central time zones began encountering a Kerberos SSPI error ("Cannot generate SSPI context"). Extensive investigation ruled out SPN duplication, encryption type mismatches, DNS misconfiguration, clock skew, and domain trust issues before Kerberos client-side logging identified the root cause: stale, cached Kerberos credentials at the machine account level on specific servers. IT initiated a targeted remediation that forced the affected servers to re-acquire current Kerberos credentials, which immediately restored functionality across all impacted servers and applications.

Incident Timeline

Impact

All connections from Mountain and Central application servers to their backend database using Windows Integrated Authentication failed for the duration of the incident, regardless of which account attempted the connection. Because the failure was identity-independent, standard troubleshooting steps (testing with elevated/domain admin credentials) did not restore service, which extended the investigation. The issue was isolated to the Mountain and Central application servers; other servers and client environments not dependent on that specific connection were unaffected.

Root Cause

The outage was caused by a stale Kerberos credential handle held by the SQL Server (MSSQLSERVER) service process. Over time, as the cached Kerberos tickets were cleared, it forced a fresh ticket request from the domain's Key Distribution Center (KDC) the next time it connected to the database. Kerberos client-side logging on Application servers captured a KRB_AP_ERR_MODIFIED error indicating that SQL process could not decrypt the ticket the KDC had issued - direct evidence that the database server’s locally cached computer account credentials no longer matched what Active Directory had on record for that account.

Standard remediation for this class of issue - resetting the secure channel trust and the computer account password via Netlogon - corrected the underlying Active Directory account state for the database, but did not resolve the outage. This is because SQL Server's own process acquires and caches its Kerberos credential handle once, at service startup, and does not automatically detect or respond to a computer account password change made at the OS level while the service is already running. As a result, sqlservr.exe continued attempting to decrypt tickets using its original (now-invalid) cached credential material even after Active Directory itself had been corrected.

The precise upstream trigger for the initial Active Directory account state discrepancy in SQL was not conclusively identified during this incident. A leading candidate is transient replication lag between domain controllers within the AWS Managed Microsoft AD directory, though no direct evidence of replication lag was found.

Resolution

We forced the SQL process to fully re-initialize and re-acquire a Kerberos credential handle reflecting the corrected, current computer account state. Immediately following the restart, Windows Integrated Authentication connections from affected Application servers to SQL succeeded without further error. Functionality was confirmed restored across all impacted applications and client environments.