Post-Mortem: Temporary Failure in External SSH Access
April 28, 2025

The Issue

On April 21st we were informed by users that SSH access to some CSE-IT managed systems was no longer available from sources external to the university. Access was available from within the university and while using the university’s VPN service. The issue was resolved on April 23rd.

Root Cause

On April 21st, CSE-IT rolled out a change to all systems running FastX (this includes the VOLE cluster, all CSELABS systems, and some personal workstations). This change modified the way SSL certificates are deployed as the method used until then was deprecated. 

While this change did not directly affect SSH, it did close a loophole where explicit permissions for external SSH access were not previously required. This change incorporated more modern policies which do require explicit declarations. Since these were not in place, external SSH access was inadvertently disabled. 

Resolution

Once CSE-IT became aware of the issue, an investigation was launched to determine the cause. Once determined, the issue was repaired. An immediate change repairing the issue was rolled out on April 23rd.

Share