Login issues

A central configuration error led to issues logging in to Linux machines between 18:03 and 18:31 today.

Due to a misconfigured LDAP object, the central authentication service at UiO failed to allow anyone to log into Linux machines at UiO today. The error passed testing due to a very specific set of assumptions that were proven to be mistaken. 

Overall, this caused authentication to our linux hosts to fail for approximately 28 minutes. As soon as the issue was detected it was escalated centrally, and it was resolved on their end by 18:16 and became active shortly afterwards.

Unfortunately, the LDAP clients (that is, our login services) are set up to cache data for up to 90 minutes, so their cache had to be manually purged. Drift@math did that locally for our login node (login.math.uio.no) and our compute/GPU nodes immediately afterwards. At the same time, a large-scale purge of all faulty LDAP cache on all hosts was pushed into production. This job finished around 18:31, returning services to a normal state.

Our apologies for any inconvenience you may have experienced.

Published Apr. 2, 2019 6:56 PM - Last modified Apr. 2, 2019 6:57 PM