On saturday evening 5/14 the Philbrook Chilled water became hot enough that the Morse rooftop was automatically triggered to replace it. Unfortunately the Morse Rooftop Chiller didn't work and replaced the warm water with a large supply of hot water. When the air supply temperature hit 100F most hosts in the Lenharth Data Center had been turned off. Including the Premise and BioMade HPC clusters.

Most LDC systems came back online later Saturday night. But it took another 12 hours to determine that a part of the Premise old Lustre is doing a filesystem check. The status tools estimate it has another 24 hours to complete it syncing.

We will email a new a new status update around noon on Monday. Until then Premise will remain offline.