Hive is down due to another partial power failure in the Data Center

Incident Report for HPC at UCD

Update

Quobyte, as well as most nodes, were brought back into service this morning. Admins are monitoring the situation.
Posted Jul 06, 2026 - 17:21 PDT

Monitoring

We have brought up the Quobyte file system and are working on maintenance tasks regarding the outage.
Posted Jul 06, 2026 - 09:01 PDT

Update

Facilities were able to restore power to the data center, but we are not able to bring up the Quobyte file system. Data center operators are not available to assist until Monday morning.
Posted Jul 05, 2026 - 20:12 PDT

Update

Admins are on-site waiting for campus facilities.
Posted Jul 05, 2026 - 15:32 PDT

Identified

Power has partially failed in the Data Center, which has brought Hive down again. Staff are in contact with the Data Center Operators and Facilities is being scheduled to investigate the power issue.
Posted Jul 05, 2026 - 10:16 PDT
This incident affects: Hive Login Node, Compute Nodes, GPU Nodes, Hive Network, Module System and Software, Hippo User Portal, OnDemand and Storage (Quobyte Parallel File System, Hive Home Directories, Legacy Storage).