Hive is down due to another partial power failure in the Data Center
Update - Quobyte, as well as most nodes, were brought back into service this morning. Admins are monitoring the situation.
Jul 06, 2026 - 17:21 PDT
Monitoring - We have brought up the Quobyte file system and are working on maintenance tasks regarding the outage.
Jul 06, 2026 - 09:01 PDT
Update - Facilities were able to restore power to the data center, but we are not able to bring up the Quobyte file system. Data center operators are not available to assist until Monday morning.
Jul 05, 2026 - 20:12 PDT
Update - Admins are on-site waiting for campus facilities.
Jul 05, 2026 - 15:32 PDT
Identified - Power has partially failed in the Data Center, which has brought Hive down again. Staff are in contact with the Data Center Operators and Facilities is being scheduled to investigate the power issue.
Jul 05, 2026 - 10:16 PDT
Uptime over the past 90 days. View historical uptime.
Hive Login Node Operational
90 days ago
99.35 % uptime
Today
Compute Nodes Operational
90 days ago
98.39 % uptime
Today
GPU Nodes Operational
90 days ago
99.03 % uptime
Today
Hive Network Operational
90 days ago
99.3 % uptime
Today
Storage Operational
90 days ago
98.88 % uptime
Today
Quobyte Parallel File System Operational
90 days ago
98.69 % uptime
Today
Hive Home Directories Operational
90 days ago
99.28 % uptime
Today
Legacy Storage Operational
90 days ago
98.68 % uptime
Today
Module System and Software Operational
90 days ago
99.35 % uptime
Today
Hippo User Portal Operational
90 days ago
99.54 % uptime
Today
OnDemand Operational
90 days ago
98.56 % uptime
Today
Farm Login Node Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Jul 21, 2026

No incidents reported today.

Jul 20, 2026

No incidents reported.

Jul 19, 2026

No incidents reported.

Jul 18, 2026

No incidents reported.

Jul 17, 2026

No incidents reported.

Jul 16, 2026

No incidents reported.

Jul 15, 2026
waltz NFS server crashing
Resolved - The crashing Waltz is NAS head has been replaced by previously retired hardware. System administrators have mounted the storage read-only so that groups can recover their data.

We urge all labs and associated staff to migrate data from Waltz before any permanent data loss occurs.
Jul 15, 18:05 PDT
Update - Based on the pattern of errors, admins believe a full failure of the NAS head is imminent. Please copy your data to a safe storage location.
Jul 13, 17:22 PDT
Identified - Waltz will be sporadically unavailable due to hardware failure. System administrators are currently looking into the issue and investigating options.
Jul 13, 16:11 PDT
Investigating - The waltz NFS server has crashed multiple times today. Admins are investigating.
Jul 13, 13:03 PDT
Emergency power re-balance within the Data Center
Completed - Data Center operators were able to rebalance the power for the Hive compute nodes without causing any nodes to crash.
Jul 15, 16:34 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 15, 13:16 PDT
Update - We will be undergoing scheduled maintenance during this time.
Jul 15, 13:14 PDT
Scheduled - Data Center staff need to perform emergency power rebalancing for the Hive compute nodes. They are going to attempt to perform this live, but there is a possibility that nodes will crash.
Jul 15, 13:13 PDT
Jul 14, 2026

No incidents reported.

Jul 13, 2026

Unresolved incident: Hive is down due to another partial power failure in the Data Center.

Jul 12, 2026

No incidents reported.

Jul 11, 2026

No incidents reported.

Jul 10, 2026

No incidents reported.

Jul 9, 2026

No incidents reported.

Jul 8, 2026

No incidents reported.

Jul 7, 2026

No incidents reported.

← Incident History Powered by Atlassian Statuspage