All Systems Operational

Uptime over the past 90 days. View historical uptime.
Cluster: Hive Operational
90 days ago
99.63 % uptime
Today
Cluster: Farm Operational
90 days ago
100.0 % uptime
Today
Cluster: Franklin Operational
90 days ago
100.0 % uptime
Today
Login Node Operational
90 days ago
99.35 % uptime
Today
Compute Nodes Operational
90 days ago
98.79 % uptime
Today
GPU Nodes Operational
90 days ago
98.79 % uptime
Today
Network Operational
90 days ago
99.35 % uptime
Today
Storage Operational
90 days ago
98.87 % uptime
Today
Quobyte Parallel File System Operational
90 days ago
98.69 % uptime
Today
Home Directories Operational
90 days ago
99.28 % uptime
Today
Legacy Storage Operational
90 days ago
98.63 % uptime
Today
Module System and Software Operational
90 days ago
99.35 % uptime
Today
Hippo User Portal Operational
90 days ago
99.54 % uptime
Today
OnDemand Operational
90 days ago
99.43 % uptime
Today
Scheduler (Slurm) Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Sep 4, 2026

No incidents reported today.

Sep 3, 2026

No incidents reported.

Sep 2, 2026

Emergency Slurm update September 2nd
Completed - Slurm has been upgraded, and all clusters are back in service.
Sep 2, 17:24 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 2, 08:00 PDT
Update - We will be undergoing scheduled maintenance during this time.
Aug 24, 15:38 PDT
Scheduled - SchedMD, the company that develops the Slurm job scheduler, has privately announced several security vulnerabilities. These vulnerabilities have not been disclosed at this time pending remediation, but have been reported to allow escalated privileges within the clusters.

SchedMD is scheduled to publish a fix on September 2, 2026. As per our commitment to follow UCOP security guidelines, we will be upgrading Slurm once the fix has been released. As a result, we will be implementing a maintenance reservation that will prevent jobs from starting if they overlap the September 2 restart. Currently running jobs that are scheduled to run past September 2 may be subject to cancellation depending on our assessed severity of the vulnerabilities once they are fully disclosed.
Aug 24, 15:32 PDT

Sep 1, 2026

No incidents reported.

Aug 31, 2026

apo file server crashed
Resolved - A bad PSU was preventing one of apo's JBODs from working. Removing that PSU has allowed apo to resume service.
Aug 31, 12:34 PDT
Update - Admins have identified dual failed power supplies in one of the JBODs and are working toward a solution.
Aug 31, 10:50 PDT
Identified - The apo file server crashed over the weekend. Admins are on-site investigating.
Aug 31, 09:04 PDT

Aug 30, 2026

No incidents reported.

Aug 29, 2026

No incidents reported.

Aug 28, 2026

No incidents reported.

Aug 27, 2026

Hive: Partial power outage in the Data Center
Resolved - All nodes and systems have been brought back into service.
Aug 27, 14:27 PDT
Update - Power has been restored, but nodes will remain down until we can reaccess power tomorrow.
Aug 26, 19:13 PDT
Identified - One of the power breakers that powers parts of both Hive and Farm tripped. Data Center operators are calling facilities.
Aug 26, 17:23 PDT

Aug 26, 2026

Aug 25, 2026

No incidents reported.

Aug 24, 2026

No incidents reported.

Aug 23, 2026

No incidents reported.

Aug 22, 2026

No incidents reported.

Aug 21, 2026

Hive is down due to another partial power failure in the Data Center
Resolved - We had the Data Center staff rebalance power across the 3 feeds they provide us, which resolved this issue.
Aug 21, 15:40 PDT
Update - Quobyte, as well as most nodes, were brought back into service this morning. Admins are monitoring the situation.
Jul 6, 17:21 PDT
Monitoring - We have brought up the Quobyte file system and are working on maintenance tasks regarding the outage.
Jul 6, 09:01 PDT
Update - Facilities were able to restore power to the data center, but we are not able to bring up the Quobyte file system. Data center operators are not available to assist until Monday morning.
Jul 5, 20:12 PDT
Update - Admins are on-site waiting for campus facilities.
Jul 5, 15:32 PDT
Identified - Power has partially failed in the Data Center, which has brought Hive down again. Staff are in contact with the Data Center Operators and Facilities is being scheduled to investigate the power issue.
Jul 5, 10:16 PDT
← Incident History Powered by Atlassian Statuspage