All systems are operational.

Last updated 1 month ago

    • Mass Storage (/pnfs)

      Grid-enabled distributed mass storage filesystem.

      Operational

    • Batch System

      Job submission infrastructure for local users as well as grid experiments.

      Operational

    • User Interfaces - mX machines

      Interfaces for the users to access the cluster’s compute and storage resources.

      Operational

    • Network

      Internal cluster network and its Internet access.

      Operational

Past incidents

No incidents reported.

No incidents reported.

No incidents reported.

No incidents reported.

No incidents reported.

No incidents reported.

issues with slow pnfs affecting all machines

1 year ago 2025-01-24 14:28 UTC
Fixed

Investigating

1 year ago 2025-01-27 15:22 UTC

Unfortunately pnfs is still very unstable.

Restarting the services or even rebooting the whole machine does not help.

We're still investigating what could cause the instability.

Fixed

1 year ago 2025-01-24 15:45 UTC

We had to restart again all the pnfs system, and that fixed everything. /pnfs is accessible again from all mX machines without a need to reboot them. A lot of worker nodes went down because of the issue unfortunately, we have remotely restarted as many as possible.

Current cluster capacity: EL7 - free: 420 + run: 3499 [1368+2131] + drain: 0 = 3919 EL9 - free: 28 + run: 5220 [5026+194] + drain: 0 = 5248

Reported

1 year ago 2025-01-24 14:28 UTC

Hello,

Unfortunately standard access to /pnfs have been slow lately. To fix this, we have been forced to restart the nfs service for pnfs.

The consequence to that is that many of our machines have lost access to /pnfs entirely. We are busy fixing that, and might have to restart mX machines.

More info will be given asap.

Sorry for the troubles, The T2B Admin Team