All systems are operational.

Last updated 1 month ago

    • Mass Storage (/pnfs)

      Grid-enabled distributed mass storage filesystem.

      Operational

    • Batch System

      Job submission infrastructure for local users as well as grid experiments.

      Operational

    • User Interfaces - mX machines

      Interfaces for the users to access the cluster’s compute and storage resources.

      Operational

    • Network

      Internal cluster network and its Internet access.

      Operational

Past incidents

No incidents reported.

No incidents reported.

Mass storage /pnfs issue

7 years ago 2019-06-28 12:47 UTC
Fixed

Fixed

7 years ago 2019-06-28 12:47 UTC

We are facing an 'Xrootd storm". This is due to a lot of remote transfers initiated probably by "ignore locality jobs". We had to restart the mass storage services. Some of your jobs might have crashed.

No incidents reported.

Batch system did not accept jobs anymore

7 years ago 2019-06-26 10:00 UTC
Fixed

Fixed

7 years ago 2019-06-26 10:00 UTC

It seems the batch system did not accept jobs anymore. The service was rebooted and is now working properly again.

/cvmfs is not working properly on m9

7 years ago 2019-06-26 08:59 UTC
Fixed

Fixed

7 years ago 2019-06-27 22:35 UTC

m9 needed a complete re-install to fix the problem. /cvmfs should be up-to-date and same as other machines now.

Identified

7 years ago 2019-06-27 11:25 UTC

To fix the issue with /cvmfs on m9, we need to empty the cache and reboot the machine. This will be done at 11PM today, Thursday 27/06, so please save your work and exit m9.

Reported

7 years ago 2019-06-26 08:59 UTC

/cvmfs seems to be outdated on m9, compared to other machines.

Incident with cooling at Computing Center

7 years ago 2019-06-25 15:15 UTC
Fixed

Fixed

7 years ago 2019-07-02 08:23 UTC

Last Friday evening, intervention on the electrical board for the cooling system fixed the issue. As the temperature was stable during the weekend, it was convened with the datacenter operators that we could bring back all our machines.

The batch system is now again at full capacity.

Identified

7 years ago 2019-06-27 22:40 UTC

The issue with the cooling has been diagnosed today as an electrical issue. As tentative to fix it will only be tried tomorrow, and in view of the heat-wave of Saturday, we will keep the batch system at half-capacity as it is right now for the weekend.

Situation will be re-evaluated Monday morning.

Watching

7 years ago 2019-06-25 17:56 UTC

Because of the temperature, the /pnfs headnode went down. It is now back so /pnfs should be accessible.

Cooling was only partially restored, so only a minimal number of job slots are available. Unless situation starts to worsen again, we will keep mX-machines and storage up. Complete availability of the batch system tomorrow will depend on whether the cooling machine can be fixed.

Reported

7 years ago 2019-06-25 15:15 UTC

There is an unknown issue with the cooling in the datacenter room. Therefore a big part of the worker nodes were forcefully stopped. Some of your jobs will have therefore failed.

If the cooling comes back, nodes will be restarted tomorrow morning. If it continues failing, the temperature will force us to stop the batch system and eventually the storage. We will update this incident regularly.

No incidents reported.