For the last few days, /pnfs is having performance issues, while some storage nodes can't serve files.
We have not identified the cause yet.
Mainly cd/ls becomes slow, and only a restart of the service makes it work fast again.
Also, a couple of storage nodes have a hard time serving files (hence why some of your files have issues), this usually solves itself after a while.
Cheers,
The T2B IT Team
Past Incidents
Sunday 28th July 2019
No incidents reported
Saturday 27th July 2019
No incidents reported
Friday 26th July 2019
No incidents reported
Thursday 25th July 2019
Complete cluster shutdown because of failed cooling
Due to the extreme outside temperature, the cooling in the datacenter failed. Therefore all hosted services were stopped, and the whole T2B cluster was shutdown.
The cluster was only restart on Monday, after the cooling hold steady the whole weekend.
Only 70% of the compute nodes were restarted in order to slowly ramp-up the stress on the cooling systems.