For the last few days, /pnfs is having performance issues, while some storage nodes can't serve files.
We have not identified the cause yet.
Mainly cd/ls becomes slow, and only a restart of the service makes it work fast again.
Also, a couple of storage nodes have a hard time serving files (hence why some of your files have issues), this usually solves itself after a while.
Cheers,
The T2B IT Team
Past Incidents
Sunday 7th July 2024
No incidents reported
Saturday 6th July 2024
No incidents reported
Friday 5th July 2024
No incidents reported
Thursday 4th July 2024
No incidents reported
Wednesday 3rd July 2024
No incidents reported
Tuesday 2nd July 2024
No incidents reported
Monday 1st July 2024
NetworkDatacenter issue impacting network
Hello,
There seems to be an issue in the datacenter, as the temperature of all servers are off-the-chart.
In particular, the NAT which allows you to access public IPv4 addresses from the batch system keeps stopping to protect itself.
Investigation is under way.
Cheers,
The T2B IT Team
Hello,
In order to protect the equipment, we decided to stop preventively about 60% of the worker nodes in the batch system.
In the meantime, a technician was dispatched and fixed the cooling, we have confirmed that the overwhole temperature is back within sensible values.
The worker nodes will be rebooted tomorrow morning to provide the whole capacity to the batch system