For the last few days, /pnfs is having performance issues, while some storage nodes can't serve files.
We have not identified the cause yet.
Mainly cd/ls becomes slow, and only a restart of the service makes it work fast again.
Also, a couple of storage nodes have a hard time serving files (hence why some of your files have issues), this usually solves itself after a while.
Cheers,
The T2B IT Team
Past Incidents
Tuesday 10th October 2023
No incidents reported
Monday 9th October 2023
Mass Storage (/pnfs)/pnfs down
Due to the many problems we had with /pnfs, we had to reboot the machine. Unfortunately, this did not go seamlessly. We are trying to recover the machine now.
Regards,
The T2B team
/pnfs is stable again since ~15 minutes.
Sunday 8th October 2023
No incidents reported
Saturday 7th October 2023
No incidents reported
Friday 6th October 2023
No incidents reported
Thursday 5th October 2023
Mass Storage (/pnfs)problems on /pnfs
Since yesterday afternoon, we have been bombarded with xrootd requests from the CMS federation, causing /pnfs to be unstable. We are looking into possible ways to mitigate the problem.
The problem was fixed by increasing the memory given to the correct service.
/pnfs should be fully operational again.