For the last few days, /pnfs is having performance issues, while some storage nodes can't serve files.
We have not identified the cause yet.
Mainly cd/ls becomes slow, and only a restart of the service makes it work fast again.
Also, a couple of storage nodes have a hard time serving files (hence why some of your files have issues), this usually solves itself after a while.
Cheers,
The T2B IT Team
Past Incidents
Saturday 7th October 2023
No incidents reported
Friday 6th October 2023
No incidents reported
Thursday 5th October 2023
Mass Storage (/pnfs)problems on /pnfs
Since yesterday afternoon, we have been bombarded with xrootd requests from the CMS federation, causing /pnfs to be unstable. We are looking into possible ways to mitigate the problem.
The problem was fixed by increasing the memory given to the correct service.
/pnfs should be fully operational again.