For the last few days, /pnfs is having performance issues, while some storage nodes can't serve files.
We have not identified the cause yet.
Mainly cd/ls becomes slow, and only a restart of the service makes it work fast again.
Also, a couple of storage nodes have a hard time serving files (hence why some of your files have issues), this usually solves itself after a while.
Cheers,
The T2B IT Team
Past Incidents
Saturday 30th March 2024
No incidents reported
Friday 29th March 2024
No incidents reported
Thursday 28th March 2024
No incidents reported
Wednesday 27th March 2024
No incidents reported
Tuesday 26th March 2024
User Interfaces - mX machinesIssues with /user
Hello,
Unfortunately we have encountered the same issue with /user as last Friday.
Connection to some mX machines can be slow, and /user storage is either slow or blocked.
We are still trying to find the source and get it fixed definitively.
Also, /pnfs is not affected.
Sorry for all the issues this causes.
Cheers,
The T2B IT Team
Hello,
We have finally found the source of the issues with /user.
It was due to a wrong workflow of a user, so after removing jobs all is fixed.
On this note, please make sure to NEVER have a single input data file that is read by all your jobs on /user.
Our /user storage system cannot cope with thousands of jobs trying to read a single file.
The correct workflow is to put the file(s) on /pnfs, then inform us so that we can make duplicates, which protects the storage system from harm.