For the last few days, /pnfs is having performance issues, while some storage nodes can't serve files.
We have not identified the cause yet.
Mainly cd/ls becomes slow, and only a restart of the service makes it work fast again.
Also, a couple of storage nodes have a hard time serving files (hence why some of your files have issues), this usually solves itself after a while.
Cheers,
The T2B IT Team
Past Incidents
Sunday 20th October 2019
No incidents reported
Saturday 19th October 2019
No incidents reported
Friday 18th October 2019
No incidents reported
Thursday 17th October 2019
No incidents reported
Wednesday 16th October 2019
No incidents reported
Tuesday 15th October 2019
No incidents reported
Monday 14th October 2019
Batch SystemBatch System is down - no qsub possible
The batch system software is currently failing to restart. We are investigating.
No local submissions are possible. Grid jobs will continue running.
Batch service is back. Unfortunately all jobs were purged during the investigation, so you will have to resubmit your jobs.
The issue was due to a remnant job file defined as array that created a segmentation fault, probably because the batch software could not handle the number of created jobs by the array.
Once the file was removed, the batch system did not crash during start anymore.