Events:
<p>
<dl>
<a name="1702911206.0.19"></a>
<dt> 2023-12-18 at 15:52

 [dardel]
<dd> Start of jobs resumed since half an hour. Accesses to files in /cfs/klemming/ expected to operate ordinary again. Please report anomalies.



<!-- gerttest -->

<a name="1702734933.0.19"></a>
<dt> 2023-12-16 at 14:49

 [dardel]
<dd> Since this morning, 2023-12-16, a fraction of accesses to select files residing on certain storage elements in /cfs/klemming/ takes very long to complete. It is not yet known whether this is related to the file-system itself, or i.e. the network and/or clients mounting the file-system.



<P> New job starts paused until further notice. (still the case / 2023-12-18 at 10:00)



<!-- gerttest -->

<a name="1702308857.1.19"></a>
<dt> 2023-12-11 at 16:33

 [dardel]
<dd> System software maintenance completed, job starts resumed since half an hour. Please report unexpected behaviour.



<!-- gerttest -->

<a name="1702303200.1.19"></a>
<dt> 2023-12-06 at 16:17

 [dardel]
<dd> System software maintenance planned to start this coming Monday 2023-12-11 at 08:00. The work is estimated to be finished within one to two days, i.e., system on-line again 2023-12-12.



<!-- gerttest -->

<a name="1697136436.0.19"></a>
<dt> 2023-10-12 at 20:40

 [dardel]
<dd> The unexpected HSN (high speed network) issues overcome, jobs running since roughly an hour. Todays HSN issue and the planned system software maintenance earlier this week did not depend on each other.



<!-- gerttest -->

<a name="1697089409.0.19"></a>
<dt> 2023-10-12 at 07:32

 [dardel]
<dd> All new job starts blocked while investigating if a changed HSN (high speed network) setting may cause HSN performance loss and/or HSN deadlock.



<!-- gerttest -->

<a name="1697036901.1.19"></a>
<dt> 2023-10-11 at 17:07

 [dardel]
<dd> System software maintenance completed, job starts resumed since half an hour. Please report unexpected behaviour.



<!-- gerttest -->

<a name="1697032800.1.19"></a>
<dt> 2023-10-03 at 13:42

 [dardel]
<dd> System software maintenance planned to start Tuesday 2023-10-10 at 09:00. The work is estimated to be finished within one to two days, i.e., system on-line again 2023-10-11.



<!-- gerttest -->

<a name="1695044479.1.19"></a>
<dt> 2023-09-18 at 15:35

 [dardel]
<dd> System software maintenance completed, job starts resumed.



<!-- gerttest -->

<a name="1695043800.1.19"></a>
<dt> 2023-09-11 at 16:17

 [dardel]
<dd> System will be unavailable for system software maintenance starting Monday 2023-09-18 at 09:00. The work is estimated to be finished within two days, i.e., system on-line again 2023-09-19.



<!-- gerttest -->

<a name="1694187386.0.19"></a>
<dt> 2023-09-08 at 17:29

 [dardel]
<dd> a so called 'container' and 'pod' containing slurmctld (scheduler master daemon) was unexpectedly restarting a couple of hours ago. During outage/restart new job submits fail. This has now been remedied, and operation is supposed to be back to normal.



<!-- gerttest -->

<a name="1693468078.1.19"></a>
<dt> 2023-08-31 at 09:45

 [dardel]
<dd> the primary login node is being restarted as it behaved poorly after system software maintenance. 

<!-- gerttest -->

<a name="1693417964.1.19"></a>
<dt> 2023-08-30 at 19:52

 [dardel]
<dd> System software maintenance completed, job starts resumed since half an hour.



<!-- gerttest -->

<a name="1693411200.1.19"></a>
<dt> 2023-08-22 at 13:33

 [dardel]
<dd> System will be unavailable for system software maintenance starting Tuesday 2023-08-29 at 09:00. The work is estimated to take two days, i.e., system on-line again 2023-08-31.



<!-- gerttest -->

<a name="1689063540.1.14"></a>
<dt> 2023-07-11 at 10:17

 [klemming]
<dd> A cable replacement will take place for a klemming disk path. During preparation/actual replacement new job starts will be blocked. Accesses to /cfs/klemming/ might experience temporary freezes.



<P>The work is anticipated to take one to two hours.



<!-- lama -->

<a name="1688931344.0.19"></a>
<dt> 2023-07-09 at 21:33

 [dardel]
<dd> the primary dardel login node is experiencing problems (some processes seem to get stuck on accessing lustre file-system) and is being rebooted shortly.



<!-- lama -->

<a name="1688463073.1.19"></a>
<dt> 2023-07-04 at 11:28

 [dardel]
<dd> An enforced failover of a flapping klemming disk path is about to be made. The failover is expected to last in the order of minutes. No new jobs allowed to start during failover. File-system accesses will freeze during failover.



<P>After a completed and ok failover, jobs will be allowed to start running again.



<!-- lama -->

<a name="1688159299.1.19"></a>
<dt> 2023-06-29 at 23:00

 [dardel]
<dd> System work completed. User jobs running since a while. Please report unexpected behaviour of batch jobs.



<!-- lama -->

<a name="1688135394.1.19"></a>
<dt> 2023-06-22 at 16:29

 [dardel]
<dd> The planned downtime after midsummer is changed to affect entire Dardel and to start 9.00, Wednesday, June 28, until June 29, with June 30 as a backup.

<!-- gert -->

<a name="1688112909.1.19"></a>
<dt> 2023-06-16 at 10:13

 [dardel]
<dd> The Lustre file systems of Dardel will have a required software update after midsummer. Klemming (used by academic and some industry users) will be down from 9.00, June 27 until June 28. The Scania disk will be down from 9.00 June 29 until June 30. We believe Dardel should only be unavailable when the filesystem you use for the home directory and data is down.



<!-- gert -->

<a name="1685052733.0.19"></a>
<dt> 2023-05-26 at 00:08

 [dardel]
<dd> The PDC login portal (https://loginportal.pdc.kth.se/) is now again online.

<!-- noname -->

<a name="1685044409.0.19"></a>
<dt> 2023-05-25 at 21:38

 [dardel]
<dd> There are issues on behalf of making use of the PDC login portal. Consider it off-line for the time being.



<!-- lama -->

<a name="1684763876.0.19"></a>
<dt> 2023-05-22 at 15:53

 [dardel]
<dd> all nodes in the GPU partition has gotten a BIOS flash update, addressing a potential Escalation of Privilege situation. Jobs allowed to run again.



<!-- lama -->

<a name="1684512660.0.19"></a>
<dt> 2023-05-19 at 18:10

 [dardel]
<dd> the internal CEPH storage in use by most internal management services now restarted. Jobs starting since half an hour. Several jobs still running since prior the outage.



<P>Some jobs likely were experiencing issues during outage.



<P>(GPUs still off, security audit pushed forward in time.)



<!-- lama -->

<a name="1684489252.0.19"></a>
<dt> 2023-05-19 at 11:36

 [dardel]
<dd> VM hosting slurm master daemon, and other services on other VMs needed to manage the system still have issues. Fault search is ongoing.



<!-- lama -->

<a name="1684477732.0.19"></a>
<dt> 2023-05-19 at 08:24

 [dardel]
<dd> the VM hosting the slurm master daemon seem stopped since 05:12 this morning. No job state changes possible since then. Investigation / restart in progress.



<!-- lama -->

<a name="1684446237.0.19"></a>
<dt> 2023-05-18 at 23:38

 [dardel]
<dd> no new job starts allowed on GPU partition while investigating a potential Escalation of Privilege situation on GPUs.



<!-- lama -->

<a name="1683952162.0.19"></a>
<dt> 2023-05-13 at 06:27

 [dardel]
<dd> the primary login has been redirected to another access node since several hours. Please report eventual anomalies.



<!-- lama -->

<a name="1683925481.0.19"></a>
<dt> 2023-05-12 at 23:02

 [dardel]
<dd> the primary login is still having issues. Work is on-going. Jobs already running or waiting in queue so far un-affected.



<!-- lama -->

<a name="1683918616.0.19"></a>
<dt> 2023-05-12 at 21:06

 [dardel]
<dd> the primary login node is unresponsive and is being rebooted.





<!-- lama -->

<a name="1683556632.1.19"></a>
<dt> 2023-05-08 at 16:33

 [dardel]
<dd> Access re-enabled, system running jobs since half an hour. Please report anomalies.



<!-- lama -->

<a name="1683489755.1.19"></a>
<dt> 2023-05-07 at 21:57

 [dardel]
<dd> As several users have experienced, the updates are not yet fully completed and verified.



Access remain disabled. Until further notice.



<!-- lama -->

<a name="1683482400.1.19"></a>
<dt> 2023-04-24 at 09:00

 [dardel]
<dd> Please find info at <a href="https://www.pdc.kth.se/about/pdc-news/downtime-for-dardel-upgrade-from-2-5-may-2023-1.1249802"> Downtime for Dardel upgrade from 2-5 May 2023 </a> on forthcoming system availability.



<!-- lama -->

<a name="1682544101.1.19"></a>
<dt> 2023-04-26 at 23:20

 [dardel]
<dd> Job queues now restarted.



<!-- lama -->

<a name="1682544033.1.19"></a>
<dt> 2023-04-26 at 19:30

 [dardel]
<dd> During the benchmark testing earlier this day we had multiple failures of the SLURM controller for unknown causes.  We hope that we can restart job queues later today.  We have also placed a number of potentially problematic compute-nodes into maintenance for further investigation.



<!-- noname -->

<a name="1681723049.0.19"></a>
<dt> 2023-04-17 at 11:13

 [dardel]
<dd> Connectivity between compute nodes and external entities operational again. Exact cause not pin-pointed, i.e., can happen again.

<!-- lama -->

<a name="1681719065.0.19"></a>
<dt> 2023-04-17 at 10:10

 [dardel]
<dd> Once again connectivity between compute nodes and external entities, such as i.e. license servers, is not reliable. Investigation in progress.

<!-- lama -->

<a name="1681463805.0.19"></a>
<dt> 2023-04-14 at 11:15

 [dardel]
<dd> Connectivity between compute nodes and external entities operational again. New job starts enabled.



<!-- lama -->

<a name="1681458590.0.19"></a>
<dt> 2023-04-14 at 09:48

 [dardel]
<dd> Connectivity between compute nodes and external entities, such as i.e. license servers, is currently not reliable. Investigation in progress.



<!-- lama -->

<a name="1681295001.0.19"></a>
<dt> 2023-04-12 at 12:21

 [dardel]
<dd> The primary login rebooted, with updated HSN firmware in use, logins enabled.



<!-- lama -->

<a name="1681290407.0.19"></a>
<dt> 2023-04-12 at 11:01

 [dardel]
<dd> Another flap of the HSN (high speed network) on the primary login node has occurred, and it's being rebooted. This time with updated firmware to potentially reduce the risk for it to happen again.



<!-- lama -->

<a name="1681117467.0.19"></a>
<dt> 2023-04-10 at 11:03

 [dardel]
<dd> A flapping HSN switch port has been reset, the login node rebooted, logins enabled again.



<!-- lama -->

<a name="1681074457.0.19"></a>
<dt> 2023-04-09 at 22:59

 [dardel]
<dd> Unfortunately the general login node still is unable to go properly on-line.



User jobs already submitted/running on any compute node piece of the system progress in order without any reported obstacle.



Investigation is continuing.





<!-- lama -->

<a name="1681063873.0.19"></a>
<dt> 2023-04-09 at 20:09

 [dardel]
<dd> the dardel general login node is very unresponsive since earlier today. Investigation on-going.



<!-- lama -->

<a name="1680552588.1.19"></a>
<dt> 2023-04-03 at 22:06

 [dardel]
<dd> login to the system re-enabled.

<P> Information on how to use the updated system sent to the dardel-users mail-list,  and can also be found at <a href="https://www.pdc.kth.se/about/pdc-news/dardel-is-updated-and-the-gpu-partition-is-open-1.1245263"> Dardel is updated and the GPU partition is open </a>



<!-- gerttest -->

<a name="1680508860.1.19"></a>
<dt> 2023-03-31 at 17:00

 [dardel]
<dd> The upgrade work is ongoing, albeit somewhat slower than anticipated. Next update will be sent Monday (April 3) the latest.



<P>(Please find <a href="https://www.pdc.kth.se/about/pdc-news/downtime-to-update-dardel-starting-on-27-march-2023-1.1238777"> Downtime to update Dardel starting on 27 March 2023 </a>  aimed at giving a description.)



<!-- lama -->

<a name="1680508800.1.19"></a>
<dt> 2023-03-20 at 14:53

 [dardel]
<dd> Reminder: Upgrade set to start on 27 March 2023 at 8.00. The system may be un-accessible/down for the entire week. A block is in place to prevent jobs to run further than that moment in time.



<P> (Please find <a href="https://www.pdc.kth.se/about/pdc-news/downtime-to-update-dardel-starting-on-27-march-2023-1.1238777"> Downtime to update Dardel starting on 27 March 2023 </a> aimed at giving a description.)



<!-- lama -->

</dl>
All flash news for 
<a href="/cgi-bin/flash/flash.py?year=2024">2024</a>, 
<a href="/cgi-bin/flash/flash.py?year=2023">2023</a>, 
<a href="/cgi-bin/flash/flash.py?year=2022">2022</a>, 
<a href="/cgi-bin/flash/flash.py?year=2021">2021</a>, 
<a href="/cgi-bin/flash/flash.py?year=2020">2020</a>, 
<a href="/cgi-bin/flash/flash.py?year=2019">2019</a>, 
<a href="/cgi-bin/flash/flash.py?year=2018">2018</a>, 
<a href="/cgi-bin/flash/flash.py?year=2017">2017</a>, 
<a href="/cgi-bin/flash/flash.py?year=2016">2016</a>, 
<a href="/cgi-bin/flash/flash.py?year=2015">2015</a>, 
<a href="/cgi-bin/flash/flash.py?year=2014">2014</a>, 
<a href="/cgi-bin/flash/flash.py?year=2013">2013</a>, 
<a href="/cgi-bin/flash/flash.py?year=2012">2012</a>, 
<a href="/cgi-bin/flash/flash.py?year=2011">2011</a>, 
<a href="/cgi-bin/flash/flash.py?year=2010">2010</a>, 
<a href="/cgi-bin/flash/flash.py?year=2009">2009</a>, 
<a href="/cgi-bin/flash/flash.py?year=2008">2008</a>, 
<a href="/cgi-bin/flash/flash.py?year=2007">2007</a>, 
<a href="/cgi-bin/flash/flash.py?year=2006">2006</a>, 
<a href="/cgi-bin/flash/flash.py?year=2005">2005</a>, 
<a href="/cgi-bin/flash/flash.py?year=2004">2004</a>, 
<a href="/cgi-bin/flash/flash.py?year=2003">2003</a>, 
<a href="/cgi-bin/flash/flash.py?year=2002">2002</a>, 
<a href="/cgi-bin/flash/flash.py?year=2001">2001</a>, 
<a href="/cgi-bin/flash/flash.py?year=2000">2000</a>, 
<a href="/cgi-bin/flash/flash.py?year=1999">1999</a>, 
<a href="/cgi-bin/flash/flash.py?year=1998">1998</a>, 
<a href="/cgi-bin/flash/flash.py?year=1997">1997</a>, 
<a href="/cgi-bin/flash/flash.py?year=1996">1996</a>, 
<a href="/cgi-bin/flash/flash.py?year=1995">1995</a>
<br><br><a href="https://www.pdc.kth.se/">Back to PDC</a>
<br><a href="https://www.pdc.kth.se/cgi-bin/flash/rss.py">Subscribe to rss</a>