From Mark.Maciolek at unh.edu  Thu Dec  6 10:27:11 2018
From: Mark.Maciolek at unh.edu (Maciolek, Mark)
Date: Thu, 6 Dec 2018 15:27:11 +0000
Subject: [Trillian-users] trillian issue
Message-ID: <a74983f4b7bc4999913abd73b395e07d@groot.ad.unh.edu>

Hi,

Jobs submitted on trillian are being queued and not running since yesterday morning.

qstat -s shows this reason

Not Running: Insufficient amount of resource arch

The only logs are from PBS mom_logs showing this:

20181205:12/05/2018 08:00:01;0080;pbs_mom;Node;alps_engine_query;ALPS ENGINE query failed with BASIL version 1.1.
20181205:12/05/2018 08:00:01;0002;pbs_mom;Node;alps_inventory;ALPS inventory request failed.
20181205:12/05/2018 08:10:01;0080;pbs_mom;Node;alps_engine_query;ALPS ENGINE query failed with BASIL version 1.1.
20181205:12/05/2018 08:10:01;0002;pbs_mom;Node;alps_inventory;ALPS inventory request failed.

The alps logs don't show any obvious issue. 

I restarted pbs on trillian, which had the effect of cancelling some jobs and restarting others.

Will keep an eye on the logs for now.

Mark

--Mark Maciolek
Network Administrator
Morse Hall Rm 338
http://www.unh.edu/research/support-units/research-computing-center



From Mark.Maciolek at unh.edu  Thu Dec  6 14:37:46 2018
From: Mark.Maciolek at unh.edu (Mark Maciolek)
Date: Thu, 6 Dec 2018 14:37:46 -0500
Subject: [Trillian-users] trillian issue
In-Reply-To: <a74983f4b7bc4999913abd73b395e07d@groot.ad.unh.edu>
References: <a74983f4b7bc4999913abd73b395e07d@groot.ad.unh.edu>
Message-ID: <00aa01d48d9b$2a142e80$7e3c8b80$@unh.edu>

Hi,

Any new jobs are going on hold or not queued. At 3:30pm I am going to cancel
any jobs running in the current que

qstat
Job id            Name             User              Time Use S Queue
----------------  ---------------- ----------------  -------- - -----
52224.sdb         trace_1          yabingwang        00:01:34 R workq
52225.sdb         trace_2          yabingwang        00:01:15 R workq
52228.sdb         trace_4          yabingwang        00:01:30 R workq
52232.sdb         trace_6          yabingwang        00:01:38 R workq
52237.sdb         trace8           yabingwang        00:01:37 R workq
52240.sdb         trace_12         yabingwang        00:01:20 R workq
52340.sdb         PBS_job_script_  salme             00:00:03 R workq
52361.sdb         trace_415        yabingwang        00:00:50 R workq
52367.sdb         2DC1.5E1.5       jhk1009           00:00:04 R workq
52368.sdb         yhout.bat        samark            00:00:01 R workq
52369.sdb         yhout.bat        samark            00:00:01 R workq

Then I will restart pbs again.

If that fails to allow new jobs to start, I will schedule a reboot of
trillian for 4PM.

Mark



-----Original Message-----
From: Trillian-users <trillian-users-bounces at lists.sr.unh.edu> On Behalf Of
Maciolek, Mark
Sent: Thursday, December 6, 2018 10:27 AM
To: 'trillian-users at lists.sr.unh.edu' <trillian-users at lists.sr.unh.edu>
Subject: [Trillian-users] trillian issue

Hi,

Jobs submitted on trillian are being queued and not running since yesterday
morning.

qstat -s shows this reason

Not Running: Insufficient amount of resource arch

The only logs are from PBS mom_logs showing this:

20181205:12/05/2018 08:00:01;0080;pbs_mom;Node;alps_engine_query;ALPS ENGINE
query failed with BASIL version 1.1.
20181205:12/05/2018 08:00:01;0002;pbs_mom;Node;alps_inventory;ALPS inventory
request failed.
20181205:12/05/2018 08:10:01;0080;pbs_mom;Node;alps_engine_query;ALPS ENGINE
query failed with BASIL version 1.1.
20181205:12/05/2018 08:10:01;0002;pbs_mom;Node;alps_inventory;ALPS inventory
request failed.

The alps logs don't show any obvious issue. 

I restarted pbs on trillian, which had the effect of cancelling some jobs
and restarting others.

Will keep an eye on the logs for now.

Mark

--Mark Maciolek
Network Administrator
Morse Hall Rm 338
http://www.unh.edu/research/support-units/research-computing-center


_______________________________________________
Trillian-users mailing list
Trillian-users at lists.sr.unh.edu
https://lists.sr.unh.edu/mailman/listinfo/trillian-users



From tjb at unh.edu  Thu Dec 13 10:04:01 2018
From: tjb at unh.edu (Thomas J. Baker)
Date: Thu, 13 Dec 2018 10:04:01 -0500
Subject: [Trillian-users] Trillian Down
Message-ID: <976B97C8-CA9A-42FA-B6AC-EA580160899D@unh.edu>


Trillian is down due to problems with the air conditioning. Someone from Dowling is scheduled to be here today to evaluate. 

Thanks,
  
tjb
--
=======================================================================
| Thomas Baker                                                        |
| Systems Programmer                            email: tjb at unh.edu    |
| Research Computing Center                    office: Morse 206      |
| University of New Hampshire                   voice: (603) 862-4490 |
| 213 Morse Hall                                  fax: (603) 862-1761 |
| Durham, NH 03824 USA                                                |
=======================================================================



From tjb at unh.edu  Thu Dec 13 15:11:50 2018
From: tjb at unh.edu (Thomas J. Baker)
Date: Thu, 13 Dec 2018 15:11:50 -0500
Subject: [Trillian-users] Trillian Back Online
Message-ID: <71F652F7-EB91-437C-8483-C2FB052649CA@unh.edu>


Trillian is back online. The queue has been cleared of previous jobs.

The AC problem was apparently due to the building controller being offline for an upgrade. RCC were not informed prior to the work that this would be occurring. 

Please send email to ops at sr.unh.edu if you encounter any problems with trillian as Mark is out until Monday.

Thanks,
  
tjb
--
=======================================================================
| Thomas Baker                                                        |
| Systems Programmer                            email: tjb at unh.edu    |
| Research Computing Center                    office: Morse 206      |
| University of New Hampshire                   voice: (603) 862-4490 |
| 213 Morse Hall                                  fax: (603) 862-1761 |
| Durham, NH 03824 USA                                                |
=======================================================================



From tjb at unh.edu  Thu Dec 13 15:22:41 2018
From: tjb at unh.edu (Thomas J. Baker)
Date: Thu, 13 Dec 2018 15:22:41 -0500
Subject: [Trillian-users] Trillian Back Online
In-Reply-To: <71F652F7-EB91-437C-8483-C2FB052649CA@unh.edu>
References: <71F652F7-EB91-437C-8483-C2FB052649CA@unh.edu>
Message-ID: <40637AD9-6C6A-436B-862C-B09267817AC3@unh.edu>

> On Dec 13, 2018, at 3:11 PM, Thomas J. Baker <tjb at unh.edu> wrote:
> 
> The AC problem was apparently due to the building controller being offline for an upgrade. RCC were not informed prior to the work that this would be occurring. 

Sorry, more information. To clarify, ?The controller was offline temporarily to your building while it was being worked on.? It?s possible this wasn?t scheduled but due to the controller malfunctioning.

Thanks,
  
tjb
--
=======================================================================
| Thomas Baker                                                        |
| Systems Programmer                            email: tjb at unh.edu    |
| Research Computing Center                    office: Morse 206      |
| University of New Hampshire                   voice: (603) 862-4490 |
| 213 Morse Hall                                  fax: (603) 862-1761 |
| Durham, NH 03824 USA                                                |
=======================================================================



From tjb at unh.edu  Fri Dec 21 10:20:30 2018
From: tjb at unh.edu (Thomas J. Baker)
Date: Fri, 21 Dec 2018 10:20:30 -0500
Subject: [Trillian-users] Trillian Back Online
Message-ID: <5222BAD5-CFC4-475E-B686-B8A4DB6430ED@unh.edu>


Trillian is back online. A momentary power interruption brought it down this morning. The queue has been cleared of previous jobs.

Thanks,
  
tjb
--
=======================================================================
| Thomas Baker                                                        |
| Systems Programmer                            email: tjb at unh.edu    |
| Research Computing Center                    office: Morse 206      |
| University of New Hampshire                   voice: (603) 862-4490 |
| 213 Morse Hall                                  fax: (603) 862-1761 |
| Durham, NH 03824 USA                                                |
=======================================================================