[Isocops] CRONology 14-Jun ** Problems with build this morning **

[Geoffrey B. Crew]

I skipped the cc to isocdev--this is ops stuff.
Some comments inlined.

-- 

		Geoff (gbc at space.mit.edu)

On Tue, Jun 15, 2010 at 03:20:20PM -0500, Mark Tapley wrote:
> Catching up is hard to do. (Apologies to Neil Sedaka).
> 
> Something is not right, starting this morning:
> 
> 12:21	Pointing file update
> 
> 10-Jun
> 
> 16:21	Cronswitch wakeup - no work.
> 20:17	Collect files from SFTP. VCID0-2, 6 .trk files, log mirror.
> 20:22	Moving files to correct directory, updating spin files
> 
> 11-Jun
> 
> 00:21	wakeup - no work.
> 02:33	rebuild /gbc tree
> 03:13	rebuild production tree
> 03:20	SVN update web
> 03:20	SVN update ena
> 03:54	SVN commit gbc
> 04:12	wakeup - no work.
> 08:21	wakeup - no work.
> 09:50	backup to giordano

I should comment that giodano.mit.edu is out of the loop these days.
I just created /home/giordano on web as I was pretty sure there would
never be a user of that name.  ;-)

> 12:17	Collect from SFTP. 3 Contact, VCID0-2, 8 .trk, one .tle.
> 	Is this OK?	IBEX_AUWA01_2010_162_14_30_VCID1.tlm is empty

1 is the back orbit buffer.  It usually holds ~ 1 MB but it is a
copy of what is in the SSR telemetry so it isn't usually dumped
unless there has been some issue (e.g. if the CEU is off).

> 12:21	Moving files to correct directories, updating orbit files.
> 16:17	Collect from SFTP site. Ephemeris products, .INP files.
> 16:21	Rebuild ephemeris files, check differences
> 20:21	wakeup - no work.
> 
> 12-Jun
> 
> 00:21	wakeup - no work.
> 02:33	rebuild /gbc tree
> 03:13	rebuild production tree
> 03:20	SVN update web
> 03:20	SVN update ena
> 03:55	SVN commit GBC
> 04:21	wakeup - no work.
> 08:21	wakeup - no work.
> 09:50	backup to giordano
> 12:21	wakeup - no work.
> 16:21	wakeup - no work.
> 20:21	wakeup - no work.
> 
> 13-Jun
> 
> 00:21	wakeup - no work.
> 02:33	rebuild /gbc tree
> 03:13	rebuild production tree
> 03:20	SVN update web
> 03:20	SVN update ena
> 03:56	SVN commit GBC
> 04:21	Built .STF file. standard stuff only, no Moon Test, no Early Descend.
> 08:18	Sent .STF to SFTP site.
> 09:45	backup to giordano
> 09:50	backup to giordano
> 12:16	SFTP retrieve new files
> 12:21	finished .STF generation sequence
> 
> 14-Jun
> 
> 02:33	rebuild /gbc tree
> 03:13	rebuild production tree
> 03:20	SVN update web
> 03:20	SVN update ena
> 03:55	SVN commit GBC
> 09:45	backup to giordano
> 09:50	backup to giordano
> 12:17	Collect from SFTP. 3 contact .INP, a .tle, and a log.
> 16:17	Collect from SFTP. an .oef and a mirror.
> 16:21	wakeup. PLOPS work.
> 
> 15-Jun
> 
> 00:16	Collect from sftp. v004 of the .stf (thanks, Chelle!) and a log.
> 02:27	rebuild /gbc tree  ****FAIL**** on isoc re-build and isoc distcheck.
> 03:08	rebuild production tree; **** FAIL **** as above.

This was a simple brain fart of mine in updating the help documents.
As you can see it is fine today.

> 03:20	SVN update web
> 03:20	SVN update ena
> 03:54	SVN commit gbc
> 09:45	backup to giordano
> 09:50	backup to giordano
> 12:17	Collect from SFTP. VCID0-2, 8 .trk files.
> 12:21	putting .trk files away, updating pointing files.
> 
> Doing OK so far?

Yes.
 
> I see Bob's comments about the failed build, but he quotes a build 
> from 02-Jun. But the same lines say "fail" in this morning's (0308 
> UTC) build:
> 
> brain wipe:     pass by 2010.166.07.00.35
> svn update:     pass by 2010.166.07.00.19
> ibex re-build:  pass by 2010.166.07.00.37
> isoc re-build:  fail by 2010.166.07.03.05
> ibex re-check:  pass by 2010.166.07.03.07
> isoc re-check:  pass by 2010.166.07.05.47
> ibex distcheck: pass by 2010.166.07.06.02
> isoc distcheck: fail by 2010.166.07.07.56
> tarballs were not installed

As with most things compiled, the first time things go off the rails
is usually the most informative.  After that it is often "collateral"
damage.  It's pretty hard (but not impossible) for distcheck to succeed
if the re-build or re-check stages have failed.

> [tapley at ena log]$ cat nightly-2010.166.06.20.01.log | grep FAIL
> XFAIL: check_ilog.sh
> XFAIL: averCarriTest.sh
> XFAIL: muvrItotTest.sh
> XFAIL: sigCXTest.sh

The "check" mechanism usually leads to PASS or FAIL.  However, it is
to be expected that in development things can be expected to FAIL for
known reasons that cannot be immediately rectified.  Thus in the Makefile
one can opt to run the tests and expect failure--this gets marked as XFAIL.

(Amusingly, if you expect it to fail and it passes, that's an error!)

If I remember correctly the latter three are tests for a development
thread that is incomplete.  The first one is a check that is useful
for verifying that the syslog facility is properly configured, however
it's not a useful thing to be running every night.

It's a simple matter to completely disable all of these.

> At 13:17 -0400 6/2/10, DeMajistre, Bob wrote:
> >Since I'm getting these logs now, I shouldn't be afraid to use them. From
> >the message below and some poking around. I find that Nathan updated the
> >Makefile and resolve lists by adding a routine that Jacob built and Ken
> >committed last week.
> >
> >For some reason I don't understand, this caused "try_skymap.sh" to fail. I
> >haven't been smart enough to reproduce this error, or to try to figure out
> >how to reproduce it. Should I be worried?
> >
> >I'm also willing to accept the answer "shut up and sit down" as an answer.
> >
> >Bob

There are a few rare probabilistic failures:  the test generates random
crap and throws it at something and almost all of the time it passes.
Likewise the nightly process has to get permission from the IDL license
server to do useful work and the pretty-please process isn't bulletproof.

So, generally speaking, it's ok to let a problem ride to see if it is
automatically repeatable.  On the other hand, when one does something in
some area and it immediately barfs, well, that's really what this is all
for....

> -- 
> 						- Mark     210-379-4635
> -----------------------------------------------------------------------
> Large Asteroids headed toward planets
> inhabited by beings that don't have
> technology adequate to stop them:
> 
> 				Think of it as Evolution in Fast-Forward.
> 
> Please restrict discussions on this email list to non-ITAR sensitive topics.
> ______________________________________________
> Isocops mailing list
> Isocops at lists.sr.unh.edu
> http://lists.sr.unh.edu/mailman/listinfo/isocops

Question for the day:

	Is Evolution in Fast-Forward an ITAR sensitive topic?