[Isocops] IBEX Time Stamp Issue: Update

[NIgel Angold]

Here's an update on the IBEX time stamp issue.

Last night's apogee contact was nominal although the  MAESTRO time 
stamps were still 1 January 1970. We verified that the apogee maneuver 
executed nominally and uplinked the orbit 540 ATS. We now have commands 
on board until 14 September. Apart from the erroneous time stamps, the 
spacecraft is in good health. And the Lo and Hi teams reported that the 
last orbit's science data looks nominal.

Many of you know that there's a table on board the spacecraft which has 
entries up until 2020 and when we got to 1 January 2021, the DOY in the 
secondary header did not reset to 1 but continued to 367 and incremented 
each day thereafter. The year remained at 2020. This had no noticeable 
effect on the spacecraft and MAESTRO created correct 2021 timestamps. 
SOC algorithms were tweaked to handle the greater than 366 DOYs. 
Northrop Grumman engineers looked into whether we would need to update 
the table at some point or whether we could live with 2020 + >366 DOYs 
for the remainder of the mission. It was thought that we could live 
without updating FSW or the on board table.

However, on August 24 i.e 2020 DOY 602 (or 2021 DOY 236), the year in 
the spacecraft secondary header jumped from 2020 to 2026 and the DOY 
jumped from 602 to 13948. 2026 DOY 13948 decodes to 9 March 2064 which 
is well past the maximum Unix time (number of seconds after 1 January 
1970 00:00 UTC) that can be represented by a 32 bit signed integer value 
(2^31-1 = 2147483647 = 19 January 2038). Since MAESTRO could not cope 
with dates/times so far in the future, it set them to 0 i.e. 1 January 
1970 00:00 UTC.

We still do not know why there was a jump in the secondary header values 
on 24 August. NG has performed some testing of the FSW code and has been 
able to generate time stamps in the far future. One possible cause is 
that the code is looking at memory outside the confines of the time 
definition lookup table. Analysis is ongoing.

We are considering a number of paths forward, including:

  * Do nothing on the spacecraft
      o what are the consequences?
          + any timing issues on board?
          + any ground timing issues e.g. MAESTRO archiving or SOC data
            processing?
      o not a comfortable option if we don't fully determine the root cause
  * Update on board table(s) or FSW
      o need to develop code fix, patch, procedures
          + several months to develop
      o need to test on Flatsat which has not been used for 5 or more years
          + NG is already working to get Flatsat up and running
      o could we poke memory locations rather than full table updates?
  * Perform a Flight Computer Reset
      o hope that this corrects the memory that changed
  * Set the on board spacecraft time to current time minus 30 years
      o this "Back to the Future" approach would put the time into the
        middle of the on board table (it starts at 1980)
      o would result in unique time stamps for archive purposes (unless
        the mission lasts more than 30 years)
      o would need to update MPS code to subtract 30 years from ATS
        command times or manually update each ATS file
      o could perform all MPS planning with current date so that flight
        dynamics processes would not be impacted

That's some food for thought!

Thanks everyone for your support.
Nigel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.sr.unh.edu/pipermail/isocops/attachments/20210901/45d8ad67/attachment.html>