[Isocops] EXT :Re: IBEX Time Stamp Issue: Follow-on Discussion

[Perry, Timothy E [US] (SP)]

Nigel,
    So, I haven’t seen any new traffic on your suggestion to create a memory map or a ram read. Is there a plan for moving forward?
       Tim


Timothy E.Perry
CRS IBEX OCO-2
Mission Operations
timothy.perry at ngc.com
Office (703) 406-5976
Cell (301) 606-1430

[cid:image001.png at 01D542D6.6047A400]


From: NIgel Angold [mailto:nangold at princeton.edu]
Sent: Thursday, September 09, 2021 3:13 PM
To: Tyler, Ryan S [US] (SP) <Ryan.Tyler at ngc.com>
Cc: Perry, Timothy E [US] (SP) <Timothy.Perry at ngc.com>; Wynn, Reese [US] (SP) <Reese.Wynn at ngc.com>; Bobbett, James E [US] (SP) <James.Bobbett at ngc.com>; ISOC at New Hampshire <isocops at lists.sr.unh.edu>; Bachir-Bouiadjra, Benahmed [US] (SP) <Benahmed.Bachir-Bouiadjra at ngc.com>; Conway, Jim [US] (SP) <Jim.Conway at ngc.com>; Leonard, Benjamin Y [US] (SP) <Benjamin.Leonard at ngc.com>; Ken Fairchild UNH (Ken.Fairchild at unh.edu) <Ken.Fairchild at unh.edu>; Dave McComas <dmccomas at princeton.edu>; Bret Hautamaki <bhautama at gmail.com>; Fleming, Ed [US] (SP) <Ed.Fleming at ngc.com>; Wesley, Sheral R [US] (SP) <Sheral.Wesley at ngc.com>; Ibex Mission Operations <ibexops at gmail.com>; Ken Fairchild (kwfairchild at gmail.com) <kwfairchild at gmail.com>; Eric Christian <eric.r.christian at nasa.gov>; Cavallo, John A [US] (SP) <John.Cavallo at ngc.com>; Chubin, Eric [US] (SP) <Eric.Chubin at ngc.com>; Cross, Walker W [US] (SP) <Walker.Cross at ngc.com>; Varia, Apurva P. (GSFC-5840) <apurva.p.varia at nasa.gov>; IBEX FDG Group (ADS-IBEX at l3harris.com) <ADS-IBEX at l3harris.com>; Scott Weidner <sweidner at princeton.edu>
Subject: EXT :Re: IBEX Time Stamp Issue: Follow-on Discussion

Thanks Ryan.

So I think the next step is to create a memory map or perform a RAM read. When do you think we can have a CAR ready for review?

Thanks,
Nigel



I’ll add my comments below in red.

-Ryan

From: NIgel Angold <nangold at princeton.edu><mailto:nangold at princeton.edu>
Sent: Wednesday, September 8, 2021 10:50 PM
To: Perry, Timothy E [US] (SP) <Timothy.Perry at ngc.com><mailto:Timothy.Perry at ngc.com>; Wynn, Reese [US] (SP) <Reese.Wynn at ngc.com><mailto:Reese.Wynn at ngc.com>; Bobbett, James E [US] (SP) <James.Bobbett at ngc.com><mailto:James.Bobbett at ngc.com>; ISOC at New Hampshire <isocops at lists.sr.unh.edu><mailto:isocops at lists.sr.unh.edu>; Bachir-Bouiadjra, Benahmed [US] (SP) <Benahmed.Bachir-Bouiadjra at ngc.com><mailto:Benahmed.Bachir-Bouiadjra at ngc.com>; Conway, Jim [US] (SP) <Jim.Conway at ngc.com><mailto:Jim.Conway at ngc.com>; Leonard, Benjamin Y [US] (SP) <Benjamin.Leonard at ngc.com><mailto:Benjamin.Leonard at ngc.com>; Ken Fairchild UNH (Ken.Fairchild at unh.edu<mailto:Ken.Fairchild at unh.edu>) <Ken.Fairchild at unh.edu><mailto:Ken.Fairchild at unh.edu>; Dave McComas <dmccomas at princeton.edu><mailto:dmccomas at princeton.edu>; Bret Hautamaki <bhautama at gmail.com><mailto:bhautama at gmail.com>; Fleming, Ed [US] (SP) <Ed.Fleming at ngc.com><mailto:Ed.Fleming at ngc.com>; Wesley, Sheral R [US] (SP) <Sheral.Wesley at ngc.com><mailto:Sheral.Wesley at ngc.com>; Tyler, Ryan S [US] (SP) <Ryan.Tyler at ngc.com><mailto:Ryan.Tyler at ngc.com>; Ibex Mission Operations <ibexops at gmail.com><mailto:ibexops at gmail.com>; Ken Fairchild (kwfairchild at gmail.com<mailto:kwfairchild at gmail.com>) <kwfairchild at gmail.com><mailto:kwfairchild at gmail.com>; Eric Christian <eric.r.christian at nasa.gov><mailto:eric.r.christian at nasa.gov>; Cavallo, John A [US] (SP) <John.Cavallo at ngc.com><mailto:John.Cavallo at ngc.com>; Chubin, Eric [US] (SP) <Eric.Chubin at ngc.com><mailto:Eric.Chubin at ngc.com>; Conway, Jim [US] (SP) <Jim.Conway at ngc.com><mailto:Jim.Conway at ngc.com>; Cross, Walker W [US] (SP) <Walker.Cross at ngc.com><mailto:Walker.Cross at ngc.com>; Varia, Apurva P. (GSFC-5840) <apurva.p.varia at nasa.gov><mailto:apurva.p.varia at nasa.gov>; IBEX FDG Group (ADS-IBEX at l3harris.com<mailto:ADS-IBEX at l3harris.com>) <ADS-IBEX at l3harris.com><mailto:ADS-IBEX at l3harris.com>; Scott Weidner <sweidner at princeton.edu><mailto:sweidner at princeton.edu>
Subject: EXT :IBEX Time Stamp Issue: Follow-on Discussion

Regarding the possible paths forward, please see my comments in blue below. Your comments are welcome.

Thanks,
Nigel

  *   Do nothing on the spacecraft

     *   what are the consequences?

        *   any timing issues on board?

           *   It doesn't look like there are any at the moment but I'm concerned about the consequences of using memory locations outside the bounds of the table to determine spacecraft functionality
           *   I agree it doesn’t look like there are, and indeed that’s been true for 1.5 years before our recent speedbump. Once we confirm the contents of memory where the table is looking, I might be satisfied.

        *   any ground timing issues e.g. MAESTRO archiving or SOC data processing?

           *   MAESTRO is not able to archive data since everything has the same 1970 time stamp
           *   If the secondary header time is still incrementing, is it possible to update the MAESTRO processing to cope with the unexpected values?
           *   I’m wondering if we can edit the binary files to fix the secondary headers based on a known conversion so that with a little (repeatable) tinkering the downlinked files can make it into the archive after all

     *   not a comfortable option if we don't fully determine the root cause

  *   Update on board table(s) or FSW

     *   need to develop code fix, patch, procedures

        *   several months to develop

           *   Would this just be increasing the size of the table to include entries for another ~10 years?
           *   2 major options:

              *   Update several rows in the table to match upcoming years and update code to make the starting year 2020 rather than 1980 – should fix all aspects of the issue, and we’ve already tested this logic
              *   Simple fix to edit only the table value in the last row of the table (2020) to make it a very large number that the GPS time will never exceed. This should lock in a 2019 year and correctly calculated number of days that can easily be converted (as MAESTRO had been doing automatically since the start of 2020).

           *   Neither of these options involve changing the table length and both should be achievable via RAM poke
           *   We need some more time to do our research on this, figure out exactly what memory locations we’re talking about.

     *   need to test on Flatsat which has not been used for 5 or more years

        *   A poke wouldn’t necessarily need to be tested on FlatSat. We’d start by reading the RAM at that location to verify we’re touching the right part of memory, abort if not confirmed. Then we poke just that same location. Since this is a RAM poke, we can always reboot if something goes wrong and come up cleanly (in 1980, a time that would be correctly interpreted), clearing out the poked RAM.
        *   NG is already working to get Flatsat up and running

           *   Has NG been able to locate the password to gain access to the Flatsat machine?

     *   could we poke memory locations rather than full table updates?

        *   Are the memory locations beyond the timing table used for anything else? Could poking new values have unintended consequences?
        *   I definitely would not want to poke beyond the table (thus, no extending the length of the table as an option)

  *   Perform a Flight Computer Reset

     *   hope that this corrects the memory that changed

        *   I think this is probably our "Hail Mary" option
        *   Only really makes sense if the test you propose below shows that now timestamps from a clock set to 3 weeks ago (prior to the recent speedbump) also turn out bad.

  *   Set the on board spacecraft time to current time minus 30 years

     *   this "Back to the Future" approach would put the time into the middle of the on board table (it starts at 1980)
     *   would result in unique time stamps for archive purposes (unless the mission lasts more than 30 years)
     *   would need to update MPS code to subtract 30 years from ATS command times or manually update each ATS file
     *   could perform all MPS planning with current date so that flight dynamics processes would not be impacted

        *   This may work but I'd like to make sure that going back in time results in MAESTRO producing good secondary header time stamps again

  *   Another Path?

     *   What have we missed?
One hypothesis is that the content of the memory just beyond the GPS timing table changed on 24 August. However, Ken points out that it's possible the content didn't change but the value is such that the time comparison wasn't exceeded until 24 August, thereafter resulting in further comparisons that were exceeded for another 6 locations until the location corresponding to year 2026 contained a value greater than the current time. If that's the case, then setting the spacecraft time back to just before the 24 August time stamp jump would result in times that MAESTRO could again process correctly, followed by another jump back to 1970. That's a test that I think I'd like to try since I'd really like to know if this issue is caused by (random) memory content changes or simply because a static memory value has been exceeded.

Ryan points out that we'd need to stop the on board ATS if we did a "back in time" test. The orbit 540 ATS runs until 2021/09/14 08:30:00 and the orbit 541 ATS starts at 2021/09:14 14:43:12. We could add a contact at around 2021/09/14 13:00-14:00 during which we'd delete the orbit 541 commands, set the spacecraft time to 24 August, watch for the time stamp jump, reset the spacecraft clock to NOW and reload the orbit 541 ATS (and orbit 542 ATS, for that matter). How does that sound? Can you think of any unintended consequences? Can we plan this for next Tuesday at ~09:00 EDT without stressing the system?

I’m in favor of such a test, but it might prove unnecessary if we can establish via memory map (and possibly a RAM read command) that Ken’s hypothesis is correct.


Feedback please!

Thanks as always,
Nigel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.sr.unh.edu/pipermail/isocops/attachments/20210913/6ad9d061/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5047 bytes
Desc: image001.png
URL: <https://lists.sr.unh.edu/pipermail/isocops/attachments/20210913/6ad9d061/image001-0001.png>