[Isocops] EXT :Re: IBEX Time Stamp Issue: Follow-on Discussion
NIgel Angold
nangold at princeton.edu
Mon Sep 13 15:11:54 EDT 2021
Hi Tim,
I am awaiting an NG assessment of how long it will take to put a CAR
together to downlink the memory contents. Once we have a CAR, we need to
review it and find a time to perform the activity. I doubt if we'll have
enough time to do anything this perigee but I hope we'll have a path
forward before the next.
Thanks,
Nigel
>
> Nigel,
>
> So, I haven’t seen any new traffic on your suggestion to create a
> memory map or a ram read. Is there a plan for moving forward?
>
> Tim
>
> Timothy E.Perry
>
> CRS IBEX OCO-2
>
> Mission Operations
>
> timothy.perry at ngc.com
>
> Office (703) 406-5976
>
> Cell (301) 606-1430
>
> cid:image001.png at 01D542D6.6047A400
>
> *From:*NIgel Angold [mailto:nangold at princeton.edu]
> *Sent:* Thursday, September 09, 2021 3:13 PM
> *To:* Tyler, Ryan S [US] (SP) <Ryan.Tyler at ngc.com>
> *Cc:* Perry, Timothy E [US] (SP) <Timothy.Perry at ngc.com>; Wynn, Reese
> [US] (SP) <Reese.Wynn at ngc.com>; Bobbett, James E [US] (SP)
> <James.Bobbett at ngc.com>; ISOC at New Hampshire
> <isocops at lists.sr.unh.edu>; Bachir-Bouiadjra, Benahmed [US] (SP)
> <Benahmed.Bachir-Bouiadjra at ngc.com>; Conway, Jim [US] (SP)
> <Jim.Conway at ngc.com>; Leonard, Benjamin Y [US] (SP)
> <Benjamin.Leonard at ngc.com>; Ken Fairchild UNH (Ken.Fairchild at unh.edu)
> <Ken.Fairchild at unh.edu>; Dave McComas <dmccomas at princeton.edu>; Bret
> Hautamaki <bhautama at gmail.com>; Fleming, Ed [US] (SP)
> <Ed.Fleming at ngc.com>; Wesley, Sheral R [US] (SP)
> <Sheral.Wesley at ngc.com>; Ibex Mission Operations <ibexops at gmail.com>;
> Ken Fairchild (kwfairchild at gmail.com) <kwfairchild at gmail.com>; Eric
> Christian <eric.r.christian at nasa.gov>; Cavallo, John A [US] (SP)
> <John.Cavallo at ngc.com>; Chubin, Eric [US] (SP) <Eric.Chubin at ngc.com>;
> Cross, Walker W [US] (SP) <Walker.Cross at ngc.com>; Varia, Apurva P.
> (GSFC-5840) <apurva.p.varia at nasa.gov>; IBEX FDG Group
> (ADS-IBEX at l3harris.com) <ADS-IBEX at l3harris.com>; Scott Weidner
> <sweidner at princeton.edu>
> *Subject:* EXT :Re: IBEX Time Stamp Issue: Follow-on Discussion
>
> Thanks Ryan.
>
> So I think the next step is to create a memory map or perform a RAM
> read. When do you think we can have a CAR ready for review?
>
> Thanks,
> Nigel
>
>
>
> I’ll add my comments below in red.
>
> -Ryan
>
> *From:*NIgel Angold <nangold at princeton.edu>
> <mailto:nangold at princeton.edu>
> *Sent:* Wednesday, September 8, 2021 10:50 PM
> *To:* Perry, Timothy E [US] (SP) <Timothy.Perry at ngc.com>
> <mailto:Timothy.Perry at ngc.com>; Wynn, Reese [US] (SP)
> <Reese.Wynn at ngc.com> <mailto:Reese.Wynn at ngc.com>; Bobbett, James E
> [US] (SP) <James.Bobbett at ngc.com> <mailto:James.Bobbett at ngc.com>;
> ISOC at New Hampshire <isocops at lists.sr.unh.edu>
> <mailto:isocops at lists.sr.unh.edu>; Bachir-Bouiadjra, Benahmed [US]
> (SP) <Benahmed.Bachir-Bouiadjra at ngc.com>
> <mailto:Benahmed.Bachir-Bouiadjra at ngc.com>; Conway, Jim [US] (SP)
> <Jim.Conway at ngc.com> <mailto:Jim.Conway at ngc.com>; Leonard,
> Benjamin Y [US] (SP) <Benjamin.Leonard at ngc.com>
> <mailto:Benjamin.Leonard at ngc.com>; Ken Fairchild UNH
> (Ken.Fairchild at unh.edu <mailto:Ken.Fairchild at unh.edu>)
> <Ken.Fairchild at unh.edu> <mailto:Ken.Fairchild at unh.edu>; Dave
> McComas <dmccomas at princeton.edu> <mailto:dmccomas at princeton.edu>;
> Bret Hautamaki <bhautama at gmail.com> <mailto:bhautama at gmail.com>;
> Fleming, Ed [US] (SP) <Ed.Fleming at ngc.com>
> <mailto:Ed.Fleming at ngc.com>; Wesley, Sheral R [US] (SP)
> <Sheral.Wesley at ngc.com> <mailto:Sheral.Wesley at ngc.com>; Tyler,
> Ryan S [US] (SP) <Ryan.Tyler at ngc.com> <mailto:Ryan.Tyler at ngc.com>;
> Ibex Mission Operations <ibexops at gmail.com>
> <mailto:ibexops at gmail.com>; Ken Fairchild (kwfairchild at gmail.com
> <mailto:kwfairchild at gmail.com>) <kwfairchild at gmail.com>
> <mailto:kwfairchild at gmail.com>; Eric Christian
> <eric.r.christian at nasa.gov> <mailto:eric.r.christian at nasa.gov>;
> Cavallo, John A [US] (SP) <John.Cavallo at ngc.com>
> <mailto:John.Cavallo at ngc.com>; Chubin, Eric [US] (SP)
> <Eric.Chubin at ngc.com> <mailto:Eric.Chubin at ngc.com>; Conway, Jim
> [US] (SP) <Jim.Conway at ngc.com> <mailto:Jim.Conway at ngc.com>; Cross,
> Walker W [US] (SP) <Walker.Cross at ngc.com>
> <mailto:Walker.Cross at ngc.com>; Varia, Apurva P. (GSFC-5840)
> <apurva.p.varia at nasa.gov> <mailto:apurva.p.varia at nasa.gov>; IBEX
> FDG Group (ADS-IBEX at l3harris.com <mailto:ADS-IBEX at l3harris.com>)
> <ADS-IBEX at l3harris.com> <mailto:ADS-IBEX at l3harris.com>; Scott
> Weidner <sweidner at princeton.edu> <mailto:sweidner at princeton.edu>
> *Subject:* EXT :IBEX Time Stamp Issue: Follow-on Discussion
>
> Regarding the possible paths forward, please see my comments in
> blue below. Your comments are welcome.
>
> Thanks,
> Nigel
>
> * Do nothing on the spacecraft
>
> o what are the consequences?
>
> + any timing issues on board?
>
> # It doesn't look like there are any at the moment
> but I'm concerned about the consequences of using
> memory locations outside the bounds of the table
> to determine spacecraft functionality
> # I agree it doesn’t look like there are, and indeed
> that’s been true for 1.5 years before our recent
> speedbump. Once we confirm the contents of memory
> where the table is looking, I might be satisfied.
>
> + any ground timing issues e.g. MAESTRO archiving or SOC
> data processing?
>
> # MAESTRO is not able to archive data since
> everything has the same 1970 time stamp
> # If the secondary header time is still
> incrementing, is it possible to update the MAESTRO
> processing to cope with the unexpected values?
> # I’m wondering if we can edit the binary files to
> fix the secondary headers based on a known
> conversion so that with a little (repeatable)
> tinkering the downlinked files can make it into
> the archive after all
>
> o not a comfortable option if we don't fully determine the
> root cause
>
> * Update on board table(s) or FSW
>
> o need to develop code fix, patch, procedures
>
> + several months to develop
>
> # Would this just be increasing the size of the
> table to include entries for another ~10 years?
> # 2 major options:
>
> * Update several rows in the table to match
> upcoming years and update code to make the
> starting year 2020 rather than 1980 – should
> fix all aspects of the issue, and we’ve
> already tested this logic
> * Simple fix to edit only the table value in the
> last row of the table (2020) to make it a very
> large number that the GPS time will never
> exceed. This should lock in a 2019 year and
> correctly calculated number of days that can
> easily be converted (as MAESTRO had been doing
> automatically since the start of 2020).
>
> # Neither of these options involve changing the
> table length and both should be achievable via RAM
> poke
> # We need some more time to do our research on this,
> figure out exactly what memory locations we’re
> talking about.
>
> o need to test on Flatsat which has not been used for 5 or
> more years
>
> + A poke wouldn’t necessarily need to be tested on
> FlatSat. We’d start by reading the RAM at that
> location to verify we’re touching the right part of
> memory, abort if not confirmed. Then we poke just that
> same location. Since this is a RAM poke, we can always
> reboot if something goes wrong and come up cleanly (in
> 1980, a time that would be correctly interpreted),
> clearing out the poked RAM.
> + NG is already working to get Flatsat up and running
>
> # Has NG been able to locate the password to gain
> access to the Flatsat machine?
>
> o could we poke memory locations rather than full table
> updates?
>
> + Are the memory locations beyond the timing table used
> for anything else? Could poking new values have
> unintended consequences?
> + I definitely would not want to poke beyond the table
> (thus, no extending the length of the table as an option)
>
> * Perform a Flight Computer Reset
>
> o hope that this corrects the memory that changed
>
> + I think this is probably our "Hail Mary" option
> + Only really makes sense if the test you propose below
> shows that now timestamps from a clock set to 3 weeks
> ago (prior to the recent speedbump) also turn out bad.
>
> * Set the on board spacecraft time to current time minus 30 years
>
> o this "Back to the Future" approach would put the time into
> the middle of the on board table (it starts at 1980)
> o would result in unique time stamps for archive purposes
> (unless the mission lasts more than 30 years)
> o would need to update MPS code to subtract 30 years from
> ATS command times or manually update each ATS file
> o could perform all MPS planning with current date so that
> flight dynamics processes would not be impacted
>
> + This may work but I'd like to make sure that going
> back in time results in MAESTRO producing good
> secondary header time stamps again
>
> * Another Path?
>
> o What have we missed?
>
> One hypothesis is that the content of the memory just beyond the
> GPS timing table changed on 24 August. However, Ken points out
> that it's possible the content didn't change but the value is such
> that the time comparison wasn't exceeded until 24 August,
> thereafter resulting in further comparisons that were exceeded for
> another 6 locations until the location corresponding to year 2026
> contained a value greater than the current time. If that's the
> case, then setting the spacecraft time back to just before the 24
> August time stamp jump would result in times that MAESTRO could
> again process correctly, followed by another jump back to 1970.
> That's a test that I think I'd like to try since I'd really like
> to know if this issue is caused by (random) memory content changes
> or simply because a static memory value has been exceeded.
>
> Ryan points out that we'd need to stop the on board ATS if we did
> a "back in time" test. The orbit 540 ATS runs until 2021/09/14
> 08:30:00 and the orbit 541 ATS starts at 2021/09:14 14:43:12. We
> could add a contact at around 2021/09/14 13:00-14:00 during which
> we'd delete the orbit 541 commands, set the spacecraft time to 24
> August, watch for the time stamp jump, reset the spacecraft clock
> to NOW and reload the orbit 541 ATS (and orbit 542 ATS, for that
> matter). How does that sound? Can you think of any unintended
> consequences? Can we plan this for next Tuesday at ~09:00 EDT
> without stressing the system?
>
> I’m in favor of such a test, but it might prove unnecessary if we
> can establish via memory map (and possibly a RAM read command)
> that Ken’s hypothesis is correct.
>
>
>
> Feedback please!
>
> Thanks as always,
> Nigel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.sr.unh.edu/pipermail/isocops/attachments/20210913/f88e029c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 5125 bytes
Desc: not available
URL: <https://lists.sr.unh.edu/pipermail/isocops/attachments/20210913/f88e029c/image-0001.png>
More information about the Isocops
mailing list