[Isocops] EXT :Re: IBEX Time Stamp Issue: Follow-on Discussion
NIgel Angold
nangold at princeton.edu
Wed Sep 15 21:07:44 EDT 2021
Thanks Ryan. This is the news I had been hoping for. I have been very
concerned that, if the time stamp jump was caused by memory values
changing, those values could change again and lead to some
non-deterministic effect on the spacecraft.
I think you have a typo in your email: the last entry in the table is
4B3D3B00 = 126230400 seconds: 1-Jan-1980 + 126230400 seconds = 1-Jan-2020
4E560000 = 1314258944: 1-Jan-1980 + 1314258944 seconds = 07:55:44 on
24-Aug-2021 (don't forget that there was another leap year in 2020).
The last "good" time stamp that we saw was 07:55:43 on 24-Aug-2021. So
that's spot on !!!!!
I also note that the next 4 locations after the one containing 4E560000
contain smaller values so a time value of 1314258944 would not force an
exit from the while loop. I'd be interested to see the next locations
(not included in your attached image) to confirm that there's a larger
value in the memory location corresponding to 2027. And if we plug in a
time value of 1314258944, it results in 2026, DOY of 13948, just like we
saw in the Secondary Header. I bet it does.
Thanks,
Nigel
>
> A bit of news on the timestamp issue:
>
> We found a software image for the flight computer and looked through
> it and were able to locate the table in question (see picture pasted
> in at the bottom of this email). The last entry in the table is
> 4B3D3B00 = 1314258944 seconds = 40.027 years (the 0.027 = 10 extra
> leap year days since 1980). Directly following this last entry in the
> table is 4E560000.
>
> Converted to years in the same way, this is 41.675, which, after
> subtracting out the extra leap year days, is 8/25/21. This is 1 day
> off of when we had the timestamp change, which I assume is just within
> the margin of error of my calculations.
>
> So…
>
> Assuming the image we found is the same as is onboard, which seems
> quite likely, at least in this portion of memory, I can confirm that
> memory has NOT changed and that we just happened to hit the date that
> was “next” in the table.
>
> At this point, it seems unnecessary to do any testing where we change
> the clock on the spacecraft. I think we can feel fairly confident that
> we’re somewhere stable, not in any particular risk of some unexpected
> consequence.
>
> The next step likely will be to attempt passive ReadRAM commands to
> the spacecraft, which we can do on any contact once we figure out
> exactly what parameters we want to use. We believe this command should
> allow us to figure out exactly where the 40^th entry in the table is.
> That part should be completely passive and risk-free (just a telemetry
> request). After that we can consider using the WriteRAM command to
> change that table entry to get us essentially back to where we were
> before 8/24 by ensuring that we never exceed the confines of the table.
>
> -Ryan
>
> *From:*NIgel Angold <nangold at princeton.edu>
> *Sent:* Thursday, September 9, 2021 3:13 PM
> *To:* Tyler, Ryan S [US] (SP) <Ryan.Tyler at ngc.com>
> *Cc:* Perry, Timothy E [US] (SP) <Timothy.Perry at ngc.com>; Wynn, Reese
> [US] (SP) <Reese.Wynn at ngc.com>; Bobbett, James E [US] (SP)
> <James.Bobbett at ngc.com>; ISOC at New Hampshire
> <isocops at lists.sr.unh.edu>; Bachir-Bouiadjra, Benahmed [US] (SP)
> <Benahmed.Bachir-Bouiadjra at ngc.com>; Conway, Jim [US] (SP)
> <Jim.Conway at ngc.com>; Leonard, Benjamin Y [US] (SP)
> <Benjamin.Leonard at ngc.com>; Ken Fairchild UNH (Ken.Fairchild at unh.edu)
> <Ken.Fairchild at unh.edu>; Dave McComas <dmccomas at princeton.edu>; Bret
> Hautamaki <bhautama at gmail.com>; Fleming, Ed [US] (SP)
> <Ed.Fleming at ngc.com>; Wesley, Sheral R [US] (SP)
> <Sheral.Wesley at ngc.com>; Ibex Mission Operations <ibexops at gmail.com>;
> Ken Fairchild (kwfairchild at gmail.com) <kwfairchild at gmail.com>; Eric
> Christian <eric.r.christian at nasa.gov>; Cavallo, John A [US] (SP)
> <John.Cavallo at ngc.com>; Chubin, Eric [US] (SP) <Eric.Chubin at ngc.com>;
> Cross, Walker W [US] (SP) <Walker.Cross at ngc.com>; Varia, Apurva P.
> (GSFC-5840) <apurva.p.varia at nasa.gov>; IBEX FDG Group
> (ADS-IBEX at l3harris.com) <ADS-IBEX at l3harris.com>; Scott Weidner
> <sweidner at princeton.edu>
> *Subject:* EXT :Re: IBEX Time Stamp Issue: Follow-on Discussion
>
> Thanks Ryan.
>
> So I think the next step is to create a memory map or perform a RAM
> read. When do you think we can have a CAR ready for review?
>
> Thanks,
> Nigel
>
>
>
> I’ll add my comments below in red.
>
> -Ryan
>
> *From:*NIgel Angold <nangold at princeton.edu>
> <mailto:nangold at princeton.edu>
> *Sent:* Wednesday, September 8, 2021 10:50 PM
> *To:* Perry, Timothy E [US] (SP) <Timothy.Perry at ngc.com>
> <mailto:Timothy.Perry at ngc.com>; Wynn, Reese [US] (SP)
> <Reese.Wynn at ngc.com> <mailto:Reese.Wynn at ngc.com>; Bobbett, James E
> [US] (SP) <James.Bobbett at ngc.com> <mailto:James.Bobbett at ngc.com>;
> ISOC at New Hampshire <isocops at lists.sr.unh.edu>
> <mailto:isocops at lists.sr.unh.edu>; Bachir-Bouiadjra, Benahmed [US]
> (SP) <Benahmed.Bachir-Bouiadjra at ngc.com>
> <mailto:Benahmed.Bachir-Bouiadjra at ngc.com>; Conway, Jim [US] (SP)
> <Jim.Conway at ngc.com> <mailto:Jim.Conway at ngc.com>; Leonard,
> Benjamin Y [US] (SP) <Benjamin.Leonard at ngc.com>
> <mailto:Benjamin.Leonard at ngc.com>; Ken Fairchild UNH
> (Ken.Fairchild at unh.edu <mailto:Ken.Fairchild at unh.edu>)
> <Ken.Fairchild at unh.edu> <mailto:Ken.Fairchild at unh.edu>; Dave
> McComas <dmccomas at princeton.edu> <mailto:dmccomas at princeton.edu>;
> Bret Hautamaki <bhautama at gmail.com> <mailto:bhautama at gmail.com>;
> Fleming, Ed [US] (SP) <Ed.Fleming at ngc.com>
> <mailto:Ed.Fleming at ngc.com>; Wesley, Sheral R [US] (SP)
> <Sheral.Wesley at ngc.com> <mailto:Sheral.Wesley at ngc.com>; Tyler,
> Ryan S [US] (SP) <Ryan.Tyler at ngc.com> <mailto:Ryan.Tyler at ngc.com>;
> Ibex Mission Operations <ibexops at gmail.com>
> <mailto:ibexops at gmail.com>; Ken Fairchild (kwfairchild at gmail.com
> <mailto:kwfairchild at gmail.com>) <kwfairchild at gmail.com>
> <mailto:kwfairchild at gmail.com>; Eric Christian
> <eric.r.christian at nasa.gov> <mailto:eric.r.christian at nasa.gov>;
> Cavallo, John A [US] (SP) <John.Cavallo at ngc.com>
> <mailto:John.Cavallo at ngc.com>; Chubin, Eric [US] (SP)
> <Eric.Chubin at ngc.com> <mailto:Eric.Chubin at ngc.com>; Conway, Jim
> [US] (SP) <Jim.Conway at ngc.com> <mailto:Jim.Conway at ngc.com>; Cross,
> Walker W [US] (SP) <Walker.Cross at ngc.com>
> <mailto:Walker.Cross at ngc.com>; Varia, Apurva P. (GSFC-5840)
> <apurva.p.varia at nasa.gov> <mailto:apurva.p.varia at nasa.gov>; IBEX
> FDG Group (ADS-IBEX at l3harris.com <mailto:ADS-IBEX at l3harris.com>)
> <ADS-IBEX at l3harris.com> <mailto:ADS-IBEX at l3harris.com>; Scott
> Weidner <sweidner at princeton.edu> <mailto:sweidner at princeton.edu>
> *Subject:* EXT :IBEX Time Stamp Issue: Follow-on Discussion
>
> Regarding the possible paths forward, please see my comments in
> blue below. Your comments are welcome.
>
> Thanks,
> Nigel
>
> * Do nothing on the spacecraft
>
> o what are the consequences?
>
> + any timing issues on board?
>
> # It doesn't look like there are any at the moment
> but I'm concerned about the consequences of using
> memory locations outside the bounds of the table
> to determine spacecraft functionality
> # I agree it doesn’t look like there are, and indeed
> that’s been true for 1.5 years before our recent
> speedbump. Once we confirm the contents of memory
> where the table is looking, I might be satisfied.
>
> + any ground timing issues e.g. MAESTRO archiving or SOC
> data processing?
>
> # MAESTRO is not able to archive data since
> everything has the same 1970 time stamp
> # If the secondary header time is still
> incrementing, is it possible to update the MAESTRO
> processing to cope with the unexpected values?
> # I’m wondering if we can edit the binary files to
> fix the secondary headers based on a known
> conversion so that with a little (repeatable)
> tinkering the downlinked files can make it into
> the archive after all
>
> o not a comfortable option if we don't fully determine the
> root cause
>
> * Update on board table(s) or FSW
>
> o need to develop code fix, patch, procedures
>
> + several months to develop
>
> # Would this just be increasing the size of the
> table to include entries for another ~10 years?
> # 2 major options:
>
> * Update several rows in the table to match
> upcoming years and update code to make the
> starting year 2020 rather than 1980 – should
> fix all aspects of the issue, and we’ve
> already tested this logic
> * Simple fix to edit only the table value in the
> last row of the table (2020) to make it a very
> large number that the GPS time will never
> exceed. This should lock in a 2019 year and
> correctly calculated number of days that can
> easily be converted (as MAESTRO had been doing
> automatically since the start of 2020).
>
> # Neither of these options involve changing the
> table length and both should be achievable via RAM
> poke
> # We need some more time to do our research on this,
> figure out exactly what memory locations we’re
> talking about.
>
> o need to test on Flatsat which has not been used for 5 or
> more years
>
> + A poke wouldn’t necessarily need to be tested on
> FlatSat. We’d start by reading the RAM at that
> location to verify we’re touching the right part of
> memory, abort if not confirmed. Then we poke just that
> same location. Since this is a RAM poke, we can always
> reboot if something goes wrong and come up cleanly (in
> 1980, a time that would be correctly interpreted),
> clearing out the poked RAM.
> + NG is already working to get Flatsat up and running
>
> # Has NG been able to locate the password to gain
> access to the Flatsat machine?
>
> o could we poke memory locations rather than full table
> updates?
>
> + Are the memory locations beyond the timing table used
> for anything else? Could poking new values have
> unintended consequences?
> + I definitely would not want to poke beyond the table
> (thus, no extending the length of the table as an option)
>
> * Perform a Flight Computer Reset
>
> o hope that this corrects the memory that changed
>
> + I think this is probably our "Hail Mary" option
> + Only really makes sense if the test you propose below
> shows that now timestamps from a clock set to 3 weeks
> ago (prior to the recent speedbump) also turn out bad.
>
> * Set the on board spacecraft time to current time minus 30 years
>
> o this "Back to the Future" approach would put the time into
> the middle of the on board table (it starts at 1980)
> o would result in unique time stamps for archive purposes
> (unless the mission lasts more than 30 years)
> o would need to update MPS code to subtract 30 years from
> ATS command times or manually update each ATS file
> o could perform all MPS planning with current date so that
> flight dynamics processes would not be impacted
>
> + This may work but I'd like to make sure that going
> back in time results in MAESTRO producing good
> secondary header time stamps again
>
> * Another Path?
>
> o What have we missed?
>
> One hypothesis is that the content of the memory just beyond the
> GPS timing table changed on 24 August. However, Ken points out
> that it's possible the content didn't change but the value is such
> that the time comparison wasn't exceeded until 24 August,
> thereafter resulting in further comparisons that were exceeded for
> another 6 locations until the location corresponding to year 2026
> contained a value greater than the current time. If that's the
> case, then setting the spacecraft time back to just before the 24
> August time stamp jump would result in times that MAESTRO could
> again process correctly, followed by another jump back to 1970.
> That's a test that I think I'd like to try since I'd really like
> to know if this issue is caused by (random) memory content changes
> or simply because a static memory value has been exceeded.
>
> Ryan points out that we'd need to stop the on board ATS if we did
> a "back in time" test. The orbit 540 ATS runs until 2021/09/14
> 08:30:00 and the orbit 541 ATS starts at 2021/09:14 14:43:12. We
> could add a contact at around 2021/09/14 13:00-14:00 during which
> we'd delete the orbit 541 commands, set the spacecraft time to 24
> August, watch for the time stamp jump, reset the spacecraft clock
> to NOW and reload the orbit 541 ATS (and orbit 542 ATS, for that
> matter). How does that sound? Can you think of any unintended
> consequences? Can we plan this for next Tuesday at ~09:00 EDT
> without stressing the system?
>
> I’m in favor of such a test, but it might prove unnecessary if we
> can establish via memory map (and possibly a RAM read command)
> that Ken’s hypothesis is correct.
>
>
>
> Feedback please!
>
> Thanks as always,
> Nigel
>
>
>
> Please restrict discussions on this email list to non-ITAR sensitive topics.
> ______________________________________________
> Isocops mailing list
> Isocops at lists.sr.unh.edu
> https://lists.sr.unh.edu/mailman/listinfo/isocops
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.sr.unh.edu/pipermail/isocops/attachments/20210915/fbe5c635/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 98794 bytes
Desc: not available
URL: <https://lists.sr.unh.edu/pipermail/isocops/attachments/20210915/fbe5c635/image-0001.png>
More information about the Isocops
mailing list