NEAR Shoemaker suffered a nearly fatal failure on December 22nd, 1998, at 22:00
UTC, as the spacecraft was executing rendezvous burn 1 (RND1). The
failure caused mission control to lose contact with the craft for 27
hours, after which NEAR was found in it's lowest safe mode,
Sun-safe-rotate. Controllers were able to figure out a way to get NEAR
to it's target, the asteroid 443 Eros, but with a significant time
penalty and a very narrow fuel margin. The only permanent damage
suffered by the craft was a contamination of the multispectral
imager's optics by propellant residues.
Because of low voltage conditions experienced by the craft during
the time it was silent, power to the solid state memory recorder was
turned off. This means that detailed telemetry and command information
had been erased and were not available to help in the reconstruction of
events during the time NEAR was not in contact with the deep space
network. The events during this time had to be reconstructed from
limited data found in processor memories, and inferences about what
was happening to the craft.
Sequence of events
NEAR used a 200 second sequence of settling burns to settle the
liquid fuel used by the main engine in it's tanks. This process
completed successfully, after which the main engine was ignited. The
main engine shut down almost immediately, apparently without
cause. Thirty seven seconds later, communications with the craft were
lost.
After the large velocity adjust (LVA) engine shutdown, the craft's
guidance and control (G&C) computers began executing a maneuver that
would bring the craft to Earth-safe mode. In Earth-safe mode, the
craft aligns itself such that it's solar panels are facing the Sun and
so that the craft is rolling about the Sun line, with its medium gain
antenna pointing toward Earth. Instead of using reaction wheels to
perform this attitude adjustment, as per normal, the 22 N thrusters
were used as they are during a delta-v maneuver. This erroneous use of
the thrusters was caused by a command script which did not properly
clean up after the LVA engine abort; it should have returned attitude
control to the reaction wheels. The use of the thrusters resulted in a
body rate of 1 degree per second toward the Sun.
Twenty four seconds after the start of the LVA burn, a script
running in the crafts command processors shut off power to the
thrusters and closed the fuel tanks. With thruster power off, the
guidance and control (G&C) system automatically started using the
reaction wheels for attitude control. The reaction control wheels couldn't
stop the slew toward the Sun started by the thrusters, and the
spacecraft overshot the Sun.
The reaction control wheels were now saturated with momentum. The
command computers began a thirty minute warm up of the thruster
catalyst beds to prepare for a momentum dump (also called an
angular momentum desaturation maneuver, AMD). This was to be a "red"
dump, as opposed to a "white" dump, which is so urgent that the dump
begins immediately, without first heating the thruster catalyst
beds. The thirty minute warm up actually took thirty seven minutes,
due to a data structure error that cause a wheel speed sensor to be
interpreted as zero instead of the wheel's maximum speed.
The first momentum dump completed at T+00:27:52 (T=0 is RND1
start), 15 seconds short of the maximum allowed dump time. G&C
signaled for the command processor to shut off thruster power, and
then began a course correction to put the craft back into Sun-safe
mode, as it had drifted off during the momentum dump. A design flaw
caused a long handshaking process to take place between G&C and the
command processor, which meant thruster power was not turned off
immediately. Since G&C was still configured to use the 22 N thrusters
for attitude control after RND1, G&C fired the thrusters, which
imparted a lot of momentum to the craft. This momentum was high enough
to initiate a "white" dump, one that occurs without first warming the
thruster catalyst beds. This dump failed to complete before the 300
second timeout for a momentum dump, which caused a switch over to the
backup attitude interface unit (AIU).
The AIU switchover caused the attitude control to switch back to
reaction control wheels, effectively cleaning up the mess left when RND1
aborted. Up to this point, ground based simulations were able to
accurately predict the series of events after the burn abort. After the
AIU switchover and the subsequent return of attitude control to the
reaction wheels, the spacecraft should have been able to recover
unless some other mode of failure was present. However, five more
momentum dumps would occur before the spacecraft experienced a quiet
period, which was followed by eight more dumps.
The spacecraft continued to switch between AIUs because momentum
dumps weren't being completed in the 300 seconds allowed. After the
fifth switchover, autonomy rules left control in the hands of AIU2 and
stopped limiting the length of the momentum dumps. As a result, some
of the dumps were extremely long, one over 1000 seconds. During this
time, the craft experienced several low voltage sense (LVS) trips,
which resulted in power being shutoff to the solid state recorder. The
spacecraft's gyros are believed to have entered their whole angle mode
(WAM) during this time. The gyroscopes would have probably entered WAM
due to high body rates, which would have resulted in very noisy
attitude readings and would have made momentum dumps extremely
inefficient.
Eight more momentum dumps occurred over the next six hours. Beginning
around T+06:10:00, the spacecraft entered a two and a half hour quiet
period that ended at T+08:31:00 with one final "white" momentum
dump. No further anomalous activity would occur. NEAR was found in
Sun-safe-rotate twenty seven hours after the abort of RND1.
Causes
The NEAR RND1 burn anomaly was caused by bugs in the software used
to control the spacecraft.
- RND1 was aborted because the startup of the engine produced an
acceleration transient that exceeded a safety threshold that was set
to low.
- The abort of the burn caused recovery scripts to execute, which
were missing a command to return attitude control to reaction
wheels.
- A total of seventeen software errors, meaning errors in compiled
code, algorithms, and data structures were found.
- Lack of adherence to procedures and lack of top level procedures were
found to contribute to the problem. Suggestions of systems engineers
were often ignored, and the original software engineer was no longer
consulted about or asked to review code changes.
- Software and hardware simulators used to test command scripts were
found to be inadequate and difficult to use.
- The effect of stored momentum in liquids (propellant slosh)was
ignored by G&C software.
No hardware failures were found. A stuck thruster or a propellant
leak was theorized to have caused the lengthy recovery, but
simulations showed that this was unlikely.
Conclusion
The NEAR operations team were awarded for thier successful efforts
to recover the spacecraft. NEAR Shoemaker was able to continue its
mission and was in orbit around 433 Eros. It has since finished its
scientific mission and was put to rest on the surface of Eros. NEAR
Shoemaker was not designed to land, but was able to return some data
to Earth after doing so.
Researched from the NEAR website, and from
The NEAR Rendezvous Burn Anomaly of December 1998, final report of
the NEAR (Near Earth Asteroid Rendezvous) Anomaly Review Board.