Red Alarm Analysis, MIPS Campaign A
Analysts: Tom Glenn and Douglas Kelly
Last Updated: 9/9/03
MIPS registered a large number of red alarms during campaign A.
The following is a detailed review of these alarms.
A large number of alarms appeared at time 03-245T01:21:47.230. This
coincides with the OFF_BOOT state transition. There is a short
time during this state transition when the CEState is indeterminate,
the telemetry values are invalid, and a large number of false red
alarms appears. The same is true during the BOOT_OPR transition
at 03-245T01:26:11.230, the MIPS_OPR_MRDY transition at 03/245-01:32:55.328,
and the MIPS_MRDY_OPR state transition at 03/245-02:57:19.289. I do
not have the timestamps for the OPR_BOOT transition during Campaign A2,
but false telemetry was expected at that time as well. The comments
below include information on when each of the telemetry values is valid.
These conditions should be incorporated into the alarm-checking software
to reduce the number of false alarms. The alarms addressed by these
state transition invalid telemetry episodes are:
I-0100 U D70OfstV_B (1 alarm entries found)
Alarm Entry: 746933588. 03/245-01:32:55.328 MIPS I-0100
U D70OfstV_B -5.10088e+00 0 REDLOW -1.000e-01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0104 U D70m5RefV (2 alarm entries found)
First Entry: 746933596. 03/245-01:33:03.289 MIPS I-0104
U D70m5RefV -4.28950e-03 32738 REDHIGH -4.960e+00
Last Entry: 746938648. 03/245-02:57:15.328 MIPS I-0104
U D70m5RefV -4.44518e-03 32737 REDHIGH -4.960e+00
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0105 U D70ILoadV (1 alarm entries found)
Alarm Entry: 746933588. 03/245-01:32:55.328 MIPS I-0105
U D70ILoadV 1.02018e+01 0 REDHIGH 4.000e-02
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0106 U D70AnalogRtn (1 alarm entries found)
Alarm Entry: 746933592. 03/245-01:32:59.289 MIPS I-0106
U D70AnalogRtn -5.10088e+00 0 REDLOW -5.000e-02
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0107 U CSM_PosOut (2 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 MIPS I-0107
U CSM_PosOut -3.38934e+04 0 REDLOW 5587
Last Entry: 746933592. 03/245-01:32:59.289 MIPS I-0107
U CSM_PosOut -3.38934e+04 0 REDLOW 5587
Comments:
Only valid when I-0028 (CSM_DriveEn) = Enable
I-0110 U CSM_Current (2 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 MIPS I-0110
U CSM_Current -3.40292e+04 0 REDLOW -2.100e+04
Last Entry: 746933592. 03/245-01:32:59.289 MIPS I-0110
U CSM_Current -3.40292e+04 0 REDLOW -2.100e+04
Comments:
Only valid when I-0028 (CSM_DriveEn) = Enable
I-0114 U D160OfstV_B (1 alarm entries found)
Alarm Entry: 746933588. 03/245-01:32:55.328 MIPS I-0114
U D160OfstV_B -5.10088e+00 0 REDLOW -1.000e-01
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
I-0120 U D160IloadV (1 alarm entries found)
Alarm Entry: 746933588. 03/245-01:32:55.328 MIPS I-0120
U D160IloadV 1.02018e+01 0 REDHIGH 4.000e-02
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
I-0121 U D160m5RefV (1 alarm entries found)
Alarm Entry: 746933592. 03/245-01:32:59.289 MIPS I-0121
U D160m5RefV -5.10088e+00 0 REDLOW -5.040e+00
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
I-0124 U D70AnHtrCur (2 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 MIPS I-0124
U D70AnHtrCur 3.38562e+01 0 REDHIGH 2.000e+01
Last Entry: 746933592. 03/245-01:32:59.289 MIPS I-0124
U D70AnHtrCur 3.38562e+01 0 REDHIGH 2.000e+01
Comments:
I-0126 U D160AnHtrCur (2 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 MIPS I-0126
U D160AnHtrCur 3.38562e+01 0 REDHIGH 2.600e+01
Last Entry: 746933592. 03/245-01:32:59.289 MIPS I-0126
U D160AnHtrCur 3.38562e+01 0 REDHIGH 2.600e+01
Comments:
I-0128 U D160QrstHiV (1 alarm entries found)
Alarm Entry: 746933588. 03/245-01:32:55.328 MIPS I-0128
U D160QrstHiV 3.60093e+00 0 REDHIGH -1.480e+00
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
I-0200 U D160DetBiasV (2 alarm entries found)
First Entry: 746933596. 03/245-01:33:03.289 MIPS I-0200
U D160DetBiasV 7.25160e-01 32821 REDHIGH 3.010e-01
Last Entry: 746938648. 03/245-02:57:15.328 MIPS I-0200
U D160DetBiasV 7.27807e-01 32804 REDHIGH 3.010e-01
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
I-0201 U D160RstDV1 (1 alarm entries found)
Alarm Entry: 746933592. 03/245-01:32:59.289 MIPS I-0201
U D160RstDV1 4.36711e+00 0 REDHIGH -7.138e-01
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
I-0203 U D160RstDV3 (1 alarm entries found)
Alarm Entry: 746933588. 03/245-01:32:55.328 MIPS I-0203
U D160RstDV3 4.36711e+00 0 REDHIGH -7.138e-01
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
I-0204 U D160RstDV4 (1 alarm entries found)
Alarm Entry: 746933592. 03/245-01:32:59.289 MIPS I-0204
U D160RstDV4 4.36711e+00 0 REDHIGH -7.138e-01
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
I-0210 U D70DetBiasV (1 alarm entries found)
Alarm Entry: 746933592. 03/245-01:32:59.289 MIPS I-0210
U D70DetBiasV 5.83464e+00 0 REDHIGH 3.010e-01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0211 U D70RstDV1 (1 alarm entries found)
Alarm Entry: 746933588. 03/245-01:32:55.328 MIPS I-0211
U D70RstDV1 4.36711e+00 0 REDHIGH -7.138e-01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0212 U D70RstDV2 (1 alarm entries found)
Alarm Entry: 746933592. 03/245-01:32:59.289 MIPS I-0212
U D70RstDV2 4.36711e+00 0 REDHIGH -7.138e-01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0214 U D70RstDV4 (1 alarm entries found)
Alarm Entry: 746933588. 03/245-01:32:55.328 MIPS I-0214
U D70RstDV4 4.36711e+00 0 REDHIGH -7.138e-01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0215 U D70RstDV5 (1 alarm entries found)
Alarm Entry: 746933592. 03/245-01:32:59.289 MIPS I-0215
U D70RstDV5 4.36711e+00 0 REDHIGH -7.138e-01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0217 U D70RstDV7 (1 alarm entries found)
Alarm Entry: 746933588. 03/245-01:32:55.328 MIPS I-0217
U D70RstDV7 4.36711e+00 0 REDHIGH -7.138e-01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0218 U D70RstDV8 (1 alarm entries found)
Alarm Entry: 746933592. 03/245-01:32:59.289 MIPS I-0218
U D70RstDV8 4.36711e+00 0 REDHIGH -7.138e-01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
K-0037 U D70JnctTmp (1 alarm entries found)
Alarm Entry: 746933600. 03/245-01:33:07.285 CE K-0037
U D70JnctTmp 1.47154e+01 32775 REDHIGH 5.000e+00
Comments:
Only valid when I-0363 (TmpSensEn) = Enable
K-0039 U D70BaseTmp (1 alarm entries found)
Alarm Entry: 746938648. 03/245-02:57:15.328 CE K-0039
U D70BaseTmp 1.52857e+01 32731 REDHIGH 7.000e+00
Comments:
Only valid when I-0363 (TmpSensEn) = Enable
K-0040 U D160BaseTmp (1 alarm entries found)
Alarm Entry: 746938652. 03/245-02:57:19.289 CE K-0040
U D160BaseTmp 1.49569e+01 32748 REDHIGH 7.000e+00
Comments:
Only valid when I-0363 (TmpSensEn) = Enable
K-0059 U D24StimCur (2 alarm entries found)
First Entry: 746932916. 03/245-01:21:43.234 CE K-0059
U D24StimCur 1.80090e+04 0 REDHIGH 6.750e+03
Last Entry: 746933180. 03/245-01:26:07.238 CE K-0059
U D24StimCur 1.80090e+04 0 REDHIGH 6.750e+03
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0061 U SCInputVolt (2 alarm entries found)
First Entry: 746932916. 03/245-01:21:43.234 CE K-0061
U SCInputVolt -6.27820e+01 0 REDLOW 2.000e+01
Last Entry: 746933180. 03/245-01:26:07.238 CE K-0061
U SCInputVolt -6.27820e+01 0 REDLOW 2.000e+01
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0063 U CS5V (2 alarm entries found)
First Entry: 746932916. 03/245-01:21:43.234 CE K-0063
U CS5V -1.24810e+01 0 REDLOW 4.600e+00
Last Entry: 746933180. 03/245-01:26:07.238 CE K-0063
U CS5V -1.24810e+01 0 REDLOW 4.600e+00
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0064 U IES5V (2 alarm entries found)
First Entry: 746932916. 03/245-01:21:43.234 CE K-0064
U IES5V -1.24810e+01 0 REDLOW 4.500e+00
Last Entry: 746933180. 03/245-01:26:07.238 CE K-0064
U IES5V -1.24810e+01 0 REDLOW 4.500e+00
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0065 U IES15V (2 alarm entries found)
First Entry: 746932916. 03/245-01:21:43.234 CE K-0065
U IES15V -2.12290e+01 0 REDLOW 1.350e+01
Last Entry: 746933180. 03/245-01:26:07.238 CE K-0065
U IES15V -2.12290e+01 0 REDLOW 1.350e+01
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0066 U IESm15V (2 alarm entries found)
First Entry: 746932920. 03/245-01:21:47.230 CE K-0066
U IESm15V -2.12290e+01 0 REDLOW -1.650e+01
Last Entry: 746933184. 03/245-01:26:11.230 CE K-0066
U IESm15V -2.12290e+01 0 REDLOW -1.650e+01
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0067 U SiDet5VDRef (2 alarm entries found)
First Entry: 746932920. 03/245-01:21:47.230 CE K-0067
U SiDet5VDRef -1.24810e+01 0 REDLOW 4.600e+00
Last Entry: 746933184. 03/245-01:26:11.230 CE K-0067
U SiDet5VDRef -1.24810e+01 0 REDLOW 4.600e+00
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0068 U SLHeaterCur (2 alarm entries found)
First Entry: 746932920. 03/245-01:21:47.230 CE K-0068
U SLHeaterCur 2.07197e+02 0 REDHIGH 1.370e+02
Last Entry: 746933184. 03/245-01:26:11.230 CE K-0068
U SLHeaterCur 2.07197e+02 0 REDHIGH 1.370e+02
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0069 U SHHeaterCur (2 alarm entries found)
First Entry: 746932920. 03/245-01:21:47.230 CE K-0069
U SHHeaterCur 2.07197e+02 0 REDHIGH 1.370e+02
Last Entry: 746933184. 03/245-01:26:11.230 CE K-0069
U SHHeaterCur 2.07197e+02 0 REDHIGH 1.370e+02
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0070 U LLHeaterCur (2 alarm entries found)
First Entry: 746932920. 03/245-01:21:47.230 CE K-0070
U LLHeaterCur 1.75863e+02 0 REDHIGH 1.100e+02
Last Entry: 746933184. 03/245-01:26:11.230 CE K-0070
U LLHeaterCur 1.75863e+02 0 REDHIGH 1.100e+02
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0071 U LHHeaterCur (2 alarm entries found)
First Entry: 746932924. 03/245-01:21:51.230 CE K-0071
U LHHeaterCur 1.75863e+02 0 REDHIGH 1.100e+02
Last Entry: 746933188. 03/245-01:26:15.234 CE K-0071
U LHHeaterCur 1.75863e+02 0 REDHIGH 1.100e+02
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0072 U D24HeaterCur (2 alarm entries found)
First Entry: 746932924. 03/245-01:21:51.230 CE K-0072
U D24HeaterCur 2.40317e+02 0 REDHIGH 1.060e+02
Last Entry: 746933188. 03/245-01:26:15.234 CE K-0072
U D24HeaterCur 2.40317e+02 0 REDHIGH 1.060e+02
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0073 U IRSStimCur (2 alarm entries found)
First Entry: 746932924. 03/245-01:21:51.230 CE K-0073
U IRSStimCur 1.42000e+03 0 REDHIGH 5.600e+02
Last Entry: 746933188. 03/245-01:26:15.234 CE K-0073
U IRSStimCur 1.42000e+03 0 REDHIGH 5.600e+02
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0074 U SiSig5VARef (2 alarm entries found)
First Entry: 746932924. 03/245-01:21:51.230 CE K-0074
U SiSig5VARef 1.26060e+01 0 REDHIGH 5.400e+00
Last Entry: 746933188. 03/245-01:26:15.234 CE K-0074
U SiSig5VARef 1.26060e+01 0 REDHIGH 5.400e+00
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0107 S WatchDogBark (1 alarm entries found)
Alarm Entry: 746932924. 03/245-01:21:51.230 CE K-0107
S WatchDogBark Silent 00 REDMASK 0X00000001
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0108 S IESStatus (1 alarm entries found)
Alarm Entry: 746932924. 03/245-01:21:51.230 CE K-0108
S IESStatus Off 00 REDMASK 0X00000001
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
K-0255 S LimCheckStat (2 alarm entries found)
First Entry: 746932916. 03/245-01:21:43.234 CE K-0255
S LimCheckStat Disable 00 REDMASK 0X00000001
Last Entry: 746933180. 03/245-01:26:07.238 CE K-0255
S LimCheckStat Disable 00 REDMASK 0X00000001
Comments:
Power On transient. Submitted Red Alarm Update Form to update persistence to 3.
-----------------------------------------------------------------------------
K-1088 U AnD24AnnealC (1083 alarm entries found)
First Entry: 746935662. 03/245-02:07:29.762 CE K-1088
U AnD24AnnealC 1.12092e+03 18512 REDHIGH 7.620e+02
Last Entry: 746938157. 03/245-02:49:04.355 CE K-1088
U AnD24AnnealC 1.06787e+03 19181 REDHIGH 7.620e+02
Comments:
This item should be removed from the telemetry verification list. The
red alarm appears because of stale telemetry. The use of ancillary
telemetry is not helpful because we never collect science data while
the anneal heater is active.
------------------------------------------------------------------------------
For the following 4 telemetry items, the out-of-limits conditions are real,
and they are expected based on the very bright background conditions. In
these bright light conditions, the Ge detector arrays quickly reach saturation
and the unit cell current draw drops precipitously. This condition should
remedy itself as we approach the 70um first light campaign. These telemetry
items are critically diagnostic of other problems, so we want to keep them
in the alarm list. We also wish to retain the current red limits levels
with the awareness that these telemetry items will be out-of-limits in the
early MIPS campaigns. These alarms are not related to the 70um readout 44
problem.
I-0101 U D70DucCurA (1220 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 MIPS I-0101
U D70DucCurA 4.22790e+02 0 REDHIGH 2.250e+02
Last Entry: 746938652. 03/245-02:57:19.289 MIPS I-0101
U D70DucCurA -1.28407e+03 32688 REDLOW 2.000e+01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-0102 U D70DucCurB (1227 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 MIPS I-0102
U D70DucCurB 3.94121e+02 0 REDHIGH 2.250e+02
Last Entry: 746938652. 03/245-02:57:19.289 MIPS I-0102
U D70DucCurB -1.29793e+03 32713 REDLOW 2.000e+01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-1101 U AnD70DucCurA (781 alarm entries found)
First Entry: 746935662. 03/245-02:07:29.762 MIPS I-1101
U AnD70DucCurA 1.32020e+01 7844 REDLOW 2.000e+01
Last Entry: 746938157. 03/245-02:49:04.355 MIPS I-1101
U AnD70DucCurA 1.35675e+01 7837 REDLOW 2.000e+01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
I-1102 U AnD70DucCurB (877 alarm entries found)
First Entry: 746935662. 03/245-02:07:29.762 MIPS I-1102
U AnD70DucCurB 1.22931e+01 7382 REDLOW 2.000e+01
Last Entry: 746938157. 03/245-02:49:04.355 MIPS I-1102
U AnD70DucCurB 1.27586e+01 7373 REDLOW 2.000e+01
Comments:
Only valid when I-0022 (D70PowerEn) = PowerOn
------------------------------------------------------------------------------
Careful examination of the following two items shows that the only times
that they are out-of-limits is during the startup and shutdown telemetry
transients.
I-0111 U D160DucCurA (5 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 MIPS I-0111
U D160DucCurA 4.60809e+02 0 REDHIGH 1.000e+02
Last Entry: 746938652. 03/245-02:57:19.289 MIPS I-0111
U D160DucCurA -1.28902e+03 32546 REDLOW -1.400e+02
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
I-0112 U D160DucCurB (5 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 MIPS I-0112
U D160DucCurB 4.36366e+02 0 REDHIGH 1.000e+02
Last Entry: 746938652. 03/245-02:57:19.289 MIPS I-0112
U D160DucCurB -1.28931e+03 32549 REDLOW -1.400e+02
Comments:
Only valid when I-0023 (D160PowerEn) = PowerOn
------------------------------------------------------------------------------
I-1110 U AnCSM_Curr (649 alarm entries found)
First Entry: 746935662. 03/245-02:07:29.762 MIPS I-1110
U AnCSM_Curr -1.63798e+04 16994 REDLOW -2.100e+03
Last Entry: 746937765. 03/245-02:42:32.727 MIPS I-1110
U AnCSM_Curr -6.15300e+03 26841 REDLOW -2.100e+03
Comments:
The REDLOW limit should be -2.100e+04, not +03.
------------------------------------------------------------------------------
K-0035 U D24TmpA (23 alarm entries found)
First Entry: 746933596. 03/245-01:33:03.289 CE K-0035
U D24TmpA 5.45257e+01 32754 REDHIGH 1.000e+01
Last Entry: 746938648. 03/245-02:57:15.328 CE K-0035
U D24TmpA 5.45339e+01 32753 REDHIGH 1.000e+01
Comments:
Only valid when K-0334 (D24SiTmpSen) = Enabled
K-0036 U D24TmpB (23 alarm entries found)
First Entry: 746933596. 03/245-01:33:03.289 CE K-0036
U D24TmpB 5.29750e+01 32765 REDHIGH 1.000e+01
Last Entry: 746938648. 03/245-02:57:15.328 CE K-0036
U D24TmpB 5.30833e+01 32752 REDHIGH 1.000e+01
Comments:
Only valid when K-0334 (D24SiTmpSen) = Enabled
K-0041 U D70TmpA (6 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 CE K-0041
U D70TmpA 1.55126e+01 32740 REDHIGH 5.000e+00
Last Entry: 746938652. 03/245-02:57:19.289 CE K-0041
U D70TmpA 1.56471e+01 32732 REDHIGH 5.000e+00
Comments:
Only valid when K-0363 (TmpSensEn) = Enabled
K-0042 U D70TmpB (6 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 CE K-0042
U D70TmpB 1.58889e+01 32749 REDHIGH 5.000e+00
Last Entry: 746938652. 03/245-02:57:19.289 CE K-0042
U D70TmpB 1.60222e+01 32743 REDHIGH 5.000e+00
Comments:
Only valid when K-0363 (TmpSensEn) = Enabled
K-0043 U D160TmpA (6 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 CE K-0043
U D160TmpA 1.49132e+01 32733 REDHIGH 5.000e+00
Last Entry: 746938652. 03/245-02:57:19.289 CE K-0043
U D160TmpA 1.49434e+01 32729 REDHIGH 5.000e+00
Comments:
Only valid when K-0363 (TmpSensEn) = Enabled
K-0045 U D160TmpB (6 alarm entries found)
First Entry: 746933588. 03/245-01:32:55.328 CE K-0045
U D160TmpB 1.48974e+01 32746 REDHIGH 5.000e+00
Last Entry: 746938652. 03/245-02:57:19.289 CE K-0045
U D160TmpB 1.49915e+01 32735 REDHIGH 5.000e+00
Comments:
Only valid when K-0363 (TmpSensEn) = Enabled