Prev   Index   Next

kadi: Thu Sep 24 08:06:10 2026

File: /proj/sot/ska/data/kadi/logs/daily.0/kadi_events.log

Task Status AgeMaxAge Last Date
kadi NOT OK 0.111.0 Thu Sep 24 08:06:10 2026

Errors:

File contents:

2026-09-23 09:06:08,010 log_run_info: ******************************************
2026-09-23 09:06:08,010 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 09:06:08,010 log_run_info: Version: 7.22.0
2026-09-23 09:06:08,010 log_run_info: Time: Wed Sep 23 09:06:08 2026
2026-09-23 09:06:08,010 log_run_info: User: aldcroft
2026-09-23 09:06:08,010 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 09:06:08,010 log_run_info: Processing args:
2026-09-23 09:06:08,011 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 09:06:08,011 log_run_info: 'delete_from_start': False,
2026-09-23 09:06:08,011 log_run_info: 'log_level': 'INFO',
2026-09-23 09:06:08,011 log_run_info: 'loop_days': 100,
2026-09-23 09:06:08,011 log_run_info: 'maude': True,
2026-09-23 09:06:08,011 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 09:06:08,011 log_run_info: 'models': None,
2026-09-23 09:06:08,011 log_run_info: 'start': None,
2026-09-23 09:06:08,011 log_run_info: 'stop': '2026:266:13:06:07.987'}
2026-09-23 09:06:08,011 log_run_info: ******************************************
2026-09-23 09:06:08,012 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 09:06:08,012 main:
2026-09-23 09:06:09,014 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2026:265:17:35:00.000'
2026-09-23 09:06:09,015 main: No new telemetry in MAUDE, skipping event updates
2026-09-23 10:06:07,289 log_run_info: ******************************************
2026-09-23 10:06:07,289 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 10:06:07,289 log_run_info: Version: 7.22.0
2026-09-23 10:06:07,289 log_run_info: Time: Wed Sep 23 10:06:07 2026
2026-09-23 10:06:07,289 log_run_info: User: aldcroft
2026-09-23 10:06:07,289 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 10:06:07,289 log_run_info: Processing args:
2026-09-23 10:06:07,289 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 10:06:07,289 log_run_info: 'delete_from_start': False,
2026-09-23 10:06:07,289 log_run_info: 'log_level': 'INFO',
2026-09-23 10:06:07,289 log_run_info: 'loop_days': 100,
2026-09-23 10:06:07,289 log_run_info: 'maude': True,
2026-09-23 10:06:07,289 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 10:06:07,289 log_run_info: 'models': None,
2026-09-23 10:06:07,289 log_run_info: 'start': None,
2026-09-23 10:06:07,289 log_run_info: 'stop': '2026:266:14:06:07.257'}
2026-09-23 10:06:07,289 log_run_info: ******************************************
2026-09-23 10:06:07,291 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 10:06:07,291 main:
2026-09-23 10:06:08,627 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2026:265:17:35:00.000'
2026-09-23 10:06:08,627 main: No new telemetry in MAUDE, skipping event updates
2026-09-23 11:06:07,509 log_run_info: ******************************************
2026-09-23 11:06:07,509 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 11:06:07,509 log_run_info: Version: 7.22.0
2026-09-23 11:06:07,509 log_run_info: Time: Wed Sep 23 11:06:07 2026
2026-09-23 11:06:07,509 log_run_info: User: aldcroft
2026-09-23 11:06:07,509 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 11:06:07,509 log_run_info: Processing args:
2026-09-23 11:06:07,509 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 11:06:07,509 log_run_info: 'delete_from_start': False,
2026-09-23 11:06:07,509 log_run_info: 'log_level': 'INFO',
2026-09-23 11:06:07,509 log_run_info: 'loop_days': 100,
2026-09-23 11:06:07,509 log_run_info: 'maude': True,
2026-09-23 11:06:07,509 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 11:06:07,509 log_run_info: 'models': None,
2026-09-23 11:06:07,509 log_run_info: 'start': None,
2026-09-23 11:06:07,509 log_run_info: 'stop': '2026:266:15:06:07.485'}
2026-09-23 11:06:07,509 log_run_info: ******************************************
2026-09-23 11:06:07,510 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 11:06:07,510 main:
2026-09-23 11:06:08,772 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2026:265:17:35:00.000'
2026-09-23 11:06:08,772 main: No new telemetry in MAUDE, skipping event updates
2026-09-23 12:06:06,706 log_run_info: ******************************************
2026-09-23 12:06:06,706 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 12:06:06,706 log_run_info: Version: 7.22.0
2026-09-23 12:06:06,707 log_run_info: Time: Wed Sep 23 12:06:06 2026
2026-09-23 12:06:06,707 log_run_info: User: aldcroft
2026-09-23 12:06:06,707 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 12:06:06,707 log_run_info: Processing args:
2026-09-23 12:06:06,707 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 12:06:06,707 log_run_info: 'delete_from_start': False,
2026-09-23 12:06:06,707 log_run_info: 'log_level': 'INFO',
2026-09-23 12:06:06,707 log_run_info: 'loop_days': 100,
2026-09-23 12:06:06,707 log_run_info: 'maude': True,
2026-09-23 12:06:06,707 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 12:06:06,707 log_run_info: 'models': None,
2026-09-23 12:06:06,707 log_run_info: 'start': None,
2026-09-23 12:06:06,707 log_run_info: 'stop': '2026:266:16:06:06.683'}
2026-09-23 12:06:06,707 log_run_info: ******************************************
2026-09-23 12:06:06,708 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 12:06:06,708 main:
2026-09-23 12:06:08,408 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2026:265:17:35:00.000'
2026-09-23 12:06:08,408 main: No new telemetry in MAUDE, skipping event updates
2026-09-23 13:06:07,143 log_run_info: ******************************************
2026-09-23 13:06:07,143 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 13:06:07,143 log_run_info: Version: 7.22.0
2026-09-23 13:06:07,143 log_run_info: Time: Wed Sep 23 13:06:07 2026
2026-09-23 13:06:07,143 log_run_info: User: aldcroft
2026-09-23 13:06:07,143 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 13:06:07,143 log_run_info: Processing args:
2026-09-23 13:06:07,143 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 13:06:07,143 log_run_info: 'delete_from_start': False,
2026-09-23 13:06:07,143 log_run_info: 'log_level': 'INFO',
2026-09-23 13:06:07,143 log_run_info: 'loop_days': 100,
2026-09-23 13:06:07,143 log_run_info: 'maude': True,
2026-09-23 13:06:07,143 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 13:06:07,143 log_run_info: 'models': None,
2026-09-23 13:06:07,143 log_run_info: 'start': None,
2026-09-23 13:06:07,143 log_run_info: 'stop': '2026:266:17:06:07.116'}
2026-09-23 13:06:07,143 log_run_info: ******************************************
2026-09-23 13:06:07,144 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 13:06:07,144 main:
2026-09-23 13:06:08,236 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2026:265:17:35:00.000'
2026-09-23 13:06:08,236 main: No new telemetry in MAUDE, skipping event updates
2026-09-23 14:06:07,901 log_run_info: ******************************************
2026-09-23 14:06:07,901 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 14:06:07,901 log_run_info: Version: 7.22.0
2026-09-23 14:06:07,901 log_run_info: Time: Wed Sep 23 14:06:07 2026
2026-09-23 14:06:07,901 log_run_info: User: aldcroft
2026-09-23 14:06:07,901 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 14:06:07,901 log_run_info: Processing args:
2026-09-23 14:06:07,901 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 14:06:07,901 log_run_info: 'delete_from_start': False,
2026-09-23 14:06:07,901 log_run_info: 'log_level': 'INFO',
2026-09-23 14:06:07,901 log_run_info: 'loop_days': 100,
2026-09-23 14:06:07,901 log_run_info: 'maude': True,
2026-09-23 14:06:07,901 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 14:06:07,901 log_run_info: 'models': None,
2026-09-23 14:06:07,901 log_run_info: 'start': None,
2026-09-23 14:06:07,901 log_run_info: 'stop': '2026:266:18:06:07.878'}
2026-09-23 14:06:07,901 log_run_info: ******************************************
2026-09-23 14:06:07,902 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 14:06:07,902 main:
2026-09-23 14:06:09,096 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2026:265:17:35:00.000'
2026-09-23 14:06:09,096 main: No new telemetry in MAUDE, skipping event updates
2026-09-23 15:06:07,523 log_run_info: ******************************************
2026-09-23 15:06:07,523 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 15:06:07,523 log_run_info: Version: 7.22.0
2026-09-23 15:06:07,523 log_run_info: Time: Wed Sep 23 15:06:07 2026
2026-09-23 15:06:07,523 log_run_info: User: aldcroft
2026-09-23 15:06:07,523 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 15:06:07,523 log_run_info: Processing args:
2026-09-23 15:06:07,523 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 15:06:07,523 log_run_info: 'delete_from_start': False,
2026-09-23 15:06:07,523 log_run_info: 'log_level': 'INFO',
2026-09-23 15:06:07,523 log_run_info: 'loop_days': 100,
2026-09-23 15:06:07,523 log_run_info: 'maude': True,
2026-09-23 15:06:07,523 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 15:06:07,523 log_run_info: 'models': None,
2026-09-23 15:06:07,523 log_run_info: 'start': None,
2026-09-23 15:06:07,524 log_run_info: 'stop': '2026:266:19:06:07.500'}
2026-09-23 15:06:07,524 log_run_info: ******************************************
2026-09-23 15:06:07,524 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 15:06:07,525 main:
Traceback (most recent call last):
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 199, in __getitem__
return super().__getitem__(item)
~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'version'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/proj/sot/ska3/flight/bin/kadi_update_events", line 8, in
sys.exit(main())
~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 539, in main
and not check_maude_server_has_new_telemetry(opt.stop)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 482, in check_maude_server_has_new_telemetry
date_last_telemetry = maude.get_last_backorbit_date(msids)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 1341, in get_last_backorbit_date
dat = get_msids(msids_5min, channel=channel, user=user, password=password)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 727, in get_msids
if msid.upper() not in MSIDS:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 130, in load_msids
msids = {msid: index for index, msid in MSID_FROM_INDEX.items()}
~~~~~~~~~~~~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 117, in load_msid_index
with open(get_data_filename("MSIDIDX", "txt"), "r") as fh:
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 92, in get_data_filename
version = MAUDE_SYSINFO["version"]
~~~~~~~~~~~~~^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 201, in __getitem__
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 156, in get_maude_sysinfo
raise IOError(
...<3 lines>...
)
OSError: request failed with status=502 for URL=https://telemetry.cfa.harvard.edu/maude/mrest/sysinfo.json and text=
502 Bad Gateway

502 Bad Gateway


nginx/1.11.13




2026-09-23 16:06:07,227 log_run_info: ******************************************
2026-09-23 16:06:07,227 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 16:06:07,227 log_run_info: Version: 7.22.0
2026-09-23 16:06:07,227 log_run_info: Time: Wed Sep 23 16:06:07 2026
2026-09-23 16:06:07,227 log_run_info: User: aldcroft
2026-09-23 16:06:07,227 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 16:06:07,227 log_run_info: Processing args:
2026-09-23 16:06:07,227 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 16:06:07,227 log_run_info: 'delete_from_start': False,
2026-09-23 16:06:07,227 log_run_info: 'log_level': 'INFO',
2026-09-23 16:06:07,227 log_run_info: 'loop_days': 100,
2026-09-23 16:06:07,227 log_run_info: 'maude': True,
2026-09-23 16:06:07,227 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 16:06:07,227 log_run_info: 'models': None,
2026-09-23 16:06:07,227 log_run_info: 'start': None,
2026-09-23 16:06:07,227 log_run_info: 'stop': '2026:266:20:06:07.203'}
2026-09-23 16:06:07,227 log_run_info: ******************************************
2026-09-23 16:06:07,228 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 16:06:07,228 main:
Traceback (most recent call last):
File "/proj/sot/ska3/flight/bin/kadi_update_events", line 8, in
sys.exit(main())
~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 539, in main
and not check_maude_server_has_new_telemetry(opt.stop)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 482, in check_maude_server_has_new_telemetry
date_last_telemetry = maude.get_last_backorbit_date(msids)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 1341, in get_last_backorbit_date
dat = get_msids(msids_5min, channel=channel, user=user, password=password)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 774, in get_msids
raise IOError(
...<3 lines>...
)
OSError: request failed with status=500 for URL=https://telemetry.cfa.harvard.edu/maude/mrest/FLIGHT/msid.bin?m=STAT_5MIN_COUNT_AOTARQT2&m=STAT_5MIN_COUNT_COBSRQID&m=STAT_5MIN_COUNT_AOATTER1&m=STAT_5MIN_COUNT_3TSCMOVE&m=STAT_5MIN_COUNT_AOTARQT1&m=STAT_5MIN_COUNT_AOPSACPR&m=STAT_5MIN_COUNT_4LPOSARO&m=STAT_5MIN_COUNT_3MRMMXMV&m=STAT_5MIN_COUNT_AOUNLOAD&m=STAT_5MIN_COUNT_3TSCPOS&m=STAT_5MIN_COUNT_AOATTER2&m=STAT_5MIN_COUNT_AOACASEQ&m=STAT_5MIN_COUNT_AOECLIPS&m=STAT_5MIN_COUNT_4MP28AV&m=STAT_5MIN_COUNT_CIUMACAC&m=STAT_5MIN_COUNT_AOTARQT3&m=STAT_5MIN_COUNT_C1SQAX&m=STAT_5MIN_COUNT_3FAMOVE&m=STAT_5MIN_COUNT_AOPCADMD&m=STAT_5MIN_COUNT_AOATTER3&m=STAT_5MIN_COUNT_AOFATTMD&m=STAT_5MIN_COUNT_3FAPOS&m=STAT_5MIN_COUNT_4HPOSARO&m=STAT_5MIN_COUNT_CTUFMTSL&m=STAT_5MIN_COUNT_AORWBIAS&m=STAT_5MIN_COUNT_CONLOFP and text=
2026-09-23 17:06:29,820 log_run_info: ******************************************
2026-09-23 17:06:29,820 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 17:06:29,820 log_run_info: Version: 7.22.0
2026-09-23 17:06:29,820 log_run_info: Time: Wed Sep 23 17:06:29 2026
2026-09-23 17:06:29,820 log_run_info: User: aldcroft
2026-09-23 17:06:29,820 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 17:06:29,820 log_run_info: Processing args:
2026-09-23 17:06:29,820 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 17:06:29,820 log_run_info: 'delete_from_start': False,
2026-09-23 17:06:29,820 log_run_info: 'log_level': 'INFO',
2026-09-23 17:06:29,820 log_run_info: 'loop_days': 100,
2026-09-23 17:06:29,820 log_run_info: 'maude': True,
2026-09-23 17:06:29,820 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 17:06:29,820 log_run_info: 'models': None,
2026-09-23 17:06:29,820 log_run_info: 'start': None,
2026-09-23 17:06:29,820 log_run_info: 'stop': '2026:266:21:06:29.764'}
2026-09-23 17:06:29,820 log_run_info: ******************************************
2026-09-23 17:06:29,824 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 17:06:29,824 main:
Traceback (most recent call last):
File "/proj/sot/ska3/flight/bin/kadi_update_events", line 8, in
sys.exit(main())
~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 539, in main
and not check_maude_server_has_new_telemetry(opt.stop)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 482, in check_maude_server_has_new_telemetry
date_last_telemetry = maude.get_last_backorbit_date(msids)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 1341, in get_last_backorbit_date
dat = get_msids(msids_5min, channel=channel, user=user, password=password)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 774, in get_msids
raise IOError(
...<3 lines>...
)
OSError: request failed with status=500 for URL=https://telemetry.cfa.harvard.edu/maude/mrest/FLIGHT/msid.bin?m=STAT_5MIN_COUNT_AOATTER3&m=STAT_5MIN_COUNT_AOTARQT2&m=STAT_5MIN_COUNT_3TSCPOS&m=STAT_5MIN_COUNT_AOPSACPR&m=STAT_5MIN_COUNT_4MP28AV&m=STAT_5MIN_COUNT_COBSRQID&m=STAT_5MIN_COUNT_CTUFMTSL&m=STAT_5MIN_COUNT_AOUNLOAD&m=STAT_5MIN_COUNT_4LPOSARO&m=STAT_5MIN_COUNT_AOTARQT3&m=STAT_5MIN_COUNT_3FAPOS&m=STAT_5MIN_COUNT_AOPCADMD&m=STAT_5MIN_COUNT_AORWBIAS&m=STAT_5MIN_COUNT_3FAMOVE&m=STAT_5MIN_COUNT_AOECLIPS&m=STAT_5MIN_COUNT_4HPOSARO&m=STAT_5MIN_COUNT_AOATTER2&m=STAT_5MIN_COUNT_3TSCMOVE&m=STAT_5MIN_COUNT_AOATTER1&m=STAT_5MIN_COUNT_CIUMACAC&m=STAT_5MIN_COUNT_3MRMMXMV&m=STAT_5MIN_COUNT_AOACASEQ&m=STAT_5MIN_COUNT_CONLOFP&m=STAT_5MIN_COUNT_AOFATTMD&m=STAT_5MIN_COUNT_AOTARQT1&m=STAT_5MIN_COUNT_C1SQAX and text=
2026-09-23 18:06:09,419 log_run_info: ******************************************
2026-09-23 18:06:09,419 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 18:06:09,419 log_run_info: Version: 7.22.0
2026-09-23 18:06:09,419 log_run_info: Time: Wed Sep 23 18:06:09 2026
2026-09-23 18:06:09,419 log_run_info: User: aldcroft
2026-09-23 18:06:09,419 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 18:06:09,419 log_run_info: Processing args:
2026-09-23 18:06:09,419 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 18:06:09,419 log_run_info: 'delete_from_start': False,
2026-09-23 18:06:09,419 log_run_info: 'log_level': 'INFO',
2026-09-23 18:06:09,419 log_run_info: 'loop_days': 100,
2026-09-23 18:06:09,419 log_run_info: 'maude': True,
2026-09-23 18:06:09,419 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 18:06:09,419 log_run_info: 'models': None,
2026-09-23 18:06:09,419 log_run_info: 'start': None,
2026-09-23 18:06:09,419 log_run_info: 'stop': '2026:266:22:06:09.383'}
2026-09-23 18:06:09,420 log_run_info: ******************************************
2026-09-23 18:06:09,421 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 18:06:09,421 main:
2026-09-23 18:06:13,423 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-23 18:06:13,423 main: No new telemetry in MAUDE, skipping event updates
2026-09-23 19:06:08,376 log_run_info: ******************************************
2026-09-23 19:06:08,376 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 19:06:08,377 log_run_info: Version: 7.22.0
2026-09-23 19:06:08,377 log_run_info: Time: Wed Sep 23 19:06:08 2026
2026-09-23 19:06:08,377 log_run_info: User: aldcroft
2026-09-23 19:06:08,377 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 19:06:08,377 log_run_info: Processing args:
2026-09-23 19:06:08,377 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 19:06:08,377 log_run_info: 'delete_from_start': False,
2026-09-23 19:06:08,377 log_run_info: 'log_level': 'INFO',
2026-09-23 19:06:08,377 log_run_info: 'loop_days': 100,
2026-09-23 19:06:08,377 log_run_info: 'maude': True,
2026-09-23 19:06:08,377 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 19:06:08,377 log_run_info: 'models': None,
2026-09-23 19:06:08,377 log_run_info: 'start': None,
2026-09-23 19:06:08,377 log_run_info: 'stop': '2026:266:23:06:08.348'}
2026-09-23 19:06:08,377 log_run_info: ******************************************
2026-09-23 19:06:08,378 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 19:06:08,378 main:
Traceback (most recent call last):
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 199, in __getitem__
return super().__getitem__(item)
~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'version'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/proj/sot/ska3/flight/bin/kadi_update_events", line 8, in
sys.exit(main())
~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 539, in main
and not check_maude_server_has_new_telemetry(opt.stop)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 482, in check_maude_server_has_new_telemetry
date_last_telemetry = maude.get_last_backorbit_date(msids)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 1341, in get_last_backorbit_date
dat = get_msids(msids_5min, channel=channel, user=user, password=password)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 727, in get_msids
if msid.upper() not in MSIDS:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 130, in load_msids
msids = {msid: index for index, msid in MSID_FROM_INDEX.items()}
~~~~~~~~~~~~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 117, in load_msid_index
with open(get_data_filename("MSIDIDX", "txt"), "r") as fh:
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 92, in get_data_filename
version = MAUDE_SYSINFO["version"]
~~~~~~~~~~~~~^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 201, in __getitem__
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 156, in get_maude_sysinfo
raise IOError(
...<3 lines>...
)
OSError: request failed with status=502 for URL=https://telemetry.cfa.harvard.edu/maude/mrest/sysinfo.json and text=
502 Bad Gateway

502 Bad Gateway


nginx/1.11.13




2026-09-23 20:06:08,380 log_run_info: ******************************************
2026-09-23 20:06:08,380 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 20:06:08,380 log_run_info: Version: 7.22.0
2026-09-23 20:06:08,380 log_run_info: Time: Wed Sep 23 20:06:08 2026
2026-09-23 20:06:08,380 log_run_info: User: aldcroft
2026-09-23 20:06:08,380 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 20:06:08,380 log_run_info: Processing args:
2026-09-23 20:06:08,380 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 20:06:08,380 log_run_info: 'delete_from_start': False,
2026-09-23 20:06:08,380 log_run_info: 'log_level': 'INFO',
2026-09-23 20:06:08,380 log_run_info: 'loop_days': 100,
2026-09-23 20:06:08,380 log_run_info: 'maude': True,
2026-09-23 20:06:08,380 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 20:06:08,380 log_run_info: 'models': None,
2026-09-23 20:06:08,380 log_run_info: 'start': None,
2026-09-23 20:06:08,380 log_run_info: 'stop': '2026:267:00:06:08.350'}
2026-09-23 20:06:08,380 log_run_info: ******************************************
2026-09-23 20:06:08,381 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 20:06:08,381 main:
Traceback (most recent call last):
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 199, in __getitem__
return super().__getitem__(item)
~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'version'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/proj/sot/ska3/flight/bin/kadi_update_events", line 8, in
sys.exit(main())
~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 539, in main
and not check_maude_server_has_new_telemetry(opt.stop)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 482, in check_maude_server_has_new_telemetry
date_last_telemetry = maude.get_last_backorbit_date(msids)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 1341, in get_last_backorbit_date
dat = get_msids(msids_5min, channel=channel, user=user, password=password)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 727, in get_msids
if msid.upper() not in MSIDS:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 130, in load_msids
msids = {msid: index for index, msid in MSID_FROM_INDEX.items()}
~~~~~~~~~~~~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 117, in load_msid_index
with open(get_data_filename("MSIDIDX", "txt"), "r") as fh:
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 92, in get_data_filename
version = MAUDE_SYSINFO["version"]
~~~~~~~~~~~~~^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 201, in __getitem__
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 156, in get_maude_sysinfo
raise IOError(
...<3 lines>...
)
OSError: request failed with status=502 for URL=https://telemetry.cfa.harvard.edu/maude/mrest/sysinfo.json and text=
502 Bad Gateway

502 Bad Gateway


nginx/1.11.13




2026-09-23 21:06:09,813 log_run_info: ******************************************
2026-09-23 21:06:09,813 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 21:06:09,813 log_run_info: Version: 7.22.0
2026-09-23 21:06:09,813 log_run_info: Time: Wed Sep 23 21:06:09 2026
2026-09-23 21:06:09,813 log_run_info: User: aldcroft
2026-09-23 21:06:09,813 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 21:06:09,813 log_run_info: Processing args:
2026-09-23 21:06:09,813 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 21:06:09,813 log_run_info: 'delete_from_start': False,
2026-09-23 21:06:09,813 log_run_info: 'log_level': 'INFO',
2026-09-23 21:06:09,813 log_run_info: 'loop_days': 100,
2026-09-23 21:06:09,813 log_run_info: 'maude': True,
2026-09-23 21:06:09,813 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 21:06:09,813 log_run_info: 'models': None,
2026-09-23 21:06:09,813 log_run_info: 'start': None,
2026-09-23 21:06:09,813 log_run_info: 'stop': '2026:267:01:06:09.760'}
2026-09-23 21:06:09,813 log_run_info: ******************************************
2026-09-23 21:06:09,815 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 21:06:09,815 main:
Traceback (most recent call last):
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 199, in __getitem__
return super().__getitem__(item)
~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'version'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/proj/sot/ska3/flight/bin/kadi_update_events", line 8, in
sys.exit(main())
~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 539, in main
and not check_maude_server_has_new_telemetry(opt.stop)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py", line 482, in check_maude_server_has_new_telemetry
date_last_telemetry = maude.get_last_backorbit_date(msids)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 1341, in get_last_backorbit_date
dat = get_msids(msids_5min, channel=channel, user=user, password=password)
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 727, in get_msids
if msid.upper() not in MSIDS:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 130, in load_msids
msids = {msid: index for index, msid in MSID_FROM_INDEX.items()}
~~~~~~~~~~~~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 87, in wrapper
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 117, in load_msid_index
with open(get_data_filename("MSIDIDX", "txt"), "r") as fh:
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 92, in get_data_filename
version = MAUDE_SYSINFO["version"]
~~~~~~~~~~~~~^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 201, in __getitem__
self._load()
~~~~~~~~~~^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/ska_helpers/utils.py", line 189, in _load
self.update(self._load_func(*self._args, **self._kwargs))
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/sot/ska3/flight/lib/python3.13/site-packages/maude/maude.py", line 156, in get_maude_sysinfo
raise IOError(
...<3 lines>...
)
OSError: request failed with status=502 for URL=https://telemetry.cfa.harvard.edu/maude/mrest/sysinfo.json and text=
502 Bad Gateway

502 Bad Gateway


nginx/1.11.13




2026-09-23 22:06:08,456 log_run_info: ******************************************
2026-09-23 22:06:08,456 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 22:06:08,456 log_run_info: Version: 7.22.0
2026-09-23 22:06:08,456 log_run_info: Time: Wed Sep 23 22:06:08 2026
2026-09-23 22:06:08,456 log_run_info: User: aldcroft
2026-09-23 22:06:08,456 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 22:06:08,456 log_run_info: Processing args:
2026-09-23 22:06:08,456 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 22:06:08,456 log_run_info: 'delete_from_start': False,
2026-09-23 22:06:08,456 log_run_info: 'log_level': 'INFO',
2026-09-23 22:06:08,456 log_run_info: 'loop_days': 100,
2026-09-23 22:06:08,456 log_run_info: 'maude': True,
2026-09-23 22:06:08,456 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 22:06:08,456 log_run_info: 'models': None,
2026-09-23 22:06:08,456 log_run_info: 'start': None,
2026-09-23 22:06:08,456 log_run_info: 'stop': '2026:267:02:06:08.428'}
2026-09-23 22:06:08,456 log_run_info: ******************************************
2026-09-23 22:06:08,457 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 22:06:08,457 main:
2026-09-23 22:06:09,662 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-23 22:06:09,662 main: No new telemetry in MAUDE, skipping event updates
2026-09-23 23:06:08,361 log_run_info: ******************************************
2026-09-23 23:06:08,361 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-23 23:06:08,361 log_run_info: Version: 7.22.0
2026-09-23 23:06:08,361 log_run_info: Time: Wed Sep 23 23:06:08 2026
2026-09-23 23:06:08,361 log_run_info: User: aldcroft
2026-09-23 23:06:08,361 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-23 23:06:08,361 log_run_info: Processing args:
2026-09-23 23:06:08,361 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-23 23:06:08,361 log_run_info: 'delete_from_start': False,
2026-09-23 23:06:08,361 log_run_info: 'log_level': 'INFO',
2026-09-23 23:06:08,361 log_run_info: 'loop_days': 100,
2026-09-23 23:06:08,361 log_run_info: 'maude': True,
2026-09-23 23:06:08,361 log_run_info: 'maude_tlm_lookback': 7,
2026-09-23 23:06:08,361 log_run_info: 'models': None,
2026-09-23 23:06:08,361 log_run_info: 'start': None,
2026-09-23 23:06:08,361 log_run_info: 'stop': '2026:267:03:06:08.332'}
2026-09-23 23:06:08,361 log_run_info: ******************************************
2026-09-23 23:06:08,362 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-23 23:06:08,362 main:
2026-09-23 23:06:10,283 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-23 23:06:10,283 main: No new telemetry in MAUDE, skipping event updates
2026-09-24 00:06:10,539 log_run_info: ******************************************
2026-09-24 00:06:10,539 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-24 00:06:10,539 log_run_info: Version: 7.22.0
2026-09-24 00:06:10,539 log_run_info: Time: Thu Sep 24 00:06:10 2026
2026-09-24 00:06:10,539 log_run_info: User: aldcroft
2026-09-24 00:06:10,539 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-24 00:06:10,539 log_run_info: Processing args:
2026-09-24 00:06:10,539 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-24 00:06:10,539 log_run_info: 'delete_from_start': False,
2026-09-24 00:06:10,539 log_run_info: 'log_level': 'INFO',
2026-09-24 00:06:10,539 log_run_info: 'loop_days': 100,
2026-09-24 00:06:10,539 log_run_info: 'maude': True,
2026-09-24 00:06:10,539 log_run_info: 'maude_tlm_lookback': 7,
2026-09-24 00:06:10,539 log_run_info: 'models': None,
2026-09-24 00:06:10,539 log_run_info: 'start': None,
2026-09-24 00:06:10,539 log_run_info: 'stop': '2026:267:04:06:10.485'}
2026-09-24 00:06:10,539 log_run_info: ******************************************
2026-09-24 00:06:10,540 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-24 00:06:10,540 main:
2026-09-24 00:06:16,028 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-24 00:06:16,028 main: No new telemetry in MAUDE, skipping event updates
2026-09-24 01:06:09,505 log_run_info: ******************************************
2026-09-24 01:06:09,505 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-24 01:06:09,505 log_run_info: Version: 7.22.0
2026-09-24 01:06:09,505 log_run_info: Time: Thu Sep 24 01:06:09 2026
2026-09-24 01:06:09,505 log_run_info: User: aldcroft
2026-09-24 01:06:09,505 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-24 01:06:09,505 log_run_info: Processing args:
2026-09-24 01:06:09,505 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-24 01:06:09,505 log_run_info: 'delete_from_start': False,
2026-09-24 01:06:09,505 log_run_info: 'log_level': 'INFO',
2026-09-24 01:06:09,505 log_run_info: 'loop_days': 100,
2026-09-24 01:06:09,506 log_run_info: 'maude': True,
2026-09-24 01:06:09,506 log_run_info: 'maude_tlm_lookback': 7,
2026-09-24 01:06:09,506 log_run_info: 'models': None,
2026-09-24 01:06:09,506 log_run_info: 'start': None,
2026-09-24 01:06:09,506 log_run_info: 'stop': '2026:267:05:06:09.462'}
2026-09-24 01:06:09,506 log_run_info: ******************************************
2026-09-24 01:06:09,509 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-24 01:06:09,509 main:
2026-09-24 01:06:12,426 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-24 01:06:12,426 main: No new telemetry in MAUDE, skipping event updates
2026-09-24 02:06:08,775 log_run_info: ******************************************
2026-09-24 02:06:08,775 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-24 02:06:08,775 log_run_info: Version: 7.22.0
2026-09-24 02:06:08,775 log_run_info: Time: Thu Sep 24 02:06:08 2026
2026-09-24 02:06:08,775 log_run_info: User: aldcroft
2026-09-24 02:06:08,775 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-24 02:06:08,775 log_run_info: Processing args:
2026-09-24 02:06:08,775 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-24 02:06:08,776 log_run_info: 'delete_from_start': False,
2026-09-24 02:06:08,776 log_run_info: 'log_level': 'INFO',
2026-09-24 02:06:08,776 log_run_info: 'loop_days': 100,
2026-09-24 02:06:08,776 log_run_info: 'maude': True,
2026-09-24 02:06:08,776 log_run_info: 'maude_tlm_lookback': 7,
2026-09-24 02:06:08,776 log_run_info: 'models': None,
2026-09-24 02:06:08,776 log_run_info: 'start': None,
2026-09-24 02:06:08,776 log_run_info: 'stop': '2026:267:06:06:08.745'}
2026-09-24 02:06:08,776 log_run_info: ******************************************
2026-09-24 02:06:08,777 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-24 02:06:08,777 main:
2026-09-24 02:06:10,156 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-24 02:06:10,156 main: No new telemetry in MAUDE, skipping event updates
2026-09-24 03:06:07,858 log_run_info: ******************************************
2026-09-24 03:06:07,858 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-24 03:06:07,858 log_run_info: Version: 7.22.0
2026-09-24 03:06:07,858 log_run_info: Time: Thu Sep 24 03:06:07 2026
2026-09-24 03:06:07,858 log_run_info: User: aldcroft
2026-09-24 03:06:07,858 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-24 03:06:07,858 log_run_info: Processing args:
2026-09-24 03:06:07,858 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-24 03:06:07,858 log_run_info: 'delete_from_start': False,
2026-09-24 03:06:07,858 log_run_info: 'log_level': 'INFO',
2026-09-24 03:06:07,858 log_run_info: 'loop_days': 100,
2026-09-24 03:06:07,858 log_run_info: 'maude': True,
2026-09-24 03:06:07,858 log_run_info: 'maude_tlm_lookback': 7,
2026-09-24 03:06:07,858 log_run_info: 'models': None,
2026-09-24 03:06:07,858 log_run_info: 'start': None,
2026-09-24 03:06:07,858 log_run_info: 'stop': '2026:267:07:06:07.786'}
2026-09-24 03:06:07,858 log_run_info: ******************************************
2026-09-24 03:06:07,864 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-24 03:06:07,864 main:
2026-09-24 03:06:09,385 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-24 03:06:09,385 main: No new telemetry in MAUDE, skipping event updates
2026-09-24 04:06:06,804 log_run_info: ******************************************
2026-09-24 04:06:06,804 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-24 04:06:06,804 log_run_info: Version: 7.22.0
2026-09-24 04:06:06,804 log_run_info: Time: Thu Sep 24 04:06:06 2026
2026-09-24 04:06:06,804 log_run_info: User: aldcroft
2026-09-24 04:06:06,804 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-24 04:06:06,804 log_run_info: Processing args:
2026-09-24 04:06:06,804 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-24 04:06:06,804 log_run_info: 'delete_from_start': False,
2026-09-24 04:06:06,804 log_run_info: 'log_level': 'INFO',
2026-09-24 04:06:06,804 log_run_info: 'loop_days': 100,
2026-09-24 04:06:06,804 log_run_info: 'maude': True,
2026-09-24 04:06:06,804 log_run_info: 'maude_tlm_lookback': 7,
2026-09-24 04:06:06,804 log_run_info: 'models': None,
2026-09-24 04:06:06,804 log_run_info: 'start': None,
2026-09-24 04:06:06,804 log_run_info: 'stop': '2026:267:08:06:06.780'}
2026-09-24 04:06:06,804 log_run_info: ******************************************
2026-09-24 04:06:06,805 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-24 04:06:06,805 main:
2026-09-24 04:06:08,164 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-24 04:06:08,165 main: No new telemetry in MAUDE, skipping event updates
2026-09-24 05:06:05,903 log_run_info: ******************************************
2026-09-24 05:06:05,903 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-24 05:06:05,903 log_run_info: Version: 7.22.0
2026-09-24 05:06:05,903 log_run_info: Time: Thu Sep 24 05:06:05 2026
2026-09-24 05:06:05,903 log_run_info: User: aldcroft
2026-09-24 05:06:05,903 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-24 05:06:05,903 log_run_info: Processing args:
2026-09-24 05:06:05,903 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-24 05:06:05,904 log_run_info: 'delete_from_start': False,
2026-09-24 05:06:05,904 log_run_info: 'log_level': 'INFO',
2026-09-24 05:06:05,904 log_run_info: 'loop_days': 100,
2026-09-24 05:06:05,904 log_run_info: 'maude': True,
2026-09-24 05:06:05,904 log_run_info: 'maude_tlm_lookback': 7,
2026-09-24 05:06:05,904 log_run_info: 'models': None,
2026-09-24 05:06:05,904 log_run_info: 'start': None,
2026-09-24 05:06:05,904 log_run_info: 'stop': '2026:267:09:06:05.879'}
2026-09-24 05:06:05,904 log_run_info: ******************************************
2026-09-24 05:06:05,905 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-24 05:06:05,905 main:
2026-09-24 05:06:07,427 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-24 05:06:07,427 main: No new telemetry in MAUDE, skipping event updates
2026-09-24 06:06:05,828 log_run_info: ******************************************
2026-09-24 06:06:05,828 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-24 06:06:05,828 log_run_info: Version: 7.22.0
2026-09-24 06:06:05,828 log_run_info: Time: Thu Sep 24 06:06:05 2026
2026-09-24 06:06:05,828 log_run_info: User: aldcroft
2026-09-24 06:06:05,828 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-24 06:06:05,828 log_run_info: Processing args:
2026-09-24 06:06:05,828 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-24 06:06:05,828 log_run_info: 'delete_from_start': False,
2026-09-24 06:06:05,828 log_run_info: 'log_level': 'INFO',
2026-09-24 06:06:05,828 log_run_info: 'loop_days': 100,
2026-09-24 06:06:05,828 log_run_info: 'maude': True,
2026-09-24 06:06:05,828 log_run_info: 'maude_tlm_lookback': 7,
2026-09-24 06:06:05,828 log_run_info: 'models': None,
2026-09-24 06:06:05,828 log_run_info: 'start': None,
2026-09-24 06:06:05,828 log_run_info: 'stop': '2026:267:10:06:05.804'}
2026-09-24 06:06:05,828 log_run_info: ******************************************
2026-09-24 06:06:05,830 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-24 06:06:05,830 main:
2026-09-24 06:06:07,493 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-24 06:06:07,493 main: No new telemetry in MAUDE, skipping event updates
2026-09-24 07:06:06,818 log_run_info: ******************************************
2026-09-24 07:06:06,818 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-24 07:06:06,818 log_run_info: Version: 7.22.0
2026-09-24 07:06:06,818 log_run_info: Time: Thu Sep 24 07:06:06 2026
2026-09-24 07:06:06,818 log_run_info: User: aldcroft
2026-09-24 07:06:06,818 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-24 07:06:06,818 log_run_info: Processing args:
2026-09-24 07:06:06,818 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-24 07:06:06,818 log_run_info: 'delete_from_start': False,
2026-09-24 07:06:06,818 log_run_info: 'log_level': 'INFO',
2026-09-24 07:06:06,818 log_run_info: 'loop_days': 100,
2026-09-24 07:06:06,818 log_run_info: 'maude': True,
2026-09-24 07:06:06,818 log_run_info: 'maude_tlm_lookback': 7,
2026-09-24 07:06:06,818 log_run_info: 'models': None,
2026-09-24 07:06:06,818 log_run_info: 'start': None,
2026-09-24 07:06:06,818 log_run_info: 'stop': '2026:267:11:06:06.794'}
2026-09-24 07:06:06,818 log_run_info: ******************************************
2026-09-24 07:06:06,819 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-24 07:06:06,820 main:
2026-09-24 07:06:08,831 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-24 07:06:08,831 main: No new telemetry in MAUDE, skipping event updates
2026-09-24 08:06:08,146 log_run_info: ******************************************
2026-09-24 08:06:08,146 log_run_info: Running: /proj/sot/ska3/flight/lib/python3.13/site-packages/kadi/scripts/update_events.py
2026-09-24 08:06:08,146 log_run_info: Version: 7.22.0
2026-09-24 08:06:08,146 log_run_info: Time: Thu Sep 24 08:06:08 2026
2026-09-24 08:06:08,146 log_run_info: User: aldcroft
2026-09-24 08:06:08,146 log_run_info: Machine: kady.cfa.harvard.edu
2026-09-24 08:06:08,146 log_run_info: Processing args:
2026-09-24 08:06:08,146 log_run_info: {'data_root': '/proj/sot/ska3/flight/data/kadi/update_events',
2026-09-24 08:06:08,146 log_run_info: 'delete_from_start': False,
2026-09-24 08:06:08,146 log_run_info: 'log_level': 'INFO',
2026-09-24 08:06:08,146 log_run_info: 'loop_days': 100,
2026-09-24 08:06:08,146 log_run_info: 'maude': True,
2026-09-24 08:06:08,146 log_run_info: 'maude_tlm_lookback': 7,
2026-09-24 08:06:08,146 log_run_info: 'models': None,
2026-09-24 08:06:08,146 log_run_info: 'start': None,
2026-09-24 08:06:08,146 log_run_info: 'stop': '2026:267:12:06:08.122'}
2026-09-24 08:06:08,146 log_run_info: ******************************************
2026-09-24 08:06:08,147 main: Event database : /proj/sot/ska3/flight/data/kadi/update_events/events3.db3
2026-09-24 08:06:08,147 main:
2026-09-24 08:06:10,315 check_maude_server_has_new_telemetry: No new telemetry in MAUDE: update_model.date='2026:265:17:35:00.000' -> date_last_telemetry='2023:365:23:55:00.000'
2026-09-24 08:06:10,315 main: No new telemetry in MAUDE, skipping event updates