Prev   Index   Next

centroid_dashboard: Mon Apr 20 13:17:50 2026

File: /proj/sot/ska/data/ska_trend/centroid_dashboard/logs/daily.0/centroid_dashboard.log

Task Status AgeMaxAge Last Date
centroid_dashboard OK 0.891.0 Mon Apr 20 13:17:50 2026

No errors

File contents:

2026-04-20 13:16:09,365 main: Processing from 2026:103:17:16:09.364 to 2026:110:17:16:09.364
2026-04-20 13:16:16,517 get_observations: Found observation 31448 at 2026:103:21:29:36.117 with 1 manvrs
2026-04-20 13:16:16,518 get_observations: Found observation 31144 at 2026:103:23:20:44.491 with 1 manvrs
2026-04-20 13:16:16,518 get_observations: Found observation 31737 at 2026:104:04:14:21.853 with 1 manvrs
2026-04-20 13:16:16,519 get_observations: Found observation 29877 at 2026:104:12:21:08.619 with 1 manvrs
2026-04-20 13:16:16,519 get_observations: Found observation 29741 at 2026:104:16:16:25.723 with 1 manvrs
2026-04-20 13:16:16,519 get_observations: Found observation 31259 at 2026:104:19:59:38.987 with 1 manvrs
2026-04-20 13:16:16,520 get_observations: Found observation 32295 at 2026:104:22:21:05.042 with 2 manvrs
2026-04-20 13:16:16,520 get_observations: Found observation 32302 at 2026:105:01:47:44.997 with 1 manvrs
2026-04-20 13:16:16,521 get_observations: Found observation 30240 at 2026:105:05:55:28.052 with 1 manvrs
2026-04-20 13:16:16,521 get_observations: Found observation 41553 at 2026:105:13:28:18.171 with 1 manvrs
2026-04-20 13:16:16,521 get_observations: Found observation 41552 at 2026:105:14:53:50.704 with 1 manvrs
2026-04-20 13:16:16,522 get_observations: Found observation 41551 at 2026:105:16:02:43.018 with 1 manvrs
2026-04-20 13:16:16,522 get_observations: Found observation 41550 at 2026:105:19:55:40.252 with 1 manvrs
2026-04-20 13:16:16,523 get_observations: Found observation 41549 at 2026:105:22:55:40.252 with 1 manvrs
2026-04-20 13:16:16,523 get_observations: Found observation 32303 at 2026:106:00:58:53.771 with 1 manvrs
2026-04-20 13:16:16,523 get_observations: Found observation 31420 at 2026:106:05:22:23.241 with 2 manvrs
2026-04-20 13:16:16,524 get_observations: Found observation 32304 at 2026:106:11:55:21.883 with 2 manvrs
2026-04-20 13:16:16,524 get_observations: Found observation 41546 at 2026:106:15:43:54.727 with 1 manvrs
2026-04-20 13:16:16,525 get_observations: Found observation 32293 at 2026:106:16:45:04.825 with 1 manvrs
2026-04-20 13:16:16,525 get_observations: Found observation 31223 at 2026:106:20:31:47.176 with 1 manvrs
2026-04-20 13:16:16,526 get_observations: Found observation 30543 at 2026:107:04:28:04.145 with 2 manvrs
2026-04-20 13:16:16,526 get_observations: Found observation 32294 at 2026:107:07:36:27.788 with 2 manvrs
2026-04-20 13:16:16,526 get_observations: Found observation 32300 at 2026:107:11:06:29.416 with 1 manvrs
2026-04-20 13:16:16,527 get_observations: Found observation 30250 at 2026:107:18:48:03.219 with 1 manvrs
2026-04-20 13:16:16,527 get_observations: Found observation 32297 at 2026:108:03:34:44.225 with 1 manvrs
2026-04-20 13:16:16,528 get_observations: Found observation 41543 at 2026:108:07:10:50.673 with 1 manvrs
2026-04-20 13:16:16,528 get_observations: Found observation 41542 at 2026:108:12:15:40.252 with 1 manvrs
2026-04-20 13:16:16,528 get_observations: Found observation 32305 at 2026:108:16:42:24.683 with 1 manvrs
2026-04-20 13:16:16,529 get_observations: Found observation 32296 at 2026:108:20:29:14.657 with 1 manvrs
2026-04-20 13:16:16,529 get_observations: Found observation 30259 at 2026:108:23:50:37.957 with 1 manvrs
2026-04-20 13:16:16,530 get_observations: Found observation 32299 at 2026:109:07:51:18.236 with 1 manvrs
2026-04-20 13:16:16,530 get_observations: Found observation 30257 at 2026:109:10:38:41.619 with 1 manvrs
2026-04-20 13:16:16,530 get_observations: Found observation 32292 at 2026:109:18:57:04.754 with 1 manvrs
2026-04-20 13:16:16,531 get_observations: Found observation 32298 at 2026:109:22:16:52.381 with 1 manvrs
2026-04-20 13:16:16,531 get_observations: Found observation 32306 at 2026:110:01:59:58.200 with 1 manvrs
2026-04-20 13:16:16,532 get_observations: Found observation 31419 at 2026:110:03:48:20.813 with 1 manvrs
2026-04-20 13:16:16,532 get_observations: Found observation 29878 at 2026:110:09:56:55.239 with 1 manvrs
2026-04-20 13:16:16,532 get_observations: Found observation 29880 at 2026:110:14:57:12.397 with 1 manvrs
2026-04-20 13:16:16,533 main: Found 38 observations
2026-04-20 13:16:16,533 main: ********************************************************************************
2026-04-20 13:16:16,533 main: Processing observation 31448
2026-04-20 13:16:16,533 main: ********************************************************************************
2026-04-20 13:16:16,578 obs_link_from_info: Found prev obsid 32301
2026-04-20 13:16:16,581 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,583 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,586 process_obs: ObsID 31448 already processed, skipping
2026-04-20 13:16:16,586 main:
2026-04-20 13:16:16,586 main: ********************************************************************************
2026-04-20 13:16:16,586 main: Processing observation 31144
2026-04-20 13:16:16,586 main: ********************************************************************************
2026-04-20 13:16:16,588 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,589 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,592 process_obs: ObsID 31144 already processed, skipping
2026-04-20 13:16:16,592 main:
2026-04-20 13:16:16,592 main: ********************************************************************************
2026-04-20 13:16:16,592 main: Processing observation 31737
2026-04-20 13:16:16,592 main: ********************************************************************************
2026-04-20 13:16:16,593 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,594 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,598 process_obs: ObsID 31737 already processed, skipping
2026-04-20 13:16:16,598 main:
2026-04-20 13:16:16,598 main: ********************************************************************************
2026-04-20 13:16:16,598 main: Processing observation 29877
2026-04-20 13:16:16,598 main: ********************************************************************************
2026-04-20 13:16:16,599 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,600 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,604 process_obs: ObsID 29877 already processed, skipping
2026-04-20 13:16:16,604 main:
2026-04-20 13:16:16,604 main: ********************************************************************************
2026-04-20 13:16:16,604 main: Processing observation 29741
2026-04-20 13:16:16,604 main: ********************************************************************************
2026-04-20 13:16:16,605 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,606 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,609 process_obs: ObsID 29741 already processed, skipping
2026-04-20 13:16:16,609 main:
2026-04-20 13:16:16,610 main: ********************************************************************************
2026-04-20 13:16:16,610 main: Processing observation 31259
2026-04-20 13:16:16,610 main: ********************************************************************************
2026-04-20 13:16:16,610 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,612 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,615 process_obs: ObsID 31259 already processed, skipping
2026-04-20 13:16:16,615 main:
2026-04-20 13:16:16,615 main: ********************************************************************************
2026-04-20 13:16:16,615 main: Processing observation 32295
2026-04-20 13:16:16,615 main: ********************************************************************************
2026-04-20 13:16:16,616 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,617 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,621 process_obs: ObsID 32295 already processed, skipping
2026-04-20 13:16:16,621 main:
2026-04-20 13:16:16,621 main: ********************************************************************************
2026-04-20 13:16:16,621 main: Processing observation 32302
2026-04-20 13:16:16,621 main: ********************************************************************************
2026-04-20 13:16:16,622 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,623 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,627 process_obs: ObsID 32302 already processed, skipping
2026-04-20 13:16:16,627 main:
2026-04-20 13:16:16,627 main: ********************************************************************************
2026-04-20 13:16:16,627 main: Processing observation 30240
2026-04-20 13:16:16,627 main: ********************************************************************************
2026-04-20 13:16:16,628 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,629 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,633 process_obs: ObsID 30240 already processed, skipping
2026-04-20 13:16:16,633 main:
2026-04-20 13:16:16,633 main: ********************************************************************************
2026-04-20 13:16:16,633 main: Processing observation 41553
2026-04-20 13:16:16,633 main: ********************************************************************************
2026-04-20 13:16:16,634 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,636 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,640 process_obs: ObsID 41553 already processed, skipping
2026-04-20 13:16:16,640 main:
2026-04-20 13:16:16,640 main: ********************************************************************************
2026-04-20 13:16:16,640 main: Processing observation 41552
2026-04-20 13:16:16,640 main: ********************************************************************************
2026-04-20 13:16:16,641 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,642 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,645 process_obs: ObsID 41552 already processed, skipping
2026-04-20 13:16:16,645 main:
2026-04-20 13:16:16,645 main: ********************************************************************************
2026-04-20 13:16:16,645 main: Processing observation 41551
2026-04-20 13:16:16,645 main: ********************************************************************************
2026-04-20 13:16:16,646 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,648 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,651 process_obs: ObsID 41551 already processed, skipping
2026-04-20 13:16:16,651 main:
2026-04-20 13:16:16,651 main: ********************************************************************************
2026-04-20 13:16:16,651 main: Processing observation 41550
2026-04-20 13:16:16,651 main: ********************************************************************************
2026-04-20 13:16:16,652 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,653 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,657 process_obs: ObsID 41550 already processed, skipping
2026-04-20 13:16:16,657 main:
2026-04-20 13:16:16,657 main: ********************************************************************************
2026-04-20 13:16:16,657 main: Processing observation 41549
2026-04-20 13:16:16,657 main: ********************************************************************************
2026-04-20 13:16:16,658 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,659 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,662 process_obs: ObsID 41549 already processed, skipping
2026-04-20 13:16:16,662 main:
2026-04-20 13:16:16,662 main: ********************************************************************************
2026-04-20 13:16:16,662 main: Processing observation 32303
2026-04-20 13:16:16,662 main: ********************************************************************************
2026-04-20 13:16:16,663 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,664 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,668 process_obs: ObsID 32303 already processed, skipping
2026-04-20 13:16:16,668 main:
2026-04-20 13:16:16,668 main: ********************************************************************************
2026-04-20 13:16:16,668 main: Processing observation 31420
2026-04-20 13:16:16,668 main: ********************************************************************************
2026-04-20 13:16:16,669 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,670 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,673 process_obs: ObsID 31420 already processed, skipping
2026-04-20 13:16:16,673 main:
2026-04-20 13:16:16,673 main: ********************************************************************************
2026-04-20 13:16:16,673 main: Processing observation 32304
2026-04-20 13:16:16,673 main: ********************************************************************************
2026-04-20 13:16:16,674 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,675 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,679 process_obs: ObsID 32304 already processed, skipping
2026-04-20 13:16:16,679 main:
2026-04-20 13:16:16,679 main: ********************************************************************************
2026-04-20 13:16:16,679 main: Processing observation 41546
2026-04-20 13:16:16,679 main: ********************************************************************************
2026-04-20 13:16:16,680 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,681 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,684 process_obs: ObsID 41546 already processed, skipping
2026-04-20 13:16:16,684 main:
2026-04-20 13:16:16,684 main: ********************************************************************************
2026-04-20 13:16:16,684 main: Processing observation 32293
2026-04-20 13:16:16,685 main: ********************************************************************************
2026-04-20 13:16:16,685 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,687 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,690 process_obs: ObsID 32293 already processed, skipping
2026-04-20 13:16:16,690 main:
2026-04-20 13:16:16,690 main: ********************************************************************************
2026-04-20 13:16:16,690 main: Processing observation 31223
2026-04-20 13:16:16,690 main: ********************************************************************************
2026-04-20 13:16:16,691 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,692 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,695 process_obs: ObsID 31223 already processed, skipping
2026-04-20 13:16:16,696 main:
2026-04-20 13:16:16,696 main: ********************************************************************************
2026-04-20 13:16:16,696 main: Processing observation 30543
2026-04-20 13:16:16,696 main: ********************************************************************************
2026-04-20 13:16:16,696 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,698 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,701 process_obs: ObsID 30543 already processed, skipping
2026-04-20 13:16:16,701 main:
2026-04-20 13:16:16,701 main: ********************************************************************************
2026-04-20 13:16:16,701 main: Processing observation 32294
2026-04-20 13:16:16,701 main: ********************************************************************************
2026-04-20 13:16:16,702 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,703 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,721 process_obs: ObsID 32294 already processed, skipping
2026-04-20 13:16:16,721 main:
2026-04-20 13:16:16,721 main: ********************************************************************************
2026-04-20 13:16:16,721 main: Processing observation 32300
2026-04-20 13:16:16,721 main: ********************************************************************************
2026-04-20 13:16:16,722 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,723 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 13:16:16,726 process_obs: ObsID 32300 already processed, skipping
2026-04-20 13:16:16,726 main:
2026-04-20 13:16:16,726 main: ********************************************************************************
2026-04-20 13:16:16,726 main: Processing observation 30250
2026-04-20 13:16:16,726 main: ********************************************************************************
2026-04-20 13:16:16,727 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 13:16:16,728 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:16:19,289 get_centroid_resids_from_file: Reading centroid residuals from /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30250/centroid_resids.pkl
2026-04-20 12:16:19,317 write_centroid_resids: Centroid residuals file exists, skipping
2026-04-20 12:16:19,317 plot_crs_time: Plot file exists, skipping
2026-04-20 12:16:19,317 plot_crs_scatter: Plot file exists, skipping
2026-04-20 12:16:20,644 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30250/n_kalman_delta_roll.png
2026-04-20 12:16:23,562 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30250/index.html
2026-04-20 12:16:23,581 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30250/info.json
2026-04-20 12:16:23,656 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/30/30250/index.html
2026-04-20 12:16:23,683 main:
2026-04-20 12:16:23,683 main: ********************************************************************************
2026-04-20 12:16:23,683 main: Processing observation 32297
2026-04-20 12:16:23,683 main: ********************************************************************************
2026-04-20 12:16:23,696 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:16:23,726 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:16:26,440 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:16:26,777 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32297/centroid_resids.pkl with 5 slots
2026-04-20 12:16:26,824 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32297/centroid_resids_time.png
2026-04-20 12:16:27,199 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32297/centroid_resids_scatter.png
2026-04-20 12:16:28,504 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32297/n_kalman_delta_roll.png
2026-04-20 12:16:29,048 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32297/index.html
2026-04-20 12:16:29,065 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32297/info.json
2026-04-20 12:16:29,082 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/32/32297/index.html
2026-04-20 12:16:29,092 main:
2026-04-20 12:16:29,092 main: ********************************************************************************
2026-04-20 12:16:29,092 main: Processing observation 41543
2026-04-20 12:16:29,092 main: ********************************************************************************
2026-04-20 12:16:29,104 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:16:29,120 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:16:31,843 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:16:32,443 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41543/centroid_resids.pkl with 8 slots
2026-04-20 12:16:32,731 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41543/centroid_resids_time.png
2026-04-20 12:16:33,147 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41543/centroid_resids_scatter.png
2026-04-20 12:16:33,351 plot_n_kalman_delta_roll: No OBC - GND deltas available, not plotting
2026-04-20 12:16:33,465 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41543/n_kalman_delta_roll.png
2026-04-20 12:16:33,784 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41543/index.html
2026-04-20 12:16:33,788 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41543/info.json
2026-04-20 12:16:33,856 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/41/41543/index.html
2026-04-20 12:16:33,863 main:
2026-04-20 12:16:33,863 main: ********************************************************************************
2026-04-20 12:16:33,863 main: Processing observation 41542
2026-04-20 12:16:33,863 main: ********************************************************************************
2026-04-20 12:16:33,872 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:16:33,883 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:16:36,416 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:16:37,180 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41542/centroid_resids.pkl with 8 slots
2026-04-20 12:16:37,254 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41542/centroid_resids_time.png
2026-04-20 12:16:37,624 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41542/centroid_resids_scatter.png
2026-04-20 12:16:37,768 plot_n_kalman_delta_roll: No OBC - GND deltas available, not plotting
2026-04-20 12:16:37,935 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41542/n_kalman_delta_roll.png
2026-04-20 12:16:38,090 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41542/index.html
2026-04-20 12:16:38,092 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/41542/info.json
2026-04-20 12:16:38,094 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/41/41542/index.html
2026-04-20 12:16:38,095 main:
2026-04-20 12:16:38,095 main: ********************************************************************************
2026-04-20 12:16:38,095 main: Processing observation 32305
2026-04-20 12:16:38,095 main: ********************************************************************************
2026-04-20 12:16:38,096 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:16:38,097 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:16:40,577 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:16:41,418 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32305/centroid_resids.pkl with 5 slots
2026-04-20 12:16:41,469 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32305/centroid_resids_time.png
2026-04-20 12:16:41,717 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32305/centroid_resids_scatter.png
2026-04-20 12:16:42,402 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32305/n_kalman_delta_roll.png
2026-04-20 12:16:42,550 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32305/index.html
2026-04-20 12:16:42,552 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32305/info.json
2026-04-20 12:16:42,554 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/32/32305/index.html
2026-04-20 12:16:42,555 main:
2026-04-20 12:16:42,555 main: ********************************************************************************
2026-04-20 12:16:42,555 main: Processing observation 32296
2026-04-20 12:16:42,555 main: ********************************************************************************
2026-04-20 12:16:42,556 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:16:42,558 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:16:44,647 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:16:45,242 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32296/centroid_resids.pkl with 5 slots
2026-04-20 12:16:45,289 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32296/centroid_resids_time.png
2026-04-20 12:16:45,533 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32296/centroid_resids_scatter.png
2026-04-20 12:16:46,244 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32296/n_kalman_delta_roll.png
2026-04-20 12:16:46,385 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32296/index.html
2026-04-20 12:16:46,387 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32296/info.json
2026-04-20 12:16:46,389 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/32/32296/index.html
2026-04-20 12:16:46,390 main:
2026-04-20 12:16:46,390 main: ********************************************************************************
2026-04-20 12:16:46,390 main: Processing observation 30259
2026-04-20 12:16:46,390 main: ********************************************************************************
2026-04-20 12:16:46,391 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:16:46,392 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:16:48,551 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:16:48,860 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30259/centroid_resids.pkl with 5 slots
2026-04-20 12:16:48,919 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30259/centroid_resids_time.png
2026-04-20 12:16:49,261 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30259/centroid_resids_scatter.png
2026-04-20 12:16:50,175 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30259/n_kalman_delta_roll.png
2026-04-20 12:16:50,322 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30259/index.html
2026-04-20 12:16:50,324 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30259/info.json
2026-04-20 12:16:50,327 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/30/30259/index.html
2026-04-20 12:16:50,327 main:
2026-04-20 12:16:50,327 main: ********************************************************************************
2026-04-20 12:16:50,327 main: Processing observation 32299
2026-04-20 12:16:50,327 main: ********************************************************************************
2026-04-20 12:16:50,329 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:16:50,330 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:16:52,370 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:16:52,674 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32299/centroid_resids.pkl with 5 slots
2026-04-20 12:16:52,717 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32299/centroid_resids_time.png
2026-04-20 12:16:53,160 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32299/centroid_resids_scatter.png
2026-04-20 12:16:53,787 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32299/n_kalman_delta_roll.png
2026-04-20 12:16:53,930 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32299/index.html
2026-04-20 12:16:53,932 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32299/info.json
2026-04-20 12:16:53,934 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/32/32299/index.html
2026-04-20 12:16:53,935 main:
2026-04-20 12:16:53,935 main: ********************************************************************************
2026-04-20 12:16:53,935 main: Processing observation 30257
2026-04-20 12:16:53,935 main: ********************************************************************************
2026-04-20 12:16:53,936 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:16:53,937 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:16:56,033 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:16:56,340 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30257/centroid_resids.pkl with 5 slots
2026-04-20 12:16:56,399 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30257/centroid_resids_time.png
2026-04-20 12:16:56,684 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30257/centroid_resids_scatter.png
2026-04-20 12:16:57,818 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30257/n_kalman_delta_roll.png
2026-04-20 12:16:57,968 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30257/index.html
2026-04-20 12:16:57,969 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/30257/info.json
2026-04-20 12:16:57,972 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/30/30257/index.html
2026-04-20 12:16:57,973 main:
2026-04-20 12:16:57,973 main: ********************************************************************************
2026-04-20 12:16:57,973 main: Processing observation 32292
2026-04-20 12:16:57,973 main: ********************************************************************************
2026-04-20 12:16:57,985 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:16:57,986 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:17:00,068 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:17:00,612 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32292/centroid_resids.pkl with 5 slots
2026-04-20 12:17:00,657 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32292/centroid_resids_time.png
2026-04-20 12:17:00,897 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32292/centroid_resids_scatter.png
2026-04-20 12:17:01,579 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32292/n_kalman_delta_roll.png
2026-04-20 12:17:01,741 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32292/index.html
2026-04-20 12:17:01,744 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32292/info.json
2026-04-20 12:17:01,747 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/32/32292/index.html
2026-04-20 12:17:01,748 main:
2026-04-20 12:17:01,748 main: ********************************************************************************
2026-04-20 12:17:01,748 main: Processing observation 32298
2026-04-20 12:17:01,748 main: ********************************************************************************
2026-04-20 12:17:01,750 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:17:01,751 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/last.html
2026-04-20 12:17:04,921 get_centroid_resids: Computing centroid residuals from telemetry
2026-04-20 12:17:05,420 write_centroid_resids: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32298/centroid_resids.pkl with 5 slots
2026-04-20 12:17:05,469 plot_crs_time: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32298/centroid_resids_time.png
2026-04-20 12:17:05,751 plot_crs_scatter: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32298/centroid_resids_scatter.png
2026-04-20 12:17:05,929 plot_n_kalman_delta_roll: No OBC - GND deltas available, not plotting
2026-04-20 12:17:06,081 plot_n_kalman_delta_roll: Writing plot file /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32298/n_kalman_delta_roll.png
2026-04-20 12:17:06,222 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32298/index.html
2026-04-20 12:17:06,229 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR1326A/32298/info.json
2026-04-20 12:17:06,233 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/32/32298/index.html
2026-04-20 12:17:06,234 main:
2026-04-20 12:17:06,234 main: ********************************************************************************
2026-04-20 12:17:06,234 main: Processing observation 32306
2026-04-20 12:17:06,234 main: ********************************************************************************
2026-04-20 12:17:06,236 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:17:06,237 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/last.html
2026-04-20 12:17:10,434 __init__: ObsID 32306 has insufficient telemetry, skipping
2026-04-20 12:17:10,436 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/32306/index.html
2026-04-20 12:17:10,464 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/32306/info.json
2026-04-20 12:17:10,465 main:
2026-04-20 12:17:10,465 main: ********************************************************************************
2026-04-20 12:17:10,465 main: Processing observation 31419
2026-04-20 12:17:10,465 main: ********************************************************************************
2026-04-20 12:17:10,467 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:17:10,469 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/last.html
2026-04-20 12:17:12,461 __init__: ObsID 31419 has insufficient telemetry, skipping
2026-04-20 12:17:12,461 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/31419/index.html
2026-04-20 12:17:12,466 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/31419/info.json
2026-04-20 12:17:12,467 main:
2026-04-20 12:17:12,467 main: ********************************************************************************
2026-04-20 12:17:12,467 main: Processing observation 29878
2026-04-20 12:17:12,467 main: ********************************************************************************
2026-04-20 12:17:12,468 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:17:12,469 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/last.html
2026-04-20 12:17:14,450 __init__: ObsID 29878 has no maneuver event in telemetry, skipping
2026-04-20 12:17:14,450 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/29878/index.html
2026-04-20 12:17:14,455 info: could not get attribute kalman_start for obsid 29878
2026-04-20 12:17:14,455 info: could not get attribute kalman_stop for obsid 29878
2026-04-20 12:17:14,455 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/29878/info.json
2026-04-20 12:17:14,456 main:
2026-04-20 12:17:14,456 main: ********************************************************************************
2026-04-20 12:17:14,456 main: Processing observation 29880
2026-04-20 12:17:14,456 main: ********************************************************************************
2026-04-20 12:17:14,457 obs_link_from_info: No /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/29880/info.json file found, obs=None
2026-04-20 12:17:14,458 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/last.html
2026-04-20 12:17:14,459 write_redirect_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/last.html
2026-04-20 12:17:16,444 __init__: ObsID 29880 has no maneuver event in telemetry, skipping
2026-04-20 12:17:16,444 write_index_html: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/29880/index.html
2026-04-20 12:17:16,449 info: could not get attribute kalman_start for obsid 29880
2026-04-20 12:17:16,449 info: could not get attribute kalman_stop for obsid 29880
2026-04-20 12:17:16,449 write_info_json: Writing /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports/2026/APR2026A/29880/info.json
2026-04-20 12:17:16,450 main:
2026-04-20 13:17:17,780 run_check: Starting centroid dashboard continuity check
2026-04-20 13:17:17,780 collect_observations: Scanning observations in /proj/sot/ska3/aca/www/ASPECT_ICXC/centroid_reports
2026-04-20 13:17:17,823 collect_observations: Processing year 2020
2026-04-20 13:17:21,353 collect_observations: Processing year 2021
2026-04-20 13:17:25,994 collect_observations: Processing year 2022
2026-04-20 13:17:31,460 collect_observations: Processing year 2023
2026-04-20 13:17:36,503 collect_observations: Processing year 2024
2026-04-20 13:17:43,352 collect_observations: Processing year 2025
2026-04-20 13:17:49,153 collect_observations: Processing year 2026
2026-04-20 13:17:50,577 collect_observations: Collected 13736 observations from 4-digit years
2026-04-20 13:17:50,581 sort_observations_by_date: Sorted observations by date
2026-04-20 13:17:50,581 check_continuity: Starting continuity check from most recent: ObsID 29880 (APR2026A) at 2026:110:14:26:51.987
2026-04-20 13:17:50,597 check_continuity: Reached chain start at ObsID 47337 (MAR3020A) at 2020:092:05:29:19.712
2026-04-20 13:17:50,597 check_continuity: Chain complete: visited 13736 observations out of 13736 total
2026-04-20 13:17:50,598 run_check: ✓ Observation chain continuity check PASSED
2026-04-20 13:17:50,598 run_check: Summary: 13736 observations from 4-digit years, all properly linked