sen2cor: IndexError at Landsat processing

[ValentinaHutter]

While testing the latest version of sen2like (v4.5.1), I noticed that because of the sys.exit(main()) in the sen2cor and sen2like implementation, the Tracebacks of the errors do not get written into the sen2like.log fully - if further development is planned, it would be useful to update the implementation, so that errors get logged. I was able to get the full traceback in a grafana dashboard, but this might not be an option for other users.

With v4.5.1, the Sentinel 2 files can be processed successfully. For Landsat files, I get the following error:

2025-04-03 13:30:43 \| P-26 \| T-140530348909184 \| INFO     \| sen2like             - Process of L8/L9 products finish |  
-- | -- | --
2025-04-03 13:30:43 \| P-26 \| T-140530348909184 \| INFO     \| sen2like             - End process for input product <LOCAL PATH>/SEN2LIKE/Landsat9/192/26/LC09_L1TP_192026_20240809_20240809_02_T1 |  
2025-04-03 13:30:43 \| P-26 \| T-140530348909184 \| INFO     \| sen2like             - Skip product LC09_L1TP_192026_20240809_20240809_02_T1 due to sen2cor failure |  
core.sen2cor_client.sen2cor_client.Sen2corError: Command '['python', <LOCAL PATH>/SEN2COR/sen2cor_3.3.0_python_3.12/SEN2COR_3/L2A_Process.py', '<LOCAL PATH>/SEN2LIKE/Landsat9/192/26/LC09_L1TP_192026_20240809_20240809_02_T1', '--output_dir', '<LOCAL PATH>/SEN2LIKE/production/1743685431/sen2cor/LC09_L1TP_192026_20240809_20240809_02_T1', '--GIP_L2A', <LOCAL PATH>/SEN2LIKE/production/1743685431/sen2cor/LC09_L1TP_192026_20240809_20240809_02_T1/sen2cor_gipp_33UUP.xml', '--work_dir', '<LOCAL PATH>/SEN2LIKE/production/1743685431/sen2cor/LC09_L1TP_192026_20240809_20240809_02_T1', '--sc_classic']' returned non-zero exit status 1. |  
raise Sen2corError(run_error) from run_error |  
File "/usr/local/sen2like/core/sen2cor_client/sen2cor_client.py", line 114, in run |  
^^^^^^^^^^^^^^^^^^^^^^^^ |  
sen2cor.run(s2l_product), |  
File "/usr/local/sen2like/./sen2like.py", line 118, in pre_process_atmcor |  
Traceback (most recent call last): |  
 |  
The above exception was the direct cause of the following exception: |  
 |  
subprocess.CalledProcessError: Command '['python', '<LOCAL PATH>/SEN2COR/sen2cor_3.3.0_python_3.12/SEN2COR_3/L2A_Process.py', '<LOCAL PATH>/SEN2LIKE/Landsat9/192/26/LC09_L1TP_192026_20240809_20240809_02_T1', '--output_dir', <LOCAL PATH>/SEN2LIKE/production/1743685431/sen2cor/LC09_L1TP_192026_20240809_20240809_02_T1', '--GIP_L2A', '<LOCAL PATH>/SEN2LIKE/production/1743685431/sen2cor/LC09_L1TP_192026_20240809_20240809_02_T1/sen2cor_gipp_33UUP.xml', '--work_dir', '<LOCAL PATH>/SEN2LIKE/production/1743685431/sen2cor/LC09_L1TP_192026_20240809_20240809_02_T1', '--sc_classic']' returned non-zero exit status 1. |  
raise CalledProcessError(retcode, process.args, |  
File "/usr/local/sen2like/core/sen2cor_client/sen2cor_client.py", line 110, in run |  
Traceback (most recent call last): |  
2025-04-03 13:30:43 \| P-26 \| T-140530348909184 \| WARNING  \| sen2like             - sen2cor raises an error |  
2025-04-03 13:30:43 \| P-26 \| T-140530348909184 \| ERROR    \| sen2cor_client       - Command '['python', <LOCAL PATH>/SEN2COR/sen2cor_3.3.0_python_3.12/SEN2COR_3/L2A_Process.py', '<LOCAL PATH>/SEN2LIKE/Landsat9/192/26/LC09_L1TP_192026_20240809_20240809_02_T1', '--output_dir', <LOCAL PATH>/SEN2LIKE/production/1743685431/sen2cor/LC09_L1TP_192026_20240809_20240809_02_T1', '--GIP_L2A', '<LOCAL PATH>/SEN2LIKE/production/1743685431/sen2cor/LC09_L1TP_192026_20240809_20240809_02_T1/sen2cor_gipp_33UUP.xml', '--work_dir', '<LOCAL PATH>/SEN2LIKE/production/1743685431/sen2cor/LC09_L1TP_192026_20240809_20240809_02_T1', '--sc_classic']' returned non-zero exit status 1. |  
2025-04-03 13:30:43 \| P-26 \| T-140530348909184 \| ERROR    \| sen2cor_client       - An error occurred during the run of sen2cor |  
IndexError: boolean index did not match indexed array along axis 0; size of axis is 3751 but size of corresponding boolean axis is 4000 |  
~~~~~~~~~^^^^^^^^^^^^^^^^^ |  
cbeta_sdw[sdw == T_Shadow] = 1  # set shadows to 1 like in the gdal hillshade output (O is for no data) |  
  | 2025-04-03 15:30:42.864 | File "<LOCAL PATH>/SEN2COR/sen2cor_3.3.0_python_3.12/SEN2COR_3/L2A_BandIO_Landsat.py", line 1101, in gdalDEM_Shade_Cast |  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |  
sdwfile = self.gdalDEM_Shade_Cast(demfile) |  
File "<LOCAL PATH>/SEN2COR/sen2cor_3.3.0_python_3.12/SEN2COR_3/L2A_BandIO_Landsat.py", line 321, in import_aux_band_list |  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |  
if self.tables.import_aux_band_list() == False: |  
File "<LOCAL PATH>/SEN2COR/sen2cor_3.3.0_python_3.12/SEN2COR_3/L2A_Process_Landsat.py", line 88, in process |  
^^^^^^^^^^^^^^ |  
File "<LOCAL PATH>/SEN2COR/sen2cor_3.3.0_python_3.12/SEN2COR_3/L2A_Process.py", line 638, in main |  
^^^^^^ |  
sys.exit(main()) |  
File "<LOCAL PATH>/SEN2COR/sen2cor_3.3.0_python_3.12/SEN2COR_3/L2A_Process.py", line 820, in <module> |  
Traceback (most recent call last): |  
2025-04-03T13:30:41.288  sen2like-h6wwn-process-182080131 003768 [ INFO] Progress[%]: 4.84 : L2A_BAND_IO: band DIGITAL_ELEVATION_MAP imported, elapsed time[s]: 3.749, total: 0:00:04.910509 |  
2025-04-03T13:30:37.538  sen2like-h6wwn-process-182080131 003768 [ INFO] Progress[%]: 3.10 : L2A_DEM: list read, elapsed time[s]: 0.021, total: 0:00:01.161080 |  
2025-04-03T13:30:37.516  sen2like-h6wwn-process-182080131 003768 [ INFO] Progress[%]: 3.09 : L2A_DEM: reading list, elapsed time[s]: 0.011, total: 0:00:01.139730 |  
2025-04-03T13:30:37.510  sen2like-h6wwn-process-182080131 003768 [WARNING] no ozone values found in input data, default (331) will be used |  
2025-04-03T13:30:37.505  sen2like-h6wwn-process-182080131 003768 [ INFO] Progress[%]: 3.08 : L2A_Tables: CAMS daily ncfile (gtco3): NOT FOUND, elapsed time[s]: 0.519, total: 0:00:01.128883 |  
2025-04-03T13:30:36.986  sen2like-h6wwn-process-182080131 003768 [ INFO] Progress[%]: 2.84 : process_landsat: start of pre processing, elapsed time[s]: 0.008, total: 0:00:00.609434 |  
2025-04-03T13:30:36.980  sen2like-h6wwn-process-182080131 003768 [ INFO] Progress[%]: 2.84 : L2A processing for Landsat, elapsed time[s]: 0.602, total: 0:00:00.601792 |  
2025-04-03T13:30:36.472  sen2like-h6wwn-process-182080131 003768 [ INFO] Sen2Cor. Version: 03.03.00, created: 2025.01.31, supporting Level-1TP Collection_1-2 Landsat_8-9 started ...

it seems like the process expects a different shape, is this the expected behaviour? IndexError: boolean index did not match indexed array along axis 0; size of axis is 3751 but size of corresponding boolean axis is 4000

[telespazio-tim]

Hi Valentina,

We think we have found the reason of your issue.
e.g. (size of axis is 3751 but size of corresponding boolean axis is 4000)
We still need to perform some tests early next week and will send you a way forward.

Best regards,

[telespazio-tim]

Hi Valentina,

We close this issue with the update of the handling of the topographic shadow library available in Sen2Cor version 3.3.1.

Best regards,
Jérôme