[BUG] [OPS] S3-PUG-STC execution failed "[code 290] [exitCode 255] [msg Task /usr/local/components/PUG-3.48/bin/PUGPostProcessor failed]" due to "Illegal Value: OLCI missing lines global info"

[suberti-ads]

Environment:

• Delivery tag:
• Platform: OPS Orange Cloud
• Configuration:
  PUG-STC 1.14.0-rc1

Traceability:

Current Behavior:
PUG-STC execution for product SY_2_SYN fall in error

[code 290] [exitCode 255] [msg Task /usr/local/components/PUG-3.48/bin/PUGPostProcessor failed]

Expected Behavior:
Production successfully done with nominal production input.

Steps To Reproduce:
Production 3%

Test execution artefacts (i.e. logs, screenshots…)
Execution logs:
pug-stc-post-core-JobOrder.131506.xml.log

Whenever possible, first analysis of the root cause
All issue were found on production with input SY_2_SYN
sample for joborder JobOrder.131506.xml
Product input :

"S3B_AX___FRO_AX_20230727T000000_20230806T000000_20230730T065449___________________EUM_O_AL_001.SEN3"
"S3B_SY_2_SYN____20230729T203841_20230729T204041_20230730T091811_0119_082_171______LN3_D_ST_002.SEN3"
"S3B_SY_2_SYN____20230729T203641_20230729T203841_20230730T093239_0119_082_171______LN3_D_ST_002.SEN3"
"JobOrder.131506.xml"

On log there was following issue during production

2023-07-30T15:21:22+00:00	{"header":{"type":"LOG","timestamp":"2023-07-30T15:21:22.400898Z","level":"INFO","line":129,"file":"TaskCallable.java","thread":"pool-35-thread-1"},"message":{"content":"Ending task /usr/local/components/PUG-3.48/bin/PUGPostProcessor with exit code 255"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ipf.execution.worker.job.process.TaskCallable"}}
2023-07-30T15:21:22+00:00	2023-07-30T15:21:22.392721 s3-pug-stc-part1-execution-worker-v1-6d7567f956-ld27p PUG_SY_2_SYN 03.48 [0000000140]: [I] PUGPostProcessor: Exiting with EXIT CODE: 255
2023-07-30T15:21:22+00:00		FATAL: All the product data unit generations exited in error!
2023-07-30T15:21:22+00:00			 must have size 1.2023-07-30T15:21:22.392605 s3-pug-stc-part1-execution-worker-v1-6d7567f956-ld27p PUG_SY_2_SYN 03.48 [0000000140]: [E] PUGPostProcessor: [PUGPostProcessor.C: execute:(367)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [140399911340224]
2023-07-30T15:21:22+00:00		acs::ManifestBuilderS3SynL1L2::ManifestBuilderS3SynL1L2Exception in ManifestBuilderS3SynL1L2.C(370) from virtual void acs::ManifestBuilderS3SynL1L2::writeSecondaryProductMetadata() thread "unnamedThread" [140399393896192]
2023-07-30T15:21:22+00:00		caused by:
2023-07-30T15:21:22+00:00		Problem found during product data unit generation -> skipping to the next, if any ...
2023-07-30T15:21:22+00:00	acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [140399911340224]
2023-07-30T15:21:22+00:00	2023-07-30T15:21:22.332761 s3-pug-stc-part1-execution-worker-v1-6d7567f956-ld27p PUG_SY_2_SYN 03.48 [0000000140]: [I] PUGPostProcessor: writeProduct DONE.

It seems to be an issue with input product.
i have 2 hypothesys:

• issue on preparation worker which not select good input
• Issue on SY_2_SYN production

Bug Generic Definition of Ready (DoR)

• The affect version in which the bug has been found is mentioned
• The context and environment of the bug is detailed
• The description of the bug is clear and unambiguous
• The procedure (steps) to reproduce the bug is clearly detailed
• The tested User Story / features is linked to the bug if available
• Logs are attached if available
• A data set attached if available

Bug Generic Definition of Done (DoD)

• the modification implemented (the solution to fix the bug) is described in the bug.
• Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
• Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
• Code is compliant with coding rules (SONAR Report as evidence)
• Acceptance criteria of the related User story are checked and Passed

[Woljtek]

@w-jka
On OPS side, we don't understand why the IPF crashes.
Could you provide to us your feedback ?

[Woljtek]

A clarification has been requested from ESA: PSC-68 PUG_SY_2_SYN ununderstood exPDUGeneratorException
We are waiting for feedback to go further.

[suberti-ads]

We had new occurrence on sample processing report on PUG-NRT
New Product type impacted: OL_1_ERR, OL_2_LRR, OL_2_LRR, OL_2_LRR, SL_1_RBT

Same error seen

2024-01-17T23:20:19+00:00	{"header":{"type":"REPORT","timestamp":"2024-01-17T23:20:19.055000Z","level":"ERROR","mission":"S3","workflow":"NOMINAL","rs_chain_name":"S3-PUG-NRT","rs_chain_version":"1.14.0"},"message":{"content":"[code 290] [exitCode 255] [msg Task /usr/local/components/PUG-3.48/bin/PUGPostProcessor failed]"},"task":{"uid":"1b73463c-7860-4955-a769-2d084e0c748a","name":"JobProcessing","event":"END","status":"NOK","output":{},"input":{"job_order_id_string":"JobOrder.146541.xml","ipf_release_string":"03.48","filename_strings":["JobOrder.146541.xml","S3B_OL_1_ERR____20240117T144238_20240117T144438_20240117T174035_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145038_20240117T145238_20240117T172440_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T150238_20240117T150438_20240117T172440_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T152038_20240117T152232_20240117T174008_0114_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T144438_20240117T144638_20240117T172440_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151438_20240117T151638_20240117T171638_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151638_20240117T151838_20240117T171639_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T144038_20240117T144238_20240117T173224_0119_088_310______LN3_D_NR_002.SEN3","S3B_AX___FRO_AX_20240114T000000_20240124T000000_20240117T064525___________________EUM_O_AL_001.SEN3","S3B_OL_1_ERR____20240117T150838_20240117T151038_20240117T171637_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T143840_20240117T144038_20240117T170840_0118_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T144638_20240117T144838_20240117T173237_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151038_20240117T151238_20240117T173226_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T150638_20240117T150838_20240117T173229_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T144838_20240117T145038_20240117T172441_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T150438_20240117T150638_20240117T173226_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145638_20240117T145838_20240117T172441_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145438_20240117T145638_20240117T174016_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T150038_20240117T150238_20240117T171637_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151838_20240117T152038_20240117T175633_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145838_20240117T150038_20240117T174834_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T145238_20240117T145438_20240117T171637_0119_088_310______LN3_D_NR_002.SEN3","S3B_OL_1_ERR____20240117T151238_20240117T151438_20240117T170840_0119_088_310______LN3_D_NR_002.SEN3"]},"quality":{},"error_code":1,"duration_in_seconds":187.733,"missing_output":[{"product_metadata_custom_object":{"product_type_string":"OL_1_ERR___","platform_serial_identifier_string":"B"},"end_to_end_product_boolean":true,"estimated_count_integer":1}]}}

Previous IPF logs error which:

2024-01-17T23:18:54+00:00	{"header":{"type":"LOG","timestamp":"2024-01-17T23:18:54.962501Z","level":"INFO","line":129,"file":"TaskCallable.java","thread":"pool-1025-thread-1"},"message":{"content":"Ending task /usr/local/components/PUG-3.48/bin/PUGPostProcessor with exit code 255"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ipf.execution.worker.job.process.TaskCallable"}}
2024-01-17T23:18:54+00:00	2024-01-17T23:18:54.954289 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [I] PUGPostProcessor: Exiting with EXIT CODE: 255
2024-01-17T23:18:54+00:00		FATAL: All the product data unit generations exited in error!
2024-01-17T23:18:54+00:00		Illegal Value: OLCI missing lines global info must have size 1.2024-01-17T23:18:54.954146 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [E] PUGPostProcessor: [PUGPostProcessor.C: execute:(367)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [140025336170688]
2024-01-17T23:18:54+00:00		acs::ManifestBuilderS3OlciL1L2::ManifestBuilderS3OlciL1L2Exception in ManifestBuilderS3OlciL1L2.C(306) from virtual void acs::ManifestBuilderS3OlciL1L2::writeSecondaryProductMetadata() thread "unnamedThread" [140024886081280]
2024-01-17T23:18:54+00:00		caused by:
2024-01-17T23:18:54+00:00		Problem found during product data unit generation -> skipping to the next, if any ...
2024-01-17T23:18:54+00:00	acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [140025336170688]

First Error seem link to this issue:

XMLException caugth: "{140024886081280} @ acs::XMLHandler::hpTrimmedString(...):[200] ": TranscodingException, invalid multi-byte sequence
2024-01-17T23:18:02+00:00	2024-01-17T23:18:02.105971 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [E] PUGPostProcessor: XMLException caugth: "{140024886081280} @ acs::XMLHandler::hpTrimmedString(...):[200] ": TranscodingException, invalid multi-byte sequence
2024-01-17T23:18:02+00:00	2024-01-17T23:18:02.105692 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [E] PUGPostProcessor: XMLException caugth: "{140024886081280} @ acs::XMLHandler::hpTrimmedString(...):[200] ": TranscodingException, invalid multi-byte sequence
2024-01-17T23:18:02+00:00	2024-01-17T23:18:02.098524 s3-pug-nrt-part1-execution-worker-v51-747c777695-bnqlj PUG_OL_1_ERR 03.48 [0000002260]: [W] PUGPostProcessor: MetadataFormatter::updatedOutputFilename : using hardwar ID got from the input filename.

Last part was not seen on another error logs.

sample log for OL_1_ERR:
Explore-logs-2024-01-19_13_19_42.gz
sample log for SL_1_RBT :
s3-pug-nrt-part1-execution-worker-v51-747c777695-cfcbm.log.gz

[suberti-ads]

New Occurrence for SL_1_RBT on PUG-NRT
Product impacted (one execution)

S3B_SL_1_RBT____20240411T140345_20240411T140845_20240411T204550_0299_091_367______LN3_O_NR_002.SEN3
S3B_SL_1_RBT____20240411T135845_20240411T140345_20240411T183010_0299_091_366______LN3_O_NR_002.SEN3

Error

2024-04-11T21:06:55+00:00	2024-04-11T21:06:55.641690 s3-pug-nrt-part1-execution-worker-v52-cf7758c9d-l7sbw PUG_SL_1_RBT 03.48 [0000001425]: [I] PUGPostProcessor: Exiting with EXIT CODE: 255
2024-04-11T21:06:55+00:00		FATAL: All the product data unit generations exited in error!
2024-04-11T21:06:55+00:00		Illegal Value: SLSTR missing lines global info must have size < 8.2024-04-11T21:06:55.641565 s3-pug-nrt-part1-execution-worker-v52-cf7758c9d-l7sbw PUG_SL_1_RBT 03.48 [0000001425]: [E] PUGPostProcessor: [PUGPostProcessor.C: execute:(367)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [139955093186752]
2024-04-11T21:06:55+00:00		acs::ManifestBuilderS3SlstrL1L2::ManifestBuilderS3SlstrL1L2Exception in ManifestBuilderS3SlstrL1L2.C(287) from virtual void acs::ManifestBuilderS3SlstrL1L2::writeSecondaryProductMetadata() thread "unnamedThread" [139954643080960]
2024-04-11T21:06:55+00:00		caused by:
2024-04-11T21:06:55+00:00		Problem found during product data unit generation -> skipping to the next, if any ...
2024-04-11T21:06:55+00:00	acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [139955093186752]

[suberti-ads]

New Occurrence for SY_2_SYN on PUG-STC
Product impacted (one execution)

S3B_SY_2_SYN____20240411T131622_20240411T131822_20240411T213708_0119_091_366______LN3_D_ST_002.SEN3
S3B_SY_2_SYN____20240411T131845_20240411T132022_20240411T173823_0096_091_366______LN3_D_ST_002.SEN3

Error:

2024-04-11T22:10:40+00:00		FATAL: All the product data unit generations exited in error!
2024-04-11T22:10:40+00:00		Illegal Value: OLCI missing lines global info must have size 1.2024-04-11T22:10:40.652555 s3-pug-stc-part1-execution-worker-v12-d7546576c-zbdtd PUG_SY_2_SYN 03.48 [0000000089]: [E] PUGPostProcessor: [PUGPostProcessor.C: execute:(367)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [140160865097920]
2024-04-11T22:10:40+00:00		acs::ManifestBuilderS3SynL1L2::ManifestBuilderS3SynL1L2Exception in ManifestBuilderS3SynL1L2.C(370) from virtual void acs::ManifestBuilderS3SynL1L2::writeSecondaryProductMetadata() thread "unnamedThread" [140160415008512]
2024-04-11T22:10:40+00:00		caused by:
2024-04-11T22:10:40+00:00		Problem found during product data unit generation -> skipping to the next, if any ...
2024-04-11T22:10:40+00:00	acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [140160865097920]

[suberti-ads]

No CAMS still created on this subject ==> To be created

[suberti-ads]

15 New occurences:
Sample for S3B_SL_1_RBT____20241010T135022_20241010T135522_20241010T150624_0299_098_267______LN3_O_NR_002

2024-10-10T17:30:55+00:00		FATAL: All the product data unit generations exited in error!
2024-10-10T17:30:55+00:00		Illegal Value: SLSTR missing lines global info must have size < 8.2024-10-10T17:30:55.757440 s3-pug-nrt-part1-execution-worker-v54-74cf49cbd4-9n9wp PUG_SL_1_RBT 03.48 [0000000615]: [E] PUGPostProcessor: [PUGPostProcessor.C: execute:(367)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [140479404943552]
2024-10-10T17:30:55+00:00		acs::ManifestBuilderS3SlstrL1L2::ManifestBuilderS3SlstrL1L2Exception in ManifestBuilderS3SlstrL1L2.C(287) from virtual void acs::ManifestBuilderS3SlstrL1L2::writeSecondaryProductMetadata() thread "unnamedThread" [140478954837760]
2024-10-10T17:30:55+00:00		caused by:
2024-10-10T17:30:55+00:00		Problem found during product data unit generation -> skipping to the next, if any ...
2024-10-10T17:30:55+00:00	acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [140479404943552]