chunk datavalue import

[gqcorneby]

📌 References

• Issue: Closes #869c4fuc1 https://app.clickup.com/t/869c4fuc1

📝 Implementation

• Issue
  • the bulk load template created for this tasks includes 11,355 dataValues if all fields are populated. This results to the system task eventually returning an empty response.
  • The root cause was that when large payloads were imported asynchronously, the ImportSummary (including errors and counts) was never being persisted to the job's store. So polling taskSummaries returned empty so don't get success or error information (https://community.dhis2.org/t/heavy-server-load-in-dhis2-instance-results-in-no-response-to-users/51763/4)
  • Chunk size 5000 still resulted to an empty response for the 1st one but 3000 worked.
• Fix
  • Reduce payload by chunking dataValues

This is the template used to test. Update the period or org unit in "Introduction" tab to test
output_test_filled.xlsx

🔥 Notes for the reviewer

🎨 Screenshots

• Chunk size 5000. This is what it would look like if there are chunks that have empty results

📑 Others

[bundlemon]

BundleMon

No change in files bundle size

Groups updated (1)

Status	Path	Size	Limits
✅	Build Folder ./**/*	1.72MB (+1.06KB +0.06%)	+20%

Final result: ✅

View report in BundleMon website ➡️

---

Current branch size history | Target branch size history

[Ramon-Jimenez]

@gqcorneby I gave it a try using American Samoa 2023 and I got the 'existing data' pop-up showing that there were 3000 DEs with data already. Just wondering if, in fact, there are 3000 DEs with data, or if there is some additional fix to be done in the 'existing data' part that is making it look like there is only 3000 DEs (because of the chunk size) when there are, in fact, more DEs with data

Also, I got this warnings:

Thanks!

[gqcorneby]

I gave it a try using American Samoa 2023 and I got the 'existing data' pop-up showing that there were 3000 DEs with data already. Just wondering if, in fact, there are 3000 DEs with data, or if there is some additional fix to be done in the 'existing data' part that is making it look like there is only 3000 DEs (because of the chunk size) when there are, in fact, more DEs with data

Hi @Ramon-Jimenez! The chunking should be isolated during saving so it shouldn't affect this part of the code. It seems like the 3000 displayed here is just a coincidence. I tried importing the the filled file again and it showed 11,355

In terms of the warning, these dataElements are indeed integer types but they have an optionSet. All of the Country Progress 3.1 dataElements are integers but some have warnings and some are saved. I'll investigate this further.

https://dev.eyeseetea.com/who-dev-41/api/dataElements?filter=name:ilike:gnarf&filter=name:ilike:3.1&fields=id,name,formName,valueType,optionSet[id,name,options[name]]