Implemented Golden EurostatData_Education_Attainment#2086
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a golden summary report and configures validation rules for the Eurostat regional education attainment statistics. The feedback highlights an incorrect relative path for the golden data file in the validation configuration, which needs to be corrected to prevent validation failures.
| "rule_id": "check_goldens_summary_report", | ||
| "validator": "GOLDENS_CHECK", | ||
| "params": { | ||
| "golden_files": "../../../../golden_data/golden_summary_report.csv" |
There was a problem hiding this comment.
The relative path specified for golden_files is incorrect. The golden file golden_summary_report.csv is located in the golden_data directory within the same folder as validation_config.json (i.e., scripts/eurostat/regional_statistics_by_nuts/education_attainment/golden_data/golden_summary_report.csv).
Using ../../../../golden_data/golden_summary_report.csv resolves to the root of the repository, which will cause the validation check to fail because the file cannot be found. It should be updated to golden_data/golden_summary_report.csv.
| "golden_files": "../../../../golden_data/golden_summary_report.csv" | |
| "golden_files": "golden_data/golden_summary_report.csv" |
This import has recurring deletion failures less than 0.01%. To tackle these small deletions, the threshold has been increased, but we also added two new golden checks. These validation checks will verify that the data contains all sets in a golden set, these are defined within the validation_config.json file, which is situated in the same directory as manifest.json. The threshold has been raised to 0.05, a value determined by historical and current deletion percentages, with a small buffer to account for these minor deletions.
PR checklist: https://docs.google.com/spreadsheets/d/1BzweR9Sj58j0H2_BweGTmfE4Z1lrjPZL8u1FS0kzCeg/edit?pli=1&gid=0#gid=0