Skip to content

feat: make json loader read dtypes correctly and also load small json values#72

Open
jokasimr wants to merge 2 commits into
mainfrom
fix-json-loader
Open

feat: make json loader read dtypes correctly and also load small json values#72
jokasimr wants to merge 2 commits into
mainfrom
fix-json-loader

Conversation

@jokasimr

@jokasimr jokasimr commented Jul 2, 2026

Copy link
Copy Markdown
Contributor
  1. Previously the json chexus reader loaded type instead of dtype which seems to be the correct field to read from the Nexus JSON templates.

  2. The json reader also didn't store any datasets, which makes sense because we don't want to run out of memory if we read large files. But that also breaks a lot of useful cases, mainly validating transformation chains.

So here I made it store only small values up to 100KB.
Unless the file has ~80K small datasets we are unlikely to run out of space with that limit.

  1. Loading bifrost json files failed before because they contain "link" entries that were not recognized by the json reader. This PR does not process such entries, but it makes them ignored instead of raising errors.

@jokasimr jokasimr removed the request for review from YooSunYoung July 6, 2026 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant