Glossary data storage

[cmutel]

AFAICT, the data storage used by the ecoinvent and Hestia glossaries isn't perfect. This is a hard problem!

We take the following as given:

• The glossary elements will be JSON-LD documents.
• There will be many elements - without compression this could easily be hundreds of megabytes (the RDF representation of the latest combined nomenclature is 138 MiB).

We need the following functionality:

• Updates shouldn't take hours to propagate to the API/UI
• The glossary can be forked for work in progress
• Forks can be reviewed and merged back into the main branch
• User can refer and link to previous versions of the glossary (depends on how releases are structured)

Open questions:

• Git versus NoSQL versus relational database versus other?
• Should the data storage be chosen for compatibility with other semantic tools?
• Should the data storage be designed for search/API use, or vice-versa?

[mklarmann]

GADM itself is already a couple of GB, which is still be fine to be kept in memory.
The glossary is limited to public data, there can be nothing client specific in there, which limits naturally its size.

We are currently considering how the GDSN would fit into this. There is a lot of public data, but it is too huge for the glossary in memory.

[cmutel]

The Global Data Synchronization Network (GDSN) is an internet-based, interconnected network of interoperable data pools governed by GS1 standards. The GDSN enables companies around the globe to exchange standardized product master data with their trading partners.

Seems quite interesting, but probably for phase 2, as it is not free data, and very specific to companies and products. I think we need to get the basics sorted first.

[cmutel]

We also need to store models in the glossary - or in something like the glossary, this would mostly contain very specific implementation details needed for orchestration.

[mklarmann]

This was also an idea we had. To register the gap-filling-models in the glossary. As they might have a bit of a fluent understanding of the data schema they can work with. As well as changing requirements regarding the properties they read and write. Also: the glossary will be a public and a first contact point for users, so it directly attaches to the documentation. Which is of course relevant for the gap-filling-models.

Or were you talking about another kind of "models" here?

[jsvgoncalves]

Or were you talking about another kind of "models" here?

Yes, any kind of computational model that can be used throughout the calculation chain should be described in the glossary (and thus it can be referred to, giving us further traceability); that can include exactly what you describe but also definitions of scope, validity, boundary conditions, meta-data etc.

[jsvgoncalves]

• How do we map terms that people bring in to existing terms in the glossary? Do we always add a new entry? Or do we create new references to other entries?
• What maps to what, do we have mapping files or map on a individual record basis?
• Do we let models be free of the terms they use and make sure that for each term
• Do we need a mapping service?
• Should we be super flexible in used terms at the risk at creating incompatibilities?
• Do we need a separate layer so that modellers don't need to redesign there models but instead create a simple facade? → We can create an example showcasing this
• Do we allow importing from external glossaries? Users can bring their own and we help them map it?
• How do we handle linking up to external systems? e.g. simapro? Between internal models it probably isn't very hard but what if we want to export to offline systems?