Description, motivation and use case
There are use cases for being able to include environment variables in the configuration file and have them resolved by the file loader.
The most important one at the moment is to be able to define the TANGO host or the EPICS prefix in examples and test without having to hard code a value which the user needs to manually change.
Proposed solution
I have looked at the resolvers in OmegaConf and think there are useful parts there to take inspiration from which would allow us to add additional resolvers in the future.
We add module wide resolvers like this:
Resolver = Callable[[str, LoadContext | None], Any]
RESOLVERS: dict[str, Resolver] = {}
and then a small decorator to register a resolver:
def resolver(name: str):
def decorate(func: Resolver) -> Resolver:
RESOLVERS[name] = func
return func
return decorate
Resolvers can then be written like this:
@resolver("env")
def resolve_env(...)
@resolver("file")
def resolve_file(...):
Question is then how the syntax for the resolvers should look like in the configuration file? In OmegaConf they use the syntax ${env:USER} to make it clear that it is something that should be resolved. I think this syntax works nicely because it makes it clear what the start and end is, allowing for example to write localhost:${env:UID} and use environment variables as part of a string.
In our case we can have:
${env:USER}${file:sub_config.yaml}- future other options...
But we also currently have two other syntax:
file:sub_config.yamlsub_config.yaml
I suggest to no longer support file:sub_config.yaml and replace it with ${file:sub_config.yaml}, but keep supporting the option to just put the file without any prefix. It will then be resolved based on the file extension being in the list of supported format similar to how it's done at the moment.
Describe alternatives you've considered
I have considered to keep supporting the file:sub_config.yaml syntax, but I think that makes it confusing for the users because it makes it unclear why there is the ${file:sub_config.yaml} option.
I have also considered to use different syntax for environment variables and files but I think long term that will cause difficulties and confusion if the number of resolvers grow. It would be better to have one generic syntax that works for all resolvers.
Still, I think we should keep supporting the syntax to just put the file and resolve based on extension since we currently have many configuration files which follows that approach.
Checklist
Description, motivation and use case
There are use cases for being able to include environment variables in the configuration file and have them resolved by the file loader.
The most important one at the moment is to be able to define the TANGO host or the EPICS prefix in examples and test without having to hard code a value which the user needs to manually change.
Proposed solution
I have looked at the resolvers in OmegaConf and think there are useful parts there to take inspiration from which would allow us to add additional resolvers in the future.
We add module wide resolvers like this:
and then a small decorator to register a resolver:
Resolvers can then be written like this:
Question is then how the syntax for the resolvers should look like in the configuration file? In OmegaConf they use the syntax
${env:USER}to make it clear that it is something that should be resolved. I think this syntax works nicely because it makes it clear what the start and end is, allowing for example to writelocalhost:${env:UID}and use environment variables as part of a string.In our case we can have:
${env:USER}${file:sub_config.yaml}But we also currently have two other syntax:
file:sub_config.yamlsub_config.yamlI suggest to no longer support
file:sub_config.yamland replace it with${file:sub_config.yaml}, but keep supporting the option to just put the file without any prefix. It will then be resolved based on the file extension being in the list of supported format similar to how it's done at the moment.Describe alternatives you've considered
I have considered to keep supporting the
file:sub_config.yamlsyntax, but I think that makes it confusing for the users because it makes it unclear why there is the${file:sub_config.yaml}option.I have also considered to use different syntax for environment variables and files but I think long term that will cause difficulties and confusion if the number of resolvers grow. It would be better to have one generic syntax that works for all resolvers.
Still, I think we should keep supporting the syntax to just put the file and resolve based on extension since we currently have many configuration files which follows that approach.
Checklist