diff --git a/docs/cli.md b/docs/cli.md index 37875bdf..e7d312a3 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -14,7 +14,6 @@ Options: Commands: alias Query remote and local aliases. config Query/update application configuration. - database Manage local simulation database. manifest Create/check manifest file. provenance Create the PROVENANCE_FILE from the current system. remote Interact with the remote SimDB service. @@ -142,7 +141,6 @@ Options: --help Show this message and exit. ``` - ## Manifest @@ -421,30 +419,38 @@ Usage: simdb remote [NAME] query [OPTIONS] [CONSTRAINTS]... NAME=[mod]VALUE Where `[mod]` is an optional query modifier. Available query modifiers are: - eq: - This checks for equality (this is the same behaviour as not providing any modifier). + eq: - This checks for equality (this is the same behaviour as not providing + any modifier). in: - This searches inside the value instead of looking for exact matches. gt: - This checks for values greater than the given quantity. agt: - This checks for any array elements are greater than the given quantity. ge: - This checks for values greater than or equal to the given quantity. - age: - This checks for any array elements are greater than or equal to the given quantity. + age: - This checks for any array elements are greater than or equal to the given + quantity. lt: - This checks for values less than the given quantity. - alt: - This checks for any array elements are less than the given quantity. + alt: - This checks for any array elements are less than the given quantity. le: - This checks for values less than or equal to the given quantity. - ale: - This checks for any array elements are less than or equal to the given quantity. + ale: - This checks for any array elements are less than or equal to the given + quantity. Modifier examples: alias=eq:foo performs exact match - summary.code.name=in:foo matches all names containing foo - summary.heating_current_drive.power_additional.value=agt:0 matches all simulations where any array element - of summary.heating_current_drive.power_additional.value is greater than 0 + summary.code.name=in:foo matches all names + containing foo + summary.heating_current_drive.power_additional.value=agt:0 matches all + simulations where any array element of + summary.heating_current_drive.power_additional.value is greater than 0 - Any string comparisons are done in a case-insensitive manner. If multiple constraints are provided then simulations - are returned that match all given constraints. + Any string comparisons are done in a case-insensitive manner. If multiple + constraints are provided then simulations are returned that match all given + constraints. Examples: - sim remote query workflow.name=in:test finds all simulations where workflow.name contains test - (case-insensitive) - sim remote query pulse=gt:1000 run=0 finds all simulations where pulse is > 1000 and run = 0 + sim remote query workflow.name=in:test finds all simulations where + workflow.name contains test + (case-insensitive) + sim remote query pulse=gt:1000 run=0 finds all simulations where pulse + is > 1000 and run = 0 Options: -m, --meta-data TEXT Additional meta-data field to print. @@ -603,20 +609,43 @@ Options: --help Show this message and exit. Commands: - delete Delete the ingested simulation with given SIM_ID (UUID or... - info Print information on the simulation with given SIM_ID (UUID... - ingest Ingest a MANIFEST_FILE. - list List ingested simulations. - modify Modify the ingested simulation. - pull Pull the simulation with the given SIM_ID (UUID or alias)... - push Push the simulation with the given SIM_ID (UUID or alias) to... - query Perform a metadata query to find matching local simulations. - validate Validate the ingested simulation with given SIM_ID (UUID or... + data Fetch IDS field data for simulation SIM_ID (UUID or alias)... + delete Delete the ingested simulation with given SIM_ID (UUID or... + info Print information on the simulation with given SIM_ID (UUID... + ingest Ingest a MANIFEST_FILE. + list List ingested simulations. + modify Modify the ingested simulation. + pull Pull the simulation with the given SIM_ID (UUID or alias)... + push Push the simulation with the given SIM_ID (UUID or alias)... + push_local Push the simulation with the given SIM_ID (UUID or alias)... + query Perform a metadata query to find matching local simulations. + validate Validate the ingested simulation with given SIM_ID (UUID or... ``` ```text -Usage: simdb simulation delete [OPTIONS] SIM_ID +Usage: simdb simulation data [OPTIONS] [REMOTE] SIM_ID IDS_PATH + + Fetch IDS field data for simulation SIM_ID (UUID or alias) from REMOTE. + + IDS_PATH format: + ids_name[:]/path/to/field + + Examples: + simdb sim data iter 4dd781b... profiles_1d[0]/grid/rho_tor_norm + simdb sim data 4dd781b... equilibrium:0/time_slice[0]/profiles_1d/psi + +Options: + --username TEXT Username used to authenticate with the remote. + --password TEXT Password used to authenticate with the remote. + --dd-version TEXT Convert IDS data to the requested Data Dictionary + version, e.g. 4.1.1. + --help Show this message and exit. +``` + + +```text +Usage: simdb simulation delete [OPTIONS] [SIM_ID] Delete the ingested simulation with given SIM_ID (UUID or alias). @@ -703,6 +732,22 @@ Options: ``` +```text +Usage: simdb simulation push_local [OPTIONS] [REMOTE] SIM_ID + + Push the simulation with the given SIM_ID (UUID or alias) to the REMOTE. + +Options: + --username TEXT Username used to authenticate with the remote. + --password TEXT Password used to authenticate with the remote. + --replaces TEXT SIM_ID of simulation to deprecate and replace. + --add-watcher Add the current user as a watcher of the simulation. + --timeout FLOAT Maximum number of seconds to wait for ingestion to + complete. [default: 600.0] + --help Show this message and exit. +``` + + ```text Usage: simdb simulation query [OPTIONS] [CONSTRAINTS]... @@ -712,7 +757,8 @@ Usage: simdb simulation query [OPTIONS] [CONSTRAINTS]... NAME=[mod]VALUE Where `[mod]` is an optional query modifier. Available query modifiers are: - eq: - This checks for equality (this is the same behaviour as not providing any modifier). + eq: - This checks for equality (this is the same behaviour as not providing any + modifier). ne: - This checks for value that do not equal. in: - This searches inside the value instead of looking for exact matches. ni: - This searches inside the value for elements that do not match. @@ -722,22 +768,26 @@ Usage: simdb simulation query [OPTIONS] [CONSTRAINTS]... le: - This checks for values less than or equal to the given quantity. For the following modifiers, VALUE should not be provided. exist: - This - returns simulations where metadata with NAME exists, regardless of the - value. + returns simulations where metadata with NAME exists, regardless + of the value. Modifier examples: responsible_name=foo performs exact match responsible_name=in:foo matches all names containing foo pulse=gt:1000 matches all pulses > 1000 - sequence=exist: matches all simulations that have "sequence" metadata values + sequence=exist: matches all simulations that have "sequence" + metadata values - Any string comparisons are done in a case-insensitive manner. If multiple constraints are provided then simulations - are returned that match all given constraints. + Any string comparisons are done in a case-insensitive manner. If multiple + constraints are provided then simulations are returned that match all given + constraints. Examples: - sim simulation query workflow.name=in:test finds all simulations where workflow.name contains test + sim simulation query workflow.name=in:test finds all simulations where + workflow.name contains test (case-insensitive) - sim simulation query pulse=gt:1000 run=0 finds all simulations where pulse is > 1000 and run = 0 + sim simulation query pulse=gt:1000 run=0 finds all simulations where + pulse is > 1000 and run = 0 Options: -m, --meta-data TEXT Additional meta-data field to print. diff --git a/docs/cli.md.in b/docs/cli.md.in index c1b8371b..adf14fa3 100644 --- a/docs/cli.md.in +++ b/docs/cli.md.in @@ -10,10 +10,6 @@ {{ config }} -## Database - -{{ database }} - ## Manifest {{ manifest }} diff --git a/docs/user_guide.md b/docs/user_guide.md index 15b927b0..1b438ed1 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -297,6 +297,54 @@ simdb simulation push This will upload all the metadata associated with your simulation to the remote server as well as taking copies of all input and output data specified. For non-IMAS data the `file` URIs will be used to locate the files to transfer, whereas for `imas` URIs SimDB will discover which files need to be transferred based on the IMAS backend specified in the URI. The files are copied to the server using an HTTP data transfer. +### Pushing Local Simulations (Optimized for Shared File Systems) + +If your local environment and the remote SimDB server share a common local file system (for example, on the ITER network where both you and the server can access same physical file paths directly), uploading large datasets over HTTP can be slow and redundant. + +In this scenario, you should use the `push_local` command: + +```bash +simdb simulation push_local +``` + +Unlike the standard `push` command, `push_local` only sends the simulation metadata and the storage file paths to the server. The remote server then: +1. Validates the simulation metadata against active schemas. +2. Queues the file copy operation in the background using an asynchronous Celery task queue. +3. Automatically completes the ingestion once background file copying finishes. + +The CLI command will block and print real-time updates while waiting for the background ingestion to complete: +```text +Waiting for ingestion to complete... queued -> copy_files -> completed +Successfully pushed simulation +``` + +#### Configuring Partitions (Shared Storage Mappings) + +To enable `push_local` to map and resolve files correctly between your local environment and the remote server, you must configure **partitions** in your `simdb.cfg` configuration file. + +Partitions define mappings between short, logical names (such as `data` or `work`) and their absolute paths on your local file system. + +##### 1. Defining Partitions on the Client +Add a `[partition]` section to your `~/.config/simdb/simdb.cfg` file, specifying the directory paths for each partition: + +```ini +[partition] +data = /home/user/my_simdb_data +work = /work/imas/shared +sdcc = / +``` + +*Note on `sdcc` partition mapping:* +In environments like the ITER network, files are often located under absolute paths like `/sdcc/projects/...`. Mapping the `sdcc` partition to the system root `/` ensures that any path beginning with `/sdcc` is correctly matched and converted to a partition-relative URI (e.g., `/sdcc/projects/my_run` becomes `sdcc:sdcc/projects/my_run`). When several partitions contain a file, the partition with the most specific (deepest) path wins, so such a catch-all mapping never shadows the other partitions. + +##### 2. How Partitions are Resolved +When you run `push_local`: +* SimDB scans the manifest's input/output files and checks if any path falls under one of your defined local partitions. +* If a match is found (e.g., `/home/user/my_simdb_data/scenarios/run1.txt` is inside `/home/user/my_simdb_data`), SimDB converts the file URI into a partition-relative scheme: `data:scenarios/run1.txt`. +* The remote server receives this logical URI. As long as the server also has the `data` partition configured (even if mounted at a different absolute path like `/mnt/shared/partition`), it resolves the URI to `/mnt/shared/partition/scenarios/run1.txt` and completes ingestion. + +This mapping mechanism allows clients and servers to share data over a network or cluster filesystem even if they mount it at different absolute paths. + ## Pulling simulations from a remote The mirror to pushing simulations is the `pull` command. This command will pull the simulation metadata from the SimDB remote to your local SimDB database and download the simulation data into a directory of your choosing. Once you have pulled a simulation it will appear in any local SimDB queries you perform. The command looks as follows: diff --git a/pyproject.toml b/pyproject.toml index 0ff24106..44a29a20 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -41,7 +41,7 @@ dependencies = [ "distro>=1.8.0", "email-validator>=1.1", "imas-python>=2.0.1", - "netCDF4>=1.5", + "netCDF4>=1.7.2", "numpy>=1.14", "pydantic>=2.10.6", "python-dateutil>=2.6", diff --git a/src/simdb/checksum.py b/src/simdb/checksum.py index 99aef230..f85aea1f 100644 --- a/src/simdb/checksum.py +++ b/src/simdb/checksum.py @@ -4,6 +4,19 @@ from simdb.imas.utils import SimDBUrl +def calculate_checksum(path: Path) -> str: + """Generate a SHA1 checksum from the file at the given path. + + :param path: the path of the file to checksum + :return: a string containing the hex representation of the computed SHA1 checksum + """ + sha1 = hashlib.sha1() + with path.open("rb") as file: + for chunk in iter(lambda: file.read(4096), b""): + sha1.update(chunk) + return sha1.hexdigest() + + def sha1_checksum(uri: SimDBUrl) -> str: """Generate a SHA1 checksum from the given file. @@ -21,8 +34,4 @@ def sha1_checksum(uri: SimDBUrl) -> str: if not path.is_file(): raise ValueError("File appears to be a directory") - sha1 = hashlib.sha1() - with path.open("rb") as file: - for chunk in iter(lambda: file.read(4096), b""): - sha1.update(chunk) - return sha1.hexdigest() + return calculate_checksum(path) diff --git a/src/simdb/cli/commands/simulation.py b/src/simdb/cli/commands/simulation.py index 59bd0bdd..11f57358 100644 --- a/src/simdb/cli/commands/simulation.py +++ b/src/simdb/cli/commands/simulation.py @@ -1,5 +1,6 @@ import contextlib import sys +import time import urllib.parse from itertools import chain from pathlib import Path @@ -14,6 +15,7 @@ from simdb.config.config import Config from simdb.database import DatabaseError, get_local_db from simdb.database.models import Simulation +from simdb.enums import IngestionStatus from simdb.query import QueryType, parse_query_arg from simdb.validation import ValidationError, Validator @@ -212,6 +214,116 @@ def parse_args(self, ctx, args): return NRequiredArgs +@simulation.command("push_local", cls=n_required_args_adaptor(1)) +@pass_config +@click.argument("remote", required=False) +@click.argument("sim_id") +@click.option("--username", help="Username used to authenticate with the remote.") +@click.option("--password", help="Password used to authenticate with the remote.") +@click.option("--replaces", help="SIM_ID of simulation to deprecate and replace.") +@click.option( + "--add-watcher", + is_flag=True, + help="Add the current user as a watcher of the simulation.", +) +@click.option( + "--timeout", + type=float, + default=600.0, + show_default=True, + help="Maximum number of seconds to wait for ingestion to complete.", +) +def simulation_push_local( + config: Config, + remote: Optional[str], + sim_id: str, + username: Optional[str], + password: Optional[str], + replaces: Optional[str], + add_watcher: bool, + timeout: float, +): + """Push the simulation with the given SIM_ID (UUID or alias) to the REMOTE.""" + + api = RemoteAPI(remote, username, password, config) + db = get_local_db(config) + + simulation = db.get_simulation(sim_id) + if simulation is None: + raise click.ClickException(f"Failed to find simulation: {sim_id}") + + if replaces: + simulation.set_meta("replaces", replaces) + + schemas = api.get_validation_schemas() + try: + for schema in schemas: + Validator(schema).validate(simulation) + except ValidationError as err: + raise click.ClickException(f"Simulation does not validate: {err}") from err + + api.push_local_simulation(simulation, add_watcher=add_watcher) + + terminal_statuses = { + IngestionStatus.COMPLETED.value, + IngestionStatus.COPY_FAILED.value, + IngestionStatus.VALIDATION_FAILED.value, + } + + max_consecutive_failures = 5 + + click.echo("Waiting for ingestion to complete...", nl=False) + last_status = None + consecutive_failures = 0 + deadline = time.monotonic() + timeout + while True: + try: + status = api.get_ingestion_status(simulation.uuid.hex) + except Exception as err: + # Tolerate transient errors: the ingestion continues server-side + consecutive_failures += 1 + if consecutive_failures >= max_consecutive_failures: + click.echo() + raise click.ClickException( + f"Failed to check ingestion status " + f"{consecutive_failures} times in a row: {err}" + ) from err + if time.monotonic() >= deadline: + click.echo() + raise click.ClickException( + f"Timed out after {timeout:g}s waiting for ingestion to " + f"complete (last status: {last_status})" + ) from err + time.sleep(1) + continue + consecutive_failures = 0 + + if status != last_status: + if last_status is not None: + click.echo(f" -> {status}", nl=False) + else: + click.echo(f" {status}", nl=False) + last_status = status + + if status in terminal_statuses: + break + + if time.monotonic() >= deadline: + click.echo() + raise click.ClickException( + f"Timed out after {timeout:g}s waiting for ingestion to complete " + f"(last status: {status})" + ) + + time.sleep(1) + + click.echo() + if status == IngestionStatus.COMPLETED.value: + click.echo(f"Successfully pushed simulation {simulation.uuid}") + else: + raise click.ClickException(f"Simulation ingestion failed with status: {status}") + + @simulation.command("push", cls=n_required_args_adaptor(1)) @pass_config @click.argument("remote", required=False) diff --git a/src/simdb/cli/remote_api.py b/src/simdb/cli/remote_api.py index 99f87363..7baaea4f 100644 --- a/src/simdb/cli/remote_api.py +++ b/src/simdb/cli/remote_api.py @@ -24,6 +24,7 @@ Optional, Tuple, Union, + cast, ) from urllib.parse import urlparse @@ -33,11 +34,13 @@ from requests.auth import AuthBase from semantic_version import Version +from simdb.checksum import calculate_checksum from simdb.config import Config from simdb.database.models import Simulation from simdb.imas.utils import SimDBUrl, imas_files from simdb.json import CustomDecoder, CustomEncoder from simdb.remote import CLIENT_API_VERSIONS, APIConstants +from simdb.remote.models import FileData, SimulationPostData from .manifest import DataType @@ -235,6 +238,77 @@ def _get_paths(file: "File") -> Iterable[Path]: return imas_files(file.uri) +def _find_partition_for_file( + file: Path, partitions: Dict[str, str] +) -> Tuple[str, Path]: + # Match the partition with the longest root so that a catch-all mapping + # (e.g. "/") does not shadow more specific partitions. + best: Optional[Tuple[str, Path]] = None + best_depth = -1 + for partition, path in partitions.items(): + root = Path(path) + try: + relative = file.relative_to(root) + except ValueError: + continue + depth = len(root.parts) + if depth > best_depth: + best = (partition, relative) + best_depth = depth + if best is None: + raise APIError( + f"File {file} is not located under any configured partition " + f"(configured partitions: {', '.join(partitions) or 'none'})" + ) + return best + + +def _file_data_for_partition( + file: FileData, source: Path, partitions: Dict[str, str] +) -> FileData: + partition, partition_path = _find_partition_for_file(source, partitions) + new_uri = SimDBUrl.build(scheme=partition, path=partition_path.as_posix()) + return FileData( + type=file.type, + uri=new_uri.encoded_string(), + checksum=calculate_checksum(source), + datetime=file.datetime, + usage=file.usage, + purpose=file.purpose, + sensitivity=file.sensitivity, + access=file.access, + embargo=file.embargo, + ) + + +def _expand_directories(files: Iterable[FileData], partitions: Dict[str, str]): + new_file_list = [] + for file in files: + file_uri = SimDBUrl(file.uri) + if file_uri.path is None: + raise APIError(f"File URI has no path: {file.uri}") + file_path = Path(file_uri.path) + if file_uri.scheme == "imas": + qs = dict(file_uri.query_params()) + path = qs.get("path") + if path is None: + raise APIError(f"IMAS URI has no path set: {file.uri}") + file_path = Path(path) + + if file_path.is_dir(): + for sub_file in file_path.iterdir(): + if sub_file.is_dir(): + raise APIError( + f"Nested directory found in {file_path}: {sub_file.name}" + ) + new_file_list.append( + _file_data_for_partition(file, sub_file, partitions) + ) + else: + new_file_list.append(_file_data_for_partition(file, file_path, partitions)) + return new_file_list + + class RemoteAPI: """ Class to represent connection to remote API. @@ -856,6 +930,30 @@ def _send_chunk( ] self.post("files", data={}, files=files) + @versioned_method("v1.3") + @try_request + def push_local_simulation(self, simulation: Simulation, add_watcher: bool = False): + sim_data = simulation.to_model(recurse=True) + + partitions = cast(Dict[str, str], self._config.get_section("partition")) + sim_data.inputs.root = _expand_directories(sim_data.inputs.root, partitions) + sim_data.outputs.root = _expand_directories(sim_data.outputs.root, partitions) + + uploaded_by = simulation.meta_dict().get("uploaded_by") + + post_data = SimulationPostData( + simulation=sim_data, + add_watcher=add_watcher, + uploaded_by=str(uploaded_by) if uploaded_by is not None else None, + ) + self.post("simulations", data=post_data.model_dump(mode="json")) + + @versioned_method("v1.3") + @try_request + def get_ingestion_status(self, sim_id: str) -> str: + res = self.get(f"simulation/status/{sim_id}") + return res.json()["status"] + @versioned_method("v1.2", "v1.3") @try_request def push_simulation( diff --git a/src/simdb/imas/utils.py b/src/simdb/imas/utils.py index 7b9ff234..016bc400 100644 --- a/src/simdb/imas/utils.py +++ b/src/simdb/imas/utils.py @@ -311,6 +311,37 @@ def _get_path(uri: SimDBUrl) -> Path: return path +def imas_backend_for_directory(directory: Path) -> str: + """ + Identify the IMAS backend of a directory by inspecting its contents. + + @param directory: a directory that may contain an IMAS dataset + @return: the backend name ("ascii", "hdf5" or "mdsplus") + @raise ValueError: if no IMAS dataset is detected + """ + children = list(directory.iterdir()) + + # ASCII heuristic + if any(child.suffix == ".ids" for child in children): + return "ascii" + + # HDF5 heuristic + if any(child.suffix == ".h5" for child in children) and any( + child.name == "master.h5" for child in children + ): + return "hdf5" + + # MDSplus heuristic + if {p.name for p in children} >= { + "ids_001.tree", + "ids_001.characteristics", + "ids_001.datafile", + }: + return "mdsplus" + + raise ValueError("IMAS backend could not be identified.") + + def imas_files(uri: SimDBUrl) -> List[Path]: """ Return all the files associated with the given IMAS URI. diff --git a/src/simdb/workers/tasks.py b/src/simdb/workers/tasks.py index 97d9213d..77ed2144 100644 --- a/src/simdb/workers/tasks.py +++ b/src/simdb/workers/tasks.py @@ -1,4 +1,3 @@ -import hashlib import itertools import logging import os @@ -8,14 +7,13 @@ from typing import Iterable, List from uuid import UUID -from pydantic import AnyUrl - +from simdb.checksum import calculate_checksum as _calculate_checksum from simdb.config import Config from simdb.database.database import get_db from simdb.database.models import File from simdb.email.server import EmailServer from simdb.enums import IngestionStatus -from simdb.imas.utils import SimDBUrl +from simdb.imas.utils import SimDBUrl, imas_backend_for_directory from simdb.remote.models import FileData, FileDataList from simdb.workers.celery import celery_app @@ -62,36 +60,14 @@ def _imas_path_to_uri(imas_path: Path) -> SimDBUrl: if imas_path.suffix == ".nc": return SimDBUrl.build(scheme="file", path=imas_path.as_posix()) - children = set(imas_path.iterdir()) - - if any(child.suffix == ".ids" for child in children): - u = SimDBUrl.build( - scheme="imas", path="ascii", query=f"path={imas_path.as_posix()}" - ) - return u + backend = imas_backend_for_directory(imas_path) - if any(child.suffix == ".h5" for child in children) and any( - child.name == "master.h5" for child in children - ): - u = SimDBUrl.build( - scheme="imas", path="hdf5", query=f"path={imas_path.as_posix()}" - ) - return u - - if {p.name for p in children} >= { - "ids_001.tree", - "ids_001.characteristics", - "ids_001.datafile", - }: - u = SimDBUrl.build( - scheme="imas", path="mdsplus", query=f"path={imas_path.as_posix()}" - ) - return u - - raise ValueError("IMAS backend could not be identified.") + return SimDBUrl.build( + scheme="imas", path=backend, query=f"path={imas_path.as_posix()}" + ) -def _resolve_uri_to_path(uri: AnyUrl, config: Config) -> Path: +def _resolve_uri_to_path(uri: SimDBUrl, config: Config) -> Path: partition = uri.scheme if not partition: raise ValueError("Partition not given") @@ -133,14 +109,6 @@ def _copy_files( shutil.copy2(source, destination) -def _calculate_checksum(path: Path) -> str: - sha1 = hashlib.sha1() - with path.open("rb") as f: - for chunk in iter(lambda: f.read(4096), b""): - sha1.update(chunk) - return sha1.hexdigest() - - def _get_imas_identifier_path(path: Path) -> Path: if path.suffix == ".nc": return path diff --git a/tests/validation/test_validator.py b/tests/validation/test_validator.py new file mode 100644 index 00000000..bcef6a40 --- /dev/null +++ b/tests/validation/test_validator.py @@ -0,0 +1,56 @@ +import numpy as np + +from simdb.validation.validator import CustomValidator + + +def test_custom_validator_min_value_max_value(): + schema = { + "field1": { + "type": "numpy", + "coerce": "numpy", + "min_value": 0.0, + "max_value": 10.0, + } + } + validator = CustomValidator(schema) + + # Test valid numpy array + assert validator.validate({"field1": np.array([1.0, 5.0, 9.0])}) + + # Test valid dictionary representing a range + assert validator.validate({"field1": {"min": 1.0, "max": 9.0}}) + + # Test numpy array out of bounds (too low) + assert not validator.validate({"field1": np.array([-1.0, 5.0, 9.0])}) + + # Test numpy array out of bounds (too high) + assert not validator.validate({"field1": np.array([1.0, 5.0, 11.0])}) + + # Test dictionary range out of bounds (min too low) + assert not validator.validate({"field1": {"min": -1.0, "max": 9.0}}) + + # Test dictionary range out of bounds (max too high) + assert not validator.validate({"field1": {"min": 1.0, "max": 11.0}}) + + +def test_custom_validator_comparisons(): + schema = { + "field_ge": {"type": "numpy", "coerce": "numpy", "ge": 0.0}, + "field_le": {"type": "numpy", "coerce": "numpy", "le": 10.0}, + } + validator = CustomValidator(schema) + + # Test valid dictionary representing a range + assert validator.validate( + {"field_ge": {"min": 0.0, "max": 5.0}, "field_le": {"min": 1.0, "max": 10.0}} + ) + + # Test invalid range for ge (min is -1, which is not >= 0) + assert not validator.validate( + {"field_ge": {"min": -1.0, "max": 5.0}, "field_le": {"min": 1.0, "max": 10.0}} + ) + + # Test invalid range for le (max is 11, which is not <= 10) + assert not validator.validate( + {"field_ge": {"min": 0.0, "max": 5.0}, "field_le": {"min": 1.0, "max": 11.0}} + ) diff --git a/uv.lock b/uv.lock index 2216c930..532738e0 100644 --- a/uv.lock +++ b/uv.lock @@ -3146,7 +3146,7 @@ requires-dist = [ { name = "imas-validator", marker = "extra == 'imas-validator'", specifier = ">=1.0.0" }, { name = "myst-parser", marker = "extra == 'build-docs'", specifier = ">=0.18.0" }, { name = "nbsphinx", marker = "extra == 'build-docs'", specifier = ">=0.8.0" }, - { name = "netcdf4", specifier = ">=1.5" }, + { name = "netcdf4", specifier = ">=1.7.2" }, { name = "numpy", specifier = ">=1.14" }, { name = "plotext", specifier = "==5.3.2" }, { name = "psycopg2-binary", marker = "extra == 'postgres'", specifier = ">=2.8.0" },