Add ability to push local (to the server) files

docs/cli.md

@@ -14,7 +14,6 @@ Options:
 Commands:
   alias       Query remote and local aliases.
   config      Query/update application configuration.
-  database    Manage local simulation database.
   manifest    Create/check manifest file.
   provenance  Create the PROVENANCE_FILE from the current system.
   remote      Interact with the remote SimDB service.
@@ -142,7 +141,6 @@ Options:
   --help  Show this message and exit.
 ```
 
-
 ## Manifest
 
 
@@ -421,30 +419,38 @@ Usage: simdb remote [NAME] query [OPTIONS] [CONSTRAINTS]...
       NAME=[mod]VALUE
 
   Where `[mod]` is an optional query modifier. Available query modifiers are:
-      eq:  - This checks for equality (this is the same behaviour as not providing any modifier).
+      eq:  - This checks for equality (this is the same behaviour as not providing
+             any modifier).
       in:  - This searches inside the value instead of looking for exact matches.
       gt:  - This checks for values greater than the given quantity.
       agt: - This checks for any array elements are greater than the given quantity.
       ge:  - This checks for values greater than or equal to the given quantity.
-      age: - This checks for any array elements are greater than or equal to the given quantity.
+      age: - This checks for any array elements are greater than or equal to the given
+             quantity.
       lt:  - This checks for values less than the given quantity.
-      alt:  - This checks for any array elements are less than the given quantity.
+      alt: - This checks for any array elements are less than the given quantity.
       le:  - This checks for values less than or equal to the given quantity.
-      ale:  - This checks for any array elements are less than or equal to the given quantity.
+      ale: - This checks for any array elements are less than or equal to the given
+              quantity.
 
   Modifier examples:
       alias=eq:foo                                                performs exact match
-      summary.code.name=in:foo                                    matches all names containing foo
-      summary.heating_current_drive.power_additional.value=agt:0  matches all simulations where any array element
-      of summary.heating_current_drive.power_additional.value is greater than 0
+      summary.code.name=in:foo                                    matches all names
+          containing foo
+      summary.heating_current_drive.power_additional.value=agt:0  matches all
+          simulations where any array element of
+          summary.heating_current_drive.power_additional.value is greater than 0
 
-  Any string comparisons are done in a case-insensitive manner. If multiple constraints are provided then simulations
-  are returned that match all given constraints.
+  Any string comparisons are done in a case-insensitive manner. If multiple
+  constraints are provided then simulations are returned that match all given
+  constraints.
 
   Examples:
-      sim remote query workflow.name=in:test       finds all simulations where workflow.name contains test
-                                                       (case-insensitive)
-      sim remote query pulse=gt:1000 run=0         finds all simulations where pulse is > 1000 and run = 0
+      sim remote query workflow.name=in:test       finds all simulations where
+                                                   workflow.name contains test
+                                                   (case-insensitive)
+      sim remote query pulse=gt:1000 run=0         finds all simulations where pulse
+                                                   is > 1000 and run = 0
 
 Options:
   -m, --meta-data TEXT  Additional meta-data field to print.
@@ -603,20 +609,43 @@ Options:
   --help  Show this message and exit.
 
 Commands:
-  delete    Delete the ingested simulation with given SIM_ID (UUID or...
-  info      Print information on the simulation with given SIM_ID (UUID...
-  ingest    Ingest a MANIFEST_FILE.
-  list      List ingested simulations.
-  modify    Modify the ingested simulation.
-  pull      Pull the simulation with the given SIM_ID (UUID or alias)...
-  push      Push the simulation with the given SIM_ID (UUID or alias) to...
-  query     Perform a metadata query to find matching local simulations.
-  validate  Validate the ingested simulation with given SIM_ID (UUID or...
+  data        Fetch IDS field data for simulation SIM_ID (UUID or alias)...
+  delete      Delete the ingested simulation with given SIM_ID (UUID or...
+  info        Print information on the simulation with given SIM_ID (UUID...
+  ingest      Ingest a MANIFEST_FILE.
+  list        List ingested simulations.
+  modify      Modify the ingested simulation.
+  pull        Pull the simulation with the given SIM_ID (UUID or alias)...
+  push        Push the simulation with the given SIM_ID (UUID or alias)...
+  push_local  Push the simulation with the given SIM_ID (UUID or alias)...
+  query       Perform a metadata query to find matching local simulations.
+  validate    Validate the ingested simulation with given SIM_ID (UUID or...
 ```
 
 
 ```text
-Usage: simdb simulation delete [OPTIONS] SIM_ID
+Usage: simdb simulation data [OPTIONS] [REMOTE] SIM_ID IDS_PATH
+
+  Fetch IDS field data for simulation SIM_ID (UUID or alias) from REMOTE.
+
+  IDS_PATH format:
+      ids_name[:<occurrence>]/path/to/field
+
+  Examples:
+      simdb sim data iter 4dd781b... profiles_1d[0]/grid/rho_tor_norm
+      simdb sim data 4dd781b... equilibrium:0/time_slice[0]/profiles_1d/psi
+
+Options:
+  --username TEXT    Username used to authenticate with the remote.
+  --password TEXT    Password used to authenticate with the remote.
+  --dd-version TEXT  Convert IDS data to the requested Data Dictionary
+                     version, e.g. 4.1.1.
+  --help             Show this message and exit.
+```
+
+
+```text
+Usage: simdb simulation delete [OPTIONS] [SIM_ID]
 
   Delete the ingested simulation with given SIM_ID (UUID or alias).
 
@@ -703,6 +732,22 @@ Options:
 ```
 
 
+```text
+Usage: simdb simulation push_local [OPTIONS] [REMOTE] SIM_ID
+
+  Push the simulation with the given SIM_ID (UUID or alias) to the REMOTE.
+
+Options:
+  --username TEXT  Username used to authenticate with the remote.
+  --password TEXT  Password used to authenticate with the remote.
+  --replaces TEXT  SIM_ID of simulation to deprecate and replace.
+  --add-watcher    Add the current user as a watcher of the simulation.
+  --timeout FLOAT  Maximum number of seconds to wait for ingestion to
+                   complete.  [default: 600.0]
+  --help           Show this message and exit.
+```
+
+
 ```text
 Usage: simdb simulation query [OPTIONS] [CONSTRAINTS]...
 
@@ -712,7 +757,8 @@ Usage: simdb simulation query [OPTIONS] [CONSTRAINTS]...
       NAME=[mod]VALUE
 
   Where `[mod]` is an optional query modifier. Available query modifiers are:
-      eq: - This checks for equality (this is the same behaviour as not providing any modifier).
+      eq: - This checks for equality (this is the same behaviour as not providing any
+            modifier).
       ne: - This checks for value that do not equal.
       in: - This searches inside the value instead of looking for exact matches.
       ni: - This searches inside the value for elements that do not match.
@@ -722,22 +768,26 @@ Usage: simdb simulation query [OPTIONS] [CONSTRAINTS]...
       le: - This checks for values less than or equal to the given quantity.
 
   For the following modifiers, VALUE should not be provided.     exist: - This
-  returns simulations where metadata with NAME exists, regardless of the
-  value.
+  returns simulations where metadata with NAME exists, regardless
+  of the value.
 
   Modifier examples:
       responsible_name=foo        performs exact match
       responsible_name=in:foo     matches all names containing foo
       pulse=gt:1000               matches all pulses > 1000
-      sequence=exist:             matches all simulations that have "sequence" metadata values
+      sequence=exist:             matches all simulations that have "sequence"
+                                  metadata values
 
-  Any string comparisons are done in a case-insensitive manner. If multiple constraints are provided then simulations
-  are returned that match all given constraints.
+  Any string comparisons are done in a case-insensitive manner. If multiple
+  constraints are provided then simulations are returned that match all given
+  constraints.
 
   Examples:
-      sim simulation query workflow.name=in:test       finds all simulations where workflow.name contains test
+      sim simulation query workflow.name=in:test       finds all simulations where
+                                                       workflow.name contains test
                                                        (case-insensitive)
-      sim simulation query pulse=gt:1000 run=0         finds all simulations where pulse is > 1000 and run = 0
+      sim simulation query pulse=gt:1000 run=0         finds all simulations where
+                                                       pulse is > 1000 and run = 0
 
 Options:
   -m, --meta-data TEXT  Additional meta-data field to print.

docs/cli.md.in

@@ -10,10 +10,6 @@
 
 {{ config }}
 
-## Database
-
-{{ database }}
-
 ## Manifest
 
 {{ manifest }}

docs/user_guide.md

@@ -297,6 +297,54 @@ simdb simulation push <SIM_ID>
 
 This will upload all the metadata associated with your simulation to the remote server as well as taking copies of all input and output data specified. For non-IMAS data the `file` URIs will be used to locate the files to transfer, whereas for `imas` URIs SimDB will discover which files need to be transferred based on the IMAS backend specified in the URI. The files are copied to the server using an HTTP data transfer.
 
+### Pushing Local Simulations (Optimized for Shared File Systems)
+
+If your local environment and the remote SimDB server share a common local file system (for example, on the ITER network where both you and the server can access same physical file paths directly), uploading large datasets over HTTP can be slow and redundant.
+
+In this scenario, you should use the `push_local` command:
+
+```bash
+simdb simulation push_local <SIM_ID>
+```
+
+Unlike the standard `push` command, `push_local` only sends the simulation metadata and the storage file paths to the server. The remote server then:
+1. Validates the simulation metadata against active schemas.
+2. Queues the file copy operation in the background using an asynchronous Celery task queue.
+3. Automatically completes the ingestion once background file copying finishes.
+
+The CLI command will block and print real-time updates while waiting for the background ingestion to complete:
+```text
+Waiting for ingestion to complete... queued -> copy_files -> completed
+Successfully pushed simulation <UUID>
+```
+
+#### Configuring Partitions (Shared Storage Mappings)
+
+To enable `push_local` to map and resolve files correctly between your local environment and the remote server, you must configure **partitions** in your `simdb.cfg` configuration file.
+
+Partitions define mappings between short, logical names (such as `data` or `work`) and their absolute paths on your local file system.
+
+##### 1. Defining Partitions on the Client
+Add a `[partition]` section to your `~/.config/simdb/simdb.cfg` file, specifying the directory paths for each partition:
+
+```ini
+[partition]
+data = /home/user/my_simdb_data
+work = /work/imas/shared
+sdcc = /
+```
+
+*Note on `sdcc` partition mapping:*
+In environments like the ITER network, files are often located under absolute paths like `/sdcc/projects/...`. Mapping the `sdcc` partition to the system root `/` ensures that any path beginning with `/sdcc` is correctly matched and converted to a partition-relative URI (e.g., `/sdcc/projects/my_run` becomes `sdcc:sdcc/projects/my_run`). When several partitions contain a file, the partition with the most specific (deepest) path wins, so such a catch-all mapping never shadows the other partitions.
+
+##### 2. How Partitions are Resolved
+When you run `push_local`:
+* SimDB scans the manifest's input/output files and checks if any path falls under one of your defined local partitions.
+* If a match is found (e.g., `/home/user/my_simdb_data/scenarios/run1.txt` is inside `/home/user/my_simdb_data`), SimDB converts the file URI into a partition-relative scheme: `data:scenarios/run1.txt`.
+* The remote server receives this logical URI. As long as the server also has the `data` partition configured (even if mounted at a different absolute path like `/mnt/shared/partition`), it resolves the URI to `/mnt/shared/partition/scenarios/run1.txt` and completes ingestion.
+
+This mapping mechanism allows clients and servers to share data over a network or cluster filesystem even if they mount it at different absolute paths.
+
 ## Pulling simulations from a remote
 
 The mirror to pushing simulations is the `pull` command. This command will pull the simulation metadata from the SimDB remote to your local SimDB database and download the simulation data into a directory of your choosing. Once you have pulled a simulation it will appear in any local SimDB queries you perform. The command looks as follows:

pyproject.toml

@@ -41,7 +41,7 @@ dependencies = [
     "distro>=1.8.0",
     "email-validator>=1.1",
     "imas-python>=2.0.1",
-    "netCDF4>=1.5",
+    "netCDF4>=1.7.2",
     "numpy>=1.14",
     "pydantic>=2.10.6",
     "python-dateutil>=2.6",

src/simdb/checksum.py

@@ -4,6 +4,19 @@
 from simdb.imas.utils import SimDBUrl
 
 
+def calculate_checksum(path: Path) -> str:
+    """Generate a SHA1 checksum from the file at the given path.
+
+    :param path: the path of the file to checksum
+    :return: a string containing the hex representation of the computed SHA1 checksum
+    """
+    sha1 = hashlib.sha1()
+    with path.open("rb") as file:
+        for chunk in iter(lambda: file.read(4096), b""):
+            sha1.update(chunk)
+    return sha1.hexdigest()
+
+
 def sha1_checksum(uri: SimDBUrl) -> str:
     """Generate a SHA1 checksum from the given file.
 
@@ -21,8 +34,4 @@ def sha1_checksum(uri: SimDBUrl) -> str:
     if not path.is_file():
         raise ValueError("File appears to be a directory")
 
-    sha1 = hashlib.sha1()
-    with path.open("rb") as file:
-        for chunk in iter(lambda: file.read(4096), b""):
-            sha1.update(chunk)
-    return sha1.hexdigest()
+    return calculate_checksum(path)