[832] Copy to beegfs and register in Tiled - #134

Merged

davramov merged 33 commits into

als-computing:mainfrom

davramov:beegfs

May 26, 2026

Commits on May 26, 2026

adding beegfs globus endpoint to config.yaml for bl832
davramov
committed
d67dfb3
updating move.py to copy to beegfs
davramov
committed
15ac2d9
adding tiled[client] to requirements
davramov
committed
4be93ed
adding beegfs globus endpoint to config.yaml for bl832
davramov
committed
ad329db
adding orchestration/tiled.py for ingesting data on beegfs
davramov
committed
6b5f297
cleaning up
davramov
committed
8b36f10
Adding todo note
davramov
committed
6f8ff1d
fixing bad import statement
davramov
committed
b1db5f5
adding beegfs endpoint to unit tests
davramov
committed
3af7f2f
pointing beegfs root_path to the correct directory
davramov
committed
d6ba153
adding tags to ingested data
davramov
committed
1502dc6
removing TILED_API_KEY from tiled.py
davramov
committed
1e7422f
removing check tags logic from main, moving it to a new method (check_tags). Also removing file paths from main, and setting as .env variables
davramov
committed
38c0d66
Removing requirements.txt, moving any missing deps to pyproject.toml, updating github action
davramov
committed
c4657f4
moving load_dotenv() to top of the module
davramov
committed
d379612
moving _apply_tags() into a prefect task, logger.info -> logger.debug, cleanup. Files and tags register with tiled, and I can see them via the Tiled python api, but not in the Tiled UI for some rea…
davramov
committed
1d91f7e
Adding register_file_to_tiled() call to move.py
davramov
committed
187977c
Adding calls to register_file_to_tiled() from nersc_recon_flow() for tiffs and zarr
davramov
committed
254a1b9
Adding TILED_URI to .env.example
davramov
committed
6184013
Adding BEAMLINE=<> to .env.example
davramov
committed
9b6579a
Making register_file_to_tiled() sync to fix async/sync nesting. Example: dispatcher (async) -> move.py (sync) -> register_file_to_tiled (async). Using Prefect's built in helper from prefect.utiliti…
davramov
committed
21536f8
adding beegfs copy to nersc.py
davramov
committed
9b7470a
create unit tests in test_tiled.py
davramov
committed
10bd2aa
Only transferring Zarrs to Beegfs and ingesting into Tiled; not transferring/ingesting raw or reconstructed tiffs
davramov
committed
239269c
pytest fixes to avoid a few warnings
davramov
committed
332a02b
Adding zarr->beegfs transfer and tiled ingestion to the ALCF reconstruction flow
davramov
committed
4d62c29
removing the scratch tag for tiled ingestion
davramov
committed
2b7dabf
fixing broken import
davramov
committed
1502807
Adding beegfs/bl832/scratch endpoint
davramov
committed
6c703f0
adding beegfs endpoints to test_globus_flow.py
davramov
committed
ee355b4
trying to reduce pytest warnings
davramov
committed
116bb97
linting
davramov
committed
210f9bc
Adding TILED_URI and TILED_API_KEY to login helper script
davramov
committed
48403d0

Navigation Menu

Uh oh!

[832] Copy to beegfs and register in Tiled - #134

[832] Copy to beegfs and register in Tiled#134
davramov merged 33 commits into
als-computing:mainfrom
davramov:beegfs

Commits on May 26, 2026

adding beegfs globus endpoint to config.yaml for bl832

updating move.py to copy to beegfs

adding tiled[client] to requirements

adding beegfs globus endpoint to config.yaml for bl832

adding orchestration/tiled.py for ingesting data on beegfs

cleaning up

Adding todo note

fixing bad import statement

adding beegfs endpoint to unit tests

pointing beegfs root_path to the correct directory

adding tags to ingested data

removing TILED_API_KEY from tiled.py

removing check tags logic from main, moving it to a new method (check_tags). Also removing file paths from main, and setting as .env variables

Removing requirements.txt, moving any missing deps to pyproject.toml, updating github action

moving load_dotenv() to top of the module

moving _apply_tags() into a prefect task, logger.info -> logger.debug, cleanup. Files and tags register with tiled, and I can see them via the Tiled python api, but not in the Tiled UI for some rea…

Adding register_file_to_tiled() call to move.py

Adding calls to register_file_to_tiled() from nersc_recon_flow() for tiffs and zarr

Adding TILED_URI to .env.example

Adding BEAMLINE=<> to .env.example

Making register_file_to_tiled() sync to fix async/sync nesting. Example: dispatcher (async) -> move.py (sync) -> register_file_to_tiled (async). Using Prefect's built in helper from prefect.utiliti…

adding beegfs copy to nersc.py

create unit tests in test_tiled.py

Only transferring Zarrs to Beegfs and ingesting into Tiled; not transferring/ingesting raw or reconstructed tiffs

pytest fixes to avoid a few warnings

Adding zarr->beegfs transfer and tiled ingestion to the ALCF reconstruction flow

removing the scratch tag for tiled ingestion

fixing broken import

Adding beegfs/bl832/scratch endpoint

adding beegfs endpoints to test_globus_flow.py

trying to reduce pytest warnings

linting

Adding TILED_URI and TILED_API_KEY to login helper script