Skip to content

feat(parser): add entrypoint infrastructure for multiple stream types - #56

Merged
cchwala merged 2 commits into
mainfrom
feat/parser-entrypoints
Jul 13, 2026
Merged

feat(parser): add entrypoint infrastructure for multiple stream types#56
cchwala merged 2 commits into
mainfrom
feat/parser-entrypoints

Conversation

@cchwala

@cchwala cchwala commented Jul 13, 2026

Copy link
Copy Markdown
Member

What

Splits the parser into pluggable entrypoints to support different data ingestion loops (SFTP push, SFTP pull, API pull) while reusing the same parser infrastructure.

Changes

New entrypoint modules

  • parser/entrypoints/sftp_push — Current logic (watch local directory for files)
  • parser/entrypoints/sftp_pull — Stub for future remote SFTP polling
  • parser/entrypoints/api_pull — Stub for future HTTP/API polling

Core changes

  • parser/main.py becomes a thin shim that delegates to sftp_push entrypoint
  • generate_config.py emits command: based on new entrypoint field in users.yml
  • Validation added for entrypoint values (sftp_push, sftp_pull, api_pull)
  • users.yml schema extended with entrypoint documentation

Tests

  • All 90 parser tests passing
  • Test imports updated to reference parser.entrypoints.sftp_push

Configuration

New optional field in users.yml:

sources:
  - id: main
    parser: demo_csv_data
    entrypoint: sftp_push  # default, can be omitted

Future stream types will use:

  • entrypoint: sftp_pull — for remote SFTP polling
  • entrypoint: api_pull — for HTTP/API polling

Backward compatibility

Existing services default to sftp_push entrypoint when not specified. No breaking changes to deploy repo configuration.

cchwala added 2 commits July 13, 2026 20:33
Split parser into pluggable entrypoints to support different ingestion loops:

- parser/entrypoints/sftp_push — current logic (watch local directory)
- parser/entrypoints/sftp_pull — stub for future SFTP polling
- parser/entrypoints/api_pull — stub for future HTTP/API polling

Changes:
- parser/main.py becomes thin shim delegating to sftp_push
- generate_config.py emits 'command:' based on new 'entrypoint' field in users.yml
- users.yml schema extended with 'entrypoint' documentation
- Tests updated to import from parser.entrypoints.sftp_push

Backward compatible: existing services default to 'sftp_push' entrypoint.
- Remove unused load_api_json_bundle import from sftp_push.py (imported locally in main() where needed)
- Apply black formatting to test_main.py for consistency
@cchwala
cchwala merged commit 0fc99da into main Jul 13, 2026
6 checks passed
@cchwala
cchwala deleted the feat/parser-entrypoints branch July 13, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant