Datavera

Datavera is a production-grade, open-source distributed data management platform built in Scala. It provides large-scale ingestion, processing, validation, and storage of structured and semi-structured data with a functional, type-safe architecture.

Features

• Configurable by design - All behavior externalized via HOCON configuration with dev/staging/production profiles
• Pluggable architecture - Extension points for ingestion connectors, transformations, storage backends, and auth providers
• Batch & streaming ingestion - CSV, JSON, Parquet files and Kafka streaming
• Functional pipelines - Composable transformations (map, filter, fold) defined in configuration
• Multi-backend storage - PostgreSQL for relational/metadata storage, S3-compatible object storage for large datasets
• REST API - http4s-based API with JWT authentication and role-based access control
• Observability - Structured logging, Prometheus metrics, OpenTelemetry tracing
• Production deployment - Docker containerization and Kubernetes manifests with HPA

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        API Layer (http4s)                   │
│              REST endpoints · Auth · RBAC                   │
├──────────────┬──────────────────┬───────────────────────────┤
│  Ingestion   │    Processing    │         Storage           │
│  Connectors  │  Pipeline Engine │  PostgreSQL · S3          │
│  Validation  │  Transformations │  Metadata · Versioning    │
├──────────────┴──────────────────┴───────────────────────────┤
│              Config · Observability · Plugin Registry       │
└─────────────────────────────────────────────────────────────┘

Quick Start

Prerequisites

• JDK 17+
• sbt 1.10+
• PostgreSQL 16+ (optional, for persistent storage)

Build

sbt compile
sbt test
sbt "project app" stage

Run locally

export DATAVERA_ENV=dev
sbt "project app" run

The API will be available at http://localhost:8080/api/v1.

Docker

sbt "project app" stage
docker compose -f docker/docker-compose.yml up --build

Configuration

Configuration files live in modules/config/src/main/resources/:

File	Purpose
application.conf	Base production-safe defaults
application-dev.conf	Development overrides
application-staging.conf	Staging environment
application-prod.conf	Production with env var substitution

Set the active profile:

export DATAVERA_ENV=dev    # dev | staging | prod

Key environment variables:

Variable	Description
DATAVERA_ENV	Active configuration profile
POSTGRES_HOST	PostgreSQL hostname
POSTGRES_PASSWORD	Database password
JWT_SECRET	JWT signing secret
KAFKA_BOOTSTRAP_SERVERS	Kafka broker addresses
S3_ENDPOINT	S3-compatible endpoint
OTEL_ENDPOINT	OpenTelemetry collector endpoint

API Endpoints

Method	Path	Description
GET	/api/v1/health	Health check
GET	/api/v1/datasets	List datasets
POST	/api/v1/datasets	Create dataset
GET	/api/v1/datasets/{id}	Get dataset metadata
POST	/api/v1/datasets/{id}/ingest	Ingest records
GET	/api/v1/datasets/{id}/query	Query dataset records
POST	/api/v1/pipelines/{id}/run	Run transformation pipeline
GET	/api/v1/metadata/pipelines	List configured pipelines

Authentication uses Bearer JWT tokens when datavera.auth.enabled = true.

Extending Datavera

Custom Ingestion Connector

Implement IngestionConnector[F] and register via configuration:

datavera.ingestion.connectors += {
  id = "my-connector"
  type = "my-connector-type"
  enabled = true
  # custom settings...
}

Custom Transformation

Implement Transformation[F] and register in TransformationRegistry.

Custom Storage Backend

Implement StorageBackend[F] and configure in datavera.storage.backends.

Project Structure

datavera/
├── modules/
│   ├── core/           # Domain types, interfaces, plugin system
│   ├── config/         # HOCON configuration loading & validation
│   ├── ingestion/      # Batch & streaming connectors
│   ├── processing/     # Pipeline engine & transformations
│   ├── storage/        # PostgreSQL & S3 backends
│   ├── observability/  # Logging, metrics, tracing
│   ├── api/            # http4s REST API
│   ├── app/            # Application entry point
│   └── integration/    # Integration tests
├── docker/             # Dockerfile & docker-compose
├── k8s/                # Kubernetes manifests
└── build.sbt

Testing

sbt test                    # All unit tests
sbt "project integration" test  # Integration tests (requires Docker)

Tests include:

• Unit tests for core logic and configuration validation
• Property-based tests with ScalaCheck for data model invariants
• Integration tests with Testcontainers (PostgreSQL, LocalStack)

Kubernetes Deployment

kubectl apply -f k8s/deployment.yaml

The deployment includes 3 replicas by default with Horizontal Pod Autoscaler (2-10 pods based on CPU).

License

Apache License 2.0 - See LICENSE for details.

Contributing

Contributions are welcome! Please read the architecture docs above and ensure:

1. All behavior is configurable, not hardcoded
2. New features implement the appropriate plugin interface
3. Tests cover critical paths
4. Configuration changes include validation