Skip to content

Large DAGs/data take a prohibitive amount of memory #388

Description

@jmafoster1

Describe the bug
Generating tests for a large DAG/dataset takes a lot of memory. I think this is because a new estimator is defined for each test case, where each estimator has its own copy of the dataset. I think it would be better to have tests as a generator.

To Reproduce
Steps to reproduce the behavior:

  1. Take a large causal DAG (50+ nodes)
  2. Take a large dataset (10k+ rows)
  3. Try to generate causal test cases
  4. Watch your memory fill up until the process is killed, or you computer becomes non-responsive.

Expected behavior
The CTF should handle larger scenarios smoothly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    In progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions