Issue with bigint columns containing null in recipe's input dataset

[alexbourret]

So some testing.

1. As a dataset tested about 1 million rows against a well known API system performance is comparable.
2. When I tried to use this in the Recipe form I got a different error. May not be related to the changes you just made. The following anallysis was written up by Claude AI.

Title: Recipe fails on a SQL-backed input when an unused column is a nullable integer

Plugin: API Connect 1.4.3 (failing) and 1.4.2 (working) — recipe.py line is identical in both, so this is not version-related
DSS: 14.7.3 local (failing), Dataiku Cloud (working)
Data Store: PostgreSQL local (failing), SPARK Parquet cloud (working.)

What happens

The recipe reads its full input with DSS schema types enforced:

input_parameters_dataframe = input_parameters_dataset.get_dataframe(infer_with_pandas=False)

pandas int64 cannot hold NA, so any int-typed column with empty values raises before
the first HTTP request — including columns the recipe never reads.
process_dataframe() only touches self.column_to_parameter_dict.

Why it looks intermittent

The same project, same recipe config, same data shape, on two backends:

Input backend	Type of the unused timestamp column	Result
S3 / Parquet via Spark	string (inferred)	Runs
PostgreSQL	bigint, nullable	ValueError

The recipe has one parameter column, id (string). The column that breaks the read is
a nullable Unix timestamp that is not a parameter column.

Traceback

File "<string>", line 56, in <module>
File ".../dataiku/core/dataset.py", line 766, in get_dataframe
    results = pd.read_table(dku_output, ...)
...
ValueError: Integer column has NA values in column 8

Steps to reproduce

1. Create a PostgreSQL-backed dataset with a nullable bigint column.
2. Add a string column to use as the API parameter, e.g. id.
3. Build an API Connect recipe on it, selecting only id as the parameter column.
4. Run.

Current work around I changed the type of the input column to string. With that type change in the input. I seem to be able to process data successfully.

Originally posted by @tom-aspireleaders in #84

[alexbourret]

Hi @tom-aspireleaders ! The issue should be fixed with v1.4.3-beta.3 of the plugin. Let me know !

[tom-aspireleaders]

Ok, I’ll test when I get a moment.

[tom-aspireleaders]

The big int with null issues seems to be resolved. Thanks You.

[tom-aspireleaders]

On a test on 7,500 records the performance seems to be about as expected.