bulkcopy() and bulkcopy_arrow() report 2x rows_copied and batch_count against Fabric Warehouse

[dlevy-msft-sql]

Describe the bug

When the target is a Fabric Warehouse (SERVERPROPERTY('EngineEdition') = 11), both cursor.bulkcopy() and cursor.bulkcopy_arrow() return statistics that are exactly double the truth.

The data written is always correct. Only the returned statistics are wrong:

field	behavior
rows_copied	2x the rows actually inserted
batch_count	2x when batch_size is set; reports 1 when batch_size=0
rows_per_second	2x, since it is derived from rows_copied
elapsed_time	correct

No exception is raised. Callers that trust rows_copied for reconciliation, logging, or progress reporting get a silently wrong number.

The doubled value also reaches the driver log:

INFO, cursor.rs:551, py-core, bulkcopy_arrow: Starting Arrow bulkcopy to table: dbo.logprobe
INFO, cursor.py:3657, Python, bulkcopy_arrow: completed - rows_copied=6000, batch_count=6, elapsed_time=2.3745249

That run copied 3,000 rows in 3 batches.

To reproduce

Needs pyarrow and azure-identity, and an az login session with write access to a Fabric Warehouse.

import pyarrow as pa
from azure.identity import AzureCliCredential
from mssql_python import connect

TABLE = "dbo.minrepro"
CONN_STR = (
    "Server=WORKSPACE_ID.datawarehouse.fabric.microsoft.com,1433;"
    "Database=WAREHOUSE_NAME;Encrypt=yes;TrustServerCertificate=no"
)

conn = connect(CONN_STR, token_provider=AzureCliCredential(process_timeout=60))
cur = conn.cursor()

cur.execute(f"IF OBJECT_ID('{TABLE}') IS NOT NULL DROP TABLE {TABLE}")
cur.execute(f"CREATE TABLE {TABLE} (id BIGINT)")
conn.commit()

result = cur.bulkcopy_arrow(TABLE, pa.table({"id": pa.array([1], pa.int64())}), timeout=600)
conn.commit()

cur.execute(f"SELECT COUNT(*) FROM {TABLE}")
print("reported:", result["rows_copied"], " actual:", cur.fetchone()[0])

Output:

reported: 2  actual: 1

Expected behavior

rows_copied equals the number of rows actually inserted, and batch_count equals the number of batches actually sent, on every engine.

Scope of the defect

Reproduced with one row, so no large dataset is needed. The 2x factor is stable and invariant to:

• row count, from 1 to 200,000
• API, bulkcopy and bulkcopy_arrow behave identically
• source type, pyarrow.Table, RecordBatchReader, DuckDBPyRelation, and list of tuples
• batch_size, including 0
• table_lock, keep_nulls, use_internal_transaction

Engine matrix, 5,000 rows, both APIs, batch_size 0 and 1000:

engine	EngineEdition	version	result
SQL Server 2025	3	17.0.1125.2	correct, 8/8 (including a clustered columnstore target)
Fabric SQL database	12	12.0.2000.8	correct, 8/8 (including a clustered columnstore target)
Fabric Warehouse	11	12.0.2000.8	2x, 4/4

Sample from the Warehouse run:

API	batch_size	requested	reported	actual	batches reported	batches expected
bulkcopy_arrow	0	1,000	2,000	1,000	1	1
bulkcopy	0	1,000	2,000	1,000	1	1
bulkcopy_arrow	1,000	5,000	10,000	5,000	10	5
bulkcopy	1,000	5,000	10,000	5,000	10	5
bulkcopy_arrow	10,000	50,000	100,000	50,000	10	5

Why this looks like a client-side bug rather than a server one

bcp was pointed at the same warehouse and the same table shape. It performs a TDS bulk load through the ODBC driver, so it exercises the same protocol path the Rust core does:

bcp dbo.bcpprobe in bcp_in.txt -S WORKSPACE_ID.datawarehouse.fabric.microsoft.com,1433 -d WAREHOUSE_NAME -c -G -U USER_UPN

Starting copy...
3 rows copied.

Actual row count afterwards was 3. So the correct count is derivable from what Warehouse sends back, and ODBC derives it correctly.

The Python layer is not involved. Cursor.bulkcopy and Cursor.bulkcopy_arrow in mssql_python/cursor.py return the dict from pycore_cursor.bulkcopy(...) unmodified, so this points at row-count accumulation in mssql_py_core.

Untested hypothesis

Warehouse is a distributed engine. bcp prints a Distributed request ID warning on the same load, which suggests the load is acknowledged by more than one layer. A factor of exactly 2 that is stable from 1 row to 200,000 rows would fit the core summing every row-count acknowledgement it sees rather than taking the final one. That would also explain batch_count doubling in lockstep.

I could not confirm this. RUST_LOG does not reach the core's log bridge, and TDS is encrypted, so I have no token-level trace. Worth checking how DONE token row counts are accumulated.

One detail that does not fit a naive "everything counted twice" model: with batch_size=0, rows_copied still doubles but batch_count reports 1 rather than 2, which suggests batch_count is derived rather than counted on the unbatched path.

Further technical details

Python version: 3.14.0 (amd64)
mssql-python: 1.13.0
mssql_py_core: 0.1.8
mssql-python-odbc: 18.6.2.1
SQL Server version: Fabric Warehouse, Microsoft Azure SQL Data Warehouse (RTM) - 12.0.2000.8, EngineEdition 11, collation Latin1_General_100_BIN2_UTF8
Operating system: Windows 11 ARM64
bcp control: 15.0.4298.1, ODBC Driver 17 for SQL Server

Additional context

Not a duplicate of #623, which was an authentication failure against Fabric Warehouse and is closed. Bulk copy connects and loads correctly here, only the returned statistics are wrong.

[github-actions]

Hi David Levy (@dlevy-msft-sql), thank you for opening this issue!

Our team will review it shortly. We aim to triage all new issues within 24-48 hours and get back to you.

If you have additional information to share, please feel free to update the issue.

Thank you for your patience!

[bewithgaurav]

David Levy (@dlevy-msft-sql) - thanks for reporting the issue
the fix is already merged in the tds core yesterday - microsoft/mssql-rs#210

it reports the client side rows written instead of trusting the server DONE count, which is what SqlBulkCopy and bcp both do, and it was validated against a real warehouse.
it isn't live yet, will be done in the next release

1.13.0 pins mssql-py-core 0.1.8 and the fix landed after that, so it goes out with the next core bump here
keeping this open for now, will close once this goes live