[DXA][SimX] GMEM responses drain in issue order unlike RTL OoO direct drain

[ForeverHYX]

Summary

The current DXA RTL and SimX model handle out-of-order GMEM responses differently.

The RTL implements out-of-order direct drain: a ready non-last cache line can drain without waiting for an older request. The cache line marked last is the exception and is deferred until all other outstanding lines have completed, because it carries the completion notification.

SimX instead always examines issued_order.front() and returns if that slot's response has not arrived. A later non-last response therefore cannot drain until every older request ahead of it has returned.

This is reproducible on master at d76b7f24e658867ab57e3942d7c648c3e6af072d.

Relevant code

SimX explicitly waits for the oldest issued slot:

vortex/sim/simx/dxa/dxa_core.cpp

Lines 575 to 588 in d76b7f2

	// ── smem_wr: drain ready slots, build LMEM MemReqs ───────────────────
	//
	// In-order: consume issued_order.front(); stall on the head until its
	// GMEM response has arrived.
	void tick_worker_smem_wr(Worker& w) {
	if (w.issued_order.empty()) {
	// Nothing in flight — if all addr_gen done, transfer is finished.
	if (w.ag_idx == w.work_list.size()) finish_worker(w);
	return;
	}
	uint32_t slot = w.issued_order.front();
	auto& s = w.inflight[slot];
	if (!s.rsp_arrived) return; // wait

uint32_t slot = w.issued_order.front();
auto& s = w.inflight[slot];
if (!s.rsp_arrived) return;

The RTL documents and implements an out-of-order direct-drain path, with only the last line deferred:

vortex/hw/rtl/dxa/VX_dxa_smem_wr.sv

Lines 14 to 24 in d76b7f2

	// DXA SMEM Writer — OOO direct drain.
	//
	// Receives CLs directly from gmem_req on the `sw_*` channel, no rsp_buf
	// in the middle. The pend slot is filled asynchronously by whatever rsp
	// (real or OOB-synthetic) gmem_req presents. The CL marked `last` is
	// special — it must drain LAST because its bus packet carries
	// notify_smem_done; we hold it in `defer_*_r` until all other CLs have
	// released, then promote it to pend.
	//
	// Drain: pend → barrel-shift → fb_data_r → SMEM_WORD beats.
	// 1 SMEM-word/cycle steady state.

Deterministic reproducer

I used standalone SimX and Verilator harnesses with a 32-bit DXA-enabled build. Both harnesses configure the same one-dimensional transfer:

GMEM base:  0x1000
SMEM base:  0x0200
size:       192 bytes
line size:   64 bytes

This produces exactly three cache-line reads:

issue order: A (0x1000), B (0x1040), C (0x1080)

The harness captures the three generated request tags, then injects only the response for B. A is deliberately left outstanding. C is the last request, so using B ensures that the injected response is not subject to the RTL's special deferred-last rule.

The essential SimX sequence is:

std::vector<MemReq> reads;
while (reads.size() < 3) {
  SimPlatform::instance().tick();
  if (!dxa->gmem_req_out.at(0).empty()) {
    reads.push_back(dxa->gmem_req_out.at(0).peek());
    dxa->gmem_req_out.at(0).pop();
  }
}

// Return B before A. B is not the last request.
dxa->gmem_rsp_in.at(0).send(MemRsp(reads.at(1).tag, 0, 1, data));
for (unsigned i = 0; i < 8; ++i)
  SimPlatform::instance().tick();

// Expected to contain B's write, but it remains empty in SimX.
assert(!dxa->lmem_req_out.at(0).empty());

The Verilator harness performs the equivalent operation through gmem_req_* and gmem_rsp_*: it captures three unique request tags, drives only the second tag on gmem_rsp, and observes lmem_req_*.

Observed results

SimX after injecting only B:

issued: 0x1000, 0x1040, 0x1080
SimX result: stalled on the first-issued request after the second response arrived

After A is subsequently injected, SimX starts with A's destination:

first LMEM write after first response: 0x200

RTL after injecting only B:

issued CL addresses: 0x40, 0x41, 0x42
RTL first LMEM write after only middle response: 0x24
PASS

The address representations differ at the interfaces: RTL GMEM addresses above are cache-line indices, and its LMEM address 0x24 is a 16-byte word index. Thus 0x24 corresponds to byte address 0x240, the expected destination for B.

Expected behavior

SimX should match the RTL behavior:

1. A ready non-last response such as B should drain even if an older request A is still outstanding.
2. The response marked last should remain deferred until all peer requests have completed, preserving the RTL completion-notification semantics.

History

Commit ad56f92e added an optional DXA_OOO_DRAIN_ENABLE SimX slot-scan path, but the macro was not enabled by a repository configuration or Makefile, so normal builds retained the in-order path.

Commit e28910a5 later reworked the RTL into its current out-of-order direct-drain pipeline without changing SimX.

Commit 48b2fa72 then removed the disabled SimX slot-scan code as dead scaffolding and retained the in-order behavior. This appears to have left SimX inconsistent with the newer RTL pipeline.

Suggested regression coverage

Two deterministic response-order tests would cover the intended behavior:

1. Return middle response B before A and verify that B reaches LMEM immediately.
2. Return last response C first and verify that it remains deferred until A and B complete, then completes the transfer last.