perf: add std.foldl object-merge fast path (avoid O(N^2) super-chain)

[szeiger]

Motivation

Building an object by composition inside std.foldl is a common pattern:

std.foldl(function(acc, x) acc { [key(x)]: x }, arr, {})

Each acc { ... } step creates a new object whose super is the previous accumulator, so the fold builds a super chain of depth N. Every key-union, field lookup, and materialization then walks that chain, making the fold O(N^2) in both time and transient memory. A std.objectHas(acc, k) / acc[k] dedup guard inside the callback rebuilds the whole key union each step, compounding the blowup.

In the Databricks monorepo (113,113 .jsonnet / .libsonnet files) there are 304 std.foldl call sites across 240 files that can be optimized. Of those, 62 folds read the accumulator mid-fold — 53 files via std.objectHas(acc, key) (insert-if-absent / dedup) and 12 via direct acc[key] indexing (dedup-with-value-check / merge-if-smaller). No fold uses objectFields / objectValues / length / in on the accumulator, so the shared key-union map is the only cross-step state these patterns need.

An internal config target (a std.foldl dedup over ~1,800 tuples, evaluated 4x) needed ~12 GB and OOM'd the 7 GB build-worker heap cap at ~2 min. With this change it evaluates in ~2 s using ~0.8 GB.

What this does

When the per-step object literal cannot observe the accumulator, the super chain is semantically inert — it exists only to enable super / late-bound self / +: reuse, none of which such a literal uses. StaticOptimizer recognizes this shape through Builtin.specialize and rewrites the call to a fast path (FoldlObjectMerge) that:

• gathers each step's own members into a single map layered over init as its only super — O(N) overall;
• keeps init intact, so its own super chain and assertions still resolve and fire exactly as under naive evaluation;
• threads the accumulator's key union through one shared, incrementally grown map, so a callback that reads the accumulator (std.objectHas(acc, k), acc[k]) resolves each lookup in O(1) per step instead of re-gathering the whole chain.

Patterns it fires on

Detection is fully static. The callback must be a 2-parameter function literal whose body — after stripping local / assert wrappers and seeing through if/else — is a tree of leaves, each one of:

• acc — a no-op step;
• acc { <object literal> } — object extension;
• acc + <object literal> — +.

…where every "delta" object literal has no +: fields, no method fields, no assertions, no super reference, and no reference to acc (transitively — any local whose right-hand side mentions acc disqualifies the fold). acc may still appear freely in assert / if conditions and messages, since those are forced immediately rather than captured into the result.

Examples that can be optimized:

// build a map
std.foldl(function(acc, x) acc { [key(x)]: x }, arr, {})

// dedup guard that reads the accumulator (stays O(1) per step)
std.foldl(function(acc, t)
  local k = tupleKey(t);
  assert !std.objectHas(acc, k) || acc[k] == t : "collision on " + k;
  acc { [k]: t }, arr, {})

// conditional insert-if-absent
std.foldl(function(acc, x)
  if std.objectHas(acc, key(x)) then acc else acc { [key(x)]: x }, arr, {})

Testing

• New FoldlObjectMergeTests — correctness (output identical to naive evaluation, including init with its own super chain / asserts, hidden-field visibility, and self-referential deltas), plus firing / non-firing shape assertions.
• Full JVM test suite green; an A/B run over hundreds of real internal jsonnet_to_json targets produced byte-identical output versus the baseline interpreter.

[CertainLach]

std.foldl(function(acc, x) acc { [key(x)]: x }, arr, {})

Isn't that just

{[key(x)]: x for x in arr}

With extra steps?..

[stephenamar-db]

std.foldl(function(acc, x) acc { [key(x)]: x }, arr, {})

Isn't that just

{[key(x)]: x for x in arr}

With extra steps?..

it's slightly different.

std.foldl allows you to override the previous value. obj comprehension will throw an error.

jsonnet -e "std.foldl(function(acc, x) acc { [x.k]: x }, [{k: 'f', v: 1}, {k: 'f', v: 2}], {})"                                                                                                                                 (stack/abc)
{
   "f": {
      "k": "f",
      "v": 2
   }
}

jsonnet -e "{[x.k]: x for x in [{k: 'f', v: 1}, {k: 'f', v: 2}]}"                                                                                                                                                         (stack/abc)
RUNTIME ERROR: Duplicate field name: "f"
	<cmdline>:1:1-53
	During evaluation

[CertainLach]

std.foldl allows you to override the previous value. obj comprehension will throw an error.

Yep, for I long time I think that object comprehensions need syntax such as

{[x]!: x for x in arr}

(exclamation point after field name - it should override the value instead of failing)

[He-Pin]

Nice, there may be more patterns that can be specialized.