Generate pointer-valued macro constants

[tamird]

The Clang macro fallback can evaluate macro forms that cexpr cannot
parse, but it discards pointer-valued expressions. Emit data pointer
macros as raw Rust pointer constants. Keep dependent expressions
eligible for fallback evaluation.

Materialize pointer macro types from a final fallback parse so
deferred type resolution does not retain cursors invalidated by a
later reparse.

Related to #3347.

[tamird]

cc @metaspace @ojeda

[tamird]

r? @emilio

[ojeda]

It could be nice to split in two commits, to see how the test changes -- from a quick test against mainline I see:

+pub const BEFORE_DECL: *mut later = 3u64 as usize as *mut later;
+pub const CONST_PTR: *const later = 4u64 as usize as *const later;
+pub const TYPEDEF_PTR: *mut later = 5u64 as usize as *mut later;
+pub const MAP_FAILED: *mut ::std::os::raw::c_void = 18446744073709551615u64 as usize
+    as *mut ::std::os::raw::c_void;
+pub const MAP_FAILED_ALIAS: *mut ::std::os::raw::c_void = 18446744073709551615u64
+    as usize as *mut ::std::os::raw::c_void;
 pub const MAP_FAILED_EQUALS_ITSELF: u32 = 1;
 pub const REDEFINED_FROM_INT: u32 = 1;
-pub const REDEFINED_ALIAS: u32 = 1;
+pub const REDEFINED_ALIAS: *mut ::std::os::raw::c_void = 2u64 as usize
+    as *mut ::std::os::raw::c_void;

Is that the expected change here?

I am surprised there was no other test that needed an update for the REDEFINED_ALIAS case.

(A couple surprising but pre-existing issues discussed elsewhere IIRC are the value of REDEFINED_FROM_INT and the type of e.g. MAP_FAILED_EQUALS_ITSELF)

[emilio]

Could you elaborate why we need this deferral thing?

View changes since the review

[tamird]

The deferral is about libclang cursor lifetimes, not just macro ordering.

The fallback translation unit is reparsed for each macro cexpr cannot evaluate. Materializing a pointer immediately can create an UnresolvedTypeRef holding a clang::Type and cursor from that translation unit. Those references are resolved only after parsing completes, but evaluating the next fallback macro invalidates them.

The regression fixture makes this concrete: BEFORE_DECL refers to struct later before its declaration has been visited, then CONST_PTR reparses the fallback translation unit. Without deferral, the first unresolved pointee would retain stale libclang handles.

Queueing pointer macros, preserving their item IDs, and materializing them together in one final reparse keeps those handles valid through type resolution. I do not see a simpler safe approach that preserves typed forward-declared pointees and the existing reusable fallback translation unit.

— tamirdex

[tamird]

Regarding the earlier review question: yes, that is the expected output change. BEFORE_DECL, CONST_PTR, TYPEDEF_PTR, MAP_FAILED, and MAP_FAILED_ALIAS become typed pointer constants. REDEFINED_ALIAS also changes from an integer to a pointer because pointer-valued macros must stay out of the cexpr value map; otherwise aliases are evaluated as integers instead of reaching the Clang fallback.

A clean review split would be:

1. Add the regression header and its current mainline expectations.
2. Add typed pointer generation, the required fallback deferral, and the updated expectations.

REDEFINED_FROM_INT and MAP_FAILED_EQUALS_ITSELF retain their existing behavior.

— tamirdex

[emilio]

So I guess clang::Cursors are not invalidated when reparsing, but the types are? Is that documented anywhere?

View changes since the review

[tamird]

Yes, but the distinction is which translation unit is reparsed, not cursors versus types.

Clang explicitly documents that clang_reparseTranslationUnit invalidates all cursors and source locations referring into the translation unit being reparsed: https://github.com/llvm/llvm-project/blob/8e76c3f0d29a713c41f1b3d0f19d965a504481cf/clang/include/clang-c/Index.h#L1042-L1054.

The saved PendingPointerMacro cursor comes from bindgen’s original, primary translation unit, which is never reparsed:

rust-bindgen/bindgen/ir/var.rs

Lines 250 to 260 in dd174a8

	// this point.
	let name = String::from_utf8(id).unwrap();
	let value =
	match value {
	MacroValue::Expr(value) => value,
	MacroValue::Pointer => {
	let id = ctx.next_item_id();
	ctx.note_pending_pointer_macro(
	PendingPointerMacro { id, name, cursor },
	);
	return Err(ParseError::Continue);

. It is the separate fallback translation unit that is reparsed for each expression:

rust-bindgen/bindgen/ir/var.rs

Lines 428 to 445 in dd174a8

	fn parse_macro_clang_fallback(
	ctx: &mut BindgenContext,
	cursor: &clang::Cursor,
	) -> Option<(Vec<u8>, MacroValue)> {
	use clang_sys::{
	CXType_FunctionNoProto, CXType_FunctionProto, CXType_Pointer,
	};
	if !ctx.options().clang_macro_fallback {
	return None;
	}
	let ftu = ctx.try_ensure_fallback_translation_unit()?;
	let name = cursor.spelling();
	let contents = format!("int main() {{ {name}; }}");
	ftu.reparse(&contents).ok()?;
	// Children of root node of AST
	let root_children = ftu.translation_unit().cursor().collect_children();

. Cursors and types obtained from that fallback translation unit must therefore not survive its next reparse. Deferring pointer materialization until the final fallback reparse keeps the primary cursor and the final pointee type valid through resolution:

rust-bindgen/bindgen/ir/var.rs

Lines 504 to 528 in dd174a8

	pub(crate) fn finish_pending_pointer_macros(ctx: &mut BindgenContext) {
	use clang_sys::{CXChildVisit_Break, CXChildVisit_Recurse, CXType_Pointer};
	let pending = ctx.take_pending_pointer_macros();
	if pending.is_empty() {
	return;
	}
	let statements = pending
	.iter()
	.map(|pointer_macro| {
	format!("{{ (unsigned long long)({}); }}", pointer_macro.name)
	})
	.collect::<Vec<_>>()
	.join(" ");
	let contents = format!("int main() {{ {statements} }}");
	let expressions = {
	let Some(ftu) = ctx.try_ensure_fallback_translation_unit() else {
	return;
	};
	if ftu.reparse(&contents).is_err() {
	return;
	}
	let root_children = ftu.translation_unit().cursor().collect_children();
	let Some(main_func) = root_children.last() else {

.

So this does not depend on a guarantee that cursors from a reparsed translation unit survive; Clang explicitly says they do not.

— tamirdex

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.