Use JSON instead of YAML for typespec-python code model serialization

.chronus/changes/http-client-python-json-codemodel-2026-7-6-16-30-0.md

@@ -0,0 +1,7 @@
+---
+changeKind: internal
+packages:
+  - "@typespec/http-client-python"
+---
+
+Switch the code model interchange format between the emitter and the Python generator from YAML to a reference-preserving JSON format, using the same `$id`/`$ref` convention as the C# emitter (System.Text.Json `ReferenceHandler.Preserve`). This removes the `js-yaml` and `PyYAML` dependencies and dramatically speeds up serialization for large specs (on the Azure network spec, end-to-end serialization dropped from ~29s to ~0.2s). The `emit-yaml-only` option is renamed to `emit-codemodel-only`.

packages/http-client-python/ARCHITECTURE.md

@@ -18,8 +18,8 @@ This document provides a comprehensive overview of the `@typespec/http-client-py
 
 The `@typespec/http-client-python` package is a TypeSpec emitter that generates Python SDK code from TypeSpec definitions. The emitter follows a two-stage architecture:
 
-1. **Emitter Stage (TypeScript)**: Processes TypeSpec definitions and generates an intermediate YAML representation
-2. **Generator Stage (Python)**: Transforms the YAML into Python SDK code using templates
+1. **Emitter Stage (TypeScript)**: Processes TypeSpec definitions and generates an intermediate JSON code model representation
+2. **Generator Stage (Python)**: Transforms the code model into Python SDK code using templates
 
 This separation allows for language-specific code generation while maintaining a clean architecture.
 
@@ -45,7 +45,7 @@ This separation allows for language-specific code generation while maintaining a
 │         │                                                       │
 │         ▼                                                       │
 │  ┌──────────────────────────────────────────────────────┐       │
-│  │         Intermediate YAML Code Model                 │       │
+│  │         Intermediate JSON Code Model                 │       │
 │  │  (contains clients, operations, models, types, etc.) │       │
 │  └──────────────────────────────────────────────────────┘       │
 └────────────────────────────┬────────────────────────────────────┘
@@ -66,7 +66,7 @@ This separation allows for language-specific code generation while maintaining a
 │  │  - Maps Python types                                     │   │
 │  └────────────────────────┬─────────────────────────────────┘   │
 │                           │                                     │
-│                           │ Enhanced YAML                       │
+│                           │ Enhanced code model                 │
 │                           │                                     │
 │  Step 2: Codegen          ▼                                     │
 │  ┌──────────────────────────────────────────────────────────┐   │
@@ -114,18 +114,18 @@ The emitter bridges TypeSpec and Python code generation.
 2. Extract client information (endpoints, credentials, operations)
 3. Build type mappings from TypeSpec to Python
 4. Convert Markdown documentation to reStructuredText format
-5. Generate intermediate YAML representation
-6. Invoke Python generator with YAML input
+5. Generate intermediate JSON code model representation
+6. Invoke Python generator with the code model as input
 
 ### 2. Generator (Python)
 
 **Location**: `packages/http-client-python/generator/pygen/`
 
-The generator transforms the YAML code model into Python SDK code through two stages:
+The generator transforms the JSON code model into Python SDK code through two stages:
 
 #### Step 1: Preprocess
 
-Enhances the YAML with Python-specific information:
+Enhances the code model with Python-specific information:
 
 - Converts names to `snake_case` and pads Python reserved words
 - Adds Python type hints and handles optional/required parameters
@@ -136,9 +136,9 @@ Enhances the YAML with Python-specific information:
 
 #### Step 2: Codegen
 
-Transforms the enhanced YAML into Python files using a three-layer architecture:
+Transforms the enhanced code model into Python files using a three-layer architecture:
 
-1. **Models** (`codegen/models/`): Python classes that parse YAML into structured objects (CodeModel, Client, Operation, ModelType, EnumType, etc.)
+1. **Models** (`codegen/models/`): Python classes that parse the code model into structured objects (CodeModel, Client, Operation, ModelType, EnumType, etc.)
 
 2. **Serializers** (`codegen/serializers/`): Orchestrate code generation using Jinja2 templates (JinjaSerializer, ClientSerializer, ModelSerializer, etc.)
 
@@ -173,28 +173,30 @@ def {{ operation.name }}(self, {{ operation.parameters }}):
    - Traverses TypeSpec AST
    - Extracts clients, operations, models, enums
    - Maps types using `types.ts`
-   - Builds YAML code model structure
+   - Builds the code model structure
 
-4. **YAML Export**
-   - Converts JavaScript objects to YAML
-   - Saves to temporary file
+4. **Code Model Export**
+   - Serializes the JavaScript code model graph to JSON using the `$id`/`$ref`
+     reference-preserving convention (the same one used by the C# emitter and
+     System.Text.Json's `ReferenceHandler.Preserve`), which preserves the cyclic/shared
+     object graph, saved to a temporary file
    - Converts Markdown descriptions to reStructuredText
 
 5. **Python Generator Invocation**
    - Emitter spawns Python process (or uses Pyodide)
-   - Passes YAML file path and options
+   - Passes the code model file path and options
    - Python generator receives control
 
 6. **Preprocessing** (`preprocess/__init__.py`)
-   - Loads YAML file
+   - Loads the code model file
    - Applies Python naming conventions
    - Pads reserved words
    - Adds type information
    - Creates operation overloads
-   - Saves enhanced YAML
+   - Saves the enhanced code model
 
 7. **Code Generation** (`codegen/__init__.py`)
-   - Loads enhanced YAML
+   - Loads the enhanced code model
    - Creates `CodeModel` object hierarchy
    - Instantiates models for all code elements
 
@@ -379,12 +381,12 @@ npm run regenerate
 
 - Use VS Code debugger with TypeScript
 - Add breakpoints in `emitter/src/`
-- Inspect YAML output in temp directory
+- Inspect code model output in temp directory
 
 **Generator debugging**:
 
 - Add Python breakpoints or print statements
-- Inspect enhanced YAML after preprocessing
+- Inspect the enhanced code model after preprocessing
 - Check generated files in output directory
 
 ## Additional Resources

packages/http-client-python/emitter/src/code-model-serializer.ts

@@ -0,0 +1,51 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+/**
+ * Serialize the code model to a JSON string using reference preservation.
+ *
+ * The code model is a cyclic object graph with heavy structural sharing (types are
+ * deduplicated and reference each other, e.g. `type.elementType`, and enums reference
+ * their own values). Plain JSON cannot represent cycles or shared references, so we use
+ * the same `$id`/`$ref` convention as the C# emitter and System.Text.Json's
+ * `ReferenceHandler.Preserve`:
+ *
+ * - The first time an object is seen it is written as `{ "$id": "N", ...properties }`.
+ * - The first time an array is seen it is written as `{ "$id": "N", "$values": [...] }`.
+ * - Any later reference to an already-seen object or array is written as `{ "$ref": "N" }`.
+ *
+ * Because the `$id` is assigned before recursing into an object's children, cycles are
+ * encoded as a `$ref` back to the enclosing object. The Python generator reads this back
+ * with a matching decoder, reconstructing an object graph with the same shared identity
+ * and cycles.
+ *
+ * @param root The code model to serialize.
+ * @returns The serialized JSON string.
+ */
+export function serializeCodeModel(root: unknown): string {
+  const ids = new Map<object, string>();
+  let counter = 0;
+
+  function encode(value: unknown): unknown {
+    if (value === null || typeof value !== "object") {
+      return value;
+    }
+    const obj = value as object;
+    const existing = ids.get(obj);
+    if (existing !== undefined) {
+      return { $ref: existing };
+    }
+    const id = (++counter).toString();
+    ids.set(obj, id);
+    if (Array.isArray(obj)) {
+      return { $id: id, $values: obj.map(encode) };
+    }
+    const result: Record<string, unknown> = { $id: id };
+    for (const key of Object.keys(obj)) {
+      result[key] = encode((obj as Record<string, unknown>)[key]);
+    }
+    return result;
+  }
+
+  return JSON.stringify(encode(root));
+}

packages/http-client-python/emitter/src/emitter.ts

@@ -1,14 +1,14 @@
 import { createSdkContext } from "@azure-tools/typespec-client-generator-core";
 import { EmitContext, emitFile, joinPaths, NoTarget } from "@typespec/compiler";
 import pkgJson from "../../package.json" with { type: "json" };
+import { serializeCodeModel } from "./code-model-serializer.js";
 import { emitCodeModel } from "./code-model.js";
 import {
   BLOB_STORAGE_BASE_URL,
   PACKAGE_NAME,
   PYGEN_WHEEL_FILENAME,
   PYODIDE_VERSION,
 } from "./constants.js";
-import { dumpCodeModelToYaml } from "./external-process.js";
 import { PythonEmitterOptions, PythonSdkContext, reportDiagnostic } from "./lib.js";
 import { runNodeEmit } from "./node-runner.js";
 import { loadPyodide, PyodideInterface } from "./pyodide-loader.js";
@@ -234,13 +234,13 @@ async function onEmitMain(context: EmitContext<PythonEmitterOptions>) {
       return;
     }
 
-    const yamlFilePath = "/yaml/python-yaml-path.yaml";
-    pyodide.FS.mkdirTree("/yaml");
+    const codeModelFilePath = "/codemodel/python-codemodel-path.json";
+    pyodide.FS.mkdirTree("/codemodel");
     pyodide.FS.mkdirTree("/output");
     clearMemfsDirectory(pyodide, "/output");
-    pyodide.FS.writeFile(yamlFilePath, dumpCodeModelToYaml(parsedYamlMap));
+    pyodide.FS.writeFile(codeModelFilePath, serializeCodeModel(parsedYamlMap));
 
-    await runPyodideGeneration(pyodide, "/output", yamlFilePath, commandArgs);
+    await runPyodideGeneration(pyodide, "/output", codeModelFilePath, commandArgs);
     await copyPyodideOutputToHost(context, pyodide, "/output");
   } else {
     await runNodeEmit({

packages/http-client-python/emitter/src/external-process.ts

@@ -2,8 +2,8 @@ import { joinPaths } from "@typespec/compiler";
 import { ChildProcess, spawn, SpawnOptions } from "child_process";
 import { randomUUID } from "crypto";
 import { mkdir, writeFile } from "fs/promises";
-import jsyaml from "js-yaml";
 import os from "os";
+import { serializeCodeModel } from "./code-model-serializer.js";
 
 const tspCodeGenTempDir = joinPaths(os.tmpdir(), "tsp-codegen");
 
@@ -12,33 +12,20 @@ export function createTempPath(extension: string, prefix: string = "") {
 }
 
 /**
- * Serialize the given codemodel to a YAML string.
+ * Save the given codemodel in a JSON file.
  *
- * The generated YAML is consumed by the Python generator, which parses it with
- * PyYAML (YAML 1.1). js-yaml, on the other hand, dumps using YAML 1.2 rules, so
- * plain scalars such as `2020_01_01` (a snake-cased enum member name) are left
- * unquoted because YAML 1.2 does not treat underscores as integer separators.
- * PyYAML would then read `2020_01_01` back as the integer `20200101`, corrupting
- * string values (e.g. enum member names, descriptions). Forcing every string
- * scalar to be quoted guarantees that PyYAML round-trips them as strings.
- * @param codemodel Codemodel to serialize
- * @return the YAML representation of the codemodel.
- */
-export function dumpCodeModelToYaml(codemodel: unknown): string {
-  return jsyaml.dump(codemodel, { forceQuotes: true, quotingType: '"' });
-}
-
-/**
- * Save the given codemodel in a yaml file.
+ * The codemodel is a cyclic object graph with shared references, so it is serialized with
+ * the `$id`/`$ref` reference-preserving convention (the same one used by the C# emitter
+ * and System.Text.Json's `ReferenceHandler.Preserve`), which preserves cycles and
+ * structural sharing. The Python generator reads it back with a matching decoder.
  * @param name Name of the codemodel. To give a guide to the temp file name.
  * @param codemodel Codemodel to save
  * @return the absolute path to the created codemodel.
  */
-export async function saveCodeModelAsYaml(name: string, codemodel: unknown): Promise<string> {
+export async function saveCodeModel(name: string, codemodel: unknown): Promise<string> {
   await mkdir(tspCodeGenTempDir, { recursive: true });
-  const filename = createTempPath(".yaml", name);
-  const yamlStr = dumpCodeModelToYaml(codemodel);
-  await writeFile(filename, yamlStr);
+  const filename = createTempPath(".json", name);
+  await writeFile(filename, serializeCodeModel(codemodel));
   return filename;
 }
 

packages/http-client-python/emitter/src/lib.ts

@@ -31,7 +31,7 @@ export interface PythonEmitterOptions {
     urls?: boolean;
   };
   "clear-output-folder"?: boolean;
-  "emit-yaml-only"?: boolean;
+  "emit-codemodel-only"?: boolean;
 }
 
 export interface PythonSdkContext extends SdkContext<PythonEmitterOptions> {
@@ -147,11 +147,11 @@ export const PythonEmitterOptionsSchema: JSONSchemaType<PythonEmitterOptions> =
       description:
         "Whether to clear the output folder before generating the code. Defaults to `false`.",
     },
-    "emit-yaml-only": {
+    "emit-codemodel-only": {
       type: "boolean",
       nullable: true,
       description:
-        "Emit YAML code model only, without running Python generator. For batch processing.",
+        "Emit the serialized code model only, without running the Python generator. For batch processing.",
     },
   },
   required: [],

packages/http-client-python/emitter/src/node-runner.ts

@@ -13,7 +13,7 @@ import path, { dirname } from "path";
 import { loadPyodide, PyodideInterface } from "pyodide";
 import { fileURLToPath } from "url";
 import { blackExcludeDirs, PYGEN_WHEEL_FILENAME } from "./constants.js";
-import { saveCodeModelAsYaml } from "./external-process.js";
+import { saveCodeModel } from "./external-process.js";
 import { PythonEmitterOptions, reportDiagnostic } from "./lib.js";
 import { runPython3 } from "./run-python3.js";
 import { quoteShellArg } from "./utils.js";
@@ -41,21 +41,22 @@ export async function runNodeEmit({
   const program = context.program;
   const outputDir = context.emitterOutputDir;
   const root = path.join(dirname(fileURLToPath(import.meta.url)), "..", "..");
-  const yamlPath = await saveCodeModelAsYaml("python-yaml-path", parsedYamlMap);
+  const codeModelPath = await saveCodeModel("python-codemodel-path", parsedYamlMap);
 
   if (program.compilerOptions.noEmit || program.hasError()) {
     return;
   }
 
-  // If emit-yaml-only mode, just copy YAML to output dir for batch processing
-  if (resolvedOptions["emit-yaml-only"]) {
+  // If emit-codemodel-only mode, just record the code model location in the output dir
+  // for batch processing.
+  if (resolvedOptions["emit-codemodel-only"]) {
     if (!fs.existsSync(outputDir)) {
       fs.mkdirSync(outputDir, { recursive: true });
     }
-    // Copy YAML to output dir with command args embedded
+    // Write a config file pointing at the code model plus the command args.
     // Use unique filename to avoid conflicts when multiple specs share output dir
-    const configId = path.basename(yamlPath, ".yaml");
-    const batchConfig = { yamlPath, commandArgs, outputDir };
+    const configId = path.basename(codeModelPath, ".json");
+    const batchConfig = { codeModelPath, commandArgs, outputDir };
     fs.writeFileSync(
       path.join(outputDir, `.tsp-codegen-${configId}.json`),
       JSON.stringify(batchConfig, null, 2),
@@ -84,10 +85,19 @@ export async function runNodeEmit({
     // mount output folder to pyodide
     pyodide.FS.mkdirTree("/output");
     pyodide.FS.mount(pyodide.FS.filesystems.NODEFS, { root: outputDir }, "/output");
-    // mount yaml file to pyodide
-    pyodide.FS.mkdirTree("/yaml");
-    pyodide.FS.mount(pyodide.FS.filesystems.NODEFS, { root: path.dirname(yamlPath) }, "/yaml");
-    await runPyodideGeneration(pyodide, "/output", `/yaml/${path.basename(yamlPath)}`, commandArgs);
+    // mount code model file to pyodide
+    pyodide.FS.mkdirTree("/codemodel");
+    pyodide.FS.mount(
+      pyodide.FS.filesystems.NODEFS,
+      { root: path.dirname(codeModelPath) },
+      "/codemodel",
+    );
+    await runPyodideGeneration(
+      pyodide,
+      "/output",
+      `/codemodel/${path.basename(codeModelPath)}`,
+      commandArgs,
+    );
     return;
   }
 
@@ -105,7 +115,7 @@ export async function runNodeEmit({
     return;
   }
   commandArgs["output-folder"] = outputDir;
-  commandArgs["tsp-file"] = yamlPath;
+  commandArgs["tsp-file"] = codeModelPath;
   const commandFlags = Object.entries(commandArgs)
     .map(([key, value]) => `--${key}=${quoteShellArg(String(value))}`)
     .join(" ");