diff --git a/.chronus/changes/http-client-python-json-codemodel-2026-7-6-16-30-0.md b/.chronus/changes/http-client-python-json-codemodel-2026-7-6-16-30-0.md new file mode 100644 index 00000000000..73292cc2342 --- /dev/null +++ b/.chronus/changes/http-client-python-json-codemodel-2026-7-6-16-30-0.md @@ -0,0 +1,7 @@ +--- +changeKind: internal +packages: + - "@typespec/http-client-python" +--- + +Switch the code model interchange format between the emitter and the Python generator from YAML to a reference-preserving JSON format, using the same `$id`/`$ref` convention as the C# emitter (System.Text.Json `ReferenceHandler.Preserve`). This removes the `js-yaml` and `PyYAML` dependencies and dramatically speeds up serialization for large specs (on the Azure network spec, end-to-end serialization dropped from ~29s to ~0.2s). The `emit-yaml-only` option is renamed to `emit-codemodel-only`. diff --git a/packages/http-client-python/ARCHITECTURE.md b/packages/http-client-python/ARCHITECTURE.md index 8f1b78d4ac9..c5eaff3a76d 100644 --- a/packages/http-client-python/ARCHITECTURE.md +++ b/packages/http-client-python/ARCHITECTURE.md @@ -18,8 +18,8 @@ This document provides a comprehensive overview of the `@typespec/http-client-py The `@typespec/http-client-python` package is a TypeSpec emitter that generates Python SDK code from TypeSpec definitions. The emitter follows a two-stage architecture: -1. **Emitter Stage (TypeScript)**: Processes TypeSpec definitions and generates an intermediate YAML representation -2. **Generator Stage (Python)**: Transforms the YAML into Python SDK code using templates +1. **Emitter Stage (TypeScript)**: Processes TypeSpec definitions and generates an intermediate JSON code model representation +2. **Generator Stage (Python)**: Transforms the code model into Python SDK code using templates This separation allows for language-specific code generation while maintaining a clean architecture. @@ -45,7 +45,7 @@ This separation allows for language-specific code generation while maintaining a │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────┐ │ -│ │ Intermediate YAML Code Model │ │ +│ │ Intermediate JSON Code Model │ │ │ │ (contains clients, operations, models, types, etc.) │ │ │ └──────────────────────────────────────────────────────┘ │ └────────────────────────────┬────────────────────────────────────┘ @@ -66,7 +66,7 @@ This separation allows for language-specific code generation while maintaining a │ │ - Maps Python types │ │ │ └────────────────────────┬─────────────────────────────────┘ │ │ │ │ -│ │ Enhanced YAML │ +│ │ Enhanced code model │ │ │ │ │ Step 2: Codegen ▼ │ │ ┌──────────────────────────────────────────────────────────┐ │ @@ -114,18 +114,18 @@ The emitter bridges TypeSpec and Python code generation. 2. Extract client information (endpoints, credentials, operations) 3. Build type mappings from TypeSpec to Python 4. Convert Markdown documentation to reStructuredText format -5. Generate intermediate YAML representation -6. Invoke Python generator with YAML input +5. Generate intermediate JSON code model representation +6. Invoke Python generator with the code model as input ### 2. Generator (Python) **Location**: `packages/http-client-python/generator/pygen/` -The generator transforms the YAML code model into Python SDK code through two stages: +The generator transforms the JSON code model into Python SDK code through two stages: #### Step 1: Preprocess -Enhances the YAML with Python-specific information: +Enhances the code model with Python-specific information: - Converts names to `snake_case` and pads Python reserved words - Adds Python type hints and handles optional/required parameters @@ -136,9 +136,9 @@ Enhances the YAML with Python-specific information: #### Step 2: Codegen -Transforms the enhanced YAML into Python files using a three-layer architecture: +Transforms the enhanced code model into Python files using a three-layer architecture: -1. **Models** (`codegen/models/`): Python classes that parse YAML into structured objects (CodeModel, Client, Operation, ModelType, EnumType, etc.) +1. **Models** (`codegen/models/`): Python classes that parse the code model into structured objects (CodeModel, Client, Operation, ModelType, EnumType, etc.) 2. **Serializers** (`codegen/serializers/`): Orchestrate code generation using Jinja2 templates (JinjaSerializer, ClientSerializer, ModelSerializer, etc.) @@ -173,28 +173,30 @@ def {{ operation.name }}(self, {{ operation.parameters }}): - Traverses TypeSpec AST - Extracts clients, operations, models, enums - Maps types using `types.ts` - - Builds YAML code model structure + - Builds the code model structure -4. **YAML Export** - - Converts JavaScript objects to YAML - - Saves to temporary file +4. **Code Model Export** + - Serializes the JavaScript code model graph to JSON using the `$id`/`$ref` + reference-preserving convention (the same one used by the C# emitter and + System.Text.Json's `ReferenceHandler.Preserve`), which preserves the cyclic/shared + object graph, saved to a temporary file - Converts Markdown descriptions to reStructuredText 5. **Python Generator Invocation** - Emitter spawns Python process (or uses Pyodide) - - Passes YAML file path and options + - Passes the code model file path and options - Python generator receives control 6. **Preprocessing** (`preprocess/__init__.py`) - - Loads YAML file + - Loads the code model file - Applies Python naming conventions - Pads reserved words - Adds type information - Creates operation overloads - - Saves enhanced YAML + - Saves the enhanced code model 7. **Code Generation** (`codegen/__init__.py`) - - Loads enhanced YAML + - Loads the enhanced code model - Creates `CodeModel` object hierarchy - Instantiates models for all code elements @@ -379,12 +381,12 @@ npm run regenerate - Use VS Code debugger with TypeScript - Add breakpoints in `emitter/src/` -- Inspect YAML output in temp directory +- Inspect code model output in temp directory **Generator debugging**: - Add Python breakpoints or print statements -- Inspect enhanced YAML after preprocessing +- Inspect the enhanced code model after preprocessing - Check generated files in output directory ## Additional Resources diff --git a/packages/http-client-python/emitter/src/code-model-serializer.ts b/packages/http-client-python/emitter/src/code-model-serializer.ts new file mode 100644 index 00000000000..f94fb99a0e5 --- /dev/null +++ b/packages/http-client-python/emitter/src/code-model-serializer.ts @@ -0,0 +1,51 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/** + * Serialize the code model to a JSON string using reference preservation. + * + * The code model is a cyclic object graph with heavy structural sharing (types are + * deduplicated and reference each other, e.g. `type.elementType`, and enums reference + * their own values). Plain JSON cannot represent cycles or shared references, so we use + * the same `$id`/`$ref` convention as the C# emitter and System.Text.Json's + * `ReferenceHandler.Preserve`: + * + * - The first time an object is seen it is written as `{ "$id": "N", ...properties }`. + * - The first time an array is seen it is written as `{ "$id": "N", "$values": [...] }`. + * - Any later reference to an already-seen object or array is written as `{ "$ref": "N" }`. + * + * Because the `$id` is assigned before recursing into an object's children, cycles are + * encoded as a `$ref` back to the enclosing object. The Python generator reads this back + * with a matching decoder, reconstructing an object graph with the same shared identity + * and cycles. + * + * @param root The code model to serialize. + * @returns The serialized JSON string. + */ +export function serializeCodeModel(root: unknown): string { + const ids = new Map(); + let counter = 0; + + function encode(value: unknown): unknown { + if (value === null || typeof value !== "object") { + return value; + } + const obj = value as object; + const existing = ids.get(obj); + if (existing !== undefined) { + return { $ref: existing }; + } + const id = (++counter).toString(); + ids.set(obj, id); + if (Array.isArray(obj)) { + return { $id: id, $values: obj.map(encode) }; + } + const result: Record = { $id: id }; + for (const key of Object.keys(obj)) { + result[key] = encode((obj as Record)[key]); + } + return result; + } + + return JSON.stringify(encode(root)); +} diff --git a/packages/http-client-python/emitter/src/emitter.ts b/packages/http-client-python/emitter/src/emitter.ts index 9e0489ee2e3..da4559528a3 100644 --- a/packages/http-client-python/emitter/src/emitter.ts +++ b/packages/http-client-python/emitter/src/emitter.ts @@ -1,6 +1,7 @@ import { createSdkContext } from "@azure-tools/typespec-client-generator-core"; import { EmitContext, emitFile, joinPaths, NoTarget } from "@typespec/compiler"; import pkgJson from "../../package.json" with { type: "json" }; +import { serializeCodeModel } from "./code-model-serializer.js"; import { emitCodeModel } from "./code-model.js"; import { BLOB_STORAGE_BASE_URL, @@ -8,7 +9,6 @@ import { PYGEN_WHEEL_FILENAME, PYODIDE_VERSION, } from "./constants.js"; -import { dumpCodeModelToYaml } from "./external-process.js"; import { PythonEmitterOptions, PythonSdkContext, reportDiagnostic } from "./lib.js"; import { runNodeEmit } from "./node-runner.js"; import { loadPyodide, PyodideInterface } from "./pyodide-loader.js"; @@ -234,13 +234,13 @@ async function onEmitMain(context: EmitContext) { return; } - const yamlFilePath = "/yaml/python-yaml-path.yaml"; - pyodide.FS.mkdirTree("/yaml"); + const codeModelFilePath = "/codemodel/python-codemodel-path.json"; + pyodide.FS.mkdirTree("/codemodel"); pyodide.FS.mkdirTree("/output"); clearMemfsDirectory(pyodide, "/output"); - pyodide.FS.writeFile(yamlFilePath, dumpCodeModelToYaml(parsedYamlMap)); + pyodide.FS.writeFile(codeModelFilePath, serializeCodeModel(parsedYamlMap)); - await runPyodideGeneration(pyodide, "/output", yamlFilePath, commandArgs); + await runPyodideGeneration(pyodide, "/output", codeModelFilePath, commandArgs); await copyPyodideOutputToHost(context, pyodide, "/output"); } else { await runNodeEmit({ diff --git a/packages/http-client-python/emitter/src/external-process.ts b/packages/http-client-python/emitter/src/external-process.ts index 8c3b0486286..196a9e4110c 100644 --- a/packages/http-client-python/emitter/src/external-process.ts +++ b/packages/http-client-python/emitter/src/external-process.ts @@ -2,8 +2,8 @@ import { joinPaths } from "@typespec/compiler"; import { ChildProcess, spawn, SpawnOptions } from "child_process"; import { randomUUID } from "crypto"; import { mkdir, writeFile } from "fs/promises"; -import jsyaml from "js-yaml"; import os from "os"; +import { serializeCodeModel } from "./code-model-serializer.js"; const tspCodeGenTempDir = joinPaths(os.tmpdir(), "tsp-codegen"); @@ -12,33 +12,20 @@ export function createTempPath(extension: string, prefix: string = "") { } /** - * Serialize the given codemodel to a YAML string. + * Save the given codemodel in a JSON file. * - * The generated YAML is consumed by the Python generator, which parses it with - * PyYAML (YAML 1.1). js-yaml, on the other hand, dumps using YAML 1.2 rules, so - * plain scalars such as `2020_01_01` (a snake-cased enum member name) are left - * unquoted because YAML 1.2 does not treat underscores as integer separators. - * PyYAML would then read `2020_01_01` back as the integer `20200101`, corrupting - * string values (e.g. enum member names, descriptions). Forcing every string - * scalar to be quoted guarantees that PyYAML round-trips them as strings. - * @param codemodel Codemodel to serialize - * @return the YAML representation of the codemodel. - */ -export function dumpCodeModelToYaml(codemodel: unknown): string { - return jsyaml.dump(codemodel, { forceQuotes: true, quotingType: '"' }); -} - -/** - * Save the given codemodel in a yaml file. + * The codemodel is a cyclic object graph with shared references, so it is serialized with + * the `$id`/`$ref` reference-preserving convention (the same one used by the C# emitter + * and System.Text.Json's `ReferenceHandler.Preserve`), which preserves cycles and + * structural sharing. The Python generator reads it back with a matching decoder. * @param name Name of the codemodel. To give a guide to the temp file name. * @param codemodel Codemodel to save * @return the absolute path to the created codemodel. */ -export async function saveCodeModelAsYaml(name: string, codemodel: unknown): Promise { +export async function saveCodeModel(name: string, codemodel: unknown): Promise { await mkdir(tspCodeGenTempDir, { recursive: true }); - const filename = createTempPath(".yaml", name); - const yamlStr = dumpCodeModelToYaml(codemodel); - await writeFile(filename, yamlStr); + const filename = createTempPath(".json", name); + await writeFile(filename, serializeCodeModel(codemodel)); return filename; } diff --git a/packages/http-client-python/emitter/src/lib.ts b/packages/http-client-python/emitter/src/lib.ts index 70d5d0d081f..4420eb7822d 100644 --- a/packages/http-client-python/emitter/src/lib.ts +++ b/packages/http-client-python/emitter/src/lib.ts @@ -31,7 +31,7 @@ export interface PythonEmitterOptions { urls?: boolean; }; "clear-output-folder"?: boolean; - "emit-yaml-only"?: boolean; + "emit-codemodel-only"?: boolean; } export interface PythonSdkContext extends SdkContext { @@ -147,11 +147,11 @@ export const PythonEmitterOptionsSchema: JSONSchemaType = description: "Whether to clear the output folder before generating the code. Defaults to `false`.", }, - "emit-yaml-only": { + "emit-codemodel-only": { type: "boolean", nullable: true, description: - "Emit YAML code model only, without running Python generator. For batch processing.", + "Emit the serialized code model only, without running the Python generator. For batch processing.", }, }, required: [], diff --git a/packages/http-client-python/emitter/src/node-runner.ts b/packages/http-client-python/emitter/src/node-runner.ts index 1d75b7c4c59..a283bb9b72c 100644 --- a/packages/http-client-python/emitter/src/node-runner.ts +++ b/packages/http-client-python/emitter/src/node-runner.ts @@ -13,7 +13,7 @@ import path, { dirname } from "path"; import { loadPyodide, PyodideInterface } from "pyodide"; import { fileURLToPath } from "url"; import { blackExcludeDirs, PYGEN_WHEEL_FILENAME } from "./constants.js"; -import { saveCodeModelAsYaml } from "./external-process.js"; +import { saveCodeModel } from "./external-process.js"; import { PythonEmitterOptions, reportDiagnostic } from "./lib.js"; import { runPython3 } from "./run-python3.js"; import { quoteShellArg } from "./utils.js"; @@ -41,21 +41,22 @@ export async function runNodeEmit({ const program = context.program; const outputDir = context.emitterOutputDir; const root = path.join(dirname(fileURLToPath(import.meta.url)), "..", ".."); - const yamlPath = await saveCodeModelAsYaml("python-yaml-path", parsedYamlMap); + const codeModelPath = await saveCodeModel("python-codemodel-path", parsedYamlMap); if (program.compilerOptions.noEmit || program.hasError()) { return; } - // If emit-yaml-only mode, just copy YAML to output dir for batch processing - if (resolvedOptions["emit-yaml-only"]) { + // If emit-codemodel-only mode, just record the code model location in the output dir + // for batch processing. + if (resolvedOptions["emit-codemodel-only"]) { if (!fs.existsSync(outputDir)) { fs.mkdirSync(outputDir, { recursive: true }); } - // Copy YAML to output dir with command args embedded + // Write a config file pointing at the code model plus the command args. // Use unique filename to avoid conflicts when multiple specs share output dir - const configId = path.basename(yamlPath, ".yaml"); - const batchConfig = { yamlPath, commandArgs, outputDir }; + const configId = path.basename(codeModelPath, ".json"); + const batchConfig = { codeModelPath, commandArgs, outputDir }; fs.writeFileSync( path.join(outputDir, `.tsp-codegen-${configId}.json`), JSON.stringify(batchConfig, null, 2), @@ -84,10 +85,19 @@ export async function runNodeEmit({ // mount output folder to pyodide pyodide.FS.mkdirTree("/output"); pyodide.FS.mount(pyodide.FS.filesystems.NODEFS, { root: outputDir }, "/output"); - // mount yaml file to pyodide - pyodide.FS.mkdirTree("/yaml"); - pyodide.FS.mount(pyodide.FS.filesystems.NODEFS, { root: path.dirname(yamlPath) }, "/yaml"); - await runPyodideGeneration(pyodide, "/output", `/yaml/${path.basename(yamlPath)}`, commandArgs); + // mount code model file to pyodide + pyodide.FS.mkdirTree("/codemodel"); + pyodide.FS.mount( + pyodide.FS.filesystems.NODEFS, + { root: path.dirname(codeModelPath) }, + "/codemodel", + ); + await runPyodideGeneration( + pyodide, + "/output", + `/codemodel/${path.basename(codeModelPath)}`, + commandArgs, + ); return; } @@ -105,7 +115,7 @@ export async function runNodeEmit({ return; } commandArgs["output-folder"] = outputDir; - commandArgs["tsp-file"] = yamlPath; + commandArgs["tsp-file"] = codeModelPath; const commandFlags = Object.entries(commandArgs) .map(([key, value]) => `--${key}=${quoteShellArg(String(value))}`) .join(" "); diff --git a/packages/http-client-python/emitter/src/types.ts b/packages/http-client-python/emitter/src/types.ts index 3f1a629f257..6fe68cd6c96 100644 --- a/packages/http-client-python/emitter/src/types.ts +++ b/packages/http-client-python/emitter/src/types.ts @@ -18,7 +18,6 @@ import { } from "@azure-tools/typespec-client-generator-core"; import { Type } from "@typespec/compiler"; import { HttpAuth, Visibility } from "@typespec/http"; -import { dump } from "js-yaml"; import { PythonSdkContext } from "./lib.js"; import { camelToSnakeCase, @@ -56,11 +55,40 @@ function isEmptyModel(type: SdkType): boolean { ); } +// Produce a deterministic key for a simple type record by stringifying with sorted keys, +// so that structurally-equal types map to the same cache entry regardless of key order. +// The type graph can be cyclic (e.g. an enum's values referencing back to the enum), so +// cycles are broken with a stable placeholder to keep this from throwing. +function stableStringify(value: unknown): string { + const seen = new WeakSet(); + const serialize = (val: unknown): string => { + if (val === null || typeof val !== "object") { + return JSON.stringify(val); + } + if (seen.has(val)) { + return '"[Circular]"'; + } + seen.add(val); + let result: string; + if (Array.isArray(val)) { + result = `[${val.map(serialize).join(",")}]`; + } else { + const entries = Object.entries(val as Record).sort(([a], [b]) => + a < b ? -1 : a > b ? 1 : 0, + ); + result = `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${serialize(v)}`).join(",")}}`; + } + seen.delete(val); + return result; + }; + return serialize(value); +} + export function getSimpleTypeResult( context: PythonSdkContext, result: Record, ): Record { - const key = dump(result, { sortKeys: true }); + const key = stableStringify(result); const value = context.__simpleTypesMap.get(key); if (value) { result = value; diff --git a/packages/http-client-python/emitter/test/code-model-serializer.test.ts b/packages/http-client-python/emitter/test/code-model-serializer.test.ts new file mode 100644 index 00000000000..ac26cbd92f1 --- /dev/null +++ b/packages/http-client-python/emitter/test/code-model-serializer.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from "vitest"; +import { serializeCodeModel } from "../src/code-model-serializer.js"; + +describe("typespec-python: code model serialization", () => { + // With the previous YAML format, a plain scalar such as `2020_01_01` (a snake-cased + // enum member name) had to be force-quoted, otherwise PyYAML (YAML 1.1) read it back as + // the integer `20200101`. JSON encodes strings unambiguously, so such values round-trip + // as strings without any special handling. + it("preserves string values that YAML 1.1 would have misinterpreted", () => { + const json = serializeCodeModel({ name: "2020_01_01", value: "2020-01-01", plain: "hello" }); + const parsed = JSON.parse(json); + expect(parsed.name).toBe("2020_01_01"); + expect(parsed.value).toBe("2020-01-01"); + expect(parsed.plain).toBe("hello"); + }); + + it("encodes shared references with $id/$ref instead of duplicating them", () => { + const shared = { kind: "string" }; + const parsed = JSON.parse(serializeCodeModel({ a: shared, b: shared })); + expect(parsed.a.$id).toBeDefined(); + expect(parsed.a.kind).toBe("string"); + // The second reference to the same object is a $ref back to the first occurrence. + expect(parsed.b).toEqual({ $ref: parsed.a.$id }); + }); + + it("encodes arrays with $id/$values and preserves cycles as $ref", () => { + const node: Record = { kind: "model" }; + node.self = node; // cyclic reference back to itself + const parsed = JSON.parse(serializeCodeModel({ items: [node] })); + expect(parsed.items.$values).toBeDefined(); + const first = parsed.items.$values[0]; + // The cycle is encoded as a $ref back to the enclosing node. + expect(first.self).toEqual({ $ref: first.$id }); + }); +}); diff --git a/packages/http-client-python/emitter/test/external-process.test.ts b/packages/http-client-python/emitter/test/external-process.test.ts deleted file mode 100644 index fd6a77135c3..00000000000 --- a/packages/http-client-python/emitter/test/external-process.test.ts +++ /dev/null @@ -1,28 +0,0 @@ -import { ok, strictEqual } from "assert"; -import { load } from "js-yaml"; -import { describe, it } from "vitest"; -import { dumpCodeModelToYaml } from "../src/external-process.js"; - -describe("typespec-python: external-process", () => { - // The Python generator parses the emitted YAML with PyYAML (YAML 1.1), where a plain - // scalar such as `2020_01_01` is interpreted as the integer `20200101`. js-yaml dumps - // using YAML 1.2 rules and would otherwise leave such string scalars unquoted, so we - // must force-quote strings to keep enum member names (and other string values) intact. - it("force-quotes string scalars that YAML 1.1 would misinterpret", () => { - const yaml = dumpCodeModelToYaml({ name: "2020_01_01" }); - // The scalar must be quoted, otherwise PyYAML reads it back as the integer 20200101. - ok(yaml.includes('"2020_01_01"'), `expected the underscore scalar to be quoted, got: ${yaml}`); - ok( - !/name:\s*2020_01_01\s*$/m.test(yaml), - `expected no unquoted underscore scalar, got: ${yaml}`, - ); - }); - - it("keeps string values as strings after a round-trip", () => { - const codeModel = { name: "2020_01_01", value: "2020-01-01", plain: "hello" }; - const roundTripped = load(dumpCodeModelToYaml(codeModel)) as Record; - strictEqual(roundTripped.name, "2020_01_01"); - strictEqual(roundTripped.value, "2020-01-01"); - strictEqual(roundTripped.plain, "hello"); - }); -}); diff --git a/packages/http-client-python/eng/scripts/ci/config/mypy.ini b/packages/http-client-python/eng/scripts/ci/config/mypy.ini index 6bd11f1f115..3cb522638d6 100644 --- a/packages/http-client-python/eng/scripts/ci/config/mypy.ini +++ b/packages/http-client-python/eng/scripts/ci/config/mypy.ini @@ -35,7 +35,4 @@ ignore_missing_imports = True [mypy-pygen.*] ignore_missing_imports = True -[mypy-yaml.*] -ignore_missing_imports = True - diff --git a/packages/http-client-python/eng/scripts/ci/dev_requirements.txt b/packages/http-client-python/eng/scripts/ci/dev_requirements.txt index adfd8faac20..164ba998b77 100644 --- a/packages/http-client-python/eng/scripts/ci/dev_requirements.txt +++ b/packages/http-client-python/eng/scripts/ci/dev_requirements.txt @@ -9,4 +9,3 @@ pytest==9.0.3 coverage==7.6.1 black==24.4.0 ptvsd==4.3.2 -types-PyYAML==6.0.12.8 diff --git a/packages/http-client-python/eng/scripts/ci/regenerate-common.ts b/packages/http-client-python/eng/scripts/ci/regenerate-common.ts index 74c0c4a7457..e14abd92cf0 100644 --- a/packages/http-client-python/eng/scripts/ci/regenerate-common.ts +++ b/packages/http-client-python/eng/scripts/ci/regenerate-common.ts @@ -61,11 +61,11 @@ export interface RegenerateContext { /** * Optional knobs for `buildTaskGroups`. Kept here so the call site in each * repo's `regenerate.ts` can opt into the upstream two-phase pipeline - * (`emitYamlOnly: true`) or the single-phase pipeline (default). + * (`emitCodeModelOnly: true`) or the single-phase pipeline (default). */ export interface BuildTaskGroupsOptions { - /** If true, ask the emitter to write YAML only and skip Python codegen. */ - emitYamlOnly?: boolean; + /** If true, ask the emitter to write the code model only and skip Python codegen. */ + emitCodeModelOnly?: boolean; } // ---- Public constants ---- @@ -515,9 +515,9 @@ export function buildTaskGroups( opts["examples-dir"] = toPosix(join(dirname(spec), "examples")); - if (options.emitYamlOnly) { - // Emit YAML only - Python processing is batched after all specs compile. - opts["emit-yaml-only"] = true; + if (options.emitCodeModelOnly) { + // Emit the code model only - Python processing is batched after all specs compile. + opts["emit-codemodel-only"] = true; } tasks.push({ spec, outputDir, options: opts }); diff --git a/packages/http-client-python/eng/scripts/ci/regenerate.ts b/packages/http-client-python/eng/scripts/ci/regenerate.ts index e6c8cee1110..1a0067ecf8d 100644 --- a/packages/http-client-python/eng/scripts/ci/regenerate.ts +++ b/packages/http-client-python/eng/scripts/ci/regenerate.ts @@ -3,8 +3,8 @@ * Regenerates Python SDK code from TypeSpec definitions. * * Two-phase pipeline: - * 1. TypeSpec compile (in-process, parallel) -> emits per-spec YAML only. - * 2. Single batched Python subprocess reads all YAMLs and writes the + * 1. TypeSpec compile (in-process, parallel) -> emits per-spec code model only. + * 2. Single batched Python subprocess reads all code models and writes the * final `.py` files. Amortizes Python-startup cost across many specs. * * Shared helpers/data live in `regenerate-common.ts` (kept identical with the @@ -185,14 +185,15 @@ async function regenerateFlavor( const allSpecs = [...azureSpecs, ...standardSpecs]; // Build task groups (tasks for same spec run sequentially to avoid state pollution). - // emitYamlOnly: true -> phase 1 emits YAML only; phase 2 (runBatchPythonProcessing) writes .py files. - const groups = buildTaskGroups(allSpecs, flags, ctx, { emitYamlOnly: true }); + // emitCodeModelOnly: true -> phase 1 emits the code model only; phase 2 + // (runBatchPythonProcessing) writes .py files. + const groups = buildTaskGroups(allSpecs, flags, ctx, { emitCodeModelOnly: true }); const totalTasks = groups.reduce((sum, g) => sum + g.tasks.length, 0); console.log(pc.cyan(`Found ${allSpecs.length} specs (${totalTasks} total tasks) to compile`)); console.log(pc.cyan(`Using ${jobs} parallel jobs\n`)); - // Run compilation (emits YAML only) + // Run compilation (emits the code model only) const startTime = performance.now(); const results = await runParallel(groups, jobs, ctx); const compileTime = (performance.now() - startTime) / 1000; diff --git a/packages/http-client-python/eng/scripts/setup/run_batch.py b/packages/http-client-python/eng/scripts/setup/run_batch.py index 9b978ec0319..c260519cf7e 100644 --- a/packages/http-client-python/eng/scripts/setup/run_batch.py +++ b/packages/http-client-python/eng/scripts/setup/run_batch.py @@ -4,7 +4,7 @@ # license information. # -------------------------------------------------------------------------- """ -Batch process multiple TypeSpec YAML files in a single Python process. +Batch process multiple TypeSpec code model files in a single Python process. This avoids the overhead of spawning a new Python process for each spec. """ @@ -34,7 +34,7 @@ def process_single_spec(config_path_str: str) -> tuple[str, bool, str]: with open(config_path, "r", encoding="utf-8") as f: config = json.load(f) - yaml_path = config["yamlPath"] + code_model_path = config["codeModelPath"] command_args = config["commandArgs"] output_dir = config["outputDir"] @@ -51,12 +51,12 @@ def _coerce(value): return False return value - pygen_args = {k: _coerce(v) for k, v in command_args.items() if k not in ["emit-yaml-only"]} + pygen_args = {k: _coerce(v) for k, v in command_args.items() if k not in ["emit-codemodel-only"]} # Run preprocess and codegen (black is batched at the end for performance) - preprocess.PreProcessPlugin(output_folder=output_dir, tsp_file=yaml_path, **pygen_args).process() + preprocess.PreProcessPlugin(output_folder=output_dir, tsp_file=code_model_path, **pygen_args).process() - codegen.CodeGenerator(output_folder=output_dir, tsp_file=yaml_path, **pygen_args).process() + codegen.CodeGenerator(output_folder=output_dir, tsp_file=code_model_path, **pygen_args).process() # Clean up the config file config_path.unlink() @@ -104,7 +104,7 @@ def collect_config_files(generated_dir: str, flavor: str) -> list[str]: def main(): - parser = argparse.ArgumentParser(description="Batch process TypeSpec YAML files") + parser = argparse.ArgumentParser(description="Batch process TypeSpec code model files") parser.add_argument( "--generated-dir", required=True, diff --git a/packages/http-client-python/generator/pygen/__init__.py b/packages/http-client-python/generator/pygen/__init__.py index b8841e4fc1a..2d7f49b0c93 100644 --- a/packages/http-client-python/generator/pygen/__init__.py +++ b/packages/http-client-python/generator/pygen/__init__.py @@ -11,7 +11,7 @@ from abc import ABC, abstractmethod from typing import Any, Iterator, Optional, Union -import yaml +from . import _codemodel_json as codemodel_json from .utils import TYPESPEC_PACKAGE_MODE, VALID_PACKAGE_MODE from ._version import VERSION @@ -285,15 +285,15 @@ def process(self) -> bool: class YamlUpdatePlugin(Plugin): - """A plugin that update the YAML as input.""" + """A plugin that update the serialized code model as input.""" def get_yaml(self) -> dict[str, Any]: # tsp file doesn't have to be relative to output folder with open(self.options["tsp_file"], "r", encoding="utf-8-sig") as fd: - return yaml.safe_load(fd.read()) + return codemodel_json.loads(fd.read()) def write_yaml(self, yaml_string: str) -> None: - with open(self.options["tsp_file"], "w", encoding="utf-8-sig") as fd: + with open(self.options["tsp_file"], "w", encoding="utf-8") as fd: fd.write(yaml_string) def process(self) -> bool: @@ -302,16 +302,16 @@ def process(self) -> bool: self.update_yaml(yaml_data) - yaml_string = yaml.safe_dump(yaml_data) + yaml_string = codemodel_json.dumps(yaml_data) self.write_yaml(yaml_string) return True @abstractmethod def update_yaml(self, yaml_data: dict[str, Any]) -> None: - """The code-model-v4-no-tags yaml model tree. + """The code-model-v4-no-tags model tree. - :rtype: updated yaml + :rtype: updated model :raises Exception: Could raise any exception, stacktrace will be sent to autorest API """ raise NotImplementedError() diff --git a/packages/http-client-python/generator/pygen/_codemodel_json.py b/packages/http-client-python/generator/pygen/_codemodel_json.py new file mode 100644 index 00000000000..0e06bf8fa21 --- /dev/null +++ b/packages/http-client-python/generator/pygen/_codemodel_json.py @@ -0,0 +1,101 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Reference-preserving JSON (de)serialization for the code model. + +The code model is a cyclic object graph with heavy structural sharing (the emitter +deduplicates types, and types reference each other, e.g. ``type.elementType``, and enums +reference their own values). Plain JSON cannot represent cycles or shared references, so +we use the same ``$id``/``$ref`` convention as the C# emitter and System.Text.Json's +``ReferenceHandler.Preserve``: + +- The first time an object is seen it is written as ``{"$id": "N", ...properties}``. +- The first time an array is seen it is written as ``{"$id": "N", "$values": [...]}``. +- Any later reference to an already-seen object/array is written as ``{"$ref": "N"}``. + +Because the ``$id`` is registered before recursing into a node's children, cycles are +encoded as a ``$ref`` back to an enclosing node. The reconstructed value has the same +shared identity and cycles as the original graph, so the rest of the generator is +unchanged. The format is interoperable with the JavaScript serializer used by the emitter. +""" + +import json +import sys +from typing import Any + +# The code model can be a deep graph; the default recursion limit (1000) is not enough. +_RECURSION_LIMIT = 100000 + + +def loads(text: str) -> Any: + """Parse a ``$id``/``$ref`` JSON string into a (possibly cyclic) Python object graph.""" + raw = json.loads(text) + id_map: dict[str, Any] = {} + old_limit = sys.getrecursionlimit() + sys.setrecursionlimit(max(old_limit, _RECURSION_LIMIT)) + try: + return _rebuild(raw, id_map) + finally: + sys.setrecursionlimit(old_limit) + + +def _rebuild(node: Any, id_map: dict[str, Any]) -> Any: + if isinstance(node, list): + # Bare arrays only appear as the content of a "$values" wrapper, but handle + # them defensively anyway. + return [_rebuild(item, id_map) for item in node] + if not isinstance(node, dict): + return node + ref = node.get("$ref") + if ref is not None: + return id_map[ref] + node_id = node.get("$id") + if "$values" in node: + arr: list[Any] = [] + if node_id is not None: + id_map[node_id] = arr # register before recursing so cycles resolve + for item in node["$values"]: + arr.append(_rebuild(item, id_map)) + return arr + obj: dict[str, Any] = {} + if node_id is not None: + id_map[node_id] = obj # register before recursing so cycles resolve + for key, value in node.items(): + if key == "$id": + continue + obj[key] = _rebuild(value, id_map) + return obj + + +def dumps(value: Any) -> str: + """Serialize a (possibly cyclic) Python object graph into a ``$id``/``$ref`` JSON string.""" + ids: dict[int, str] = {} + counter = [0] + old_limit = sys.getrecursionlimit() + sys.setrecursionlimit(max(old_limit, _RECURSION_LIMIT)) + try: + return json.dumps(_encode(value, ids, counter)) + finally: + sys.setrecursionlimit(old_limit) + + +def _encode(value: Any, ids: dict[int, str], counter: list) -> Any: + # Scalars (including strings) are inlined. bool is a subclass of int, so it is covered. + if value is None or isinstance(value, (bool, int, float, str)): + return value + if not isinstance(value, (dict, list, tuple)): + return value + existing = ids.get(id(value)) + if existing is not None: + return {"$ref": existing} + counter[0] += 1 + node_id = str(counter[0]) + ids[id(value)] = node_id + if isinstance(value, dict): + result: dict[str, Any] = {"$id": node_id} + for key, val in value.items(): + result[key] = _encode(val, ids, counter) + return result + return {"$id": node_id, "$values": [_encode(item, ids, counter) for item in value]} diff --git a/packages/http-client-python/generator/pygen/codegen/__init__.py b/packages/http-client-python/generator/pygen/codegen/__init__.py index 74e3031729c..cf9083d831f 100644 --- a/packages/http-client-python/generator/pygen/codegen/__init__.py +++ b/packages/http-client-python/generator/pygen/codegen/__init__.py @@ -5,10 +5,10 @@ # -------------------------------------------------------------------------- import logging from typing import Any -import yaml from .. import Plugin +from .. import _codemodel_json as codemodel_json from ..utils import parse_args from .models.code_model import CodeModel from .serializers import JinjaSerializer @@ -62,7 +62,7 @@ def remove_cloud_errors(yaml_data: dict[str, Any]) -> None: def get_yaml(self) -> dict[str, Any]: # tsp file doesn't have to be relative to output folder with open(self.options["tsp_file"], "r", encoding="utf-8-sig") as fd: - return yaml.safe_load(fd.read()) + return codemodel_json.loads(fd.read()) def get_serializer(self, code_model: CodeModel): return JinjaSerializer(code_model, output_folder=self.output_folder) diff --git a/packages/http-client-python/generator/setup.py b/packages/http-client-python/generator/setup.py index cb08938e73d..47156755692 100644 --- a/packages/http-client-python/generator/setup.py +++ b/packages/http-client-python/generator/setup.py @@ -50,7 +50,6 @@ "black==24.4.0", "docutils>=0.20.1", "Jinja2==3.1.6", - "PyYAML==6.0.1", "tomli==2.0.1", "setuptools==75.3.2", ], diff --git a/packages/http-client-python/package-lock.json b/packages/http-client-python/package-lock.json index e3197d5be9a..b70402f799e 100644 --- a/packages/http-client-python/package-lock.json +++ b/packages/http-client-python/package-lock.json @@ -10,7 +10,6 @@ "hasInstallScript": true, "license": "MIT", "dependencies": { - "js-yaml": "^4.2.0", "marked": "^15.0.6", "pyodide": "0.26.2", "semver": "~7.6.2", @@ -23,7 +22,6 @@ "@azure-tools/typespec-azure-resource-manager": "~0.69.1", "@azure-tools/typespec-azure-rulesets": "~0.69.1", "@azure-tools/typespec-client-generator-core": "~0.69.1", - "@types/js-yaml": "~4.0.5", "@types/node": "~25.0.2", "@types/semver": "7.5.8", "@typespec/compiler": "^1.13.0", @@ -2110,12 +2108,6 @@ "integrity": "sha512-2QF/t/auWm0lsy8XtKVPG19v3sSOQlJe/YHZgfjb/KBBHOGSV+J2q/S671rcq9uTBrLAXmZpqJiaQbMT+zNU1w==", "dev": true }, - "node_modules/@types/js-yaml": { - "version": "4.0.9", - "resolved": "https://registry.npmjs.org/@types/js-yaml/-/js-yaml-4.0.9.tgz", - "integrity": "sha512-k4MGaQl5TGo/iipqb2UDG2UwjXziSWkh0uysQelTlJpX1qGlpUZYm8PnO4DxG1qBomtJUdYJ6qR6xdIah10JLg==", - "dev": true - }, "node_modules/@types/json-schema": { "version": "7.0.15", "resolved": "https://registry.npmjs.org/@types/json-schema/-/json-schema-7.0.15.tgz", @@ -3073,7 +3065,9 @@ "node_modules/argparse": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz", - "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==" + "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==", + "dev": true, + "peer": true }, "node_modules/array-buffer-byte-length": { "version": "1.0.2", @@ -5385,6 +5379,7 @@ "version": "4.2.0", "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.2.0.tgz", "integrity": "sha512-ePWsvanv0DWuDRsW8dnt+R4jQ31SCRCQ7hhNcPXZPsoBZiemuZNYGf7adZdqX2D86j6rvKp3RpCxVTSb8WQlOw==", + "dev": true, "funding": [ { "type": "github", @@ -5396,6 +5391,7 @@ } ], "license": "MIT", + "peer": true, "dependencies": { "argparse": "^2.0.1" }, diff --git a/packages/http-client-python/package.json b/packages/http-client-python/package.json index d220de9eb89..2986da0a0b5 100644 --- a/packages/http-client-python/package.json +++ b/packages/http-client-python/package.json @@ -95,7 +95,6 @@ ] }, "dependencies": { - "js-yaml": "^4.2.0", "marked": "^15.0.6", "pyodide": "0.26.2", "semver": "~7.6.2", @@ -120,7 +119,6 @@ "@typespec/streams": "~0.83.0", "@typespec/xml": "~0.83.0", "@typespec/http-specs": "0.1.0-alpha.39-dev.2", - "@types/js-yaml": "~4.0.5", "@types/node": "~25.0.2", "@types/semver": "7.5.8", "c8": "^10.1.3", diff --git a/packages/http-client-python/tests/requirements/typecheck.txt b/packages/http-client-python/tests/requirements/typecheck.txt index 6e09ee898c7..57549d515b9 100644 --- a/packages/http-client-python/tests/requirements/typecheck.txt +++ b/packages/http-client-python/tests/requirements/typecheck.txt @@ -2,4 +2,3 @@ -r base.txt pyright==1.1.407 mypy==1.19.1 -types-PyYAML==6.0.12.8