fix: modernize OpenAI and LangChain support for GPT-5.1

[ranadeepsingh]

Summary

Modernize SynapseML's Azure OpenAI tests, examples, packages, and LangChain integration for gpt-5.1 and gpt-5-mini on the synapseml-openai-3 resource. Remove OpenAI SDK 0.x compatibility, use current Azure OpenAI v1 APIs, and update release compatibility coverage to Spark 3.5 and Spark 4.1.

Why

The previous coverage depended on old deployments, SDKs, and request behavior that is incompatible with current GPT-5 reasoning models:

• Reasoning models reject explicit temperature and top_p.
• Deprecated clients and token parameters no longer represent supported APIs.
• Small completion budgets can be consumed by reasoning tokens and return no visible content.
• Responses API output can contain reasoning items before the final message.
• Modern OpenAI clients contain non-picklable HTTP transport state and cannot be safely reconstructed on Spark workers without changing authentication or transport semantics.

What changed

• Replaced older live-test deployments with gpt-5.1 or gpt-5-mini.
• Switched shared tests and examples to synapseml-openai-3 / openai-api-key-3.
• Removed reasoning-incompatible sampling parameters and migrated to max_completion_tokens / max_output_tokens.
• Migrated examples to Azure OpenAI v1 endpoints and current OpenAI/LangChain clients.
• Removed explicit openai<1.0 exception handling, legacy openai.api_* global configuration, and legacy LangChain chain-loading fallback.
• Updated LangchainTransformer to support Spark-picklable Runnables, normalize modern LangChain string outputs, capture current OpenAIError exceptions, and fail clearly when a chain captures a non-picklable OpenAI client.
• Updated the LangChain notebook to use LangChain batching on the driver; distributed network inference should use SynapseML's native OpenAI transformers.
• Upgraded active Databricks OpenAI/LangChain dependencies.
• Updated pipeline.yaml triggers and release compatibility checks to support only Spark 3.5/JDK 11 and Spark 4.1/JDK 17.
• Updated Responses API extraction to select the final message instead of assuming the first output item contains text.
• Reset mutable OpenAI defaults between tests to prevent cross-test state leakage.

Validation

• sbt "core/Test/compile" scalastyle test:scalastyle
• sbt cognitive/Test/compile
• OpenAIDefaultsSuite and OpenAIV1EndpointSuite
• Targeted Chat Completions, Responses, and OpenAIPrompt tests
• Offline Spark tests for Runnable execution, modern OpenAI error handling, persistence, and non-picklable client rejection
• black --check --extend-exclude 'docs/' .
• Notebook JSON and Python syntax validation
• Pipeline YAML and Spark/JDK matrix validation
• Final staged code review

Dependency changes

• langchain: 0.0.152 -> 1.3.14
• openai: 0.27.5 -> 2.47.0
• Added langchain-classic==1.0.8
• Added langchain-community==0.4.2
• Added langchain-openai==1.4.0

Compatibility notes

• OpenAI SDK versions below 1.0 are no longer supported.
• LangchainTransformer no longer configures OpenAI through module-global state.
• Captured OpenAI clients that are not Spark-picklable are rejected before execution rather than reconstructed with incomplete settings.
• Release branch compatibility coverage is limited to Spark 3.5 and Spark 4.1.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey @ranadeepsingh 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
No pipelines were found matching this branch/path.

[Copilot]

Pull request overview

This PR migrates SynapseML’s Azure OpenAI examples and test suites to the synapseml-openai-3 resource, updating model coverage to GPT‑5.1 / GPT‑5‑mini and modernizing OpenAI + LangChain usage (OpenAI v1 endpoints, reasoning token parameters, and Runnable pipelines).

Changes:

• Updated Python dependencies to OpenAI SDK v2 and modern LangChain packages (incl. langchain-openai, langchain-community, and langchain-classic).
• Migrated notebooks to GPT‑5.1 / GPT‑5‑mini deployments, OpenAI v1 endpoint patterns, and reasoning-model parameter conventions (max_completion_tokens, reasoning_effort).
• Updated Scala/Python OpenAI and LangChain tests to remove unsupported sampling params for reasoning models and align expected payloads/output handling.

Show a summary per file

File	Description
environment.yml	Updates OpenAI + LangChain dependency versions to the new stack.
docs/Explore Algorithms/OpenAI/Quickstart - Understand and Search Forms.ipynb	Moves to OpenAI SDK v2 client + v1 endpoint style; updates reasoning token parameters.
docs/Explore Algorithms/OpenAI/Quickstart - OpenAI Embedding.ipynb	Switches resource/key to synapseml-openai-3.
docs/Explore Algorithms/OpenAI/Quickstart - OpenAI Embedding and GPU based KNN.ipynb	Switches resource/key to synapseml-openai-3.
docs/Explore Algorithms/OpenAI/Quickstart - Multimodal OpenAI Prompter with Responses API.ipynb	Updates deployment to gpt-5.1 and new secret/resource.
docs/Explore Algorithms/OpenAI/OpenAI.ipynb	Updates deployments, removes sampling defaults for reasoning models, and modernizes Responses output extraction.
docs/Explore Algorithms/OpenAI/Langchain.ipynb	Migrates to LangChain Runnable pipelines + ChatOpenAI and updates endpoint/key usage.
docs/Explore Algorithms/AI Services/Quickstart - Document Question and Answering with PDFs.ipynb	Updates LangChain imports and AOAI config to GPT‑5.1 + v1 endpoint base URL.
core/src/test/scala/com/microsoft/azure/synapse/ml/Secrets.scala	Updates the OpenAI test secret name to openai-api-key-3.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIV1EndpointSuite.scala	Updates expected request URL/model mapping tests to GPT‑5.1.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIResponsesSuite.scala	Removes temperature usage and updates model strings/output expectations.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPromptSuite.scala	Removes temperature usage and updates prompt text/headers/tests to GPT‑5.x naming.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIPromptResponsesSuite.scala	Switches live Responses tests to GPT‑5.1 and GPT‑5‑mini.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIEmbeddingsSuite.scala	Updates default non-embedding deployment used in defaults test to GPT‑5‑mini.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIDefaultsSuite.scala	Adds per-test default resets and updates globals usage for reasoning models.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIChatCompletionSuite.scala	Removes temperature usage and retargets reasoning/sampling behavior tests to new deployments.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/openai/OpenAIAPIKey.scala	Updates env var names and standard test deployment names to GPT‑5.1 / GPT‑5‑mini.
cognitive/src/test/python/synapsemltest/services/openai/test_StructuredOutput.py	Updates secret/resource/deployment and resets unsupported sampling defaults.
cognitive/src/test/python/synapsemltest/services/openai/test_OpenAIDefaults.py	Updates secret/resource/deployment and adds teardown resets for defaults.
cognitive/src/test/python/synapsemltest/services/langchain/test_LangchainTransform.py	Migrates tests to Runnable pipelines + ChatOpenAI with reasoning token parameters.
cognitive/src/main/python/synapse/ml/services/langchain/LangchainTransform.py	Updates serialization to langchain-core dumps/loads and supports Runnable invoke() with result normalization.

Review details

Comments suppressed due to low confidence (1)

cognitive/src/main/python/synapse/ml/services/langchain/LangchainTransform.py:267

• The OpenAI <1.0 Azure configuration path no longer sets openai.api_type = "azure". For legacy openai SDK versions, omitting api_type can cause requests to be routed as non-Azure (or otherwise misconfigured), breaking existing LangchainTransformer usage when users rely on the transformer's subscriptionKey/url/apiVersion wiring.

            else:
                if self.isSet(self.subscriptionKey):
                    openai.api_key = self.getSubscriptionKey()
                if self.isSet(self.url):
                    openai.api_base = self.getUrl()

• Files reviewed: 21/21 changed files
• Comments generated: 1
• Review effort level: Low

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
No pipelines were found matching this branch/path.