When vertexai.init (or aiplatform.init) is called with location="global" together with an explicit api_endpoint, the SDK silently discards the api_endpoint and sends traffic to the hard-coded public host aiplatform.googleapis.com. The same api_endpoint is honored for every regional location. There is no error or warning, so requests intended for a custom endpoint (reverse proxy, API gateway, Private Service Connect) quietly leave for the public endpoint instead
This makes it impossible to route global-location traffic through a custom endpoint using the documented api_endpoint parameter, and it is surprising from a data-egress standpoint since the traffic goes somewhere other than what was configured
Environment details
- OS type and version: Linux (reproduces on any OS; the defect is in pure endpoint-resolution logic)
- Python version: 3.11.x
- pip version: any
google-cloud-aiplatform version: 1.133.0 (also present on main at time of filing)
Steps to reproduce
- Call
vertexai.init(project=..., location="global", api_endpoint="https://my-proxy.example.com/vertex", api_transport="rest")
- Create any client, for example
GenerativeModel(...) and inspect the prediction client's transport host, or call initializer.global_config.get_client_options(...) directly
- The resolved host is
aiplatform.googleapis.com instead of the configured endpoint. Repeat with location="us-central1" and the configured endpoint is honored
Code example
Hermetic; no network, credentials, or GCP project required
import vertexai
from vertexai.generative_models import GenerativeModel
from google.auth.credentials import AnonymousCredentials
CUSTOM_ENDPOINT = "https://my-proxy.example.com/vertex"
def resolved_host(location: str) -> str:
vertexai.init(
project="my-project",
location=location,
api_endpoint=CUSTOM_ENDPOINT,
api_transport="rest",
credentials=AnonymousCredentials(),
)
return GenerativeModel("gemini-1.5-flash")._prediction_client.transport._host
print(resolved_host("us-central1"))
print(resolved_host("global"))
Output with 1.133.0:
https://my-proxy.example.com/vertex
https://aiplatform.googleapis.com
The same can be shown at the root, independent of the generative models surface and of any transport:
from google.cloud.aiplatform import initializer
cfg = initializer._Config()
cfg.init(project="my-project", location="global",
api_endpoint="https://my-proxy.example.com/vertex", api_transport="rest")
print(cfg.get_client_options(location_override="global", prediction_client=True).api_endpoint)
# aiplatform.googleapis.com
Stack trace
None; the endpoint is replaced silently with no warning or error
Root cause
_Config.get_client_options in google/cloud/aiplatform/initializer.py (v1.133.0 permalink):
api_endpoint = self.api_endpoint
if (
api_endpoint is None
and not self._project
and not self._location
and not location_override
) or (self._location == "global"):
# Default endpoint is location invariant if using API key or global
# location.
api_endpoint = "aiplatform.googleapis.com"
Since and binds tighter than or, the condition is (api_endpoint is None and ...) or (self._location == "global"). The first operand is correctly guarded by api_endpoint is None; the global operand is not, so whenever location == "global" the branch is taken and the user's explicit api_endpoint is overwritten. The intent of defaulting global to the location-invariant host seems right, but it should only apply when the caller did not provide an endpoint, for example:
if api_endpoint is None and (
(not self._project and not self._location and not location_override)
or self._location == "global"
):
api_endpoint = "aiplatform.googleapis.com"
Note that location="global" already forces api_transport="rest" (grpc is rejected for global), but the resolution defect itself is transport-independent since it lives in get_client_options, which is shared by clients created through both aiplatform.init and vertexai.init
When
vertexai.init(oraiplatform.init) is called withlocation="global"together with an explicitapi_endpoint, the SDK silently discards theapi_endpointand sends traffic to the hard-coded public hostaiplatform.googleapis.com. The sameapi_endpointis honored for every regional location. There is no error or warning, so requests intended for a custom endpoint (reverse proxy, API gateway, Private Service Connect) quietly leave for the public endpoint insteadThis makes it impossible to route global-location traffic through a custom endpoint using the documented
api_endpointparameter, and it is surprising from a data-egress standpoint since the traffic goes somewhere other than what was configuredEnvironment details
google-cloud-aiplatformversion: 1.133.0 (also present onmainat time of filing)Steps to reproduce
vertexai.init(project=..., location="global", api_endpoint="https://my-proxy.example.com/vertex", api_transport="rest")GenerativeModel(...)and inspect the prediction client's transport host, or callinitializer.global_config.get_client_options(...)directlyaiplatform.googleapis.cominstead of the configured endpoint. Repeat withlocation="us-central1"and the configured endpoint is honoredCode example
Hermetic; no network, credentials, or GCP project required
Output with 1.133.0:
The same can be shown at the root, independent of the generative models surface and of any transport:
Stack trace
Root cause
_Config.get_client_optionsingoogle/cloud/aiplatform/initializer.py(v1.133.0 permalink):Since
andbinds tighter thanor, the condition is(api_endpoint is None and ...) or (self._location == "global"). The first operand is correctly guarded byapi_endpoint is None; the global operand is not, so wheneverlocation == "global"the branch is taken and the user's explicitapi_endpointis overwritten. The intent of defaulting global to the location-invariant host seems right, but it should only apply when the caller did not provide an endpoint, for example:Note that
location="global"already forcesapi_transport="rest"(grpc is rejected for global), but the resolution defect itself is transport-independent since it lives inget_client_options, which is shared by clients created through bothaiplatform.initandvertexai.init