Private-mode dev tunnel connect fails for owner: resolver requires a Connect token the Management API never returns (contradicts design #19)

[JoshuaRowePhantom]

Summary

The tunnel owner ΓÇö on the same machine, same GitHub account, and the owner of the tunnel ΓÇö cannot connect a PrivateΓÇæmode dev tunnel by name. Connecting a "playspace 3 debug" client to the "playspace 3" host fails at startup with:

Failed to connect: The Management API did not return a Connect-scope tunnel token. Ensure the GitHub identity has access to the tunnel.

Because the connecting identity is the tunnel owner on the same box/account, this is not an identity/access misconfiguration ΓÇö the owning identity still receives a null Connect token and the app throws. DevTunnelEndpointResolver.ResolveAsync requires a ConnectΓÇæscope token for any nonΓÇæAnonymous (Private) access mode and throws when it is null. That directly contradicts design point #19 in docs/design/dev-tunnel-host-service.md, which states that Private connect is identityΓÇæderived and returns a null tunnelAuthToken (the client connects using its GitHub identity, with no AccessTokenSource).

Impact: Private playspace connect (e.g. "playspace 3 debug" ΓåÆ "playspace 3") is broken for the owner. The parallel explicitΓÇæaccessΓÇæpoint path (WebRepositorySource with UseGitHubAuthToken) already works by sending the GitHub identity token as X-Tunnel-Authorization, so only the tunnelΓÇæname/Private path regressed.

Regressed by #517 (d233d835 "use API-issued connect token instead of env-var fallback") and #521 (fccf4c8e "DevTunnel access control end-to-end").

---

Root Cause

1. The resolver treats a null Connect token as fatal in Private mode

features\Phantom.Workspaces\Services\DevTunnel\DevTunnelEndpointResolver.cs:44-55

// Both Private and Token modes require X-Tunnel-Authorization: tunnel <connect-token>.
var tunnelAuthToken = accessMode switch
{
    DevTunnelAccessMode.Anonymous => null,
    _ => lookup.ConnectToken
        ?? throw new InvalidOperationException(
            "The Management API did not return a Connect-scope tunnel token. " +
            "Ensure the GitHub identity has access to the tunnel."),
};

For any nonΓÇæAnonymous mode (Private, and the retired Token mode) a null ConnectToken throws the exact userΓÇævisible message. Per design #19 a null token in Private mode is expected and valid, not fatal.

2. The connect token is obtained from the wrong Management API call, so it is null by construction

ClientΓÇæside lookup goes through LookupByNameAsync / DiscoverSingleAsync ΓåÆ ListWorkspacesTunnelsAsync ΓåÆ ListTunnelsAsync(...), then reads the token off the listed tunnel:

features\Phantom.Workspaces\Services\DevTunnel\DevTunnelManagementClientWrapper.cs:260-284

private async Task<IReadOnlyList<Tunnel>> ListWorkspacesTunnelsAsync(
    TunnelRequestOptions requestOptions, CancellationToken cancellationToken)
{
    var ownedTunnels = await this.managementClient
        .ListTunnelsAsync(null, null, requestOptions, true, cancellationToken)   // LIST, not GET
        .ConfigureAwait(false);
    ...
}

private static DevTunnelLookupResult ToLookupResult(Tunnel tunnel)
{
    ...
    string? connectToken = null;
    tunnel.AccessTokens?.TryGetValue(TunnelAccessScopes.Connect, out connectToken);  // null on a list result
    ...
}

CreateConnectRequestOptions sets TokenScopes = [TunnelAccessScopes.Connect] (DevTunnelManagementClientWrapper.cs:317-324), so the intent is that ListTunnelsAsync mints and returns a perΓÇætunnel Connect token in Tunnel.AccessTokens. The Dev Tunnels SDK does not populate access tokens on a list result.

SDK evidence (Microsoft.DevTunnels.Management / .Contracts 1.3.50):

• TunnelRequestOptions.TokenScopes XML doc: "an optional list of token scopes that are requested when retrieving a tunnel or tunnel port object. ΓǪ The service issues an access token for each scope ΓǪ and returns the token(s) in the Tunnel.AccessTokens ΓǪ dictionary. If the caller does not have permission to get a token for one or more scopes then a token is not returned but the overall request does not fail." ΓÇö i.e. tokens are keyed to singleΓÇæobject retrieval (GET), and a missing token is silent (null), never an error.
• ITunnelManagementClient.ListTunnelsAsync XML doc lists only Labels and IncludePorts as options that affect the returned tunnels; it says nothing about access tokens, and the SDK implementation calls SendRequestAsync<TunnelListByRegionResponse> without PreserveAccessTokens, unlike GetTunnelAsync which passes ReadAccessTokenScopes and calls PreserveAccessTokens(tunnel, result) (cs/src/Management/TunnelManagementClient.cs). The list endpoint returns tunnel metadata, not minted perΓÇætunnel tokens.

So tunnel.AccessTokens[Connect] is null by construction on the list path ΓÇö regardless of the caller's permissions. The owner is not specialΓÇæcased; nobody gets a token here.

Notably, the host side already fetches the connect token correctly, via GetTunnelAsync with a Connect scope:

features\Phantom.Workspaces\Services\DevTunnel\DevTunnelManagementClientWrapper.cs:185-194

var withConnectToken = await this.managementClient
    .GetTunnelAsync(
        this.currentTunnel,
        new TunnelRequestOptions { TokenScopes = [TunnelAccessScopes.Connect] },
        cancellationToken)
    .ConfigureAwait(false);

string? connectToken = null;
withConnectToken?.AccessTokens?.TryGetValue(TunnelAccessScopes.Connect, out connectToken);
return connectToken;

The client path was never updated to use GetTunnelAsync the same way ΓÇö it reads tokens off a list result that never carries them.

3. Private mode writes an empty ACL ΓÇö the architecture is identityΓÇæbased, not tokenΓÇæbased

features\Phantom.Workspaces\Services\DevTunnel\DevTunnelManagementClientWrapper.cs:159-172

tunnel.AccessControl = new TunnelAccessControl
{
    Entries = accessMode == DevTunnelAccessMode.Anonymous
        ?
        [
            new TunnelAccessControlEntry
            {
                Type = TunnelAccessControlEntryType.Anonymous,
                Subjects = [],
                Scopes = [TunnelAccessScopes.Connect],
            },
        ]
        : [],   // Private: EMPTY ACL ΓÇö no Connect ACE for anyone
};

Only the Anonymous branch adds a Connect ACE. The Private branch writes an empty ACL, and DevTunnelAccessMode.Token is retired (WorkspacesConfiguration.cs:32-38). An empty ACL plus retired Token mode is the signature of an identityΓÇæbased Private connect: the owner connects using their own GitHub identity, not a shared Connect token. This matches how the working explicitΓÇæaccessΓÇæpoint path authorizes ΓÇö it sends the GitHub auth token as X-Tunnel-Authorization: tunnel <github-token> (EntityRepository.cs:100-116, WebClientDataAccessLayer.cs:42-46).

4. Design contradiction

features\docs\design\dev-tunnel-host-service.md:485-487 (point #19):

Private connect needs no token: in Private mode IDevTunnelEndpointResolver returns a null tunnelAuthToken (identity-derived) and the client connects without any AccessTokenSource; the token-source UI is hidden for Private and shown only for Token mode.

The current resolver requires a Connect token in Private mode and throws when null ΓÇö the exact opposite of the design.

Why it fails for the owner specifically

Same machine / same account / tunnel owner rules out any accessΓÇæcontrol remediation: there is nothing to grant. The failure is mechanical and universal on the tunnelΓÇæname Private path ΓÇö ListTunnelsAsync never returns a Connect access token, so lookup.ConnectToken is always null, so ResolveAsync always throws. The owner hits it because the owner is the one connecting; adding ACEs or reΓÇæauthenticating cannot help because the token is never minted on the list call in the first place, and per design the owner shouldn't need a Connect token at all.

Downstream, DevTunnelEndpointResolution.TunnelAuthToken flows to WebClientDataAccessLayer (EntityRepository.cs:129-137). A null token there simply omits the X-Tunnel-Authorization header (WebClientDataAccessLayer.cs:42-46), so an identityΓÇæbased Private connect must additionally supply the GitHub identity token/refresh resolver the way the UseGitHubAuthToken path already does (EntityRepository.cs:100-116).

---

Affected Files

File	Line(s)	Contribution
features\Phantom.Workspaces\Services\DevTunnel\DevTunnelEndpointResolver.cs	44-55	Throws for nonΓÇæAnonymous modes when ConnectToken is null ΓÇö the throw site; contradicts design #19.
features\Phantom.Workspaces\Services\DevTunnel\DevTunnelManagementClientWrapper.cs	230-284	LookupByNameAsync/DiscoverSingleAsync → ListTunnelsAsync → ToLookupResult reads AccessTokens[Connect] off a list result that never carries minted tokens.
features\Phantom.Workspaces\Services\DevTunnel\DevTunnelManagementClientWrapper.cs	317-324	CreateConnectRequestOptions sets TokenScopes=[Connect] on a list request, expecting a token that list does not return.
features\Phantom.Workspaces\Services\DevTunnel\DevTunnelManagementClientWrapper.cs	159-195	ApplyAccessModeAsync: Private writes an empty ACL (identityΓÇæbased); the correct GetTunnelAsyncΓÇæwithΓÇæConnectΓÇæscope token pattern already lives here (host side).
features\Phantom.Workspaces\EntityRepository.cs	100-141	Wires the resolver result into WebClientDataAccessLayer; the UseGitHubAuthToken path shows the identityΓÇætoken mechanism the Private tunnelΓÇæname path should reuse.
features\Phantom.Workspaces.Data.Web.Client\WebClientDataAccessLayer.cs	42-46, 104-111	A null token omits X-Tunnel-Authorization; identity connect needs the GitHub token + 401 refresh resolver.
features\docs\design\dev-tunnel-host-service.md	485-487	Design point #19: Private connect is identityΓÇæderived and returns a null token.
features\Phantom.Workspaces.Tests\DevTunnelEndpointResolverTests.cs	37-45	..._WhenConnectTokenIsNull_Throws currently codifies the buggy behavior; must be updated.

---

Design / Fix

Fix A ΓÇö Align with design #19: null Connect token is valid in Private mode (RECOMMENDED)

Treat a null ConnectToken in Private mode as valid: return a null tunnelAuthToken and let the client connect using its GitHub identity. Remove the throw for Private mode.

// DevTunnelEndpointResolver.ResolveAsync
var tunnelAuthToken = accessMode switch
{
    DevTunnelAccessMode.Anonymous => null,
    // Private connect is identity-derived (design #19): a null Connect token is expected
    // and valid ΓÇö the client authorizes via its GitHub identity, not a shared token.
    DevTunnelAccessMode.Private => lookup.ConnectToken,   // may be null; do NOT throw
#pragma warning disable CS0618
    DevTunnelAccessMode.Token => lookup.ConnectToken
        ?? throw new InvalidOperationException(
            "Token access mode requires a pre-shared Connect token."),
#pragma warning restore CS0618
    _ => lookup.ConnectToken,
};

Because a null tunnelAuthToken causes WebClientDataAccessLayer to omit X-Tunnel-Authorization entirely (WebClientDataAccessLayer.cs:42-46), the Private tunnelΓÇæname path must also supply the GitHub identity token the same way the explicitΓÇæaccessΓÇæpoint path does today ΓÇö i.e. in EntityRepository.CreateDevTunnelNameDataAccessLayerAsync, when the token is null in Private mode, pass the GitHub auth token and a refresh resolver:

buildDataAccessLayer: resolution =>
{
    var (token, resolver) = resolution.TunnelAuthToken is { Length: > 0 } t
        ? (t, (Func<string?>?)null)                               // explicit Connect/Token
        : (GitHubAuthTokenResolver.Resolve(),                     // identity-derived (Private)
           (Func<string?>?)(() => GitHubAuthTokenResolver.Resolve()));
    return new WebClientDataAccessLayer(resolution.BaseUri.ToString(), token, resolver);
},

This mirrors the alreadyΓÇæworking WebRepositorySource + UseGitHubAuthToken flow (EntityRepository.cs:100-116), keeps Token mode behavior intact, and matches the emptyΓÇæACL/retiredΓÇæToken architecture. Recommended because the empty ACL, the retirement of Token mode, and design #19 all point at identityΓÇæbased Private connect.

Fix B ΓÇö Obtain the Connect token via the correct API (complementary / fallback)

If a minted Connect token is genuinely required, fetch it with the API that actually returns tokens ΓÇö GetTunnelAsync with TokenScopes = [Connect] ΓÇö instead of reading AccessTokens off a ListTunnelsAsync result that never carries them. This mirrors the host side (DevTunnelManagementClientWrapper.cs:185-194):

// After locating the tunnel by label via ListTunnelsAsync, mint the Connect token via GET:
var withConnectToken = await this.managementClient
    .GetTunnelAsync(
        located,
        new TunnelRequestOptions { TokenScopes = [TunnelAccessScopes.Connect], IncludePorts = true },
        cancellationToken)
    .ConfigureAwait(false);

string? connectToken = null;
withConnectToken?.AccessTokens?.TryGetValue(TunnelAccessScopes.Connect, out connectToken);

ACL note: For Fix B to yield a Connect token, the tunnel must grant the caller the Connect scope. Today the Private branch of ApplyAccessModeAsync writes an empty ACL (DevTunnelManagementClientWrapper.cs:159-172); if a token is required, the host would need to add a Connect ACE for the owner/identity rather than an empty ACL. That contradicts the identityΓÇæbased intent, which is why Fix A is preferred. Even under Fix B, the resolver should not throw on a null token in Private mode (Fix A's throw removal still applies).

Recommendation: Implement Fix A (remove the PrivateΓÇæmode throw + reuse the GitHub identity token in the tunnelΓÇæname Private path). Optionally adopt Fix B's GetTunnelAsync correction if telemetry shows a minted Connect token is still wanted for crossΓÇæaccount distribution ΓÇö but the architecture (empty ACL, retired Token mode, design #19) is identityΓÇæbased, so Fix A is the correct primary fix.

---

Expected Tests

Test Name	Class	What It Verifies
ResolveAsync_PrivateMode_WhenConnectTokenIsNull_DoesNotThrow	DevTunnelEndpointResolverTests	Private mode with a null Connect token returns a resolution instead of throwing (replaces ..._WhenConnectTokenIsNull_Throws).
ResolveAsync_PrivateMode_WhenConnectTokenIsNull_YieldsNullTunnelAuthToken	DevTunnelEndpointResolverTests	Private mode with a null Connect token yields a null/identity TunnelAuthToken (identityΓÇæderived connect per design #19).
ResolveAsync_PrivateMode_ConnectsAsOwner_WithoutConnectToken	DevTunnelEndpointResolverTests	Owner connect succeeds (base URI resolved, no exception) when no Connect token is available.
ResolveAsync_AnonymousMode_ReturnsNullToken	DevTunnelEndpointResolverTests	Anonymous mode still resolves with a null token (unchanged).
LookupByNameAsync_PrivateMode_FetchesConnectTokenViaGetTunnelAsync	DevTunnelManagementClientWrapperTests	(Fix B only) When a Connect token is fetched, it comes from GetTunnelAsync with TokenScopes=[Connect], not from a ListTunnelsAsync result.
ApplyAccessModeAsync_AnonymousMode_ReturnsAnonymousConnectToken	DevTunnelManagementClientWrapperTests	Anonymous mode still grants a Connect ACE and returns an anonymous Connect token (unchanged).
CreateDevTunnelNameDataAccessLayer_PrivateMode_WithNullToken_UsesGitHubIdentityHeader	DevTunnelHostServiceTests	(Fix A wiring) The Private tunnelΓÇæname path with a null token authorizes via the GitHub identity token / 401 refresh resolver rather than sending no X-Tunnel-Authorization.

---

Diagnosis references: throw at DevTunnelEndpointResolver.cs:44-55; token misΓÇæfetch via ListTunnelsAsync in DevTunnelManagementClientWrapper.cs:260-284; emptyΓÇæACL Private branch at DevTunnelManagementClientWrapper.cs:159-172; design contradiction at dev-tunnel-host-service.md:485-487. Regressing commits: #517 (d233d835), #521 (fccf4c8e). SDK: Microsoft.DevTunnels.Management/.Contracts 1.3.50 ΓÇö TokenScopes tokens are returned only when retrieving a single tunnel object (GET), never on a list result.

Required Integration Test

In addition to the unit tests above, this fix must be covered by an end-to-end integration test that reproduces the reported scenario (owner, same machine, same GitHub account, Private-mode tunnel) and would have caught this regression:

Test Name	Class	What It Verifies
DevTunnelPrivateConnect_OwnerSameIdentity_ConnectsWithoutConnectToken	DevTunnelIntegrationTests	Starting a Private-mode host tunnel and then resolving + connecting a client as the same owning GitHub identity succeeds end-to-end — the client connects using the identity-derived X-Tunnel-Authorization: tunnel <github-token> path and no Connect-scope token is required (asserts ResolveAsync returns a null tunnelAuthToken in Private mode and the relay connect completes / a request round-trips). This is the exact "playspace 3 debug" → "playspace 3" flow that currently throws "The Management API did not return a Connect-scope tunnel token."

Requirements for the integration test:

• Exercise the real DevTunnelManagementClientWrapper + DevTunnelEndpointResolver against the live DevTunnels service (not a mock), authenticated via the machine's GitHub identity (GITHUB_TOKEN / gh auth token), so it validates the actual ListTunnelsAsync/GetTunnelAsync token behavior rather than a stubbed ConnectToken.
• Host side creates/hosts the Private tunnel and applies access mode; client side resolves by tunnel name and connects. Assert a successful connect (and, ideally, a minimal request/response over the tunnel), proving the owner does not need a Connect-scope token in Private mode.
• Gate it so it is skippable when no GitHub tunnel credentials are available in the environment (e.g. [Trait]/skip-if-no-credentials), consistent with how other network-dependent integration tests in the repo are guarded — but it must run in the environment where tunnel credentials exist.
• A companion assertion that Anonymous mode still returns an anonymous Connect token and connects, so the fix does not regress anonymous tunnels.

[JoshuaRowePhantom]

When implementing this, run the integration test to verify we have done this correctly. If it turns out we need a Connect scoped token, transition to that model.

[JoshuaRowePhantom]

Verification result

Status: ✅ Verified

Commit checked: f071aa81 Fix #1082: Private-mode dev tunnel connect works for owner without a Connect token

Assessment

• Design chosen is Fix A (identity-derived Private connect): The issue body explicitly designates Fix A as RECOMMENDED / "the correct primary fix" (empty ACL + retired Token mode + design Bump ModelContextProtocol and ModelContextProtocol.AspNetCore #19 all point at identity-based connect), and the owner comment authorises transitioning to a Connect-token model only if the integration test shows it is needed. Fix A is correctly implemented.
• Resolver no longer throws in Private mode: DevTunnelEndpointResolver.ResolveAsync now returns lookup.ConnectToken (may be null) for Private; only the retired Token mode still throws when null (guarded with #pragma warning disable CS0618). Matches design Bump ModelContextProtocol and ModelContextProtocol.AspNetCore #19.
• Identity wiring: New DevTunnelClientAuthorization.Resolve selects the auth token — explicit Connect token verbatim; GitHub identity token + 401-refresh resolver for Private-with-null; no header for Anonymous. EntityRepository.CreateDevTunnelNameDataAccessLayerAsync uses it and passes the refresh resolver to WebClientDataAccessLayer, mirroring the working UseGitHubAuthToken path.
• Docs: design point Bump ModelContextProtocol and ModelContextProtocol.AspNetCore #19 in dev-tunnel-host-service.md clarified (null Connect token not fatal; identity-derived; 401-refresh resolver).
• Expected tests present: ResolveAsync_PrivateMode_WhenConnectTokenIsNull_DoesNotThrow (replaces the buggy ..._Throws), ..._YieldsNullTunnelAuthToken, ResolveAsync_PrivateMode_ConnectsAsOwner_WithoutConnectToken, ResolveAsync_AnonymousMode_ReturnsNullToken (unchanged), ApplyAccessModeAsync_AnonymousMode_ReturnsAnonymousConnectToken (unchanged). ResolveAsync_TokenMode_WhenConnectTokenIsNull_Throws preserves the retired-mode branch. The CreateDevTunnelNameDataAccessLayer_PrivateMode_WithNullToken_UsesGitHubIdentityHeader wiring test is present in a new DevTunnelClientAuthorizationTests class (the wiring was extracted into the DevTunnelClientAuthorization helper) with the requested method name, plus two extra branch tests (explicit-token verbatim / anonymous-no-header).
• Required Integration Test present: DevTunnelIntegrationTests.DevTunnelPrivateConnect_OwnerSameIdentity_ConnectsWithoutConnectToken exists, uses the real DevTunnelEndpointResolver + DevTunnelServiceFactory against a hosted Private tunnel via InProcessDevTunnelPrivateFixture, asserts a null TunnelAuthToken + identity-derived authorization, includes the Anonymous companion assertion, and is [IntegrationFact]-gated on PHANTOM_INTEGRATION_GITHUB_TOKEN (skips gracefully without credentials). It compiles against pre-existing fixture/factory types.

Adjudicated deviations

• Fix A chosen, Fix B's LookupByNameAsync_PrivateMode_FetchesConnectTokenViaGetTunnelAsync omitted (accepted): That test is explicitly marked "(Fix B only)" in the issue's Expected Tests table. Since the design mandated is Fix A, there is no GetTunnelAsync-token fetch to assert, and the omission is legitimate.
• Wiring test relocated to a new class (accepted): placed in DevTunnelClientAuthorizationTests because the token-selection logic was extracted into the DevTunnelClientAuthorization helper; the method name from the issue is preserved.

Outstanding gaps

None — all criteria satisfied.

Tests

Fast suite: run 1 = 4147/4148 (single unrelated #1068-test flake), run 2 = 4148/4148 clean. Unit tests (DevTunnelEndpointResolverTests, DevTunnelClientAuthorizationTests) green in both runs. The mandated end-to-end owner-connect integration test is credential-gated and cannot run in the fast suite — applying needs-slow-tests so the dev-tunnel/network integration coverage is exercised at checkin (per the diagnosis and the maintainer comment: run the integration test to confirm the identity-derived model is correct).

[JoshuaRowePhantom]

Cross-reference: #1089 tracks a DISTINCT dev-tunnel failure — an HTTP 401 'Tunnel service response status code: Unauthorized' returned by the DevTunnels service during the HOSTING flow (DevTunnelHostService.StartAsync), rejecting the GitHub-scheme management identity. That is mechanically different from this issue's null Connect-scope token thrown by the client resolver before any request is made. Filed separately.

[JoshuaRowePhantom]

Checked in

Commit: 6c56669
Branch fast-forwarded: features → 6c56669

Landed as part of batch/chat1.