net: add createPipe() and createSocketPair()

[kingces95]

Add createPipe() to net module returning a readable and writable endpoint owned by the parent process.

The endpoints may be passed to child_process.spawn() stdio. This lets the parent lend a pipe endpoint to a child without turning the parent stream itself into child-owned stdio. The parent can then reclaim unread bytes or lend the same endpoint to a later child.

This is useful for bash-like partial consumption of long-lived streams. For example, a parent can keep ownership of a Server Sent Event stream while delegating bounded reads to external tools, then continue parsing from the exact byte where the child stopped.

Endpoints created by pipe.createPipe() are rejected by spawnSync(), and an endpoint may only be leased to one child process at a time.

net.createPipe

Child process 'sips' just a single character from the pipe leaving parent to drain what's left.

const { spawn } = require('node:child_process');
const { createPipe } = require('node:net');
const { text } = require('node:stream/consumers');

const { readable, writable } = createPipe();
const child = spawn(process.execPath, ['-e', `
  const fs = require('node:fs');
  const buffer = Buffer.alloc(1);
  const count = fs.readSync(0, buffer, 0, 1, null);
  fs.writeSync(1, buffer.subarray(0, count));
`], {
  stdio: [readable, 'pipe', 'inherit'],
});

const output = text(child.stdout);
writable.end('abc');

child.on('close', async () => {
  console.log(await output); // Prints: a
  console.log(await text(readable)); // Prints: bc
});

net.createSocketPair()

The net.createPipe() and net.createSocketPair() could be two separate PRs and can be easily cleaved on request. I included both because taken together they form a tidy family of handle backed streams.

• net.createPipe() exposes directional OS-backed endpoints: { readable, writable }.
• net.createSocketPair() exposes symmetric duplex OS-backed endpoints: [left, right].

const { spawn } = require('node:child_process');
const { createSocketPair } = require('node:net');

const [left, right] = createSocketPair();

const child = spawn(process.execPath, ['-e', `
  process.on('message', (message, socket) => {
    socket.on('data', (chunk) => {
      socket.write(chunk.toString().toUpperCase());
    });
    socket.resume();
    process.send('ready');
  });
`], {
  stdio: ['ignore', 'inherit', 'inherit', 'ipc'],
});

child.once('message', () => {
  left.write('hello');
});

left.once('data', (chunk) => {
  console.log(chunk.toString()); // Prints: HELLO
  left.destroy();
  child.kill();
});

child.send('socket', right, { keepOpen: false });

Why leasing?

Node has policy which disallows pipes used to construct pipelines between children from being passed to other children; A pipe created by spawning a child process is marked with the symbol kIsUsedAsStdio the first time it is passed as stdio to another child process so an error can be raised should the pipe be passed to a third child process.

To abide by that policy, the pipes returned from createPipe() can be marked with kLeasedTo which points to the child currently using the pipe. When the child exits the lease is released and the pipe can be handed to another child process. In this way, only one child is reading or writing at a time in accordance with Node policy. The new ability (limited to pipes created by createPipe()) to repass a pipe to a subsequent child allows emulation of the common shell idiom which is the impetus for the feature.

Under The Hood

Under the hood, createPipe() and createSocketPair() sit on uv_pipe and uv_socketpair respectively. While those libuv APIs do not appear to be used directly by Node’s own src/ today, their data types (uv_pipe_t and uv_os_sock_t) have been mapped into Javascript as Node Pipe(). This makes exposing their functionality to Javascript straightforward: Create a pair of Pipe(), initialize them with an internal call to pairPipes(pipe, pipe) or pairSockets(pipe, pipe), wrap in Node Socket(), and return the result. See lib/net.js.

A stream descriptor of type wrap is duplicated into a to child process without invoking any logic implying ownership by the node process abstraction for the child. For example, this happens when a child producer's stdout is passed as a child consumer's stdin. The producer "leases" its stream to the consumer. Pipes generated by createPipe() follow this same code path. They are "leased" by the parent to the child. See process_wrap.cpp. The difference is the former lease is never released while the latter is released when the child exits.

A stream returned by createSocketPair() follows the same wrapping model a TCP-backed net.Socket stream and makes use of use of the same IPC handle-transfer story between cooperating Node processes.

Thank you!

Thank you for considering my pull request!

[marco-ippolito]

I dont think we should create a new module

[jasnell]

+1, there's no justification for a new top-level module.

[kingces95]

Yep. I'll move it to net.