Add HTTP.blocklist for denying requests by IP/host

.mutant.yml

@@ -16,6 +16,10 @@ requires:
 mutation:
   operators: full
   timeout: 10.0
+  ignore_patterns:
+    # getaddrinfo resolves the same addresses regardless of the socktype and
+    # protocol arguments, so mutating them cannot change behavior.
+    - send{selector=getaddrinfo}
 
 matcher:
   subjects:
@@ -44,6 +48,10 @@ matcher:
     - "HTTP::Response#init_body"
     - "HTTP::Response#init_request"
     # Equivalent mutations — semantically identical code that mutant cannot distinguish
+    # `is_a?` -> `instance_of?`; Blocklist is never subclassed
+    - "HTTP::Blocklist.new"
+    # `downcase` -> `upcase`; case folding is applied to both rules and queries
+    - "HTTP::Blocklist#normalize_host"
     - "HTTP::Response::Body#loggable?"
     - "HTTP::Response::Body#read_contents"
     - "HTTP::Response::Body#to_s"

CHANGELOG.md

@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- `HTTP.blocklist` denies requests to hostnames and IP addresses you don't want
+  reachable, raising `HTTP::BlockedHostError`. `IPAddr` entries are matched
+  against every address the request host resolves to, and the socket connects to
+  the address that was validated, so DNS cannot answer differently between the
+  check and the connect. `String` entries are hostnames, matching the request
+  host and its subdomains. An optional `deny:` callable receives each resolved
+  address as an `IPAddr` and blocks it by returning true, e.g.
+  `HTTP.blocklist(deny: ->(address) { address.loopback? || address.private? })`.
+  Every redirect hop is checked. Also available as a per-request and constructor
+  option, either as a list of rules or as a Hash:
+  `HTTP.get(url, blocklist: [IPAddr.new("127.0.0.0/8")])`,
+  `HTTP.get(url, blocklist: { entries: [...], deny: ->(a) { a.loopback? } })`.
+  Note that requests using a blocklist connect to a single validated address, so
+  dual-stack fallback (Happy Eyeballs) does not apply to them. A blocklist
+  cannot be enforced through a proxy — the proxy resolves the target and makes
+  the connection — so combining the two warns once and the checks are advisory.
+
 ### Fixed
 
 - Building a default `Host` header now raises `HTTP::RequestError` when the

README.md

@@ -146,6 +146,50 @@ HTTP.base_uri("https://api.example.com/v1").persistent do |http|
 end
 ```
 
+### Blocklist
+
+When fetching user-supplied URLs, block hosts you don't want reachable:
+
+```ruby
+http = HTTP.blocklist(
+  IPAddr.new("169.254.0.0/16"), # link-local, incl. cloud metadata
+  "internal.example.com",       # by hostname
+  deny: ->(address) { address.loopback? || address.private? }
+)
+
+http.get("http://127.0.0.1/")   # raises HTTP::BlockedHostError
+http.get("https://example.com") # allowed
+```
+
+Entries are classified by type, and the block decides anything they can't:
+
+- **`IPAddr`** entries are checked against *every* address the request host
+  resolves to, so a hostname pointing at `127.0.0.1` is blocked too. The socket
+  then connects to the address that was validated, so DNS cannot return a
+  different answer between the check and the connect.
+- **`String`** entries are hostnames, matched against the request host and its
+  subdomains, case-insensitively: `internal.example.com` also blocks
+  `db.internal.example.com`. A String is *always* a hostname — write
+  `IPAddr.new("127.0.0.0/8")`, not `"127.0.0.0/8"`.
+- **`deny:`** receives each resolved address as an `IPAddr` and blocks it by
+  returning true, like `retriable`'s `should_retry:`. It composes with the
+  entries: an address is blocked if either matches. In the `blocklist:` option
+  form, pass a Hash — `blocklist: { entries: [...], deny: ->(a) { ... } }`.
+
+Every redirect hop is checked, so a redirect into a blocked host raises rather
+than being followed. `HTTP::BlockedHostError` is an `HTTP::RequestError`, not a
+`ConnectionError`, so `retriable` will not retry it.
+
+**A blocklist cannot be enforced through a proxy**, and warns once when the two
+are combined. The proxy resolves the target and makes the connection, so the
+addresses checked here are not necessarily the ones reached, and a target that
+only resolves from the proxy's network will fail to resolve locally. The rules
+are still applied on a best-effort basis; treat the result as advisory.
+
+Note that with a blocklist configured http.rb connects to a single validated
+address rather than letting the OS try each one, so dual-stack fallback
+(Happy Eyeballs) does not apply to those requests.
+
 ### Thread Safety
 
 Configured sessions are safe to share across threads:

lib/http.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "http/errors"
+require "http/blocklist"
 require "http/timeout/null"
 require "http/timeout/per_operation"
 require "http/timeout/global"

lib/http/blocklist.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+require "ipaddr"
+require "socket"
+
+module HTTP
+  # Denies requests to hostnames and IP addresses matching configured rules
+  #
+  # @example
+  #   HTTP::Blocklist.new([IPAddr.new("127.0.0.0/8"), "internal.example.com"])
+  #   HTTP::Blocklist.new(deny: ->(address) { address.loopback? || address.private? })
+  class Blocklist
+    # Returns existing Blocklist or creates a new one
+    #
+    # @example
+    #   HTTP::Blocklist.new([IPAddr.new("127.0.0.0/8"), "localhost"])
+    #
+    # @param [HTTP::Blocklist, Array<IPAddr, String>, String] entries
+    # @param [#call, nil] deny predicate called with each resolved address
+    # @return [HTTP::Blocklist]
+    # @api public
+    def self.new(entries = [], deny: nil)
+      return entries if entries.is_a?(Blocklist)
+
+      super
+    end
+
+    # Initializes a blocklist from address rules, hostname rules, and a predicate
+    #
+    # @example
+    #   HTTP::Blocklist.new([IPAddr.new("169.254.0.0/16")], deny: ->(ip) { ip.loopback? })
+    #
+    # @param [Array<IPAddr, String>, String] entries address and hostname rules
+    # @param [#call, nil] deny called with each resolved address, truthy blocks it
+    # @return [HTTP::Blocklist]
+    # @api public
+    def initialize(entries, deny:)
+      rules = Array(entries) #: Array[IPAddr | String]
+      hosts = rules.grep(String) #: Array[String]
+
+      @addresses = rules.grep(IPAddr)
+      @hosts     = hosts.map { |entry| ".#{normalize_host(entry)}" }
+      @deny      = deny
+    end
+
+    # Whether a hostname matches any hostname rule
+    #
+    # @example
+    #   blocklist.blocked_host?("api.example.com")
+    #
+    # @param [String] host hostname to check
+    # @return [Boolean]
+    # @api public
+    def blocked_host?(host)
+      name = ".#{normalize_host(host)}"
+      @hosts.any? { |rule| name.end_with?(rule) }
+    end
+
+    # Whether an address matches any address rule or is denied by the predicate
+    #
+    # IPv4-mapped IPv6 addresses are reduced to their native IPv4 form first,
+    # so `::ffff:127.0.0.1` cannot slip past a `127.0.0.0/8` rule, and `deny`
+    # sees the same normalized address.
+    #
+    # @example
+    #   blocklist.blocked_address?(IPAddr.new("127.0.0.1"))
+    #
+    # @param [IPAddr] address address to check
+    # @return [Boolean]
+    # @api public
+    def blocked_address?(address)
+      ip   = address.native
+      deny = @deny
+
+      return true if @addresses.any? { |rule| rule.include?(ip) }
+      return false unless deny
+
+      deny.call(ip) ? true : false
+    end
+
+    # Resolves a hostname, denying it if the name or any address is blocked
+    #
+    # Hostname rules are checked before resolving, so a blocked name never
+    # generates DNS traffic. Every resolved address is checked, and the first
+    # one is returned so the caller can connect to a validated address rather
+    # than resolving a second time.
+    #
+    # @example
+    #   blocklist.validate!("example.com") # => "93.184.216.34"
+    #
+    # @param [String] host hostname or IP address to resolve
+    # @return [String] the validated address to connect to
+    # @raise [HTTP::BlockedHostError] when the host or an address is blocked
+    # @api public
+    def validate!(host)
+      raise BlockedHostError, "blocked host: #{host}" if blocked_host?(host)
+
+      addresses = Addrinfo.getaddrinfo(host, nil, nil, :STREAM).map(&:ip_address)
+
+      addresses.each do |address|
+        next unless blocked_address?(IPAddr.new(address))
+
+        raise BlockedHostError, "#{host} resolves to blocked address: #{address}"
+      end
+
+      addresses.first
+    end
+
+    # Warns once that a blocklist cannot be enforced through a proxy
+    #
+    # @example
+    #   blocklist.warn_proxy_incompatible
+    #
+    # @return [void]
+    # @api private
+    def warn_proxy_incompatible
+      return if @warned
+
+      @warned = true
+      warn "HTTP::Blocklist: a blocklist cannot be enforced through a proxy, because the proxy " \
+           "resolves the target and makes the connection; the addresses checked here are not " \
+           "necessarily the ones reached"
+    end
+
+    private
+
+    # Normalizes a hostname for comparison
+    #
+    # @param [String] host hostname to normalize
+    # @return [String] downcased hostname without a trailing dot
+    # @api private
+    def normalize_host(host)
+      host.downcase.chomp(".")
+    end
+  end
+end

lib/http/chainable.rb

@@ -166,6 +166,23 @@ def via(*proxy)
     end
     alias through via
 
+    # Deny requests to blocked hostnames and IP addresses
+    #
+    # @example
+    #   HTTP.blocklist(IPAddr.new("127.0.0.0/8"), "localhost").get("http://example.com")
+    #
+    # @example Deciding per address
+    #   HTTP.blocklist(deny: ->(address) { address.loopback? || address.private? })
+    #
+    # @param [Array<IPAddr, String>] entries address and hostname rules
+    # @param [#call, nil] deny called with each resolved address, truthy blocks it
+    # @return [HTTP::Session]
+    # @see HTTP::Blocklist
+    # @api public
+    def blocklist(*entries, deny: nil)
+      branch default_options.with_blocklist(entries: entries.flatten, deny: deny)
+    end
+
     # Make client follow redirects
     #
     # @example

lib/http/client.rb

@@ -47,13 +47,15 @@ def request(verb, uri,
                 response: nil, encoding: nil, follow: nil, ssl: nil, ssl_context: nil,
                 proxy: nil, nodelay: nil, features: nil, retriable: nil,
                 socket_class: nil, ssl_socket_class: nil, timeout_class: nil,
-                timeout_options: nil, keep_alive_timeout: nil, base_uri: nil, persistent: nil)
+                timeout_options: nil, keep_alive_timeout: nil, base_uri: nil, persistent: nil,
+                blocklist: nil)
       opts = { headers: headers, params: params, form: form, json: json, body: body,
                response: response, encoding: encoding, follow: follow, ssl: ssl,
                ssl_context: ssl_context, proxy: proxy, nodelay: nodelay, features: features,
                retriable: retriable, socket_class: socket_class, ssl_socket_class: ssl_socket_class,
                timeout_class: timeout_class, timeout_options: timeout_options,
-               keep_alive_timeout: keep_alive_timeout, base_uri: base_uri, persistent: persistent }.compact
+               keep_alive_timeout: keep_alive_timeout, base_uri: base_uri, persistent: persistent,
+               blocklist: blocklist }.compact
       opts = @default_options.merge(opts)
       builder = Request::Builder.new(opts)
       req     = builder.build(verb, uri)

lib/http/connection.rb

@@ -255,7 +255,7 @@ def check_premature_eof(eof)
     # @api private
     def connect_socket(req, options)
       @socket = options.timeout_class.new(**options.timeout_options)
-      @socket.connect(options.socket_class, req.socket_host, req.socket_port, nodelay: options.nodelay)
+      @socket.connect(options.socket_class, connect_address(req, options), req.socket_port, nodelay: options.nodelay)
 
       send_proxy_connect_request(req)
       start_tls(req, options)

lib/http/connection/internals.rb

@@ -34,6 +34,24 @@ def flush_or_close_response(response)
         end
       end
 
+      # Resolve the address to connect to, enforcing the blocklist
+      #
+      # @param [HTTP::Request] req
+      # @param [HTTP::Options] options
+      # @return [String] host or validated address to connect to
+      # @raise [HTTP::BlockedHostError] when the request target is blocked
+      # @api private
+      def connect_address(req, options)
+        blocklist = options.blocklist
+        return req.socket_host unless blocklist
+
+        proxied = req.using_proxy?
+        blocklist.warn_proxy_incompatible if proxied
+        address = blocklist.validate!(req.host)
+
+        proxied ? req.socket_host : address
+      end
+
       # Sets up SSL context and starts TLS if needed
       # @param (see Connection#initialize)
       # @return [void]

lib/http/errors.rb

@@ -17,6 +17,9 @@ class SocketWriteError < ConnectionError; end
   # Generic Request error
   class RequestError < Error; end
 
+  # Error raised when host matches a blocked host/ip
+  class BlockedHostError < RequestError; end
+
   # Generic Response error
   class ResponseError < Error; end
 

lib/http/options.rb

@@ -152,7 +152,8 @@ def initialize(
       retriable: nil,
       base_uri: nil,
       persistent: nil,
-      ssl_context: nil
+      ssl_context: nil,
+      blocklist: nil
     )
       assign_options(binding)
     end

lib/http/options/definitions.rb

@@ -90,6 +90,25 @@ def retriable=(value)
         end
     end
 
+    def_option :blocklist, reader_only: true
+
+    # Sets the blocklist of denied hostnames and addresses
+    #
+    # Accepts a {Blocklist}, a list of rules, or a Hash of `entries` and `deny`.
+    #
+    # @param [HTTP::Blocklist, Array<IPAddr, String>, String, Hash, nil] value
+    # @api private
+    # @return [HTTP::Blocklist, nil]
+    def blocklist=(value)
+      @blocklist =
+        if value.respond_to?(:to_hash)
+          hash = value #: Hash[Symbol, untyped]
+          Blocklist.new(hash[:entries], **hash.except(:entries))
+        elsif value
+          Blocklist.new(value)
+        end
+    end
+
     def_option :base_uri, reader_only: true
 
     # Sets the base URI for resolving relative request paths

lib/http/session.rb

@@ -108,14 +108,16 @@ def request(verb, uri,
                 response: nil, encoding: nil, follow: nil, ssl: nil, ssl_context: nil,
                 proxy: nil, nodelay: nil, features: nil, retriable: nil,
                 socket_class: nil, ssl_socket_class: nil, timeout_class: nil,
-                timeout_options: nil, keep_alive_timeout: nil, base_uri: nil, persistent: nil, &block)
+                timeout_options: nil, keep_alive_timeout: nil, base_uri: nil, persistent: nil,
+                blocklist: nil, &block)
       merged = default_options.merge(
         { headers: headers, params: params, form: form, json: json, body: body,
           response: response, encoding: encoding, follow: follow, ssl: ssl,
           ssl_context: ssl_context, proxy: proxy, nodelay: nodelay, features: features,
           retriable: retriable, socket_class: socket_class, ssl_socket_class: ssl_socket_class,
           timeout_class: timeout_class, timeout_options: timeout_options,
-          keep_alive_timeout: keep_alive_timeout, base_uri: base_uri, persistent: persistent }.compact
+          keep_alive_timeout: keep_alive_timeout, base_uri: base_uri, persistent: persistent,
+          blocklist: blocklist }.compact
       )
       client = persistent? ? nil : make_client(default_options)
       res    = perform_request(client, verb, uri, merged)