Typecheck unreachable code

[A5rocks]

• Fixes [regression] no type checking beyond perceived isinstance guard failure #18329
• Maybe type narrowing leads to "unreachable" code being actually reachable #12598
• Maybe <nothing> object is not iterable #4330
• Fixes "Cannot determine type" if first assignment to attribute is unreachable #9709
• Fixes assert_type fails to fail for NoReturn #16319
• Fixes Union of Never can't be used #18779

Mypy doesn't typecheck unreachable code and this is surprising to many. This is an attempt at fixing that. It's a draft until:

• I list out the issues fixed (done)
• I fix the bugs raised by the test cases (done)
• I fix the bugs raised by mypy primer (done)
• I get feedback that this is fine (skipped for now)
• I put this behind a flag (we can bundle this in --strict maybe and hopefully enable it by default in 2.0) (skipped for now, I will add after reviews)

Problem areas based on test cases:

• currently we use None for unreachable type maps but we want to know what is unreachable to mark it as Never (fixed what was relevant in tests. narrowing is not guaranteed by mypy and we can fix things in the future -- I'd rather have context for any fixes)
  • it doesn't seem too bad to mark unreachability based on if a variable has a Never type
• analyzing a function multiple times for type var values (fixed)
  • I think I can disable this new behavior based on if we're in a function like that?
• reporting unreachability multiple times (block in a block) (fixed)
  • blocks could store unreachability report status
• exports could change (see check-incremental.test's changes)
  • I think this is fine?
• Never doesn't support operations (e.g. <Never>.f(), <Never> > 5) (fixed)
  • it should return Never for these, I think?

Also maybe reveal_type(<Never>) should be non-trivial. (fixed)

Generally I tried enabling --warn-unreachable in a test case if it looked like people were relying on un-checked unreachable code as a test. Also obviously many of the test case changes are wrong I just did not have the energy to fix all the bugs (and honestly maybe should think through the two things above -- the first especially was a source of issues).

I also have no clue what mypyc does so the test change there could be awful.

[JukkaL]

Thanks for the PR! Not a full review yet, just a few comments. I probably won't have time to review this in the next few days at least, so if anybody else has bandwidth, please have a look.

[JukkaL]

Also there's now a merge conflict.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

beartype (https://github.com/beartype/beartype)
+ beartype/_util/cache/utilcachecall.py:575: error: Incompatible return value type (got "Callable[..., Any]", expected "CallableT")  [return-value]

manticore (https://github.com/trailofbits/manticore)
+ manticore/wasm/types.py:56: error: Unused "type: ignore" comment  [unused-ignore]

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/server/models/deployments.py:744: error: "Never" has no attribute "model_dump"  [attr-defined]

pandas (https://github.com/pandas-dev/pandas)
- pandas/io/formats/style.py:1903: error: "Never" has no attribute "shape"  [attr-defined]
- pandas/io/formats/style.py:1906: error: "Never" has no attribute "shape"  [attr-defined]

spark (https://github.com/apache/spark)
+ python/pyspark/core/context.py:818: error: Unused "type: ignore" comment  [unused-ignore]
+ python/pyspark/core/context.py:819: error: Unused "type: ignore" comment  [unused-ignore]

xarray (https://github.com/pydata/xarray)
+ xarray/core/variable.py:900: error: Unused "type: ignore" comment  [unused-ignore]
+ xarray/core/variable.py:902: error: Unused "type: ignore" comment  [unused-ignore]
+ xarray/core/variable.py:2786: error: Unused "type: ignore" comment  [unused-ignore]
+ xarray/core/variable.py:2788: error: Unused "type: ignore" comment  [unused-ignore]
- xarray/computation/apply_ufunc.py: note: In function "apply_variable_ufunc":
- xarray/computation/apply_ufunc.py:845: error: "Never" has no attribute "ndim"  [attr-defined]
- xarray/computation/apply_ufunc.py:848: error: "Never" has no attribute "ndim"  [attr-defined]

jinja (https://github.com/pallets/jinja)
+ src/jinja2/runtime.py:903: error: Statement is unreachable  [unreachable]

websockets (https://github.com/aaugustin/websockets)
+ src/websockets/legacy/protocol.py:1077: error: Argument 1 to "join" of "str" has incompatible type "list[str | bytes]"; expected "Iterable[str]"  [arg-type]
+ src/websockets/legacy/protocol.py:1077: error: Argument 1 to "join" of "bytes" has incompatible type "list[str | bytes]"; expected "Iterable[Buffer]"  [arg-type]

Tanjun (https://github.com/FasterSpeeding/Tanjun)
- tanjun/parsing.py:588: error: "Never" has no attribute "__iter__" (not iterable)  [attr-defined]
- tanjun/parsing.py:1632: error: Statement is unreachable  [unreachable]
- tanjun/parsing.py:1642: error: Statement is unreachable  [unreachable]
- tanjun/dependencies/limiters.py:851: error: "Never" has no attribute "unlock"  [attr-defined]
+ tanjun/dependencies/limiters.py:852: error: Statement is unreachable  [unreachable]
- tanjun/commands/slash.py:820: error: Statement is unreachable  [unreachable]
- tanjun/commands/slash.py:1787: error: Statement is unreachable  [unreachable]
- tanjun/clients.py:2294: error: "Never" has no attribute "__iter__" (not iterable)  [attr-defined]
+ tanjun/clients.py:2295: error: Statement is unreachable  [unreachable]

ibis (https://github.com/ibis-project/ibis)
- ibis/common/bases.py:231: error: Need type annotation for "field"  [var-annotated]

jax (https://github.com/google/jax)
+ jax/_src/interpreters/partial_eval.py:97: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/_src/interpreters/partial_eval.py:100: error: No overload variant of "get" of "dict" matches argument types "int | DArray | Tracer | Var | DBIdx | InDBIdx | OutDBIdx", "int | DArray | Tracer | Var | DBIdx | InDBIdx | OutDBIdx"  [call-overload]
+ jax/_src/interpreters/partial_eval.py:100: note: Possible overload variants:
+ jax/_src/interpreters/partial_eval.py:100: note:     def get(self, Name, None = ..., /) -> DBIdx | None
+ jax/_src/interpreters/partial_eval.py:100: note:     def get(self, Name, DBIdx, /) -> DBIdx
+ jax/_src/interpreters/partial_eval.py:100: note:     def [_T] get(self, Name, _T, /) -> DBIdx | _T
+ jax/_src/interpreters/batching.py:235: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/_src/interpreters/batching.py:237: error: No overload variant of "get" of "dict" matches argument types "int | DArray | Tracer | Var | DBIdx | InDBIdx | OutDBIdx", "int | DArray | Tracer | Var | DBIdx | InDBIdx | OutDBIdx"  [call-overload]
+ jax/_src/interpreters/batching.py:237: note: Possible overload variants:
+ jax/_src/interpreters/batching.py:237: note:     def get(self, Name, None = ..., /) -> DBIdx | None
+ jax/_src/interpreters/batching.py:237: note:     def get(self, Name, DBIdx, /) -> DBIdx
+ jax/_src/interpreters/batching.py:237: note:     def [_T] get(self, Name, _T, /) -> DBIdx | _T
+ jax/_src/interpreters/batching.py:261: error: Unused "type: ignore" comment  [unused-ignore]

core (https://github.com/home-assistant/core)
+ homeassistant/components/rfxtrx/__init__.py:455: error: Statement is unreachable  [unreachable]

graphql-core (https://github.com/graphql-python/graphql-core)
+ tests/execution/test_subscribe.py:212: error: Unused "type: ignore" comment  [unused-ignore]

pytest (https://github.com/pytest-dev/pytest)
+ testing/code/test_source.py:238: error: Unused "type: ignore" comment  [unused-ignore]

[A5rocks]

Wow those conflicts are nasty. I should I probably just remake the PR rather than try to resolve them...

[hauntsaninja]

Yeah, this is a remake situation 😅
Several of the test changes could also be in their own PR.

[A5rocks]

I'm just going to close this! I kept forgetting to remake it. (I'll put the very basic changes into a new PR and... maybe I'll figure out how to make a stacked PR? Who knows...)

[A5rocks]

Ok, check out #21834!