Make the walkthrough seed opt-in: unseeded FrotzEnv is stochastic (v4.0)

README.md

@@ -43,6 +43,42 @@ python -m spacy download en_core_web_sm
 - [Utilities](https://jericho-py.readthedocs.io/en/latest/util.html)
 - [Defines](https://jericho-py.readthedocs.io/en/latest/defines.html)
 
+## Breaking changes in Jericho 4.0
+
+Prior to version 4.0, creating an environment without specifying a seed would silently
+use the game's walkthrough seed (when known), making episodes deterministic. As described
+in the [Jericho paper](http://arxiv.org/abs/1909.05398), a fixed random seed is a *handicap*
+that should be chosen and disclosed explicitly. Starting with version 4.0:
+
+- `FrotzEnv(rom)` (i.e. without a seed) is now stochastic: a fresh random seed is drawn for
+  each episode. The seed actually used is reported in `reset()`'s info dict and as
+  `FrotzEnv.episode_seed`, so any episode can be reproduced after the fact.
+- `FrotzEnv.reset()` accepts a `use_walkthrough_seed` argument to seed the emulator with the
+  game's walkthrough seed, which is needed to reproduce the walkthrough. It raises `ValueError`
+  if the game has no known walkthrough seed (check `env.walkthrough_seed is None`).
+- `FrotzEnv.walkthrough_seed` returns the game's walkthrough seed, if it is known, otherwise `None`.
+- An `ImplicitRandomSeedWarning` is issued (once per environment) when the first episode of a
+  game that has a walkthrough seed begins — via `reset()` or a direct `step()` — without an
+  explicit seeding choice. Providing any seed (e.g. `seed=-1` to explicitly request random episodes),
+  calling `env.seed()`, or resetting with `use_walkthrough_seed=True` silences it.
+
+To keep the old behavior (e.g. to reproduce results published with Jericho ≤ 3.x), either pin
+`pip install 'jericho<4'` or seed explicitly: `env.seed(env.walkthrough_seed)` before `env.reset()`.
+
+```python
+from jericho import FrotzEnv
+
+env = FrotzEnv("zork1.z5")  # Stochastic (a random seed is drawn per episode).
+obs, info = env.reset()     # info['seed'] (also env.episode_seed) is the drawn seed.
+replay = FrotzEnv("zork1.z5", seed=info['seed'])  # Reproduces the episode above.
+
+env = FrotzEnv("zork1.z5", seed=-1)  # Stochastic, explicitly (no warning).
+env = FrotzEnv("zork1.z5", seed=42)  # Deterministic with seed 42.
+
+env.reset(use_walkthrough_seed=True)  # Deterministic, reproduces env.get_walkthrough().
+print(env.walkthrough_seed)           # 12
+```
+
 ## Agents
 
 - [Reading Comprehension Deep Q-Network (RCDQN)](https://github.com/XiaoxiaoGuo/rcdqn)

docs/source/tutorial_quick.rst

@@ -56,6 +56,7 @@ Jericho implements a reinforcement learning interface in which the agent provide
 
                 from jericho import *
                 # Create the environment, optionally specifying a random seed
+                # (by default, a fresh random seed is drawn for each episode).
                 env = FrotzEnv("z-machine-games-master/jericho-game-suite/zork1.z5")
                 initial_observation, info = env.reset()
                 done = False
@@ -127,12 +128,15 @@ One of the most common difficulties with parser-based text games is identifying
 Walkthroughs
 ------------
 
-Jericho provides walkthroughs for supported games using :meth:`jericho.FrotzEnv.get_walkthrough`. To use the walkthrough, it is necessary to reset the environment with the desired seed:
+Jericho provides walkthroughs for supported games using :meth:`jericho.FrotzEnv.get_walkthrough`. To reproduce a walkthrough, it is necessary to reset the environment with the game's walkthrough seed, which is available via :attr:`jericho.FrotzEnv.walkthrough_seed`:
 
 .. code-block:: python
 
                 >>> from jericho import *
                 >>> env = FrotzEnv("z-machine-games-master/jericho-game-suite/zork1.z5")
                 >>> walkthrough = env.get_walkthrough()
+                >>> env.reset(use_walkthrough_seed=True)  # Applies the walkthrough seed to this episode only.
                 >>> for act in walkthrough:
                 >>>     env.step(act)
+
+.. note:: Since Jericho 4.0, an environment created without an explicit seed is stochastic: a fresh random seed is drawn for each episode. The seed actually used is reported in the info dict returned by :meth:`jericho.FrotzEnv.reset` and via :attr:`jericho.FrotzEnv.episode_seed`, so any episode can be reproduced after the fact. Seeding the emulator (e.g. with the walkthrough seed) is a *handicap*, as defined in the `Jericho paper <https://arxiv.org/abs/1909.05398>`_, and should be disclosed when reporting results.

jericho/jericho.py

@@ -15,8 +15,10 @@
 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
 
 import os
+import random
 import shutil
 import tempfile
+import operator
 import warnings
 import hashlib
 
@@ -368,17 +370,49 @@ class TruncatedInputActionWarning(UserWarning):
     pass
 
 
+class ImplicitRandomSeedWarning(UserWarning):
+    pass
+
+
+def _resolve_seed(seed):
+    '''
+    Resolves a user-provided seed to the int stored on the environment.
+
+    The emulator receives the seed as a C int; without a range check, a value
+    like 2**32-1 (e.g. from np.random.randint(2**32)) would silently wrap to
+    the -1 "stochastic" sentinel, making an explicitly seeded env stochastic.
+    '''
+    if seed is None:
+        return -1
+    seed = operator.index(seed)  # Accepts any integer type; rejects e.g. floats.
+    if not -2**31 <= seed < 2**31:
+        raise ValueError("seed must fit in a signed 32-bit integer, got {}.".format(seed))
+    return seed
+
+
+# Episode seeds must stay distinct across forked workers (get_valid_actions'
+# mp.Pool) and independent of random.seed(); SystemRandom draws from OS
+# entropy with no process-local state, guaranteeing both.
+_SYSTEM_RNG = random.SystemRandom()
+
+
 class FrotzEnv():
     """
     The Frotz Environment is a fast interface to Z-Machine games.
 
     :param story_file: Path to a Z-machine rom file (.z3/.z5/.z6/.z8).
     :param seed: Seed the random number generator used by the emulator.
-                 Default: use walkthrough's seed if it exists,
-                          otherwise use value of -1 which changes with time.
+                 Default: None (equivalent to -1), i.e. a fresh random seed is
+                 drawn for each episode, making episodes stochastic. The seed of the
+                 current episode is available as :attr:`jericho.FrotzEnv.episode_seed`.
     :type story_file: path
     :type seed: int
 
+    .. note:: Since Jericho 4.0, the seed needed to reproduce a game's walkthrough
+              is no longer used by default. To reproduce a walkthrough, either call
+              :meth:`jericho.FrotzEnv.reset` with `use_walkthrough_seed=True` or
+              provide :attr:`jericho.FrotzEnv.walkthrough_seed` as the `seed` argument.
+
     """
     def __init__(self, story_file, seed=None):
         self._cache = {}
@@ -397,8 +431,9 @@ def load(self, story_file, seed=None):
 
         :param story_file: Path to a Z-machine rom file (.z3/.z5/.z6/.z8).
         :param seed: Seed the random number generator used by the emulator.
-                    Default: use walkthrough's seed if it exists,
-                            otherwise use value of -1 which changes with time.
+                    Default: None (equivalent to -1), i.e. a fresh random seed is
+                    drawn for each episode, making episodes stochastic. The seed of the
+                    current episode is available as :attr:`jericho.FrotzEnv.episode_seed`.
         :type story_file: path
         :type seed: int
         '''
@@ -425,41 +460,165 @@ def load(self, story_file, seed=None):
 
         rom, self._bindings, self.act_gen = self._cache[story_file]
 
-        self.seed(seed)
-        self.frotz_lib.setup(self.story_file, self._seed, rom, len(rom))
+        # Track seed explicitness here rather than via seed(): a direct call
+        # to seed() is always an explicit choice, but the constructor default
+        # (seed=None) is not.
+        self._seed_is_explicit = seed is not None
+        self._seed = _resolve_seed(seed)
+        self._warned_implicit_seed = False
+        self._episode_seed_implicit = not self._seed_is_explicit
+        self.frotz_lib.setup(self.story_file, self._next_episode_seed(), rom, len(rom))
         self.player_obj_num = self.frotz_lib.get_self_object_num()
 
+    def _next_episode_seed(self):
+        '''
+        The concrete seed to hand to the emulator for the episode being started.
+
+        The stochastic sentinel (-1) must never reach the emulator: it would
+        trigger the `time(0)` fallback in `os_random_seed`, whose one-second
+        resolution makes e.g. parallel unseeded envs play identical episodes.
+        Resolving it here to a drawn seed also keeps every episode reproducible
+        after the fact (see :attr:`jericho.FrotzEnv.episode_seed`).
+        '''
+        if self._seed == -1:
+            self._episode_seed = _SYSTEM_RNG.getrandbits(31)
+        else:
+            self._episode_seed = self._seed
+        return self._episode_seed
+
+    def _maybe_warn_implicit_seed(self, stacklevel):
+        '''
+        Warns (at most once per loaded game) when an episode is played without
+        an explicit seeding choice for a game whose walkthrough seed would
+        have been silently applied prior to Jericho 4.0. Called at the start
+        of the first episode interaction — reset() or, since stepping is
+        possible without calling reset(), the first step() — so that correct
+        usage such as `FrotzEnv(rom)` followed by
+        `reset(use_walkthrough_seed=True)` is never flagged.
+        '''
+        if not self._episode_seed_implicit or self._warned_implicit_seed:
+            return
+        if self.walkthrough_seed is None:
+            return
+        # Mark as warned *before* warning: under warnings.simplefilter("error")
+        # the user gets a single exception, not one per reset()/step() forever.
+        self._warned_implicit_seed = True
+        msg = ("Since Jericho 4.0, the walkthrough seed ({}) of game '{}' is no longer used"
+               " by default, i.e. this episode is stochastic (randomly drawn seed: {})."
+               " Call reset(use_walkthrough_seed=True) to reproduce the walkthrough,"
+               " or make stochasticity explicit (e.g. FrotzEnv(rom, seed=-1) or env.seed(-1))"
+               " to silence this warning.").format(self.walkthrough_seed, self.story_file.decode(),
+                                                   self._episode_seed)
+        warnings.warn(msg, ImplicitRandomSeedWarning, stacklevel=stacklevel)
+
     def seed(self, seed=None):
         '''
         Changes seed used for the emulator's random number generator.
 
         :param seed: Seed the random number generator used by the emulator.
-                     Default: use walkthrough's seed if it exists,
-                              otherwise use value of -1 which changes with time.
-        :returns: The value of the seed.
+                     Default: None (equivalent to -1), i.e. a fresh random seed is
+                     drawn for each episode, making episodes stochastic. The seed of the
+                     current episode is available as :attr:`jericho.FrotzEnv.episode_seed`.
+        :returns: The value of the seed (-1 stands for "draw one per episode").
 
         .. note:: :meth:`jericho.FrotzEnv.reset()` must be called before the seed takes effect.
 
+        .. note:: Since Jericho 4.0, calling this method without a seed no longer
+                  silently uses the game's walkthrough seed. Use
+                  :attr:`jericho.FrotzEnv.walkthrough_seed` or
+                  :meth:`jericho.FrotzEnv.reset` with `use_walkthrough_seed=True`
+                  to reproduce a walkthrough.
+
+        .. note:: Calling this method counts as an explicit seeding choice, even
+                  without an argument (i.e. deliberately requesting stochastic
+                  episodes), so subsequent episodes do not raise
+                  :class:`jericho.ImplicitRandomSeedWarning`.
+
+        '''
+        self._seed_is_explicit = True
+        self._seed = _resolve_seed(seed)
+        return self._seed
+
+    @property
+    def walkthrough_seed(self):
+        '''
+        Seed needed to reproduce this game's walkthrough, if it is known.
+
+        :returns: The walkthrough's seed, or `None` if the game has no known walkthrough seed.
+
+        :Example:
+
+        >>> import jericho
+        >>> env = jericho.FrotzEnv('zork1.z5')
+        >>> env.walkthrough_seed
+        12
+        >>> env.reset(use_walkthrough_seed=True)  # Applies the walkthrough seed to this episode only.
+
+        '''
+        return self.bindings.get('seed')
+
+    @property
+    def episode_seed(self):
+        '''
+        The seed the emulator was seeded with at the start of the current episode.
+
+        For a stochastic env this is the randomly drawn seed of the episode, so any
+        episode can be reproduced after the fact, e.g. with
+        `FrotzEnv(rom, seed=env.episode_seed)`.
+
+        :returns: The current episode's seed.
+
+        .. note:: Restoring a mid-episode state with
+                  :meth:`jericho.FrotzEnv.set_state` does not update this value:
+                  the restored state carries the RNG registers of the episode it
+                  was captured from, not a seed.
         '''
-        seed = seed or self.bindings.get('seed', -1)
-        self._seed = seed
-        return seed
+        return self._episode_seed
 
-    def reset(self):
+    def reset(self, use_walkthrough_seed=False):
         '''
         Resets the game.
 
         :param use_walkthrough_seed: Seed the emulator to reproduce the walkthrough.
+                                     Default: `False`, i.e. use the seed set with
+                                     :meth:`jericho.FrotzEnv.seed` (a randomly drawn
+                                     seed, unless one was explicitly provided).
+        :type use_walkthrough_seed: bool
         :returns: A tuple containing the initial observation,\
-        and a dictionary of info.
+        and a dictionary of info (including the `seed` used for this episode).
         :rtype: string, dictionary
+        :raises ValueError: If `use_walkthrough_seed=True` but no walkthrough seed
+                            is known for this game. Check
+                            `env.walkthrough_seed is None` to handle such games.
+
+        .. note:: Using `use_walkthrough_seed=True` makes the game deterministic.
+                  As described in the Jericho paper, this is a *handicap* that
+                  should be disclosed when reporting results.
+
+        .. note:: `use_walkthrough_seed=True` applies to this episode only: it does
+                  not modify the seed set with :meth:`jericho.FrotzEnv.seed`, so a
+                  subsequent plain `reset()` reverts to that seed. To make the
+                  walkthrough seed persistent, use `env.seed(env.walkthrough_seed)`.
 
         '''
+        if use_walkthrough_seed:
+            if self.walkthrough_seed is None:
+                raise ValueError(
+                    "No walkthrough seed is known for game '{}'. Check"
+                    " `env.walkthrough_seed is None` before requesting"
+                    " use_walkthrough_seed=True.".format(self.story_file.decode()))
+            seed = self._episode_seed = self.walkthrough_seed
+        else:
+            seed = self._next_episode_seed()
+
+        self._episode_seed_implicit = not (self._seed_is_explicit or use_walkthrough_seed)
+        self._maybe_warn_implicit_seed(stacklevel=3)
+
         self.close()
         rom, _, _ = self._cache[self.story_file.decode()]
-        obs_ini = self.frotz_lib.setup(self.story_file, self._seed, rom, len(rom)).decode('cp1252')
+        obs_ini = self.frotz_lib.setup(self.story_file, seed, rom, len(rom)).decode('cp1252')
         score = self.frotz_lib.get_score()
-        return obs_ini, {'moves':self.get_moves(), 'score':score}
+        return obs_ini, {'moves':self.get_moves(), 'score':score, 'seed':seed}
 
     def step(self, action):
         '''
@@ -476,6 +635,9 @@ def step(self, action):
         Note:
         - The action is converted to bytes and truncated to 198 characters.
         '''
+        # The env is playable without calling reset() first, so the implicit-seed
+        # warning must also cover episodes that begin with a step().
+        self._maybe_warn_implicit_seed(stacklevel=3)
         action_bytes = action.encode('utf-8')
         if len(action_bytes) > INPUT_BUFFER_SIZE:
             action_bytes = action_bytes[:INPUT_BUFFER_SIZE]
@@ -579,7 +741,7 @@ def set_state(self, state):
         '''
         Sets the game's internal state.
 
-        :param state: Tuple of (ram, stack, pc, sp, fp, frame_count, rng) as\
+        :param state: Tuple of (ram, stack, pc, sp, fp, frame_count, opcode, rng, narrative) as\
         obtained by :meth:`jericho.FrotzEnv.get_state`.
         :type state: tuple
 
@@ -607,7 +769,7 @@ def get_state(self):
         Returns the internal game state. This state can be subsequently restored
         using :meth:`jericho.FrotzEnv.set_state`.
 
-        :returns: Tuple of (ram, stack, pc, sp, fp, frame_count, rng).
+        :returns: Tuple of (ram, stack, pc, sp, fp, frame_count, opcode, rng, narrative).
 
         >>> from jericho import *
         >>> env = FrotzEnv(rom_path)
@@ -636,9 +798,24 @@ def get_max_score(self):
         return self.frotz_lib.get_max_score()
 
     def copy(self):
-        ''' Forks this FrotzEnv instance. '''
+        ''' Forks this FrotzEnv instance.
+
+        The copy replays the current game faithfully (the emulator's RNG
+        registers are part of the copied state), but like the original, a
+        subsequent :meth:`jericho.FrotzEnv.reset` uses the seed set with
+        :meth:`jericho.FrotzEnv.seed` — not the seed of the episode being
+        copied, if that episode was started with `reset(use_walkthrough_seed=True)`.
+        '''
         state = self.get_state()
         env = FrotzEnv(self.story_file.decode(), seed=self._seed)
+        # Passing seed= above would make the copy count as explicitly seeded;
+        # carry over the original's bookkeeping instead.
+        env._seed_is_explicit = self._seed_is_explicit
+        env._warned_implicit_seed = self._warned_implicit_seed
+        env._episode_seed_implicit = self._episode_seed_implicit
+        # set_state() below makes the fork replay the original's episode, so
+        # it must report the original's episode seed, not the one load() drew.
+        env._episode_seed = self._episode_seed
         env.set_state(state)
         return env
 

jericho/version.py

@@ -1 +1 @@
-__version__ = '3.3.1'
+__version__ = '4.0.0'