Reference Bayesian open games [hedges-etal-bayesian-games]

Abstract

  • October 8, 2019
  • Joe Bolt, Jules Hedges, Philipp Zahn

This paper generalises the treatment of compositional game theory as introduced by the second and third authors with Ghani and Winschel, where games are modelled as morphisms of a symmetric monoidal category. From an economic modelling perspective, the existing notion of an open game is not expressive enough for many applications. This includes stochastic environments, stochastic choices by players, as well as incomplete information regarding the game being played. The current paper addresses these three issue all at once. To achieve this we make significant use of category theory, especially the "coend optics" of Riley. 

 @article
{hedges-etal-bayesian-games, title={Bayesian open games}, url={http://arxiv.org/abs/1910.03656}, journal={arXiv:1910.03656 [cs, math]}, author={Bolt, Joe and Hedges, Jules and Zahn, Philipp}, year={2019}, month={Oct},language={en} }

Backlinks

Markov Fibrations and Stochastic Lenses › Introduction [efr-X0ZV]

Let Kl(\Delta ) denote the Kleisli category of the discrete (countable) distribution monad on \mathsf {Set}. This is a simple setting for working with probability theory---sufficient for many applications. In order to study compositional Bayesian game theory (Reference [hedges-etal-bayesian-games], Reference [towards-cybercat]) one studies the category \mathsf {Optic}(Kl(\Delta )) of optics in Kl(\Delta ). These have the right level of expressivity to talk about players taking random actions, and where payoff depends stochastically on players' decisions.

In \mathsf {Optic}(Kl(\Delta )), {R \choose X} + {R \choose Y} \cong {R \choose X+Y}. Games with this codomain naturally describe the situation of a player who has a binary choice between X or Y. We call coproducts of this form the "good" coproducts---note that also {* \choose *} + {\emptyset \choose *} = {\emptyset \choose 2}, but this is considered somewhat pathological, since it relies on the nonexistence of any morphism {\emptyset \choose X} \to {A \choose Y} when A and X are nonempty.

In order to describe this structure on \mathsf {Optic}(Kl(\Delta )), it would be useful if it had all coproducts. Unfortunately this is not the case. \mathsf {Optic}(\mathsf {Set}) = \mathsf {Lens}(\mathsf {Set}) has a well-known extension with all coproducts, given by the fiberwise opposite of the fibration \mathsf {Fam}(\mathsf {Set}) \to \mathsf {Set} (this is Proposition [efr-VTPS] in the case \mathcal {C} = \mathsf {Set}). Extending this to stochastic maps would be the obvious way of constructing such a category of "dependent optics".

Consider a category Kl(\Delta )^\to defined as follows. Its objects are the objects of \mathsf {Set}^\to ---that is, they are indexed families of sets. A map in Kl(\Delta )^\to is a stochastic map in the base X \to Y \in Kl(\Delta ), and a stochastic map on the total spaces \bar {X} \to \bar {Y} which is compatible with it. Note that if \bar {X} \to X is surjective, the map on the base is fully determined by the map on the fibers, which must merely satisfy the condition that the distribution of the indexing point in Y depends only on the indexing point in X, not the specific point in the fiber \bar {X}_x.

We claim Kl(\Delta )^\to is a reasonable notion of "stochastic charts". Recall that by "chart" we mean something like "lenses where both maps go forward". If stochastic lenses are supposed to include optics as the full subcategory spanned by the "non-dependent" objects, then the charts should include "co-optics"---that is, maps between X' \otimes X \to X and Y' \otimes Y \to Y should be given by the coend \int ^M \operatorname {\mathrm {Hom}}(X, M \otimes Y) \times \operatorname {\mathrm {Hom}}(X' \otimes M, Y')

And in fact this is the case: Clearly there is a map from this coend to maps in Kl(\Delta )^\to . By taking M = X \otimes Y and conditioning on Y, we see this is surjective. Finally, by restricting to the support of the forwards part inside X \otimes Y, we obtain a representative for each element of the coend which is uniquely determined by X \to Y and X' \times X \to Y' (since the conditional is well-defined on the support).

Note: This relies both on the fact that Kl(\Delta ) has conditionals, and on the existence of supports. We've previously seen these defined in abstract Markov categories (Definition [efr-AB57])--supports in Kl(\Delta ) of a morphism p: A \to B are simply given by those b so that p(b | a) > 0. Note that the existence of both conditionals and supports is a very strong assumption---the only categories we are aware of with both properties are those whose probability distributions have a discrete character, like Kl(\Delta ) and \mathsf {FinStoch}. Neither will be essential to the theory, but both will play a role in certain theorems---we will see more of this later.

The goal of the theory of Markov fibrations is to give a notion of "fiberwise opposite" which can be applied to the codomain functor Kl(\Delta )^\to \to Kl(\Delta ) to give a reasonable notion of "stochastic lenses". In particular, we should recover the usual category of optics in the previous case.

It is clear that the codomain functor is not a (Grothendieck) fibration, since this would require Kl(\Delta ) to have pullbacks, which can only hold for a Cartesian Markov category. However, we can do some things. Namely, given a Cartesian (pullback) square in \mathsf {Set}

and a map {\bar {X} \choose X} \to {\bar {B} \choose B} where the base map X \to B is deterministic, for each deterministic factorizing map X \to A there is a unique lift \bar {X} \to \bar {A}. In other words, the pullback over \mathsf {Set} \hookrightarrow Kl(\Delta ) is a fibration---in fact, it is simply the family fibration \mathsf {Fam}(\mathsf {Kl}(\Delta )).

Furthermore, if \bar {X} \to \bar {B} is itself deterministic, there is such a unique lift even without assuming that the factorization X \to A is deterministic.

Moreover, we can factor any map in Kl(\Delta )^\to as such an induced map followed by a map over a deterministic base, as follows:

This gives us a hope that we can, in some way, control the category Kl(\Delta )^\to using the pullback over \mathsf {Set}, which is a fibration, and some information somehow given by these extra maps. Note also that the diagram above is equivalent to giving: a span X \leftarrow M \to Y and a section X \to M, which all lives in the base, and a map p^*\bar {X} \to p^{'*}\bar {Y} in the fiber over M. Thus it would seem to be very amenable to fiberwise dualization.

Analogous to our argument above that "co-optics" are equivalent to maps in Kl(\Delta )^\to , we can do the following:

Suppose given two tuples (M_0,p_0,p_0',s_0,\phi _0), (M_1,p_1,p_1',s_1,\phi _1) as above. Suppose there exists a map f: M_0 \to M_1 over X,Y, so that f s_0 = s_1. Then there is a canonical map p_0^*\bar {X} \to p_1^*\bar {X} over f, because pullbacks commute. If the triangle

moreover commutes, then these two triples represent the same map in Kl(\Delta )^\to

These equivalence relations correspond to "sliding" for deterministic morphisms M \to M'. Note that the condition here can be checked just on the fibration Kl(\Delta )^\to \times _{Kl(\Delta )} \mathsf {Set} \to \mathsf {Set}.

To obtain the full set of sliding equations, we will need to use stochastic maps M \to M', and thus leave that fibration behind. However, we are tantalizingly close to realizing Kl(\Delta )^\to as being presented by some sort of additional structure on the fibration \mathsf {Fam}(\mathsf {Kl}(\Delta )). (For a general monad T acting on C, the category \mathsf {Optic}_{\mathcal {C}}(Kl(T),Kl(T)), of effectful optics up to sliding of pure morphisms, was studied by Riley in Reference [riley-optics], section 4.9, and by Hedges in Reference [hedges-blog-optics-effect])

In this chapter we will indeed provide such a structure, and analyze it. In § [efr-2IMZ], we'll axiomatise the lifting property of Kl(\Delta )^\to discussed above into a property we call a Markov prefibration (Definition [efr-0019]). In § [efr-GO6R], we exhibit a free Markov prefibration associated to a fibration---its morphisms are precisely spans of the form seen above. Naturally, given a prefibration \mathcal {D} \to \mathcal {C}, its underlying fibration on \mathcal {C}_\mathrm {det} becomes an algebra for the monad of this adjunction. In § [efr-U4RA], we characterize the class of prefibrations which are presented by their underlying algebra in this way---these are the Markov fibrations (Definition [efr-HVUT]). Since the monad commutes with fiberwise opposites, this yields a notion of fiberwise opposite for Markov fibrations.

Following this, we review a few properties of the theory of Markov fibrations, including the existence of coproducts in the fibration (Proposition [efr-QAV2]), the stability of Markov fibrations under limits (§ [efr-HWAZ]), and induced monoidal structures (§ [efr-LTEL]). Combining these, we can prove:

Theorem [efr-OSH4]

There exists a (strong) symmetric monoidal functor \mathsf {Optic}(Kl(\Delta )) \to \mathsf {SLens}(Kl(\Delta )^\to ) which

  1. Is fully faithful.
  2. Preserves the good coproducts.
  3. So that the image has all coproducts, and every object of the image is a coproduct of objects in \mathsf {Optic}(Kl(\Delta ))
  4. And so that moreover the monoidal structure on the image is distributive

Markov Fibrations › Introduction [efr-HQ73]

In applying category theory to fields as diverse as game theory (Reference [hedges-etal-comp-gametheory],Reference [hedges-etal-bayesian-games], Reference [hedges-etal-graph-games]), machine learning (Reference [bruno-etal-categorical-learning-2021], Reference [bruno-thesis-fundamental-components]), dynamical systems (Reference [myers-cst], Reference [lynch-myers-rischel-staton-stoch-clocks]), and server design (Reference [videla-capucci-servers]), people have found it useful to study categories whose morphisms describe processes or functions that take place in two "stages", where the second is dependent on the first, but composes in the other direction---so-called lenses. In many of these domains, stochastic phenomena play a role. There is a useful generalization of lenses to categories of stochastic maps---the so-called optics of Riley, Reference [riley-optics]---but this does not accommodate another useful generalization, so called dependent lenses where not only the backwards process but the set it takes values in is indexed over the base. It has been a long-standing problem to develop a suitable common generalization, "dependent optics", of these two ideas. In this thesis, we solve this problem by developing a theory of Markov fibrations (§ [efr-O088]).

A Markov fibration is a weakening of the notion of (Grothendieck) fibration to include (subject to some assumptions) Markov categories of indexed families of objects (given by deterministic functions E \to X) and compatible stochastic maps (given by commutative squares). The main point of Markov fibrations is that they admit fiberwise opposites, which generalize the fiberwise opposites of ordinary categories. Just as the fiberwise opposite of the codomain fibration of a finitely complete category describes dependent lenses (see § [efr-ZCTD]), the fiberwise opposite of these codomain Markov fibrations give a good notion of stochastic lens (see Theorem [efr-K6NM]).

We will apply our theory chiefly to two problems. First, we will use them to generalize open games (Hedges, Reference [hedges-towards-compositional-thesis]) to a larger class of interfaces (namely, indexed families of sets) while at the same time considering possibly-stochastic maps. This will allow us to construct the so-called external choice operator on open games, which describes branching.

Secondly, we will generalize Myers' categorical dynamical systems theory to allow for stochastic maps in the base. When combined with another generalization of these systems, to general parametrized lenses, this gives a more natural way of modeling certain stochastic dynamical systems, such as those associated to the training dynamics of machine learning models, see eg Example [efr-IMZ3].

A brief introduction to lenses

If \mathcal {C} is a category with finite products, the category \mathsf {Lens}(\mathcal {C}) of lenses in \mathcal {C} has objects pairs \binom {A}{X} of objects in \mathcal {C}, and morphisms \binom {A}{X} \leftrightarrows \binom {B}{Y} given by pairs f: X \to Y, f^\#: B \times X \to A. (Note that it is of course the morphisms, not the objects, that are called lenses). One way to generalize this is to ask for \mathcal {C} to have pullbacks, and consider an object given by a more general map A \to X, and let a map be f: X \to Y, f^\#: B \times _Y X \to A (so that the triangle over X commutes). (Note that this recovers the "simple lenses" when the objects are of the form A \times X \to X). Under the interpretation of a map A \to X as a family of objects A_x indexed over the elements of X, we see this as a "dependent lens" (since A is a "dependent type"). There are various variations of this idea, which ultimately all fit the pattern of taking a Grothendieck fibration \mathcal {D} \to \mathcal {C} and forming the fiberwise opposite---for the dependent lenses as above, this is the codomain fibration \mathcal {C}^\to \to \mathcal {C}.

Meanwhile, another wide-ranging generalization of \mathsf {Lens}(\mathcal {C}) are the optics of Riley (Reference [riley-optics]). For \mathcal {C} a monoidal category, the objects of \mathsf {Optic}(\mathcal {C}) are again pairs \binom {A}{X} of objects, but now the morphisms are elements of the coend \int ^{M \in \mathcal {C}}\mathcal {C}(X, M \otimes Y) \times \mathcal {C}(M \otimes B, A). That is, to give an optic \binom {A}{X} \leftrightarrows \binom {B}{Y} is to give an object M \in \mathcal {C} and morphisms l: X \to M \otimes Y, r: M \otimes B \to A, up to the equivalence relation generated by, whenever s: N \to M, l: X \to N \otimes Y, r: M \otimes B \to A, identifying the two optics given by (M, (s \otimes 1_Y) \circ l, r) and (N, l, r \circ (s \otimes 1_B)). One can show that, in the case where \mathcal {C} is a Cartesian monoidal category, this set can be identified with the set of (simple) lenses.

A vector field on a smooth manifold X is simply a (smooth) section of the tangent bundle TX. A section is a bundle map from the trivial bundle X \to X. This gives the idea of considering a parametrized dynamical system as consisting of a map X \to A to some other smooth manifold, a bundle E \to A (the points of E are the parameters) and a bundle map E \times _A X \to TX. As we will see, such a bundle map is exactly a dependent lens in a category of bundles. Building on this idea, Myers (Reference [myers-cst]) described a highly abstract theory of open dynamical systems, given by lenses out of tangent bundles (where the notion of space, bundle and tangent bundle are generalized to any fibration).

It has been observed that, although a lens from the tangent bundle TS \leftrightarrows A may indeed be said to describe an open dynamical system with state space S, the same can be said for a lens of type TS \otimes A \leftrightarrows I---although a dynamical system of a different kind (essentially, the first type are the Moore machines, the second the Mealy machines---see the introduction to § [efr-000D] for more on this). It is natural to consider parametrized maps TS \otimes A \leftrightarrows B as a common generalization of these two concepts---and in fact, special cases of this idea, lenses parametrized by tangent bundles, have already been considered many times, for example in the semantics of gradient descent (see Reference [bruno-etal-categorical-learning-2021], Reference [towards-cybercat], Reference [backprop-as-functor]).

Since parametrized maps compose in an obvious way, we now have three different notions of morphisms between bundles---lenses, charts, and "generalized systems". These should form some sort of symmetric monoidal triple category, but the right axiomatisation of this concept is somewhat elusive.

Lenses, in their various incarnations, have been widely used in applied category theory. For example, Hedges and his collaborators have developed a compositional approach to game theory (Reference [hedges-string-games], Reference [hedges-etal-comp-gametheory], Reference [hedges-etal-institutions], Reference [hedges-etal-graph-games], others). Recent work by Hedges and Sakamoto bring this idea to reinforcement learning (Reference [sakamoto-reinforcement-lenses]). We briefly mentioned Myers' categorical systems theory above, and we will see much more of it later. As we discussed above, there is also a literature using lenses to study gradient descent in an abstract sense. This is without even mentioning their original role in the theory of functional programming (this is the origin of the somewhat confusing term lens), or their prehistory in Gödel's dialectica interpretation---see eg Reference [jules-lenses-blogpost] for a survey of this.

Dependent Optics

An open game, in the sense of Hedges, has as its interface two objects in the category of lenses (of sets) \binom {S}{X}, \binom {R}{Y} (that is, two pairs of sets). An open game with this interface consists of a set \Sigma of strategies equipped with a function \Sigma \to \mathsf {Lens}(\mathsf {Set})(\binom {S}{X},\binom {R}{Y}), and a subset Eq(x, k) \subset \Sigma of equilibrium strategies for each x \in X and k: Y \to R---note that such an x is exactly a lens \binom {*}{*} \leftrightarrows \binom {S}{X}, and such a k is exactly a lens \binom {R}{Y} \to \binom {*}{*}. This view of open games makes the composition rule much simpler to define, although we will not delve into the details here, see eg Reference [hedges-etal-comp-gametheory]. The idea is that the open game represents the strategies and preferences of a player or set of players---\sigma \in \Sigma are the possible strategies, x \in X is the information revealed to the player before they make their decision of which moves to make, the resulting y \in Y is the choice the player "sends" in response to x, and the r \in R is the "utility", the value they eventually learn, depending on their choice y, which they have some preferences over.

The above description gives a theory of deterministic games, but of course, it is completely essential for game theory to model both random decisions (mixed strategies) and decisions made under uncertainty. This motivates the replacement of lenses in this definition with optics in a category of probability kernels (the form of the equilibrium relation must be modified as well, see Reference [hedges-etal-bayesian-games], Reference [towards-cybercat])

In the abstract study of open games, it is desirable to define a "choice" operation \oplus on the objects (the pairs of sets) so that maps into \binom {R}{X} \oplus \binom {R'}{X'} represent players who are faced with some binary choice between X and X', after which play may proceed for a time in one of two branches. However, this is immediately problematic because the category of lenses do not have coproducts---inside the category of dependent lenses, however, we can form the coproduct simply as R \times X + R' \times X' \to X + X'---that is, we get a family which is R over X, and R' over X'. This indeed does the job, but for game theory the extension to stochastic maps---optics---is absolutely essential. This motivates the search for a theory of dependent optics, a common generalization of optics and dependent lenses.

There have been a number of attempts at this---see eg Reference [fibre-optics-2021], Reference [hedges-braithwaite-dep], Reference [milewski-polylens], Reference [vertechi-dep-optics]. While these efforts have to some extent succeeded in describing a theory that is general enough to contain the desired examples, it is generally very ad hoc. The construction of optics as a special case of Vertechi's dependent optics (Reference [vertechi-dep-optics]) gives them as fibre optics with the indexing bicategory being (the delooping of) a monoidal category \mathcal {M}, but embeds lenses using a bicategory of spans. Thus simple lenses admit two distinct encodings in this theory. Moreover, Vertechi's example of dependent monoidal optics fails to describe a supercategory of \mathsf {Optic}(\mathcal {C}) when applied to Markov categories, and are thus not suitable for, for example, game theory.

Categorical Cybernetics

The observation that both open games and machine learning systems can be fruitfully described using lenses, naturally led to the idea that this could be used to develop a more thorough analogy between the two. After all, a machine learning system is also a player in a kind of game (with the objective of minimizing loss). Hedges coined the phrase categorical cybernetics to describe this locus of ideas (Reference [towards-cybercat], also used in Reference [sakamoto-reinforcement-lenses]).

Myers' categorical systems theory also describes dynamical systems as lenses. However, where a cybernetic system in the sense of Reference [towards-cybercat] is a parametrized lens \Sigma \otimes {\bar {X} \choose X} \leftrightarrows {\bar {Y} \choose Y}, and thus has two different inputs - x \in X, which is the information on which basis they are allowed to choose their decision y \in Y, and y^\sharp \in \bar {Y}, which is their "utility", the information they are allowed to care about.

It is straightforward to describe the machine learning part of the categorical cybernetics analogy in terms of Myers' theory---a "learner" is simply a lens of the above form with \Sigma replace with a tangent bundle TS. This idea (described in these terms, although not with the analogy to machine learning) has already been described by Reference [energy-driven-systems]. However, for game theory, it is necessary that the output of a given system can be stochastically chosen (a player must be able to randomize their strategy). At the same time, for a machine learning system, the x \in X is usually some sort of sample from a training distribution---ie, it is random. Hence to really describe the training dynamics of machine learning systems using this theory, we are again naturally drawn to consider systems theories (that is, fibrations) which have stochastic maps in the base, not merely the fiber.

Overview and contributions

We begin the thesis in § [efr-POD2] with a review of some preliminary material. This chapter can largely be skipped for readers already familiar with the material (Markov categories, double categories, lenses and optics, fibrations, and categorical dynamical systems theory). The chapter on Markov categories introduces a few novel auxiliary notions, but these can be referred back to as necessary (they are primarily regularity conditions which hold in most Markov categories of interest).

In § [efr-O088], we give the main technical contribution of the thesis by developing a theory of Markov fibrations. Their fiberwise opposites generalize both the fiberwise opposites of codomain fibrations (dependent lenses in a classical sense) and optics in Markov categories. Theorem [efr-K6NM] provides the statement of the latter (the former is straightforward). We also give a description of monoidal structures on Markov fibrations (which pass to monoidal structures on the resulting categories of optics).

In § [efr-000D], we give a construction of the double category of parametrized morphisms for a category action (or actegory)---in fact, we do this internally to any 2-category. This has the advantage of allowing us to construct more highly-structured versions of this double category by carrying out the construction internally to higher-structured actions. (Bicategorical versions of \mathsf {Para} are a relatively old idea, and the double categorical version is a straightforward extension which has existed for some time as folklore, but the fully-internal construction here is novel).

In § [efr-GFG0], we apply Markov fibrations to compositional game theory, and solve a longstanding problem by giving a category of stochastic open games with a general "external choice" operator. This construction is largely abstract over the particular category, and so can be applied to different Markov fibrations to describe games defined with different notions of stochasticity. We also review some previous work with Capucci, Hedges, and Gavranovic on abstract constructions of categories of open games (Reference [towards-cybercat]).

In § [efr-ZRUZ], we describe how to extend Myers' categorical systems theory to Markov fibrations, to give theories of systems which may have stochastic maps not merely in the fiber (as was already described by Myers in Reference [myers-cst]) but also in the base. We also leverage our construction from § [efr-000D] to give a description of a symmetric monoidal triple category whose three types of morphisms, are lenses, charts, and what we call bisystems, that is morphisms TS \otimes A \leftrightarrows B - these are open dynamical systems which have two directions of interaction with the environment. We also describe a particular systems theory of smooth manifolds and smooth stochastic maps between them.

Acknowledgements [efr-0008]

I would like to extend a great thanks to the Mathematically Structured Programming group at the University of Strathclyde, my fellow PhD students in particular, for making my time there such a great one. A good chunk of this work was completed while I worked at the Compositional Systems Laboratory at the Tallinn University of Technology---I owe everyone there a great thanks as well.

For conversations which contributed to this work, and to my mathematical development in general, I would like to thank Dylan Braithwaite, Matteo Capucci, Elena di Lavore, Tobias Fritz, Davidad, Tomáš Gonda, Bruno Gavranović, Diana Kessler, Owen Lynch, Jade Master, David Jaz Myers, Riu Nakamura, Chad Nester, Evan Patterson, Paolo Perrone, Mario Roman, Brandon Shapiro, Toby Smithe, Pawel Sobocinski, David Spivak, and Andre Videla. A special thanks to Dylan, Matteo, and Bruno, for serving their time with me in the dependent optics mines.

To Neil Ghani, Radu Mardare, and Jules Hedges, a massive thanks for serving as my advisors during the somewhat tumultuous process that was the production of this thesis.

Finally, I thank my wonderful family. Now you can stop asking me about it.