A major goal of deep learning is to build neural nets and use them to make predictions. Much of the field employs the mathematics of linear algebra in Euclidean space to encode data as vectors embedded into sufficiently high dimensional $\mathbf R^n$. Gradient descent becomes an indispensable tool, because "learning" in these contexts is defined to be 'gradient descent of embedded vector data over a parameter manifold.' And in that sense, learning, in one's neural net, is precisely achieved by successfully exercising this descent. More specifically, one must determine optimal step sizes that ensure the descent indeed converges to local/global minimas on the parameter manifold. A foundational method gives us the competence to realize this convergence stochastically, and it is called 'stochastic gradient descent' (SGD). Mainly it allows one to minimize a model's loss function by updating parameters using a single randomly chosen training sample instead of the entire dataset.
Although modern neural architectures exploit the well-understood underlying mathematical principles quite successfully, there are some serious problems with the current paradigm, namely model interpretability. Practitioners are accustomed to studying their neural nets in what's called 'parameter space.' Parameter space is high dimensional, and highly redundant. There are many parameters of the neural net that are copies of others, but which occupy different spatial locations in the parameter space, hence the redundancy. In the direction of making headway on the problem of interpretability, one would like to be able to trace a given 'computation' of a neural net, back to a specified region, layer, or strata, independent of statistical noise. That is, one would like to be privy to what is going on 'under the hood' of a given model, and not rely purely on heuristics, or guesswork.
A promising, and relatively recent approach to extracting model interpretability is to leverage the notion of a 'function space.' Roughly speaking, a function space is a quotient space of a parameter space. The classes of redundant parameters are identified, and the resulting quotient space admit degenerate behavior in the generic case. There are several closely related fields of study in this direction, namely, neuro-manifolds and neuro-varieties.
This degenerate behavior is important. In parameter space, redundant parameters are generically mutually exclusive or disconnected, whereas in function space they may form complicated intersection topologies. The understanding of such pathological regions and how they coexist within the broader 'regular part' of the function space is paramount.
Following this vein, let us assume our function space takes the form:
$$\mathcal I = \bigg(\bigsqcup_{i=1}^4 X_i\bigg)\big/\sim_{\Gamma}, \qquad X_i \cong S^2, \quad|\Gamma|\cong K_{2,2,2}.$$
The parameter space at our disposal, is simply a disjoint union of spheres, under the assumption that we ignore any 'extra' structure such as differentiable or analytic structure, and purely approach this at the topological level. Upon gluing these separate $X_i$ sheets together along the seam graph $\Gamma$, we obtain our function space, $\mathcal I$.
The next order of business is to tokenize $\mathcal I$. Consider organizing our function space into 'buckets' which we will call 'strata.' Evidently, there are at least three strata to meaningfully organize around. But first we will make the additional assumption that each $X_i$ has a pair of cone points. We still have topological spheres, but they are now geometrically generalized. The eight cone points will constitute input/output tokens. To commence every computation of this neural net, we feed some type of source data into one, some, or all of these cone points.
Upon feeding our source data of choice into the input nodes, we arrive at the next region of $\mathcal I$ to be tokenized. This layer is the 'regular part' of the function space. It is a smooth manifold when considered away from the cone points or the seam graph $\Gamma$. The regular part acts as the mediator between input data and $\Gamma$-data, which will be formalized next. The regular part acts like a bridge that allows input types to be converted into flows that eventually reach the seam graph 'junction.' The seam graph then receives this 2D signal and reads it off as a compressed 1D type along its edges. Thus, one should not think of $\Gamma$ as a lone Graph Neural Net (GNN), but as a GNN embedded into our function space as a singular skeleton, which computes based on the type readouts from the regular parts of the function space.
No comments:
Post a Comment