Implementation of kernels on the edge space of graphs or simplicial complexes

[cookbook-ms]

This pull requests an implementation of kernels on the Edge space of a graph or a simplicial 2-complex.
I implemented the following:

1. the discrete space of edge space: written as a class in geometric_kernels/spaces/graph_edge.py, which allows backends based on numpy, pytorch and jax.
2. tutorials on this, as how graph kernel tutorial looks like. This includes:

• numpy backend:

an example of the edge space of a graph, written in jupyter notebook in notebooks/EdgeSpaceGraph.ipynb
an example of the edge space of a simplicial 2-complex, written in notebooks/EdgeSpaceSimplicialComplex

• jax and pytorch backends:

an example of the edge space of a graph, written in notebooks/backends/Jax_EdgeSpaceGraph.ipynb and notebooks/backends/PyTorch_EdgeSpaceGraph.ipynb
an example of the edge space of a simplicial 2-complex, written in notebooks/backends/Jax_EdgeSpaceSimplicialComplex and notebooks/backends/PyTorch_EdgeSpaceSimplicialComplex

3. testing on the space of GraphEdge: written in "tests/spaces/test_graph_edge.py"

Note that:

• The above 8 files are the ones I wrote. All the modifications elsewhere are only about that I added few lines to look for the correct path of the geometric_kernels module (without installing it, since I need to test my own GraphEdge space).
• The current implementation in fact does not contain Hodge-compositional kernels, instead, they are using the one set of kernel parameters on the whole eigenspace. To implement the Hodge-compositional kernels, we can define the three sets of eigenspaces individually, then allowing different sets of parameters.

[stoprightthere]

Hi @cookbook-ms this looks cool! I've only had a brief skim over the code and left a couple of minor comments. Didn't check the maths though.

[cookbook-ms]

Sorry for the late updates.

In this update, we have the following changes:

1. the space GraphEdge takes a graph and optional triangle list as inputs and builds a simplicial 2-complex. If no triangle list is provided, we look for all the 2-cliques. Of course, one can choose only using graph instead of simplicial 2-complex.
2. the space GraphEdge gives an eigenbasis together with their Hodge type specified, so to enable Hodge-compositional kernel
3. a Hodge-compositional kernel is enabled in a separate class MaternKarhunenLoeveKernel_HodgeCompositionEdge defined in geometric_kernels/kernels/karhunen_loeve.py which takes parameters for three subspaces as inputs.
4. minimal changes are made in geometric_kernels/kernels/matern_kernel.py, as well as in geometric_kernels/feature_maps/deterministic.py and geometric_kernels/sampling/samplers.py, to enable the option of using Hodge-compositional kernel.
5. tutorials in numpy, jax and pytorch are provided in notebooks/SimplicialComplex.ipynb, notebooks/backends/JAX_SimplicialComplex.ipynb, and notebooks/backends/PyTorch_SimplicialComplex.ipynb which give how to define Hodge-compositional edge kernels in a simplicial 2-complex, as well as Hodge feature maps.

Please have a look at your convenience. Thanks :)

[vabor112]

Thanks a lot, Maosheng! I plan to take a look in a week or two, after the deadline streak has passed.

[vabor112]

Hi @cookbook-ms,

Thank you so much for implementing the Hodge compositional kernel!

I remember in the past we discussed a somewhat different design, which can hopefully relief the confusion of edges being implicitly oriented and which could better fit into the existing structure of the library. Please tell me if you would be willing to implement (perhaps some parts of) this proposal. What you won't do, I'll get around to doing myself in some time.

I'm copying a proposal from one of my old emails below. Well, rather a somewhat more detailed version of it.

GraphEdge space
— inputs are pairs (i, j) where i and j are indices of nodes. Internally, we have canonical orientation of edges and triangles (i < j and i < j < k), but interface-wise a user provides oriented edges as inputs and doesn't know of the internal orientation;
— by default (if no triangles are provided explicitly), triangles are all possible triangles;
— random samples edges as pairs (i, j), always in the canonical orientation (i.e. i < j);
— eigenpairs are ordered as usual, so the space is fully compatible with MaternKarhunenLoeveKernel, however, we store a mask for div-free, curl-free and harmonic pairs, which can be used by HodgeCompositionalMaternKernel;
— goes together with the OrientedEigenfunctionsFromEigenvectors class.

OrientedEigenfunctionsFromEigenvectors eigenfunctions class
— inherits from EigenfunctionsFromEigenvectors, its __call__ calls the parent's __call__ with the right transform of X, X2 (so that they are converted from pairs (i, j) to the edge index in canonical orientation we use internally for the Laplacian).

HodgeCompositionalMaternKernel
— only works with the GraphEdge space;
— the kernel uses different hyperparameters for different groups of eigenfunctions, using the mask.

In addition to the above classes, add a documentation page for theory of Hodge-compositional kernels on graphs/simplicial complexes.

Feel free to ping me if you want to discuss this more.

[cookbook-ms]

Hi Slava, please check my reply per item.

Hi @cookbook-ms,

Thank you so much for implementing the Hodge compositional kernel!

I remember in the past we discussed a somewhat different design, which can hopefully relief the confusion of edges being implicitly oriented and which could better fit into the existing structure of the library. Please tell me if you would be willing to implement (perhaps some parts of) this proposal. What you won't do, I'll get around to doing myself in some time.

I'm copying a proposal from one of my old emails below. Well, rather a somewhat more detailed version of it.

GraphEdge space — inputs are pairs (i, j) where i and j are indices of nodes. Internally, we have canonical orientation of edges and triangles (i < j and i < j < k), but interface-wise a user provides oriented edges as inputs and doesn't know of the internal orientation; — by default (if no triangles are provided explicitly), triangles are all possible triangles; — random samples edges as pairs (i, j), always in the canonical orientation (i.e. i < j); — eigenpairs are ordered as usual, so the space is fully compatible with MaternKarhunenLoeveKernel, however, we store a mask for div-free, curl-free and harmonic pairs, which can be used by HodgeCompositionalMaternKernel; — goes together with the OrientedEigenfunctionsFromEigenvectors class.

• The space now takes a graph as the input where the orientations are not required, and the canonical orientations are given inside the class.
• An extra option of using SC or not. If True: the edge space is in an SC; otherwise, in a graph.
• The triangles are optional. If provided (and use SC), we use those to construct SCs; otherwise, we look for all possible triangles to construct SC.
• Eigenpairs are ordered as usual now. I added an extra label which indicates the Hodge subspace it belongs to, so that it can be used for 'MaternKarhunenLoeveKernel_HodgeCompositionEdge'.
• I didn't implement the 'random' sampling of edges as pairs of nodes. I'll fix this together with OrientedEigenfunctionsFromEigenvectors eigenfunctions class below.

OrientedEigenfunctionsFromEigenvectors eigenfunctions class — inherits from EigenfunctionsFromEigenvectors, its __call__ calls the parent's __call__ with the right transform of X, X2 (so that they are converted from pairs (i, j) to the edge index in canonical orientation we use internally for the Laplacian).

HodgeCompositionalMaternKernel — only works with the GraphEdge space; — the kernel uses different hyperparameters for different groups of eigenfunctions, using the mask.

'MaternKarhunenLoeveKernel_HodgeCompositionEdge' should be implemented as discussed already, which takes three sets of parameters as inputs.

In addition to the above classes, add a documentation page for theory of Hodge-compositional kernels on graphs/simplicial complexes.

I planned to do that after you pass the above.

Feel free to ping me if you want to discuss this more.

[vabor112]

I see. I want to finish #149 asap, so, if that's okay, I'll take another look after you do

I didn't implement the 'random' sampling of edges as pairs of nodes. I'll fix this together with OrientedEigenfunctionsFromEigenvectors eigenfunctions class below.

[cookbook-ms]

I see. I want to finish #149 asap, so, if that's okay, I'll take another look after you do

I didn't implement the 'random' sampling of edges as pairs of nodes. I'll fix this together with OrientedEigenfunctionsFromEigenvectors eigenfunctions class below.

Now it samples random edges returning both the edges as pairs of nodes, also their indices. But it is much easier to use indices of edges to obtain the eigenvectors at the corresponding locations, thus computing the kernels. For this, I included a method to obtain the indices of given input edges.

[vabor112]

Some random comments. Also, don't forget to run make lint and make test before pushing.

[aterenin]

I had a full read-over the code and ran the notebook. Largely LGTM, except there are a few places in GraphEdges.ipynb where I would suggest a small number of changes. This is mostly:

• A missing random seed in one of the calls.
• A few places where certain package design choices are pointed out to the user, for instance handling outputscale, but no link to additional documentation is provided. I suggest providing links so that a user who is only interested in graph edge functionality can easily find what they need.

Otherwise LGTM!

[aterenin]

This should have a random seed.

[vabor112]

This RandomState is only used to tell GraphEdges.from_adjacency which backend to use for internal computations. It's never actually used for random number generation, thus there's no need to set a seed. I changed the line to

type_reference = np.random.RandomState()  # defines the backend, never used for random number generation

Hopefully, this makes things more clear.

[aterenin]

Why is the exact same line used later with seeds inputted?

[vabor112]

Other RandomState-s are actually used for random number generation. Where this is the case, the variable is called key rather than type_reference, as in

key = np.random.RandomState(1234)

in contrast to

type_reference = np.random.RandomState()  # defines the backend, never used for random number generation

[aterenin]

Draw attention to difference from previous figure - that there is one less triangle.

[vabor112]

Done

[aterenin]

Add a link to somewhere which shows an example doing this - otherwise someone who is interested only in the graph part may not find it.

[vabor112]

Done.

[aterenin]

Visualizing where these edges are would help.

[vabor112]

Done.