If you have JAX installed, here's how one gets the jacobian of a Linear layer with stax: One example is stax, which models Modules as an (init_fn, apply_fn) pair. There have been a number of NN module designs in JAX that work with a functional API.

We may want something like this if we're planning on having a way to make our NN modules more functional anyways. However, users use the NN API because it is handy: carrying around all of the parameters manually sometimes unwieldy, especially if you built a model out of some number of nested layers and have a large number of parameters to keep track ofĪ stretch goal that may be out of scope for this is 'An alternative, functional NN Module API in addition to our current NN API'. One alternative to a NN Module functional API is to tell PyTorch users to always use the nn.functional variant of a module if they wish to use the autograd.functional APIs (e.g., use F.conv2d instead of nn.Conv2d). We can certainly do this for most of them, though. I'm not sure that we can actually remove state from all of our nn.Modules. How does this work with modules that mutate internal state, like nn.BatchNorm?ĭeepmind/dm-haiku has some interesting ideas for the above two questions that we can look to for inspiration.

What about buffers that are registered on modules? Do we pull them out as well?.how should 'Params' look like? How does the user create a different set of Params or operate over them?.We've omitted some details of how this API should work: jacobian( partial( func, params = params), data) # Compute jacobian w.r.t to module inputs jacobian( partial( func, data), params) # Compute jacobian w.r.t to module parameters F.

Set model params nn torch