updated deprecated parameter specs

Author: zhaozewang

README.md

@@ -11,7 +11,7 @@ permalink: /
 </div>
 
 <p style="display: flex; align-items: center;">
-  <img src="{{ '/assets/images/icons/github.png' | relative love }}" width="20" alt="GitHub">
+  <img src="{{ '/assets/images/icons/github.png' | relative_url }}" width="20" alt="GitHub">
   <a href="https://github.com/NN4Neurosim/nn4n" style="margin-left: 5px; color: #333; text-decoration: underline">GitHub Repository</a>
 </p>
 

_collection_1/quickstart.md

@@ -7,19 +7,11 @@ layout: post
 order: 2
 ---
 
-## Initialize a Continuous-Time RNN
-```python
-import torch
-from nn4n.model import CTRNN
-
-rnn = CTRNN(input_dim=1, hidden_size=10, output_dim=1)
-optimizer = torch.optim.Adam(rnn.parameters(), lr=0.001)
-```
-
-
 ## Define a Task
 The input/output signals to train the RNN can be any time series data. Let $X$ be the input signal and $Y$ be the output signal. $X$ should of shape `(n_timesteps, batch_size, input_dim)` and $Y$ should be of shape `(n_timesteps, batch_size, output_dim)`. These signals could be representations of many cognitive tasks, such as working memory, decision making, or motor control, etc. Here we use a simple sin wave as an example.
+
 ```python
+import torch
 import numpy as np
 import matplotlib.pyplot as plt
 
@@ -38,6 +30,14 @@ plt.show()
   <img src="{{ '/assets/images/results/sin_wave.png' | relative_url }}" width="400" alt="Sin Wave">
 </p>
 
+## Initialize a Continuous-Time RNN
+```python
+from nn4n.model import CTRNN
+
+rnn = CTRNN(input_dim=1, hidden_size=10, output_dim=1)
+optimizer = torch.optim.Adam(rnn.parameters(), lr=0.001)
+```
+
 ## Train the RNN
 ```python
 losses = []

_collection_2/layer_methods.md

@@ -0,0 +1,26 @@
+---
+title: Layer Methods
+author: Zhaoze Wang
+date: 2024-06-16
+category: docs
+layout: post
+order: 4
+---
+
+# layer.set_weight( )
+
+##### Description
+Set the weight of the layer. Only applicable to the hidden_layer and linear_layer.
+
+##### Parameters
+> `weight`: (torch.Tensor), required. The weight tensor to be set. The shape of the weight tensor should be `(output_dim, input_dim)`.
+
+##### Usage
+```python
+import torch
+from nn4n.model import CTRNN
+
+ctrnn = CTRNN()
+weight = torch.rand(100, 1)
+ctrnn.layers[0].set_weight(weight)
+```

_collection_2/methods.md

@@ -206,3 +206,30 @@ for _ in range(100):
     loss.backward()
     optimizer.step()
 ```
+
+
+# CTRNN.layers
+
+##### Description
+Get a list of the network layers.
+
+##### Returns
+> `layers`: (list) A list of the network layers.
+
+##### Usage
+```python
+from nn4n.model import CTRNN
+
+ctrnn = CTRNN()
+layers = ctrnn.layers
+
+for layer in layers:
+    print(layer)
+```
+
+##### Output
+```
+LinearLayer()
+HiddenLayer()
+LinearLayer()
+```

_collection_2/parameters.md

@@ -12,16 +12,12 @@ These parameters primarily determine the structure of the network. It is recomme
 
 <div class="table-wrapper" markdown="block">
 
-|Parameter|Default|Type|Description|
+| Parameter | Default | Type | Description |
 |:-:|:-:|:-:|:-:|
-| input_dim | 1 | `int` | Input dimension |
-| output_dim | 1 | `int` | Output dimension |
-| hidden_size | 100 | `int` | Number of hidden nodes |
-| scaling | 1.0 | `float` | Scaling factor for the hidden weights, it will scale the hidden weight by $\frac{scaling}{\sqrt{N\_{hid}}}$. Won't be affected when the HiddenLayer distribution is `uniform`. |
-| self_connections | False | `boolean` | Whether a neuron can connect to itself |
-| activation | `relu` | `string` | Activation function, could be `relu`/`tanh` / `sigmoid`/`retanh` |
-| layer_distributions | `uniform` | `string`/`list` | Layer distributions. Either `string` or a `list` of three elements. The `string` or `list` element must be either `uniform`, `normal`, or `zero`. If the given value is a `string`, it will be broadcasted to all layers. If the provided value is a `list`, its length must match the number of layers in the network and contains only valid distribution values. |
-| layer_biases | `True` | `boolean` or `list`  | Whether to use bias in each layer. Either a `boolean` or a `list` of three `boolean`s. If the given value is a list, its length must match the number of layers in the network and contains only `boolean` values. |
+| dims | [1, 100, 1] | `list` | Dimensions of the network |
+| activation | 'relu' | `string` | Activation function, could be 'relu', 'tanh', 'sigmoid', or 'retanh' |
+| biases | `None` | `None`, `string`, or `list` | Use bias or not for each layer. A single value is broadcasted to a list of three values, which can be `None` (not using bias); 'zero' or 0 (bias initialized to 0 but could change during training); 'normal' (normal distribution), or ;'uniform' (bias initialized from a uniform distribution). If a list of three values is passed, each can be as described or a numpy array/torch tensor specifying the bias. |
+| weights | 'uniform' | `string` or `list` | Distribution of weights for each layer. A single string is broadcasted to a list of three strings. Possible values: 'normal' (weights initialized from a normal distribution), 'uniform' (weights initialized from a uniform distribution). If a list of three values is passed, each can be as described or a numpy array/torch tensor specifying the weights. |
 
 </div>
 
@@ -41,16 +37,17 @@ These parameters primarily determine the training process of the network. The `t
 
 </div>
 
-# Constraint Parameters
-These parameters primarily determine the constraints of the network. By default, the network is initialized using the most lenient constraints, i.e., no constraints being enforced.
+# Mask Parameters
+When modeling the brain with neural networks, both the connections between neurons (synapses) and the neuron's non-linear activation functions are crucial components. Synapses, in particular, provide numerous degrees of freedom within the network. The connectivity matrix, for example, determines the network's structure, while various properties of synapses—such as their plasticity, whether they are excitatory or inhibitory, their strength, and the potential for new synapses to form—add layers of complexity and control. Here, we use masks to manage these characteristics.
 
 <div class="table-wrapper" markdown="block">
 
-| Parameter | Default | Type | Description |	
-|:-:|:-:|:-:|:-:|
-| positivity_constraints | False | `boolean`/`list` | Whether to enforce Dale's law. Either a `boolean` or a `list` of three `boolean`s. If the given value is a list, from the first element to the last element, corresponds to the InputLayer, HiddenLayer, and ReadoutLayer, respectively. |
-| sparsity_constraints  | True | `boolean`/`list` | Whether a neuron can grow new connections. See [constraints and masks](#constraints-and-masks). If it's a list, it must have precisely three elements. Note: this must be checked even if your mask is sparse, otherwise the new connection will still be generated. |
-| layer_masks | `None` or `list` | `list` of `np.ndarray` | Layer masks if `sparsity_constraints/positivity_constraints is set to true. From the first to the last, the list elements correspond to the mask for Input-Hidden, Hidden-Hidden, and Hidden-Readout weights, respectively. Each mask must have the same dimension as the corresponding weight matrix. See [constraints and masks](#constraints-and-masks) for details. |
+| Parameter             | Default | Type                     | Description |
+|:-:                    |:-:      |:-:                       |:-:          |
+| sparsity_masks        | None    | `None` or `list`         | Use `sparsity_masks` or not. A single `None` will be broadcasted to a list of three `None`s. If a list of three values is passed, each value can be either `None` or a numpy array/torch tensor specifying the `sparsity_masks`. |
+| ei_masks              | None    | `None` or `list`         | Use `ei_masks` or not. A single `None` will be broadcasted to a list of three `None`s. If a list of three values is passed, each value can be either `None` or a numpy array/torch tensor specifying the `ei_masks`. |
+| plasticity_masks      | None    | `None` or `list`         | Use `plasticity_masks` or not. A single `None` will be broadcasted to a list of three `None`s. If a list of three values is passed, each value can be either `None` or a numpy array/torch tensor specifying the `plasticity_masks`. |
+| synapse_growth_masks  | None    | `None` or `list`         | Use `synapse_growth_masks` or not. A single `None` will be broadcasted to a list of three `None`s. If a list of three values is passed, each value can be either `None` or a numpy array/torch tensor that directly specifies the probability of growing a synapse at the selected location if there is no synapse. |
 
 </div>
 
@@ -104,11 +101,12 @@ Whether a neuron can connect to itself. This feature is enforced along with the
 
 | Method | Description |
 |:-:|:-:|
-| [`forward()`]({{ site.baseurl }}/rnn/methods/#ctrnnforward-) | Forward pass |
+| [`forward()`]({{ site.baseurl }}/rnn/methods/#ctrnnforward-) | Forward pass. |
 | [`save()`]({{ site.baseurl }}/rnn/methods/#ctrnnsave-) | Save the network to a given path. |
 | [`load()`]({{ site.baseurl }}/rnn/methods/#ctrnnload-) | Load the network from a given path. |
-| [`print_layers()`]({{ site.baseurl }}/rnn/methods/#ctrnnprint_layers-) | Print the network architecture and layer-by-layer specifications |
+| [`print_layers()`]({{ site.baseurl }}/rnn/methods/#ctrnnprint_layers-) | Print the network architecture and layer-by-layer specifications. |
 | [`train()`]({{ site.baseurl }}/rnn/methods/#ctrnntrain-) | Set the network to training mode, training will be performed and constraints will be enforced. Also, during training, the recurrent noises (preact_noise and postact_noise) won't be added. |
-| [`eval()`]({{ site.baseurl }}/rnn/methods/#ctrnneval-) | Set the network to evaluation mode, no training will be performed and no constraints will be enforced |
+| [`eval()`]({{ site.baseurl }}/rnn/methods/#ctrnneval-) | Set the network to evaluation mode, no training will be performed and no constraints will be enforced. |
+| [`layers`]({{ site.baseurl }}/rnn/methods/#ctrnnlayers) | Return a list of the network layers. |
 
 </div>

_collection_4/fr_constraints.md

@@ -13,27 +13,35 @@ order: 6
 
 Firing rate loss function. The Firing rate loss function is defined as:
 
-$$ \mathcal{L}_{fr} = \frac{\lambda_{fr}}{N_{batch} T N_{hid}} \sum_{b,t,n=0}^{N_{batch}, T, N_{hid}} fr_{btn}^2 $$
+$$ \mathcal{L}_{fr} = \frac{\lambda_{fr}}{B \times T \times N_{hid}} \sum_{b,t,i=0}^{B, T, N_{hid}} r_{b,t,i}^2 $$
 
-where $N_{batch}$ is the batch size, $T$ is the number of time steps, $N_{out}$ is the number of output neurons, $fr_{btn}$ is the firing rate of neuron $n$ at time $t$ in the $i$-th batch.
+- $ B $: number of batches.
+- $ T $: number of time steps.
+- $ N_{hid} $: number of hidden neurons.
+- $ r_{b,t,i} $: firing rate of the $i$-th neuron at time $t$ in the $b$-th batch.
 
 
 # Firing Rate Constraint (SD)
 **Lambda Key:** `lambda_fr_sd`
 
-Regularize the SD of the HiddenLayer firing rates such that all neurons will fire at approximately the same rate. The Firing rate SD loss function is defined as:
+Regularize the standard deviation of the firing rate.
 
-$$ \mathcal{L}_{fr\_sd} = \lambda_{fr\_sd} \sqrt{\frac{1}{N_{hid}} \sum_{n=0}^{N_{hid}} \left(\sum_{b,t=0}^{N_{batch}, T}\frac{fr_{bt}}{N_{batch} T} - \mu \right)^2} $$
+$$ \mathcal{L}_{fr\_sd} = \lambda_{fr\_sd} \sqrt{\frac{1}{N_{hid}} \sum_{n=0}^{N_{hid}} \left(\frac{1}{B \times T} \sum_{b,t=0}^{B,T} r_{b,t} - \mu \right)^2} $$
 
-$$ \mu = \frac{\lambda_{fr}}{N_{batch} T N_{hid}} \sum_{b,t,n=0}^{N_{batch}, T, N_{hid}} fr_{btn} $$
+$$ \mu = \frac{1}{B \times T \times N_{hid}} \sum_{b,t,i=0}^{B, T, N_{hid}} r_{b,t,i} $$
 
-where $N_{batch}$ is the batch size, $T$ is the number of time steps, $ y_{bt} $ is the predicted firing rate of the neuron at time $t$ in the $i$-th batch, and $f_{bt}$ is the ground truth firing rate of the neuron at time $t$ in the $i$-th batch.
+- $ B $: number of batches.
+- $ T $: number of time steps.
+- $ N_{hid} $: number of hidden neurons.
+- $ r_{b,t,i} $: firing rate of the $i$-th neuron at time $t$ in the $b$-th batch.
+- $ \mu $: mean firing rate.
 
 # Firing Rate Constraint (CV)
 **Lambda Key:** `lambda_fr_cv`
 
-Regularize the firing rate of a single neuron such that all neurons will fire at approximately the same rate. The Firing Rate Regularization loss function is defined as:
+Regularize the coefficient of variation of the firing rate.
 
 $$ \mathcal{L}_{fr\_cv} = \lambda_{fr\_cv} \frac{\sigma}{\mu} $$
 
-where $\sigma$ is the standard deviation of the firing rate of the neuron, and $\mu$ is the network mean firing rate.
+- $ \sigma $: standard deviation of the firing rate of the neuron.
+- $ \mu $: network mean firing rate.

_collection_4/readout_constraints.md

@@ -13,4 +13,6 @@ ReadoutLayer constraint. The ReadoutLayer sparsity loss function is defined as:
 
 $$ \mathcal{L}_{out} = \frac{\lambda_{out}}{N_{out} N_{hid}} ||W_{out}||_F^2 $$
 
-- $N_{out}$: number of output neurons.
+- $ N_{out} $: number of readout neurons.
+- $ N_{hid} $: number of hidden neurons.
+- $ \textbf{W}_{out} $: readout layer weight matrix.

_collection_5/base_struct.md

@@ -34,7 +34,7 @@ Methods that are shared by all structures.
 | [`get_readout_idx()`]({{ site.baseurl }}/mask/methods/#maskget_readout_idx-)  | Get indices of neurons that readout from.           |
 | [`get_non_input_idx()`]({{ site.baseurl }}/mask/methods/#maskget_non_input_idx-) | Get indices of neurons that don't receive input. |
 | [`visualize()`]({{ site.baseurl }}/mask/methods/#maskvisualize-)              | Visualize the generated masks.                      |
-| [`masks()`]({{ site.baseurl }}/mask/methods/#maskmasks-)                      | Return a list of np.ndarray masks. It will be of length 3, where the first element is the input mask, the second element is the hidden mask, and the third element is the readout mask. For those structures that do not have specification for a certain mask, it will be an all-one matrix. |
+| [`get_masks()`]({{ site.baseurl }}/mask/methods/#maskget_masks-)                      | Return a list of np.ndarray masks. It will be of length 3, where the first element is the input mask, the second element is the hidden mask, and the third element is the readout mask. For those structures that do not have specification for a certain mask, it will be an all-one matrix. |
 | [`get_areas()`]({{ site.baseurl }}/mask/methods/#maskget_areas-)              | Get a list of areas names.                 | 
 | [`get_area_idx()`]({{ site.baseurl }}/mask/methods/#maskget_area_idx-)        | Get indices of neurons in a specific area. The parameter `area` could be either a string from the `get_areas()` or a index of the area. |
 

_collection_5/methods.md

@@ -50,7 +50,7 @@ Output:
 </p>
 
 
-# mask.masks( )
+# mask.get_masks( )
 ##### Description
 Return layer masks in a list.
 
@@ -68,7 +68,7 @@ params = {
     "output_dim": 2,
 }
 multiarea = MultiArea(**params)
-multiarea.masks()
+multiarea.get_masks()
 ```
 
 Output

_collection_5/multi_area.md

@@ -53,4 +53,60 @@ $W$ may not matter if your connectivity matrix is symmetric. But if it's not, yo
 <p align="center">
 <img src="{{ '/assets/images/basics/Multi_Area.png' | relative_url }}" width="600">
 </p>
-<br>
+
+
+## Example
+
+```python
+from nn4n.mask import MultiArea
+
+area_connectivities = np.array([
+    [1.0, 0.1, 0.0, 0.0],
+    [0.1, 1.0, 0.1, 0.0],
+    [0.0, 0.1, 1.0, 0.1],
+    [0.0, 0.0, 0.1, 1.0],
+])
+
+struct_params = {
+    "n_areas": 4,
+    "area_connectivities": area_connectivities,
+    "input_areas": [0],
+    "readout_areas": [2, 3],
+    "dims": [1, 100, 1],
+}
+
+network_struct = MultiArea(**struct_params)
+network_struct.plot_masks()
+```
+
+###### Output:
+<p>
+<img src="{{ '/assets/images/mask/results/multi_area_input_mask.png' | relative_url }}" width="480">
+</p>
+<p>
+<img src="{{ '/assets/images/mask/results/multi_area_hidden_mask.png' | relative_url }}" width="480">
+</p>
+<p>
+<img src="{{ '/assets/images/mask/results/multi_area_output_mask.png' | relative_url }}" width="480">
+</p>
+
+##### Use It as a Sparsity Mask
+```python
+from nn4n.model import CTRNN
+
+rnn = CTRNN(sparsity_masks=network_struct.get_masks())
+layer = rnn.layers[1]
+optimizer = torch.optim.Adam(rnn.parameters(), lr=0.001)
+rnn.plot_layers()
+```
+
+###### Output:
+<p>
+<img src="{{ '/assets/images/mask/results/multi_area_input_weight.png' | relative_url }}" width="500">
+</p>
+<p>
+<img src="{{ '/assets/images/mask/results/multi_area_hidden_weight.png' | relative_url }}" width="500">
+</p>
+<p>
+<img src="{{ '/assets/images/mask/results/multi_area_output_weight.png' | relative_url }}" width="500">
+</p>

_collection_6/v1.1.0.md

@@ -16,10 +16,11 @@ order: 2
 
 #### Todos:
 - [ ] Add custom constraint wrapper.
-- [ ] Change `visualize()` in the `nn4n.structure` to `plot_masks()`.
-- [ ] Change `masks()` in the `nn4n.structure` to `get_masks()`.
-- [ ] Change the loss to functional forms. (e.g. function(model) -> loss)
+- [x] Change `visualize()` in the `nn4n.structure` to `plot_masks()`.
+- [x] Change `masks()` in the `nn4n.structure` to `get_masks()`.
+- [ ] Change the loss to functional forms. (e.g. function(model) -> loss). It might be better to also have a function that gets the specific model attributes (i.e., an interface cuz the model might change).
 - [ ] Rename `criterion` to `constraint`.
 - [ ] Change quickstart example to a more complex example.
-- [ ] Change `BaseStruct` to `BaseMask`.
-- [ ] Change top icons to actual links
+- [x] Change `BaseStruct` to `BaseMask`.
+- [ ] Change top icons to actual links
+- [ ] The EIRNN example need to be change to the actual example.

assets/dadras_attractor.png

@@ -159,9 +159,11 @@
 
 .book.font-family-0 {
     font-family: Georgia, serif;
+    font-feature-settings: "calt" 0, "liga" 0, "dlig" 0, "kern" 0;
 }
 .book.font-family-1 {
     font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
+    font-feature-settings: "calt" 0, "liga" 0, "dlig" 0, "kern" 0;
 }
 
 /* Render wide tables */

assets/gitbook/custom.css

@@ -8,37 +8,10 @@ require(['gitbook', 'jquery'], function(gitbook, $) {
     var fontState;
 
     // Default themes
-    var THEMES = [
-        {
-            config: 'white',
-            text: 'White',
-            id: 0
-        },
-        {
-            config: 'sepia',
-            text: 'Sepia',
-            id: 1
-        },
-        {
-            config: 'night',
-            text: 'Night',
-            id: 2
-        }
-    ];
+    var THEMES = [];
 
     // Default font families
-    var FAMILIES = [
-        {
-            config: 'serif',
-            text: 'Serif',
-            id: 0
-        },
-        {
-            config: 'sans',
-            text: 'Sans',
-            id: 1
-        }
-    ];
+    var FAMILIES = [];
 
     // Return configured themes
     function getThemes() {
@@ -157,7 +130,7 @@ require(['gitbook', 'jquery'], function(gitbook, $) {
 
         $book[0].className = $book[0].className.replace(/\bfont-\S+/g, '');
         $book.addClass('font-size-'+fontState.size);
-        $book.addClass('font-family-'+fontState.family);
+        // $book.addClass('font-family-'+fontState.family);
 
         if(fontState.theme !== 0) {
             $book[0].className = $book[0].className.replace(/\bcolor-theme-\S+/g, '');