![logo]()
Feeds information from the yellow input cells through layers of blue recurrent cells to the red output cells.
FFNNs with a time twist: they are not stateless; they have connections between passes, connections through time.
Neurons are fed information not just from the previous layer but also from themselves from the previous pass, storing their previous weight.
This means that the order in which you feed the input and train the network matters: feeding it "milk" and then "cookies" may yield different results compared to feeding it "cookies" and then "milk".
One big problem with this network is the vanishing (or exploding) gradient problem where, depending on the activation functions used, information rapidly gets lost over time, just like very deep FFNNs lose information in depth.
Intuitively this wouldn't be much of a problem because these are just weights and not neuron states, but the weights through time is actually where the information from the past is stored; if the weight reaches a value of 0 or 10^N, the previous state won't be very informative.
In principle be used in many fields where data doesn't have a timeline (i.e. like sound or video) but can be represented as a sequence (A picture or string fed in one pixel or character at a time).
Here, the time (state) dependent weights are used for what came before in the sequence, not actually from what happened x seconds before. Therefore, these networks are generally a good choice for advancing or completing information, such as autocompletion. Feeds information from the yellow input cells through layers of hidden cells to the red matching input output cells.
In a way, the opposite of their family. Instead of teaching a network to represent a bunch of information in less "space" or nodes, we try to encode information in more space.
So instead of the network converging in the middle and then expanding back to the input size, we blow up the middle. These types of networks can be used to extract many small features from a dataset.
If one were to train this network the same way as an AE, you would in almost all cases end up with a pretty useless identity network (as in what comes in is what comes out, without any transformation or decomposition). To prevent this, instead of feeding back the input, we feed back the input plus a sparsity driver.
This sparsity driver can take the form of a threshold filter, where only a certain error is passed back and trained, the other error will be "irrelevant" for that pass and set to zero. In a way this resembles spiking neural networks, where not all neurons fire all the time (and points are scored for biological plausibility). Feeds information from the pink kernels and convolutions/pools, and again through layers of green hidden cells until the red output cell layer.
These are quite different from most other networks. They are primarily used for image processing but can also be used for other types of input such as as audio.
A typical use case is image classification; feeding the network images and the network classifies the data, e.g. it outputs "cat" if you give it a cat picture and "dog" when you give it a dog picture.
These tend to start with an input "scanner" which is not intended to parse all the training data at once. For example, to input an image of 200 x 200 pixels, you wouldn't want a layer with 40 000 nodes. Rather, you create a scanning input layer of say 20 x 20 which you feed the first 20 x 20 pixels of the image (usually starting in the upper left corner). Once you passed that input (and possibly use it for training) you feed it the next 20 x 20 pixels: you move the scanner one pixel to the right. Note that one wouldn't move the input 20 pixels (or whatever scanner width) over, you're not dissecting the image into blocks of 20 x 20, but rather you're crawling over it.
This input data is then fed through convolutional layers instead of normal layers, where not all nodes are connected to all nodes. Each node only concerns itself with close neighbouring cells (how close depends on the implementation, but usually not more than a few). These convolutional layers also tend to shrink as they become deeper, mostly by easily divisible factors of the input (so 20 would probably go to a layer of 10 followed by a layer of 5). Powers of two are very commonly used here, as they can be divided cleanly and completely by definition: 32, 16, 8, 4, 2, 1.
Besides these convolutional layers, they also often feature pooling layers. Pooling is a way to filter out details: a commonly found pooling technique is max pooling, where we take say 2 x 2 pixels and pass on the pixel with the most amount of red.
For audio, you basically feed the input audio waves and inch over the length of the clip, segment by segment. Real world implementations of these often glue an FFNN to the end to further process the data, which allows for highly non-linear abstractions. These networks are called DCNNs but the names and abbreviations between these two are often used interchangeably. Feeds information from the yellow input cells, through pink convolutions/pools, again through layers of kernels until the red output cell layer.
Also called inverse graphics networks (IGNs), are reversed convolutional neural networks.
Imagine feeding a network the word "cat" and training it to produce cat-like pictures, by comparing what it generates to real pictures of cats. DNNs can be combined with FFNNs just like regular CNNs, but this is about the point where the line is drawn with coming up with new abbreviations.
Note that in most applications one wouldn't actually feed text-like input to the network, more likely a binary classification input vector. Think <0, 1> being cat, <1, 0> being dog and <1, 1> being cat and dog. The pooling layers commonly found in CNNs are often replaced with similar inverse operations, mainly interpolation and extrapolation with biased assumptions (if a pooling layer uses max pooling, you can invent exclusively lower new data when reversing it). Feeds information from the yellow input cells, through a layer of pink kernels until pink convolutions/pools, converging to a layer of green hidden probabilistic cells, and then out again through layers of kernels and convolutions until the red output cell layer.
These have a somewhat misleading name, as they are actually VAEs but with CNNs and DNNs for the respective encoders and decoders.
These networks attempt to model "features" in the encoding as probabilities, so that it can learn to produce a picture with a cat and a dog together, having only ever seen one of the two in separate pictures. Similarly, you could feed it a picture of a cat with your neighbours' annoying dog on it, and ask it to remove the dog, without ever having done such an operation.
Demos have shown that these networks can also learn to model complex transformations on images, such as changing the source of light or the rotation of a 3D object. These networks tend to be trained with back-propagation. The yellow input cells feed their weights via biases into green hidden cells, and again to the red output cell. This framework of weights and biases becomes the 'solution' of a traditional NN.
This means by which a neural network determines neuron weights and biases, the framework by which it maps input and output values, and applies to data after training to 'predict' output values from unknown input values.
In the case of the upper neuron, the weights matrix is the input layer [1, 0.5] and the bias matrix relating it to that neuron is [-5.8, 2.1]. When these are multiplied together we get -2.9 and 2.1, which we sum again get the final parameter -0.8.
This taken into one of the hidden green cells, which activates here via the sigmoidal (S-curve) function to get the result 0.3, which becomes the weight of this function.
In the case of the lower neuron, the weights matrix is still [1, 0.5] (the input layer) but the bias matrix to this neuron is now [1.2, 0.2]. When these are multiplied together summed we get 1.3, which is put into the sigmoidal function to 'activate' the neuron with the 1.0 weight.
For the final red output cell, the same weight and bias process happens again, this time with the weights of the newly activated green hidden layer [0.3, 1.0] and biases [1.2, 0.2], which is solved and summed again to get 0.6.
This is how the input values of 1.0 and 0.5 are associated to the 0.6 output value.
In training, this NN will probably categorize values near {0.5, 1} to be near the 0.6 output. Every NN is different and will use it's weights and biases differently (even arbitrarily without training).
However, it is these weights and biases that describe a network, and is what is retained by the network after it is 'taught' via backpropogation (the transformation of these weights and biases to map inputs and ouputs via error minimization).
Therefore, the best NN is the one with the weights and biases trained via sample data that has lowest error mapping inputs and outputs, especially given actual population data. 
