We are finally set to analyze the impacts of noise on our particular model of recurrent computation - reservoir computing with echo state networks. In the previous post, we implemented a GPU simulator together with an extremely simple noise model by adding Gaussian noise to each element of the output signal from the reservoir. In principle, the dynamics of a system will have more complicated noise based on the details of the computation being done, but in our case using such a simple model will let us explore intuitively why we should expect a substantial degradation in performance in the first place. Based on our discussion in the first post, we have additionally considered all products of the output signals, and are assessing the performance of the reservoir by using it for the NARMA10 task.

In particular, we are going to analyze the impact of the Gaussian noise by considering a simple model of how the signals become corrupted. We naively expect an exponential number of product signals because the number of possible product signals from a set of signals is given by the cardinality of its power set - there are an exponential number of possible combinations from any set. We will group these product signals by the number of terms in each product, and compute the probability that the product signal is uncorrupted from the noise. Doing this, we will be able to reason about the expected number of useful signals, and establish that with noise the expected number of signals should scale polynomially, rather than exponentially.

Once we have established that the expected number of signals in this simple model is approximately a monomial in the system size, rather than exponential, we will plot how we expect the degree of the monomial to vary with noise. To think about the limits of this plot - when the noise becomes very large, the echo state network will be unable to learn, and hence their performance will be independent of the number of signals, giving a degree of zero. When the noise goes to zero, then we can expect the reservoir will be able to make use of the exponentially-many signals, and the approximation of a monomial dependence becomes invalid (the monomial degree becomes arbitrarily large). Let's give the code a look.

Acknowledgements

A special thanks to Alex Meiburg, André Melo and Eric Peterson for feedback on this post!