Background:
Classical AI has a consistent problem with fragility
and difficulty of learning, which I trace in large
part to the use of discrete data spaces (in particular,
boolean and small integer), which among other things
prevent the application of hill-climbing algorithms.
I'm interested in understanding how one programs
using continuous data spaces and types.
Focus: State storage
At the moment, I'm particularly interested in how
one replaces conventional von Neuman memories which
use small integers as keys and values with memories
which use continuous functions as keys and values.
Inspiration:
Holography. Many of the suggested applications of
holography to computing in general and intelligence
in particular strike me as addle-pated, but this is
one case in which they seem to clearly fit the bill
with no mysticism involved.
(Leaves me with the odd feeling that I've inadvertantly wandered from artificial intelligence into the design and implementation of all-optical computers *blink*...)
It is easy and natural to store continuous functions using holography: A conceptually continuous wavefront is precisely what is being stored and reconstructed, using a conceptually continuous medium, even if it all reduces to discrete atoms and quantum mechanics in the end.
Further, it is easy and natural to store multiple datasets in the same medium using holography, by using multiple reference beams to store and recall.
Even better, it is perfectly natural to control recall from a holographic storage system using a continuous control function. For example, a continuous set of field values can be stored into a holographic-style data store using a continuously varying reference beam angle.
Current holographic memory proposals use discrete beam positions to store and retrieve a discrete number of datasets composed of a discrete number of bits, but this is clearly an artifact of the percieved current market, not something drawn from the inherent nature of the medium.
Thus, it is immediately clear that if you can classify your input data using a 1 or 2 dimensional continuous function, you can use this function to select the reference beam read/write angle, and can afterwards retrieve data from "similar" previous writes by searching around the read angle thus derived.
I'd need to think about it a bit (perhaps Jon has insights) but I'd suspect that one can encode the stored dataset using more parameters than just read/write beam angle, if one desired: Read/write wavelength is an obvious third parameter which can be used, allowing one to write values in a 3D classification space (in principle at least, if not perhaps as easily in practice). I suspect that a little thought would show that the query/recall space can be further increased in dimensionality without much conceptual effort if one wishes: Should it not be possible to store a dataset by writing using a reference beam following some path, rather than a fixed angle, and then recall by integrating results as the read beam follows a path, rather than a fixed angle? This would let one store datasets using 2D curves as the read/write keys, instead of just 2D points, and thus to recall values stored using "similar" curves, rather than values stored using "similar" points, no?
I'd presume that what can be done in time can equally easily be done in space, so a static image of a curve should also be a viable holographic read/write key, instead of a point tracing a curve over time, and in fact one should be able to use holography to build an memory which uses essentially arbitrary images/functions as both key and value, no? I.e., use an arbitrary image as the reference field, and another arbitrary image as the value field, and recall the value image by re-presenting the key image.
Further, holographic memory has a finite capacity, and in principle one can keep writing new images in more or less constantly, and have old images fade away exponentially, no? This is very characteristic of biological memory, very convenient for intelligence applications, and a property very hard to achieve in conventional von Neumann memories. No?
I'm not necessarily actually thinking of using literal optical holographic storage in practice (although it begins to sound tempting), since I'm a software hacker, not a hardware hacker (shall we join forces and get rich, Jon? :), but these systems -do- provide examples of system which possess the sorts of properties which we desire for programming in continuous domains, which I conjecture to be nearly essential for implementing intelligent systems.
The basic scheme of using convolution to store many continuous datasets in a single large shared store can also be applied on available von Neuman machines, of course, and in fact existing Bloom filters can be understood as a primitive form of this kind of storage. (Bloom filters use multiple independent hash functions defined over a common bit vector to probabilistically encode boolean information such as whether a given string is part of a given set of strings -- spelling dictionaries are the canonical application.)
So, any time you can produce a continuous function expressing your measure of "similarity" over some data space, you should be able to store elements from that dataspace (which can be as complex as a movie with soundtrack, of course), in a holographic-style memory, and later query that memory for any entries "similar" to a given one. No?
This is all pathetically primitive compared to the system you'd design if you thoroughly understood the problem and the implentation possibilities, I'm sure, but it seems to be taking big steps in the right direction from our current von Neuman rut...
(In particular, large complex data are being stored as single entities here, under a single key; A viable system should perhaps be more like a linklist, in which complex data such as a movie are broken into significant chunks -- "scenes", say -- and a given scene stored as the continuous equivalent of a doubly-linked list, with the keys for the next and previous scene stored along with the scene itself.
One can then query the memory for any scenes "similar" to a given scene, and if a scene is found, follow the pointers included in it to retrieve the test of the scenes in the movie.
This design would have the further advantage of allowing higher-quality readout: Since we are initially reading a scene by using a key "similar" but not identical to the key used to write that scene, the first retrieval is likely to be noisy and low-quality -- and quite possibly contaminated with other, less similar possibilities. (Does this at all remind you of trying to get something out of human memory when a false hit is blocking the actually desired datum, anyone?) But if the retrieval key used to write the scene is written with the scene, and the initial read suffices to reconstruct a better approximation of the key than the initial one, we can use that key to re-read with higher fidelity, obtain yet a better approximation to the true write key as a result, and exponentially (?) converge to a near-optimal readout of the stored scene.
Since it seems clear that a holographically stored datum can contain more data than the read/write key, I see no reason why the above sort of read convergence should not work. (Jon? Jerry?)
Okie, that's this afternoon's rant. Is it time to apply for a patent and get rich, yet? :) I'll bet the above properties will get patented and used by someone sooner or later just for mundane stuff like the example movie storage and playback, even if those aren't the point of the current discussion...
So, the continuous-computing analogue to standard von Neuman recursive function calls with parameter substitution remains the next item on the agenda...
(Afterthought, minutes later:) In the specific case of movie storage and retrieval, the sound track might make (part of?) a sensible read/write key. It has the desired characteristic of being fairly variable, and of containing much less information than the scene as a whole, and also has the advantage of being relatively easy to use as a key: People who have difficulty painting a face or scene and asking "Do you have any movies with scenes like that?" might still be able to say, "Do you have any movies with scenes where they say, 'Frankly, my dear, I don't -give- a damn'". ?)
This needn't preclude including other components in the key, or using other keys for silent scenes or such, of course... for example, a single still has much less information than a 5-second clip, and thus might be a perfectly good key. As always, indices separate from the main datum store are also practical and probably usually sensible...