a dynamical systems approach to learning

investigating the role of fast vs. slow components in learning

2006-04-07T16:57:00.000+01:00

I've been working during the last days but haven't been able to report on any of it. Mainly because I have not made as much straightforward progress as I can easily explain.

I've been working on evolving learning on a continuum in the abstract scenario. I have managed to evolve circuits that can 'memorise' features on a continuum without the need to introduce any parameter-changing rules.

How do they memorise the features? The key to answering this question is in the interaction at very different time-scales between the 'neurons' of the circuit.

Currently I am performing experiments in this one-shot remembering features on a continuum where the delay can vary greatly (between 10 and 100 units of time).

From 20 evolutionary runs using 10 node CTRNNs I observed that there were, roughly, 2 possible types of evolved behaviors: discrete categorisers or generalisers. The discrete categorising circuits are unable to differentiate most of the features within the continuum and rather they make up 2 or 3 categories for them. Either small and large, or small, medium and large type of features. The ones that make 3 categories score better than the ones that only make 2 categories. The generalisers do much better and can distinguish between many more signals, without making any obvious categories a priori.

In terms of the mechanisms available, there's 10 nodes and they can act in very different time-scales. The fastest ones have a time-parameter of 1 while the slowest can be as slow as 140 units of time. This parameter describes the way in which state leaks out of this component. One thing is common among the categorisers, they do not make use of slow acting neurons, but only fast-acting ones. Whereas the generalisers make use of both.

For the purpose of determining the role of fast versus slow acting components in the circuit in this memorisation task, the best evolved agent's time parameters were studied and from it we defined what slow acting neuron ranges and what fast acting neuron ranges where needed to solve the task. Slow acting then corresponded to neurons with time parameters between (50 and 100) while fast acting nodes where defined to be between (1, 7).

To observe the role of this different components I am running evolutionary runs while the circuit is constrained to have different numbers of fast/slow neurons. I am using a 10 node CTRNN so there are 11 cases. In one extreme is the case when all neurons are restricted to the fast range, then there is the case when one is restricted to be slow and the rest restricted to be fast, and so on until all neurons are restricted to be slow. I am running 10 evolutionary runs using different seeds for each case. The intuition is that neither of the extremes do well, and that the optimum for the learning task lays somewhere in-between, possibly with more fast neurons than slow neurons. The resulting curve should be interesting.

On the other hand, I have not managed to evolve circuits that can continue to remember the feature after they have had to make a first decision. I am working on changes to the current evolutionary methodology that will lead to success here. One of the changes that I will try might be to add a feedback input (I have to explain why I think this is important but will do so later on). Also I must simply wait longer (more generations) as well as try larger circuits (currently the biggest has been a 10 node one).

On a completely different note, I have started to design the experiments in the embodied version of this memorisation task loosely inspired on behavioral plasticity observed in C. elegans. A similar form of learning to imprinting has been observed in these organisms. In particular, it has been observed that the animals that were cultivated normally with food at temperatures ranging from 15C to 25C migrate to the cultivation temperature on a temperature gradient and move isothermally at that temperature. By contrast, the animals migrate away from the temperature at which they were previously starved. They don't call it imprinting, but it is a very appropriate behaviorual paradigm to continue to study learning and memory in the embodied version. I will say some more on this soon.

results from noise-less experiments: success

2006-03-15T16:04:00.000Z

This post will describe results from 20 evolutionary runs using 5 and 10 node CTRNNs (10 runs each). Each evolutionary run uses a different seeds and stops after 5000 generations [roughly around 11 hours in the cluster]. The experimental set-up is as follows: [1] parents and test individuals are picked at random from a uniform distribution (no incremental beta distribution), [2] the random initialisation has been taken off - CTRNNs now begin from the same state, [3] there is a fixed delay at the start and between presentations - no longer is there a random component, [4] Gaussian-weighted evaluation is used, [5] traditional gen-phen mapping is used and [6] number of trials per fitnes went up from 100 to 200. The main idea of these experiments will be to see whether I can evolve agents for the continuum irreversible learning task without noise. There is still a lot of inter-trial variability.

Most (7/10) runs with 5 node CTRNNs end up categorizing ‘just’ two signals – indifferent to the differences between the ones in-between. Some make very strict categories from the inputs, others less. See this figure for an example of this kind of 'discrete' learning performance. As in this blog you will encounter this particular type of figure all over the place I will provide once the explanation of it. The vertical axis represents the different ‘parent’ signals that the agent can receive. The horizontal axis represents the different ‘test’ signals. Notice that both of these axes provide information about the continuum of that feature. The line along the diagonal, then, represents the trials where the parent and test individual are the same. The colour shading represents the agent's decision to accept as their parent or not, the test individual. The colormap goes from blue (i.e. 'this is not my parent') to red (i.e. 'this is my parent').

Out of the 5node runs, number 10 did much better than the rest. See its performance here in this figure. The activation of the nodes and the input for several trials can be seen in this figure for 10 random trials. You will probably see this style of figure often as well around here so I will explain it once first. The figure shows all ‘neuronal’ activations over time. The signals at the bottom represent the input presented to the circuit at that particular time. The vertical dashed lines represent the end and beginning of a different trial. At the beginning of each trial the circuit’s state is re-initialised. The activation in the shaded area is the ‘designated’ output node and the black boxes around the end of each trial represent the evaluation period as well as what the ‘correct’ of the output node should be for the previously presented individuals. This five node CTRNN is using all of its nodes to generate this behaviour, but the ‘strategy’ we cannot tell from this view alone. We can see that the agent makes mistakes, in for example, the 5th trial, where two different individuals are presented but the agent classifies them as the same. We can also appreciate from that figure, nevertheless, the relative proximity in ‘appearance’ of these two individuals.

Out of the 10-node runs, 2 did particularly well [runs 3 and 9]. I will show here their performance (3, 9) and the circuit’s activation for 10 random trials (3 and 9), but I will not analyse them much further yet.

It is likely that the agents use the transients and the precise timing of the presentations and delays in this particular scenario. Experiments introducing delays in the in-between periods for these best evolved agents show that, delays do affect their performance (see for example how the performance shifts as delays are introduced in one of the 10 node scenarios in this figure). One question of interest is whether we can re-evolve the successful agents to cope with random delays? The same applies for the random initialisation of the state of the circuit? If we can, then an incremental approach to noise would be rather useful.

variants on current experimental set-ups

2006-03-14T18:11:00.000Z

Evolving to remember and distinguish a feature within a continuum is truly hard for abstract dynamical systems, even if irreversible, at least in comparison with its discrete counterpart. I will detail ahead a number of several variants that I am experimenting with, in order to get something that works.

First of all, by abstract I mean disembodied and non-situated in an environment and accordingly where the system itself cannot vary the way the feature is being given, at least not directly.

[1] Gradual increment from discrete to continuum during evolution: The ‘zen’ approach would be to pick parents and individuals randomly from a uniform distribution every time from the beginning of evolutionary time until the end. But it is not easy to be ‘zen’ (particularly when you are not making as much progress as you would like to be), so we put our engineering hats on. One could think that it would be easier for evolution to gradually change the task into ‘more complex’ as the population gets better – this is probably most of the times the case, but I’m not so sure it applies in this particular scenario. As the title suggests the idea is to evolve agents for the 2 most different parents first and as the agent gets good at doing this, more closely related parents are presented. One way to do this is drawing parents randomly from a distribution that gradually shifts from the binary 2-parent case towards the complete continuum. The Beta distribution is quite ideal for this, it has two parameters and changing the parameters from 0.1 to 1 causes the distribution to change from ‘almost’ discrete to ‘almost’ uniform.

This figure shows the histogram of random samples taken a beta distribution for parameters 0.01, 0.11, 0.21, 0.31, 0.41, 0.51, 0.61, 0.71, 0.81. As you can guess, when the parameter (alpha) reaches 1, the distribution is effectively uniform across the continuum. The motivation for this is to encourage the agent to distinguish between the more different individuals first and later on to learn the smaller differences. This distribution can be used to pick both the parent and the test individual.

I’ve used this for a while and have run comparisons with and without this ‘incremental’ approach. I have not made a thorough comparison (yet), nor have I performed sufficiently big tests (basically compared 10 runs with the dynamic Beta dist and 10 with the simple uniform dist), nevertheless (and interestingly enough indeed), it does not seem to improve evolvability in any way. It actually made things worst as far as I can tell. Again, I have not studied this in depth but hope to do so at some point.

[2] Gradual increment from noise-less to noise-full: For the behaviour to be interesting it has to be able to cope with several forms of noise: [a] random initialisation of the activation of the nodes, [b] random delays at the beginning of the run and between presentation of individuals and [c] inter-trial variability. There are also other types of noise that I am not considering at the moment (e.g. [d] inter-node, sensory and motor node noise).

[3] Gaussian-weighted Evaluation of agent’s output: This idea is taken directly from (Phattanasri et al., 2002, 2006). The agent is evaluated for 10 units of time after the presentation of a test individual. Instead of simply taking the area under the output node’s activation, a Gaussian-weighted area is taken. This takes away importance to the beginning and end of the evaluation period and concentrates on the middle region. It helped them, since mine is not evolving I figured I’d use it until it works – then I would run tests to see whether this in particular helped or not.

Finally, there are several parameters we know (from common sense or experience) to be crucial:

[4] Genotype-phenotype mapping:
[i] Weights: mapped linearly from [0,1] to something like [-6, 6]. I’m quite happy with this.
[ii] Time-parameters: Map exponentially from [0,1] to [e0,e3]. The important thing here is that the smallest possible is around 10 times bigger than the time-step of integration (usually 0.1) and the largest will depend on how long a trial is – definitely not longer than that. The exponential mapping simply provides more precision for the small time-scales (where more precision might be needed), and less as the time-parameter becomes bigger. I’m quite happy with this as well.
[iii] Biases: generally I map it linearly from [0,1] to something like [-10,10] but I’m not happy with this one at all. One idea (which could be very fruitful to CTRNN evolution in general) is to map the genotype value always relative to the centre of activation of that node (which would depend on its incoming weights). Furthermore, this could be extended to include an exponential mapping so that there’s more precision around the centre-crossing region and less towards the outer parts,

[5] Number of trials per fitness (this is related to the inter-trial variability): Each agent is tested 200 times every time it is chosen during selection. I’ve done 100 until now so this is an experiment to see whether there’s any difference.

[6] Evolutionary operators: mutation and recombination: For mutation I used the common Gaussian vector mutation, where basically each value in the genotype gets perturbed with a random number drawn from a Normal distribution around 0 with very small standard deviation. I’m happy with this. For the recombination I take x/N genes from the winner of the tournament and (N-x)/N genes from the loser. I’m less happy about this method. It makes a lot of sense for the discrete (e.g. binary) genotypes but less so for continuous ones. Ideally the new individual should be made from a combination between winner and loser that is not constrained to the gene dimension. One way would be to take a new point in genotype space using regular Euclidean n-space that is an x proportion away from the winner and a (1-x) proportion away from the loser. Not sure though.

[7] Finally, the very obvious, number of nodes in the CTRNN: I have played around with circuits between 3 and 10. As there are 5 node CTRNNs whose nodes are all active but which are not doing the full* task then I will be running experiments with 10 node CTRNNs and when it works I’ll see how many are (if any) saturated on/off and take it from there.

project finally begun, first update.

2006-03-11T23:53:00.000Z

In this post I will give the overview of the broad plan (more or less again), followed by the prioritization that I have planned for my experiments. I will then provide the details of the first experimental set-up followed by the first update on the experiments that begun 4 days ago.

The long-term plan to answer the questions posted previously will be to evolve and thoroughly analyze dynamical system agents in several different experimental set-ups: two different tasks (i) reversible and (ii) irreversible learning on a continuum; and two experimental set-ups: (a) a disembodied/non-situated one and (b) an embodied/situated one. The reversible learning further subdivides in classical conditioning or operant conditioning.

This could add altogether to 6 different experimental scenarios. However, they will not be tackled all at once. I am prioritising the set-ups in terms of relatively simplicity and in terms of the main goal of my research. Although eventually I would like to understand all of these in terms of dynamical systems, I will give priority to the abstract scenarios over the embodied/situated ones because my main interest is in the analysis of the dynamics underlying learning and memory.

Furthermore, I will prioritize these abstract set-ups in terms of relative simplicity. The first set-up to be tackled will be concerned irreversible learning. Second, I will tackle the reversible learning scenarios.

The first experimental set up is to do with evolving agents that can learn once during their lifetime. They require that the agent ‘memorise’ the presentation of a feature in the environment and be able to make a decision concerning this feature after a time delay. There are a number of examples of irreversible forms of learning in animals; one which is particularly interesting is parental imprinting in birds as studied by Konrad Lorenz.

The tasks are similar to the imprinting scenarios that I have played with until now but they extend in very important directions: [1] So far the learning on a continuum has only been evolved and tested on few successive presentations of test individuals (i.e. one or two). One important direction is to extend this imprinting-like-learning over many successive presentations of test individuals. This will address questions regarding the persistence of memory.

The agent is a fully connected CTRNN. There’s one sensory signal that is being fed to all nodes in the CTRNN via a set of weights that are evolved along with the rest of the CTRNN parameters. The feature that the agent has to remember is a signal between [1, 2] provided for a fixed length of time (10 units of time). At the beginning of a trial, a random delay in introduced ([10, 20] ut). The first individual is then presented and this should be interpreted as the ‘parent’ individual in the imprinting metaphor of the task. Random delays are introduced again. A second signal is then produced. This can be of the same value or a different one to the first. This is interpreted as a ‘test’ individual. The agent has one output node. The output of this node is interpreted as ‘this is my parent’ when 1 and ‘this is not my parent’ when the output is 0. The agent is evaluated after the test individual is presented and a successful agent must produce the correct output for any number of individuals presented after the ‘parent’.

The learning is irreversible in this case because the agent cannot relearn a new parent at any particular point during ‘its life’ only at the beginning. At the same time, this learning is interesting because the agent has to hold on to its ‘memory’ of the first individual for the longest time possible.

So, I finally started on Tuesday (09.03.06). I have further subdivided the first set-up into 4 stages: [i] 2-possible-parents irreversible learning with only one evaluation, [ii] 2-possible-parents irreversible learning with several successive evaluations, [iii] possible parents on a continuum irreversible learning with only one evaluation, and [iv] possible parents on a continuum with several successive evaluations.

I was able to evolve on the same day I started 2 and 3 node CTRNNs for stage [i]. I have not had the chance to analyse these circuits nor their evolutionary dynamics for two reasons. First, because all of it has evolved so quickly, but also because the plan is to wait until successfully evolving the full task (i.e. [iv]) before stopping and analysing.

Using an incremental approach I was also able to evolve 3 node CTRNNs for stage [ii]. The incremental approach was very simple (and inspired loosely in Phattanasri's work): the parent individual is always presented at first, after a random delay the first test individual is presented. Once one agent in the population achieves 95% fitness score the fitness trial changes to include an extra test individual after some other random delay. So on, until an agent can discriminate up to 5 test individuals one after another.

In the time that I get between evolving, I am building up some tools (in C and Matlab) to help me visualise the performance of the agents and analyse both the evolutionary dynamics and the CTRNN dynamics.

Now I've set-up the evolutionary runs for the continuum and successive case [iv] at once. There are two parallel incremental approaches at work for these runs. The first incremental approach is the same one as the already described to go from discriminating the first test individual correctly to identifying successive individuals, until it generalises. The second incremental approach concerns the shift from 2-possible-parents to a continuum of possibilities. This approach is a bit more subtle and I will describe later on.

There are a rather large number of questions to be answered in this first scenario (including questions in i, ii, iii and iv). These have begun to come up as I evolve the simpler milestones. I will write about the questions in the next update.

current directions of research

2006-03-08T18:08:00.000Z

The broad motivation of my research is to understand the mechanisms underlying learning and memory of features on a continuum in dynamical systems.

In particular I'm interested in understanding, how can dynamical systems ‘record’ a feature from the environment within a continuum and later ‘make-use’ of this ‘stored-information’ to make a decision? For how long can such a memory persist? And what is required to make it persist for longer or indefinitely? I'm interested in these issues from an evolutionary perspective as well, so questions like: what sort of evolutionary pressure is needed to evolve agents with the capacity to retain a particular memory throughout its lifetime? This corresponds to certain aspects of learning irreversibility and the evolution of critical periods. But also, this work will be concerned with understanding how this learning can be made reversible. So, what sort of dynamical mechanisms does a system 'need' to be able to re-learn a feature in a continuum from the environment over and over?

Currently there are (at least) three different directions my research could take. This has become clearer from discussions. Which directions to emphasize depend a great deal on what the main goal that I have is
[a] If the interest is analyzing the dynamics underlying learning and memory, then I could initially focus on abstract tasks.
[b] If the interest is exploring the role of embodiment in learning, then I should focus on embodied/situated tasks.
[c] If the interest is demonstrating that a CTRNN/evolutionary approach can produce a wide variety of learning phenomena, then I might want to focus more on designing a set of evolutionary experiments that illustrate a wide range of learning behavior.

still planning

2006-03-02T11:46:00.000Z

The stepping-back-&-planning-ahead phase is almost over. For the last weeks I have resorted to writing on my (physical) research notebook, making concrete the cognitively interesting tasks that I will evolve and analyse, figuring out how they relate to each other and what the big question that they are answering is. This is, obviously, a crucial step in the making of my PhD thesis. But before I leap into the experiments themselves I am running these ideas by my advisor and a couple of other people. I hope to publish here the overall plan and motivation once it's all there.

in their shoes

2006-02-28T19:26:00.000Z

I have been terribly busy with the Journal of Adaptive Behavior special issue that Fernando and I are editing. This issue is based on work from our ECAL2005 workshop. I must say that the issue is looking very good and most importantly we should be wrapping up this work very soon. The complete process has been incredibly draining and time-consuming but the experience has been undeniably fruitful. Putting myself in the place of the reviewers has proven to be an excellent academic exercise because it gives me a very good notion of how to improve the writing of my own papers - which things to do and which things not to. Particularly when comparing my reviews of somebody’s work with the reviews made from other (more experienced) researchers in the field.

deciding on tasks to be evolved and analysed

2006-02-05T21:43:00.000Z

Deciding on which tasks to evolve and analyse agents on as soon as possible will allow me to work on the interesting bits (see bellow) sooner and longer. For this reason I will sketch ahead (in brief only) the range of tasks that I have in mind. They are all to do with evolving learning behaviour and they involve (I argue) minimally cognitive behaviours. I will attempt to provide motivation and the ups and downs of each one later on.

[1] The first is an abstract disembodied scenario requiring the agent to learn from a continuum the first frequency and then recall which frequency is the same as given before.
[2] The second one is a circle/diamond/bar discrimination task, where the agent must discriminate differently according to which was presented previously.
[3] The last one, is a circle size discrimination task, where again the agent must be imprinted with the first.

These tasks are interesting because they involve memorisation behaviour as well as decision making.

Chosing the tasks will allow me to focus on:
(i) Studying how the agents evolve and what can be done to facillitate this process. In particular, there is much work still to be done with studying nearly neutral networks in the evolution of continuous-time dynamical systems (cf. my master thesis). I should be able here to develop the evolutionary techniques for synthetising agents capable of adaptive behaviour.
(ii) IN-DEPTH (cf. phattanasri) understanding of evolved networks. For each task, the best evolved network should be analyzed such that all of its behaviour is fully understood. I should be able here to develop the dynamical system tools for analysis of the mechanisms underlying adaptive behaviour.

Having this should allow me to work out the outline of my thesis.

Be right back

2006-02-04T21:20:00.000Z

It’s amazing how hard it has been to force myself to start writing again. The winter term ended and my wife and I went home after a 2 and a half year absence. Had an incredible time yet couldn't wait to get back. I have been preparing for an interview for a junior position in Indiana University – will be there next week. I am 'competing' with my lab-mate Rob Vickerstaff and ex-easy-master Josh Bongard. The process of applying to this position and the preparation for the interview has been very beneficial for my PhD work. I've had the chance to really step out of the details of my work, looking down and ahead on the things I've done and the things I want to do next. Most importantly it has made me realise the path that I should take for getting a thesis underway. From next week on I will begin my research notebook blog again.

Even more efforts to move away from the curse of GOFAI!!!!

2005-12-01T16:33:00.000Z

Many of the 'cognitively interesting' tasks that we design remain to be greatly influenced by GOFAI. Furthermore, it may be the case that the more GOFAI they are the more cognitively interesting they seem. Which is obviously utterly stupid.
Evolutinary robotics, and artificial life in general, has the chance, philosophically, conceptually and technically, to change this, and so we must.
This is particularly the case in experiments involving learning. They generally involve discrete signals, discrete experimenter-determined phases, and evaluations that presume that things like a 'decision' needs to be discrete.
A good example of how agents' decisions need not be discrete is the case of the circle-diamond discriminating agent who as it moves in time it makes up its mind and changes its decision and so on back and forth until it actually catches or avoids.
Tasks can be further improved to encourage this type of analysis and understanding by making learning (and in general any other cognitive) tasks more dynamical, more interactive, more situated, and less GOFAI. More on this later, definitely.

GA/CTRNN Exercise for ALife Course

2005-11-21T10:25:00.000Z

We've set-up an arguably minimal task where a CTRNN can be evolved to use its ability to integrate over time mildly interestingly as an exercise for EASy master's students.
The general aim of the activity is to get students to play with the evolution of dynamical systems controllers. The activity is incremental in that for their first activity they implemented a GA for a maxOne optimisation problem.

The big picture of the task is the following. You have to evolve a CTRNN to be able to calculate (or at least approximate) the rate of change of an input that comes into one 'input node' - and signal its 'best guess' by the activation on another node designated as the 'output node'. There may be some other internal nodes as well, and they may all be fully connected together. The evolutionary technique's job is to find a set of weights, biases and time-parameters for links and nodes, that make this an effective CTRNN at doing this job.

The task is sufficiently interesting because it requires the CTRNN to perform some integration over time, as opposed to (a) purely reactive behaviour (e.g. a phototactic robot) and (b) as opposed to merely computation (e.g. a 'xor' logic gate).

For more information on the task and my solution in C see here .

artificial coupling devices

2005-11-16T12:13:00.000Z

very interesting meeting to discuss sensory substitution held by the dysturb reading group based on the approach taken in Compiegne.

it may or may not be obvious that research using artificial coupling devices lies very closely to the dynamical systems approach to learning, there were several points in the reading that I found most relevant some of which were brought to the discussion.

Of particular interest is the point raised from sensory substitution experiments about the richness of perception depending as much on the quality of movements as in that of the sensations. Yet I am a bit surprised that much of the actual research lies on the side of supplementing sensations, as opposed to supplementing motor activity. If perceptual supplementation is to be taken seriously then motor-substitution should have similar effects on our perception as well as the more commonly explored sensory-substitution. The thing might be that we do supplement our motors in all ways since we are very young of age, with bikes, skateboards, rollerblades, eventually cars, etc. So, do these motor-supplementation change our perception? they do. But we are too used to them to take much 'scientific' notice. So, how do these motor-supplementations change our perception?.. I'm thinking that this field would benefit from minimal studies experimenting with motor and sensor supplementing and see how equivalent they are. It would be very interesting if one could even compensate our perception with motor and sensor changes such that one suppresses the other's effect.

Another point of interest is the possibility of studying 'values' from such experiments, without resource to philosophy and pschocology. It is interesting to know that for a person that experiments such 3D perception from a camera mounted on its head which was born blind the 3D perception has little or no meaning. This points towards values not being something that is attached to objects out in the world but rather of values being a historical construction from the interaction with perceptions over time.

Many other really interesting points, but the last one I'll mention here is that of the plasticity shown by organisms' nervous systems and bodies. The capacity of take in perturbations from the environment and couple with them in an increasingly meaningful way over time. One thing which is of interest is to know where this plasticity is taking place. For example, say somebody trains to 'see' with his stomach and then the patch is displaced or placed in a completely different place (its back), how long does it take to re-learn/re-adapt to 'see' again? I think that depending on how long it takes one to re-learn would say something about where the plasticity is coming from. If it took very little time, once learnt, to swap between any part of the body the patch and still see then if would be a plasticity at a very high level in the nervous system. On the other hand, if one had to learn almost from scratch again then it would imply that a lot of it is happening at the level of the body... does that make any sense?

anyway, hopefully more on this later on..

the reading was: Lenay C., Gapenne O., Hanneton S., Marque C. et Genouëlle C. Traduction anglaise (2003) Sensory Substitution, Limits and Perspectives, in Touch for Knowing, John Benjamins Publishers, Amsterdam.

Alergic talk Wed 9th Nov, 16:30 in ARUN-401

2005-11-07T20:41:00.000Z

I will present:
Learning on a Continuum in Evolved Dynamical Node Networks: an Imprinting Scenario

Conventionally, learning behaviour has been associated with the modification of neural network's parameters (e.g. synaptic weights). A more integrated view of learning behaviour has been presented where the agent's behaviour and the modification of the behaviour are both generated by the same dynamical systems controller. All of the work (so far) in this field, however, involves learning behaviour as swapping between, in practice, n (eg 2) different modes of interaction according to which environment the agent finds itself in. There's a different form of learning behaviour that involves having to act differently according to features from a continuum in the environment. This latter form is also related to the ability to cope with never-before-experienced scenarios. A good example of which is parental imprinting in birds. This talk will provide results being submitted to the Alife X conference on the evolution and analysis of embodied and situated agents in a task loosely inspired on imprinting-like behaviour.

Link to the slides in PDF

conferences ahead

2005-11-01T18:57:00.000Z

you know you don't have a proper life when you haven't finished the paper that it's due in roughly 5 days and you are already excited about another submission deadline 3, 4 months away. anyway, amazingly appropriate conference for further work along this line will come soon! international conference on development and learning (ICDL), chaired by linda smith and running almost in parallel to alifeX. very exciting! hope to extend my work to address more developmental questions by then. deadline is for feb. 06. 2006.

another conference will be due a month away from ICDL. one where the work that i've left behind with regards to evolving minimal controllers will definitely be picked up. this is the simulation of adaptive behavior / from animal to animats conference for the 20th of march, 2006.

yes. a lot of writing up. still onto it. 80% there.. title: 'learning on a continuum on evolved dynamical node networks'. link to first complete draft coming soon.

writing up

2005-10-27T20:09:00.000+01:00

have been quite busy writing up the paper for alifeX. the results are there and look reasonably nice - more importantly they convey exactly what I want to say, which is: the evolution of learning behaviour from a continuum of features in the environment using dynamical 'neural' networks without explicitly building in weight-changing mechanisms and actually with no weight-changing allowed whatsoever. the interest in this work is in the analysis of how such learning behaviour can be produced from the interaction of the agent embedded in its body with its environment. what role do the internal dynamics play? what role does the body play? and what role does the interaction with the environment play? the paper should be ready soon! fingers crossed.

teacher training has been very time consuming. a need to remember to do a number of assignments in order to practice what we have learned as well as to get the accreditation for the course, all after submission for alife. one cool thing that i realised today, although obvious, is that all this reflecting (for example this blog) about the things that I'm doing and the activities that i am carrying out is very beneficial for my development, in general - or at least my development in the things that i am writing about. my research. my teaching. let's see..

i have decided to leave the evolving minimal controllers point with the circle-diamond discrimination task for slightly later. either directly for a journal with added explanations or for SAB coming up in march. this means i may not pick this work up until january.

a lot of talk about embodiment and what it means to be embodied and what it is that 'embodimentalists' stand for, etc, etc, in the philosophical discussion group at uni. although i find it bearable i don't think they are getting anywhere.

have joined simon's first session on the discussion of whether AI is or can be of any help to developing nations. although i'm obviously no expert, my questions and interest were mainly with regards to what it means to be developed and whether what we want is to help develop countries or help them in their developing. the first issue is related with whether being developed means being faster/stronger/bigger at the things the country does or whether being developed means actually being smarter, living a more balanced way of life, taking the bike instead of a car, this sort of things. The second issue is with regards to the way of helping. there's two ways of helping. the two that I mention are quite different from each other. in the first scenario you develop them, and they are left more needy and dependent, thus for me less developed. whereas in the second scenario, they are capable of doing things on their own.

minimal controller for circle-diamond discrimination task

2005-10-18T17:01:00.000+01:00

here's an update on these experiments...

changes:
1. binary genotype added on top of the real-valued genotype so as to turn on/off internode connections and also stateless/statefull nodes. 30 binary genes in total. 5 for the 5 time-constants of the 5 inter-nodes which are modified to same value as the time-step of integration in the case of 0 and to the mapped value on the real in the case of 1. 25 values for the inter-node connections between the 5 inter-nodes. 0 corresponds to no connection and 1 corresponds to a connection with the strength of the real-valued mapped gene.
2. sensory and motor nodes have no state (time-constant = time-step of integration).

sets of experiments:
[a] gradual fitness: smaller weights and smaller time-constants improve fitness.
[b] threshold fitness: fitness only improves if inter-connection weights are off and time-constants are off - smaller doesn't count.

results so far:
out of 5 runs 1 was fully successful. it evolved the first 1000 generations with fitness function from set [a] and the last 100 with the fitness function from set [b] given that it needed to prune out only one more recurrent connection which was already very low (i.e. 0.48). (will add some figures soon!)

plan:
plan to leave 20 runs for fitness function [b] and cross my fingers.
that should be it.
hopefully at least a couple of reactive solutions evolve in which case i shall analyse them and report them to our friends at indiana ;)

embodied imprinting task: changes and results

2005-10-18T16:58:00.000+01:00

here's an update on the embodied imprinting task.

changes:
(1) objects now fall from same offset always: why? because (i) it was becoming too difficult to evolve and (ii) it was making analysis of evolved agents harder. More generally we can safely say that different offsets are not adding anything in particular to this task, at least so far.
(2) networks are not symmetrically constrained: why? the circles fall straight above from the agent so symmetric agents would need a symmetry breaker. having a non-symmetrically constrained agent makes it easier because (i) there's no need for a symmetry breaking mechanism and (ii) the agent can use more 'dimensions' to solve the task - it can actually make the task easier by being able to use both sides of the object.
(3) sensor and motor nodes are now reactive (time-constant = time-step of integration): why? basically (i) sensor and motor nodes can act as place holders given that the truly recurrent and interesting bit is in the inter-nodes anyway, (ii) makes evolved network easier to analyse and (iii) makes for slightly faster simulations.

results so far :
have left 10 runs on star cluster and 4 on my machine, this last ones evolved towards ~80% of fitness after 500 gens - much better than any previous result, but with plenty room for improvement.

meeting with advisor

2005-10-13T19:02:00.000+01:00

met with my advisor today for an hour and a half and talked to another member of faculty yesterday for a couple of minutes all which has left me incredibly inspired to continue the things i'm working on... if anything faster! my advisor was very excited about both projects: (a) imprinting and (b) minimal controllers.

regarding project (a) the plan is: (1) profile the code so as to optimise* as much as possible, (2) add trivial-1D-geographical-demes to microbial GA (in hopes of being favoured by the gods) and (3) change a couple of parameters, including range of sizes of the radi of circles to [10,20], so as to make the kick-start of a solution ever-more-likely. all of which means everything else including my implementation of the 'zen approach' sounded reasonably well to him - that's not generally easy to accomplish in itself.

regarding project (b) ezequiel suggested a number of things which i will mention in a next post.

in theory, whatever results come out first (or come out at all), will be considered for an alifeX paper (ahhhhh.. submission deadline in around three weeks!!!)

despite common sense, i managed to have had my arm twisted to do alergic, next week!, in exchange of not having to do seminars.. it shouldn't be too much work given that it will be mostly based on what i presented in elio tuci's workshop with whatever update i may have by then, which i fear i can't promise there need be any update at all.

finally, i have to mention (as a reminder to myself) that i also need to do the corresponding planning for the next activate.d sessions.. i've thought of a couple of topics that would be quite appropriate.. but haven't pinned any down so far.

* regarding optimisation:
(i) non-optimised => -O3 (~58% time saved)
(ii) exp() in sigmoid => look-up table (~36% time saved)
(iii) added geography (at less than 6% time cost).. ?

update: 10 evolutionary runs have been left in the linux cluster overnight (pop=50, rep=50, gens=400, ~14 hours) and 2 runs on my machine should be done by the time i arrive tomorrow (pop=50, rep=50, gens=1000, ~12hours)... fingers very much crossed for anything to go above 60% fitnes.. ?

on adding bells and whistles to CTRNNs

2005-10-13T05:23:00.000+01:00

the discussion about reasons for and against adding bells and whistles to CTRNNs has come to be almost a daily thing in our lab. i will try to expose here first what the types of extensions to the model can be, then i will try to explain what i think is at least a subset of reasons for people wanting the whistles and the bells followed by my reasons for thinking they are either wrong or asking a question which is not of interest (at least to me).

[1] types of bells and whistles

[2] why some want to add them

[3] why this is not needed

note: will be writing this in for a couple of days..

on evolving minimal controllers

2005-10-12T11:44:00.000+01:00

I now know the circle-diamond (fixed size) task can be solve by a reactive controller. One important implication of this result can be illustrated by always attempting to evolve for the simplest controller.

An interesting way of making this point is by using the exact same set up as Randy in his circle-diamond paper but change the fitness function to add two more costs: (a) cost for recurrent connections and (b) cost for big time constants.

So the fitness of an individual is given by:
f = 0.5A + 0.25B + 0.25C
where A is the fitness function used by Randy (i.e. points for catching circles and avoiding diamonds), B attempts to minimise the absolute value of the weights which are recurrent towards nil while C attempts to minimise the value of the time constant towards its minimum (i.e. one).

In order to cope with the unlikeliness of having a weight which is exactly zero, the simulation is changed so that all weights between -1 and 1 are effectively treated as zero. Similarly for time-constants, as it is unlikely to have a time-constant which is exactly 1.0, time-constants smaller than 1.5 are changed to 1.0.

As a test it was checked when the weight of A was reduced to zero the evolved network would straightforwardly go towards a reactive one which it did.

There is one very significant doubt which I have about the time-step of simulation which is 0.2 at the moment. As this value is fixed (obviously not evolved) there is no chance for changing it in the middle of the experiment.. Would a network which has no recurrent connections and all time-constants set to 1 be reactive regardless of the time-step of integration being less than 1? If not, is there at least the possibility that, once evolved, that network be able to perform when the simulation is changed to a time-step of 1?

alife seminars (i)

2005-10-11T21:15:00.000+01:00

back to teaching alife seminars. this year's group is much smaller and that's actually a good thing. two hour discussion on richard dawkins' "evolution of evolvability" isn't as straightforward as it seems. the perspectives that the students take on the subject can be quite diverse and at times unintelligible. there's a number of interesting points that this paper raises but there's also an inmenly loose treatment of an artificial model to answer any scientifically relevant questions.

the main points raised during the seminar discussions were: (a) the use of artificial life models, either as models of real life or as generator of insight; (b) being clear about the assumptions that are being built into your alife model and trying to reduce these to the bearest minimum necessary or possible; (c) the use of embriology in artificial evolution as a helpful constraint/canalisation of useful forms; (d) whether his model gives insight towards the evolution of evolvability? (e) and in which ways would this model be extended or improved either to answer questions regarding the evolution of evolvability or other questions.

making embodied imprinting simulation as efficient as possible

2005-10-10T11:08:00.000+01:00

plan to leave a number of evolutionary runs for the embodied parental imprinting task which now has a more 'zen' approach (will detail it later). the thing is runs take relatively much longer than my usual runs (for several reasons but around 30 mins for 100 gens with pop size 10). so i have had to work on making it as simple and efficient as possible, getting rid of a number of things which i generally have laying around in my code (e.g. the non-symmetric genes, the binary connectivity genes, etc).

teacher training

2005-10-06T22:32:00.000+01:00

after a couple of false starts i finally began the mandatory course on learning to be a teacher as part of my doctoral training... surprisingly, it wasn't as bad as people said it would be. it's been quite intensive, about 7 hours in 3 days, all compensated by interesting readings, activities and discussions. somehow this course has reminded me of how much i would like to become a 'good' teacher. how much i'd like to inspire people to do something.. as others have done for me. the only thing i now know better than before is that being an inspiring and ethical teacher is not easy.

circle-diamond categoriser with a reactive-controller

2005-10-03T18:42:00.000+01:00

after implementing diamond-shaped objects into my circle-falling simulations (a la Beer) and intensive debugging i have gotten the simulation to work properly!!! also have managed to evolve so far reactive controllers as well as non-reactive ones for the task!!!

let's see how it goes.. there's much work to be done still. i shall remember to leave mass/inertia out of this one for simplicity. most likely the explanation of the evolved system is again one of historical interactions between the agent and its environment. the interesting question here is, what rules can be introduced into the evolutionary algorithm to select for simplest networks?

and would also be interesting to attempt evolution while vaying sizes of objects - mainly to find out (a) which is the simplest agent that can solve that task and (b) what is the dynamical systems explanation of the best-and-simplest evolved agent.

embodied imprinting

2005-09-30T23:32:00.000+01:00

these last two days i have been trying to work on the embodied imprinting task.. but the complete task that i want seems a bit too hard to tackle from scratch so i've started a very slow incremental approach by trying simpler tasks so as to validate any small details of the simulation...

1. can the agent distinguish between two circles of different sizes? yes... this was relatively easy.
2. what's the smallest difference that it can evolve to distinguish? it seems like for a 4 inter-node CTRNN a radius size difference of 1 was the smallest it could evolve to discriminate.
3. can an agent catch circles one after the other? this would be needed for my task. also relatively easy.
4. can an agent catch circles of size A and avoid circles of size B when presented one after the other in different orders? this was a bit harder.. and got me to solve a couple of bugs but does work.

the things i've learned so far from this start are:
a) perfect symmetry helps a great amount in this symmetric task... will stick with it for the moment.
b) noise and perhaps more importantly inertia are helping break the symmetry in those undesirable cases,
c) i can see that getting the full task to work won't be easy..

on a different note, aisha, alex and chrisantha submitted their theses today.. i think ian
must have as well..

bits and pieces on ecal2005

2005-09-29T22:53:00.000+01:00

I have to mention that there was quite a good reception in ECAL for (a) the paper in Tuci's workshop on evolving learning in the imprinting scenario, (b) the activate.d workshop, in particular the roundtable discussion and (c) the paper in collaboration with Ezequiel in the main conference on whether an agent which is embodied can ever be purely reactive. A couple of things about this last one....

I have been thinking more and more about the implications of the examples shown and discussed in that paper as well as a number of other similar examples. I will mention here two examples that I have in mind.

The first one is very straightforward and involves the use of inertia: catching a circle despite going blind after it starts falling. Beer suggested in his 1996 paper (Toward the evolution of dynamical neural networks for minimally cognitive behavior) that this task needs internal state, but the embodied examples shows that with only inertia (without internal state) it is also possible (I have experiments showing that already).

The second example is a bit more complex, it involves trying to get a reactive agent to solve the exact same problem of Beer's more recent (2003) paper in adaptive behaviour (The dynamics of active categorical perception in an evolved model agent). I am almost sure it could be possible that an agent with a purely reactive network can solve the task when the circle and the diamond are of fixed sizes (this is the same paper which you replicated while varying the sizes of the circles/diamonds, in which case I would not think could be solved with a reactive controller). This has two implications, the first one would be along the lines that we have been talking about, the importance of embodiment and situatedness. But the second one is a little bit different, it prompts the question, is the methodology that we are using all right? what are missing? when Beer evolves this agent to categorise between circles and diamonds (of fixed size), there is no pressure to evolve the simplest solution. And if the interest is in the analysis of the resulting dynamical system, as is the case in that paper (and is the case for my motivation), then, one ends up analysing something that is much more complex than could be needed. So, if it is the case that a reactive agent evolves to solve the task, placing the explanations on the agent/environment's historical interaction and not on the particular activations of the neurons, then the dynamical systems explanation that Randy offers is of less use. The particular suggestion to change our approach would be to have a cost associated to complexity in our methodology of evolving agents, time-constants should tend to be as small as possible by default (as reactive as possible), having slower acting neurons being costly.. , similar for other parameters such as connections, which may need to be costly as well, so networks tend to be as small as possible.. etc.

Anyway, new but related ideas and discussions are ongoing with Ezequiel which may end up in a journal publication (fingers crossed)...

evolving path integration on 2D agent

2005-09-28T22:33:00.000+01:00

At the moment I am working on evolving a 2 dimensional khepera-style agent to perform photo taxis followed by path integration back to the place where it started from, controlled by a 2-5-2 node bilaterally symmetric continuous-time recurrent neural network (CTRNN). The simulation is a very fast version of a 2 dimensional world with simplified physics. It has however proven to transfer successfully to the real khepera robot.

There are a number of reasons why the path integration task is important for a dynamical systems approach to learning and memory but I will not go into it just yet. Today I've left running 6 sets of experiments. 5 evolutionary runs per set. 2000 generations per run. Results will be due in around 30 hours... this should give me time enough to work on my primary task the imprinting scenario which I will describe later on.