Forum

> > CS2D > Scripts > Learning / Evolving Bots | Machine Learning
Forums overviewCS2D overview Scripts overviewLog in to reply

English Learning / Evolving Bots | Machine Learning

22 replies
Page
To the start Previous 1 2 Next To the start

old Learning / Evolving Bots | Machine Learning

Loooser
User Off Offline

Quote
Some weeks ago I stumbled upon a youtube video by SethBling where a computer program "learns" how to play a level in Super Mario World. In this video he explains how he uses a process called neuroevolution to make the program finish the level.

So I started asking myself if I could write a program that "learns" how to play cs2d.

I took the code from SethBling and rewrote it so I could use it on cs2d bots. He was using NEAT (neuroevolution of augmenting topologies).

Before I continue, I recommend you to watch his video, where he also explains how it works:
https://www.youtube.com/watch?v=qv6UVOQ0F44

> How it works:
NEAT basically generates a fixed number of neural networks.

Neural networks are working similar to our brain. They get Inputs (information) and calculate Outputs (e.g. cs2d controls) based on the structure of the neural network.

All those generated neural networks get tested and rated with a fitness value based on how good they did on the given task (e.g killing enemies). When all neural networks have been tested the best with the highest fitness value get selected and a new generation of mutated neural networks gets generated. This new generation gets tested again -> best get selected -> new generation and so on...

Each generation the neural networks (should) get better.

> Implementation:
When I finished rewriting the code I realized that it would be highly unrealistic to make bots fully controlled by neural nets.
Especially pathfinding is far to complex. So I decided to start with the enemy engage section of the bot code (when the bot has to fight an enemy).

Currently the neural network has the following inputs:
1 - Can I hit an enemy? Freeline and Angle (0 to 1 | no to yes)
2 - Nearest target angle (-1 to 1 | left to right)
3 - Target Distance (0 to 1 | near to far)
4 - Angle the enemy is aiming at relative me (-1 to 1 | left to right)

The neural network can give the following outputs:
1 - Intelligent Attack! (if output > 0.5)
2- Rotate (-1 to 1 | left to right)
3 - Change aim distance (0 to 1 | near to far)
4 - move X relative to target. (-1 to 1 | left to right)
5 - move Y relative to target. (-1 to 1 | go away to get closer)

fitness evaluation
:
Every time a bot takes longer than 200 frames to engage(timeout) or the bot respawns (bot has most likely died) the neural network gets evaluated and the next neural network gets tested.

fitness value is calculated with:
1 - Number of times the bot fires the right direction.
2 - Number of Scores / Kills
3 - Engagement time
4 - How close the bot was on aiming in the right direction

Here a video of a training process:
https://youtu.be/dk3VvSt6DXs

I also implemented a save an load functionality so that you don’t have to restart training every time the server gets restarted. It creates a save file after each generation.

>Where to go from here?
• I want to encourage everyone who is interested in trying it out and maybe make something cool or improve the current implementation.
Here is the Github Repository: https://github.com/LoooserDev/CS2D-NEAT-Bots/

• It would be great if the results could be improved with better inputs and outputs. Maybe there is a potential to make great bots with this approach! Also a functionality to play the best neural network only is missing.

• A visualisation of the neural network and stats on ai_debug screen would also be cool

• Try different neural networks that also have memory cells. These could be useful for decision making that is not only based on the current frame. This would most likely be a Recurrent Neural Network. This is not so easy as it has to be implemented in a evolution process.

• Make a great Video that hopefully gets many views to get more people interested in CS2D. SethBlings video has 4.5 mio. views and there are just a few videos on this topic. Also CS2D would be special since it’s a online shooter and real players can play against these Bots.
edited 1×, last 02.07.17 11:01:09 am

old Re: Learning / Evolving Bots | Machine Learning

Avo
User Off Offline

Quote
Wonderful to see someone made an attempt of making genetically evolving neural network for CS2D bots. Well done, sir.

OP has written
I want to encourage everyone who is interested in trying it out and maybe make something cool or improve the current implementation. (I will upload the files soon)
Repository would be good, especially if you want to find some helpers.

user Starkkz has written
I never got to know how to use correctly the back propagation.
Same here.

old Re: Learning / Evolving Bots | Machine Learning

SQ
Moderator Off Offline

Quote
@user _Yank: Yeah, this bot would not come up with strategy and tactics. Performance wise also it's not the best solution due to that it's written in Lua. At the moment the worse part of the bots are the movement, they are always looking at the direction they are going and the patterns of all bots are the same. I thought of a quite challenging AI system, however I have no time for that.

old Re: Learning / Evolving Bots | Machine Learning

GeoB99
Moderator Off Offline

Quote
Definitely this is amazing to see such a system like neuroevolution for CS2D bots. Out of curiosity, how's the performance like with the implementation of NEAT system in CS2D considering the complexity and the great amount of neural networks?

old Re: Learning / Evolving Bots | Machine Learning

Pagyra
User Off Offline

Quote
Simple evolution is an ineffective selection of decisions and actions, it is not dependent on tactics, strategy, human behavior, player characteristics.
After all, in end we want to get bots that are similar and behave like a team of real live players, and not as terminators that destroy everything they see around them on their way to goal point.
Therefore, I think that you should try to combining existing code of regular user Dousea's and old user Loooser's bots (meaning those parts that help bot manage its character) and this new neural network but in that stage we need add some functions(human factors and parameters) to change final behavior, decisions and actions of bot.
May be we need a coders team to make correct scripts for bots.
And perhaps you need to add an analysis function of replays (or something like that), so that bot is trained based on data of real player's behavior.

old Re: Learning / Evolving Bots | Machine Learning

Loooser
User Off Offline

Quote
Great to hear your opinions on this topic! I don't believe neuroevolution can replace the logic our current AI. But if improved neuroevolution has the opportunity to give the bots a more natural feel through its variation.

In order to improve the evolution speed I made every bot to test a different neural network. The benefit is also that the bots behave differently.

@user Dousea: Don't stop developing your bots. Im excited to see what you come up with. And maybe you can make use of this.

@user Pagyra: yes evolution takes a while to deliver good results. How good they get is determined by many factors inputs, outputs, fitness evaluation and how the evolution process is written. Simple "tactics" can be developed. But I agree an strategy or a team can't be developed. Thats way a mix of all would be great.

@user SQ: What kind of AI system where you thinking about?

@user GeoB99: As far as I can remember I had 31 bots running at the same time and the performance was good. Except for when the server had to generate a new generation of 300 neural networks and save it into a file, that's where the server lagged. But running neural networks without evolution should be fine from the performance perspective.

old Re: Learning / Evolving Bots | Machine Learning

Dousea
User Off Offline

Quote
Well glad to hear. But if you don't post your code around this time, I wouldn't be able to make the bots compatible with this "necroevolution", but let's see what's ahead of us. Good luck with this.

old Re: Learning / Evolving Bots | Machine Learning

Hador
User Off Offline

Quote
I like the idea (Machine Learning being currently my main field of study at university), however I do not believe learning to play cs2d is currently possible within a reasonable amount of time, I would imagine you need a crapload of runs before this might work well (thinking in the millions here, even though you got it down to a nice amount of useful parameters).

However, I wish you good luck, and I am looking forward to watching the process!

old Re: Learning / Evolving Bots | Machine Learning

Pagyra
User Off Offline

Quote
user Pagyra has written
And perhaps you need to add an analysis function of replays (or something like that), so that bot is trained based on data of real player's behavior.

I think that is really good idea for neuroevolution.

old Re: Learning / Evolving Bots | Machine Learning

Avo
User Off Offline

Quote
user Dousea has written
Shit. I lost. This might be a better solution. Really want to see this happening.

Hey, don't say anything about losing! Keep going with your AI, the time will show us which one is better. Or maybe in the end it will appear that synergy of your and user Loooser's work will make an effective AI for CS2D bots.
To the start Previous 1 2 Next To the start
Log in to reply Scripts overviewCS2D overviewForums overview