Forum

> > CS2D > Scripts > Save huge table in textfile fast
Forums overviewCS2D overview Scripts overviewLog in to reply

English Save huge table in textfile fast

3 replies
To the start Previous 1 Next To the start

old Save huge table in textfile fast

Bowlinghead
User Off Offline

Quote
Hey us nrealSoftware,

I was testing some Record/Replay script to create demos of your games you can watch later.

Idea:
This script creates a big table for every "tick". This table includes every information of all the relevant hooks, including almost all parameters. This table gets saved into a textfile periodicly - the demo file.

Problem:
The problem with saving this periodicly using normal io-functions into a textfile: You get a noticable freeze when it saves duo to the huge amount of lines to print, with just ~4 players.

My solutions so far:
Saving everything for a round in the table and only write on roundstart into the textfile would sort-of decreese the freeze problem (a lag on roundstart is less impactful than mid-round) BUT the save process takes over 30seconds for like 5 players (which depends on roundtime and amount of hooks fired listed in Addhooks.lua).

How did I structure my code:
In the config.lua you will only find constant variables to make the script easy modifiable.
The Assembler.lua helps me to create strings (Used in Addhooks.lua).
The GameState.lua is every data I collect for the current tick.
The GameReplay.lua is every data for every tick (It contains an "actualState"-instance of GameState which swaps out every tick).
Addhooks.lua contains every addhook command. Every addhook command saves the data about the parameters in GameReplay.actualState.


The source code can be found here:https://github.com/Bowlinghead/cs2drecorder
Dofile "dofileRecord.lua" to let the script run; then write "!record <name>" to start record/ "!stop" for stop.

Essential code parts are quoted here:
More >


Note that this script runs perfect on my side, except for the long write-time (tested with bots). I dont know the magic behind the io.- maybe I use it completly wrong. The file remains open during the whole recording - what does it mean, to (io.)open a file essentially?

TL;DR
- How do I save big table data into a text file the fastest?
- On which factors does the speed to io.write depend on mainly? (Example: Is 1 io.write call with a big string better than 100 small io.writes?)

old Re: Save huge table in textfile fast

Cebra
User Off Offline

Quote
Can you save incremental?
Should reduce the data, but need more calculation (maybe each tick, depending on the implementation)

I did a small test and maybe will do some more research: for a small dataset of 100 kb the single-write-op is 10-times faster then the multi-wirte-op (the raw writing-operation, not include the creation of the diffrent types of data)

old Re: Save huge table in textfile fast

MikuAuahDark
User Off Offline

Quote
user Bowlinghead has written
How do I save big table data into a text file the fastest?

I've left my code review to the GitHub issue https://github.com/Bowlinghead/cs2drecorder/issues/1 and you probably want to read it as I highlight important bits in your code.

user Bowlinghead has written
On which factors does the speed to io.write depend on mainly?

io.write
is actually fast, assume with default file buffering. If you disable file buffering, then it's better to write large strings in one go.

Optional >
To the start Previous 1 Next To the start
Log in to reply Scripts overviewCS2D overviewForums overview