Automate Gekko’s Strats parameters backtesting (with Gekkoga)

We saw in previous posts how to install gekko, use it, and customize our first strategy.

But, as we figured out, every strategy, shall it be your own custom one or any Strat you will find on Internet with excellent backtests results showed by its creator, also needs to be tweaked, for a specific market, currency, asset, and it means we need to find the good parameters to be used with this specific Strat. And you will need a lot of backtesting, then news tests on the live market with simulated orders (paperTrader mode), before being launched “live”.

Note that finding the perfect parameters for a backtest (the ones which will provide you the best profit and best sharpe ratio) does not mean that it will perform well on a live market, as trends and volumes can simply not be known by advance. Would be too easy. Therefore, the tools we will use here do have a strong limitation: they will help you to find the best parameters for a Strat, using data from the past, but in no way it means it will perform well in the future (see overfitting or curvefitting).

So, first of all, we need to define a good backtest strategy, whatever the way (automated or not) we will find and test parameters. IMO a good testing strategy -this is what is done in AI learning & testing phases- is to split your backtest dataset in several parts: one long dataset to make a general backtest and reach good profit & sharpe; then test it on smaller datasets, of course still from the same market/currency/asset, but with different kind of trends. This way we will be able to understand how well will perform the Strat with parameters X or Y on this or that kind of trend.

We could also run Gekkoga sessions on datasets “specialized” in a kind of trend, and check if the optimized parameters found will change a lot and how between each dataset.
With those kind of results and knowledge, we could imagine implement a strat which would dynamically change and auto-adapt its parameters to the current trend, if it’s a long term one. Remember the parameters won’t only depend on the trend, depending on the indicators used, it could also depends on the market prices or other things.

In any case, each “good” test should require a stronger -manual- analysis from you: you will need to study the trades (when they were made). Is it accurate or not ? Were the larges losses controlled by a stop-loss implementation or not ? If you change a little bit one parameter, won’t it make your Strat less profitable on your past dataset, but also less risky and more profitable for the future ? The main key is probably to control large market losses. Then to add some bonuses to the Strat.

Let’s come back to this post: I wanted to double my theoretical studies on various indicators -to better understand them and eventually find an appropriate way to mix them- with technical tools to improve the backtesting phase. When I test a Strat, I need to test it a lot of time; therefore I naturally searched for tools which would allow me to automate that, and I found -among others- Gekkoga.

Gekkoga is told to be a Genetic Algorithm (GA) trainer, it means that it will:

1. Automagically test random parameters using controlled backtests it will launch through your Gekko installation,
2. Automagically try to mix some of the parameters which seemed to perform well and launch new tests, and study the results, and/or mutate them or others,
3. Log the best result it finds in terms of profit (may not be the most accurate target !) with the associated parameters used during the test,
4. Until … I don’t know yet if it actually can ends sometime ! And I did not check the code to understand it yet, I simply used it.

Gekkoga Installation

Enough talks … Let’s install it. It’s quite simple, BUT you need a fully functionnal Gekko. Do not try to use Gekkoga if you don’t have a working Gekko and if you don’t masterize its use yet.

cd <gekko_installdir>

git clone https://github.com/gekkowarez/gekkoga.git && cd gekkoga

Now we need to deploy a fix made to make Gekkoga compatible with latest Gekko v0.6x we installed previously, as some changes were made in its API.

git fetch origin pull/49/head:49
git checkout 49

We manually download a fix in index.js to support nested Gekko’s parameters and fix something in mutations

mv index.js index.js.orig
curl -L -O https://raw.githubusercontent.com/gekkowarez/gekkoga/stable/index.js

We manually download a fix in package.json to support nested config-parameters

mv package.json package.json.orig
curl -L -O https://raw.githubusercontent.com/gekkowarez/gekkoga/stable/package.json

Then we install it. Once again, beware: don’t run ‘npm audit fix’ as suggested at the end of the npm install command below; it would break things.

npm install

Note: Gekkoga will need either Gekko’s full UI mode to be launched (use the PM2 startup script start_ui.sh we created in Gekko’s installation post), or the API server which is found in <gekko_installdir>/web, and it will make an intensive use of it. This is why in Gekko’s installation post, I recommended to raise the stock timeouts in <gekko_installdir>/web/vue/dist/UIconfig.js and in
<gekko_installdir>/web/vue/public/UIconfig.js to 600000.

Gekkoga Configuration

Gekkoga’s configuration file is located in <gekko_installdir>/config/. We will copy the original one to a new one dedicated to our previously customized strategy (MyMACD) and symlink it with the name of the config file we defined in the start.sh script. It will make our life easier later when we will have new strats to backtest: we will just need to copy in gekkoga/config one config file with the filename containing the name of the strat used, and update the symbolic link config/config-backtester.js to point on this specific config.file.

cp <gekko_installdir>/gekkoga/config/sample-config.js
gekko_installdir>/gekkoga/config/config-MyMACD-backtester.js

ln -s
<gekko_installdir>/gekkoga/config/config-MyMACD-backtester.js
<gekko_installdir>/gekkoga/config/config-backtester.js

Now we will edit <gekko_installdir>/gekkoga/config/config-MyMACD-backtester.js. It is not complicated BUT we will need to define EXACTLY the same parameters as in your gekko config file or toml file. Otherwise Geekkoga will start, but with no trades if anything is wrong. Beware of the typos, beware of the type of data you will use and their type (integer vs float) & eventual decimals.

Hint: try to use as much integers as possible in your Strat parameters, and avoid floats when you can. This is why in our customized MACD Strat, I defined the stoploss percentage as an integer, and then in MyMACD.js when we need to use it, we divide it by 100. If we used a float to allow very accurate stoploss, it would have force us to tell Gekkoga to generate randomized floats, and even if we can try to fix the number of decimals used, the number of possible combinations and subsequent backtests to perform would be exponential. also, the .toFixed(2) we will sometimes used in Gekkoga conf file is an artefact: the library used to generate the random numbers will actually generate floats with a much higher precision than 2 decimals, but we will artificially truncate or round it to 2 digits. It means that Gekkoga will indeed perform a lot of backtests with the same float rounded to 2 digits, because the floats it actually generated in backend were indeed not equals.

First we change the config section, once again we want it to reflect EXACTLY our gekko config file. Same parameters, same values.

const config = {
stratName: ‘MyMACD‘,
gekkoConfig: {
watch: {
exchange: ‘kraken‘,
currency: ‘EUR,
asset: ‘ETH‘
},

We use the scan functionnality to automatically the daterange of the dataset to use, as we only have one dataset for kraken; and for now we want to test Gekkoga on the whole dataset. Later on when we know Gekkoka works, you will be able to change that in order to to reduce the dataset and reflect the testing strategy I explained before.

daterange: ‘scan’,

/*
daterange: {
from: ‘2018-01-01 00:00’,
to: ‘2018-02-01 00:00’
},
*/

Now we update our balance and fees.

simulationBalance: {
‘asset’: 0,
‘currency’: 100 //note that I changed this since initial confs in other posts
},

slippage: 0.05,
feeTaker: 0.16,
feeMaker: 0.26,
feeUsing: ‘taker’, // maker || taker

The apiURL should be OK.

apiUrl: ‘http://localhost:3000’,

We won’t change the standard populationAmt, variation, mutateElements, minSharpe or mainObjective.

parallelqueries needs to be updated to reflect your CPU configuration, as it is the number of parallel backtests Gekkoga will be able to launch. the more CPU you have, the better it is. But align this on your number of CPU. If you have 4 CPU or vCPUs, use 3 or 4. With 4, your whole CPU capacity will be filled by Gekkoga, it could make your computer almost unusable for other tasks while Gekkoga is running (and your CPU fan will start to make noise). If you have a dedicated Gekoga computer this is fine, if you don’t, this may be a problem so consider a lower value. It’s up to you.

In my case,

• On my regular laptop, I have 2 CPUs but 4 seen by the OS thanks to hyper threading, so I’ll use 3 as I wan’t one CPU to be available for other tasks;
• At the time I’m writing this article, I tried to run Gekkoga on an Amazon EC2 t2.micro with this setting to 1, I lost control on the VM and had to restart it;
• For this test I will launch it on my Intel NUC VM, powered with 2 vCPU, but I’ll keep the setting to 1 to not stress it too much as a NUC is not designed for intensive CPU computations (I’m afraid the fan won’t cool enough the case & CPU).

parallelqueries: 1,

I don’t use emails notifications for now so I leave it to false.

Now we enter the interesting part. We will explain Gekkoga all the parameters we need our Strat to be filled with, and their values. If you enter a fixed value, Gekkoga will use this value all the time, in every backtest. It won’t change it. But we can define:

• Some ranges, by using arrays, eg. [5,10,15,30,60,120,240]
• Randomized values, by using functions as randomExt.integer(max, min) or randomExt.float(max, min).toFixed(2)

Did you notice the .toFixed(2) ? It forces the randomized float to be rounded to 2 decimals and this is the rounded value which will be used in Gekko’s backtest. But keep in mind that the float will still be generated with a higher number of digits, and 2.0003 or 2.0004 will in both cases be rounded to 2.00. It will lead to duplicate backtests. This is why I recommanded you to use as much integers as possible instead of floats.

First, the candleValues. I’m not really confident about short candles, but as the tool will test it for us, why not. Let’s extend it a little bit and remove a few values.

candleValues: [5,15,30,60,120,240,480,600,720],

Now the Strat parameters …

getProperties: () => ({

historySize: randomExt.integer(50, 0),

short: randomExt.integer(30,5),
long: randomExt.integer(100,15),
signal: randomExt.integer(20,6),

thresholds: {
//up: randomExt.float(20,0).toFixed(2),
//down: randomExt.float(0,-20).toFixed(2),
up: randomExt.integer(400,0)/100,
down: randomExt.integer(0,-400)/100,
persistence: randomExt.integer(9,0),
stoploss: randomExt.integer(50,0),
},

Let’s give it a try …. we will need to carefully check the console information.

gekko@bitbot:~/gekkoga/gekkoga$ node run.js -c config/config-MyMACD-backtester.js
No previous run data, starting from scratch!
Starting GA with epoch populations of 20, running 1 units at a time! node run -c config/config-MyMACD-backtester.js

Woohoo ! It started. Now I stop it and will create a nice PM2 startup script as I want to easily make it run in the background and easily get information on it.

echo #!/bin/bash > start.sh
echo “rm logs/*” >> start.sh
echo “pm2 start run.js –name gekkogaMyMACD –log-date-format=\”YYYY-MM-DD HH:mm Z\” -e logs/err.log -o logs/out.log — -c config/config-MyMACD-backtester.js max_old_space_size=8192 >> start.sh

chmod 755 start.sh

We restart it using this script …

./start.sh

Let’s check its logs …

It’s running in the background, good. Now let’s have a look at Gekko’s UI logs as it is supposed to receive API calls from Gekkoga:

We can see some calls to the backtest API, perfect. Now let’s check other informations, while we are waiting for some first results:

Conclusion: Gekkoga is pushing hard on one of the 2 CPU, this is conform to what we defined in conf. Memory consumption is low.

And finally, 21 minutes later, the first epoch completed:

You can find an explanation of an epoch here. What we see here is the winner of this epoch, and Gekkoga keeps running to compute more and compare them. It logs the best combination found in <gekkoga_installdir>/results in a JSON format, so to display it we will use jq (run ‘apt-get install jq’ as root if you don’t have it yet):

So we see here that the winner for now used a long value of 65, short 29, signal 9, 5mn candlesize, 15 candles history size, an up threshold of 2.44, a down threshold of -2.13, a stoploss of 16% and a persistence of 0. Our Gekkoga config file using only integers is well formated !

Now, the sharpe is very high, but in terms of estimated profits, on this whole dataset we actually performed better (1101%) than the market (817%), with 68 trades.

Now the problem is that it will take a very very .. very … long time to run. so, for now, I don’t have much more to say, we have to wait. At the same time we can continue to work on improving our knowledge about indicators and how they work, and imagine improvements to our Strats.

The only way to optimize the runtime seems to be to make it run on a higher number of CPUs and adapt the parallelqueries setting. I’ve never done that before but it gave me the idea to try to make it run on an Amazon EC2 machine. this will be detailed in another article.