Ошибка no data for testing

Файл fxt с папки tester/history/… не удаляется.

Попробовал запустить без галочки «использовать дату».
Надпись в журнале «no data for testing» исчезла, но появилась новая
картина:

При тестировании на fxt файле, сгенерированном из csv файла тиковой
истории, постоянно идет сообщение «invalid price». Не открывается
ни одной сделки, журнал выглядит так:

При тестировании на fxt файле созданном при помощи обычной истории
минуток все нормально работает. На демо тоже все нормально работает,
никаких «invalid price» нет. Журнал выглядит так:

Чтобы убедится что проблема не в коде эксперта, протестировал
еще одного, который у меня работает болле 2-х месяцес на счете,
и делает 10-25 сделок в сутки. При тестировании на обычной минутной
истории эксперт нормально работал. А при тестировании на fxt
файле сгенерированом при помощи simple_csv2fxt.mq4 эксперт не сделал
ни одной сделки. Хотя по алгоритму эксперта он должен был выставить
отложенный ордер в случае если iOpen(NULL,0,1)!=iClose(NULL,0,1). Создается
впечетление, как будто тики есть, а баров нет, или каждый тик
является отдельным баром. И еще, при запуске с визуализацией
на сгенерированном fxt файле, никакие бары не рисуются, черный
экран, и линия «бид» прыгает в соответствии с тиками.

У меня есть 4 варианта:
1. Я что-то не так делаю.
2. У меня какой-то глючный файл истории.
3. Неправильно работает simple_csv2fxt.mq4
4. Неправильно работает тестер стратегий.

Прикрепляю fxt файл сгенерированный при помощи simple_csv2fxt из файла
csv тиковой истории. Файл csv прикрепить не получается, 6 Мб в заархивированном
виде. Посмотрите пожалуйста, ничего не могу понять. Может у меня
руки, а может и что-то серьезнее.

Спасибо.

---

---

---

---

---

Return to “Bug reports”

Table of contents

• Converting to FXT
  1. When running CSV2FXT, I get the message “Could not open input file”. What should I do?
  2. How can I change the fixed spread in an existing FXT file?
  3. How can I change [some other setting] in an existing FXT file?
  4. I cannot find the resulting FXT and HST files. Where are they?!
  5. Backtesting an EA with my newly created FXT results in Order Send Error 131. Why?
  6. What do I do if I get an alert saying Bad CSV Format when I try to convert the CSV I exported from JForex?
  7. Is it possible to append tick data to an existing FXT file (and the accompanying HST files)?
  8. Is it possible to append tick data to an existing CSV file?
  9. Why does the CSV2FXT script get stuck as soon as it is started? The progress indicator is not displayed on the chart and the only message displayed in the experts log is “The date column appears to be 0. Sample…”
  10. How can I change the timeframe of an existing FXT file?
  11. Can I create range bars or Renko bars with CSV2FXT?
• Using the tick data
  1. Why are my FXT files being overwritten?
  2. Why is my terminal creating an FXT file when I start backtesting?
  3. My backtest runs but I do not get 99% modeling quality. Why?
  4. My backtest stops as soon as I click the start button. The backtesting journal tab displays TestGenerator: internal error because the file not opened. What is wrong?
  5. Why does my terminal crash as soon as I press “start” in the backtest user interface?
  6. Why do I get a red bar at the end of the backtest even though my modeling quality is 99%?
  7. What can I do if my optimization results window is empty even though MT4 appears to run multiple passes correctly?
  8. I am getting empty results (no trades) in all my optimization passes when using the Walk Forward Analyzer. Is there a workaround?
  9. Why are all the volumes in my tick data backtest smaller than 1?
  10. Why are all the volumes in my tick data backtest huge numbers?
  11. Why does my backtest fail as soon as I attempt to start it with the backtest journal saying No data for testing.?
  12. My backtest stops somewhere in the middle, even though the FXT file I created spans a longer time period. How can I fix this?
  13. How come my backtest results differ from an optimization cycle results if I am using the same parameters?

Converting to FXT

1. When running CSV2FXT, I get the message “Could not open input file”. What should I do?

That can only be caused by the script trying to open a file that is not there so you either copied the CSV to the wrong folder, named it incorrectly or typed its name incorrectly. Check the following things:

• You are using the correct CSV2FXT version. v0.43 and below are only suited for MT4 builds 509 and below. You must use a newer CSV2FXT with MT4 builds 545 or higher.
• You put the CSV file in the correct folder (expertsfiles for MT4 build 509 or lower respectively MQL4Files for MT4 builds 545+).
• If using MT4 build 509 or earlier, you put the CSV file in the correct MT4 installation folder (if you have more than one).
• If using MT4 build 545+, you put the CSV file in the correct MT4 data folder (head to File -> Open Data Folder to find out where it is).
• If you left the CsvFile field blank, check that the file is named as your pair and has a csv extension.
• If you typed something in the CsvFile field, make sure that you typed the full file name including its extension.
• If your explorer is configured to hide the file extension for known files, your file might be named EURUSD.CSV.CSV and explorer might just show EURUSD.CSV, or your file might be correctly named EURUSD.CSV and your explorer might just show EURUSD. In Windows 7, to work around this issue you can hit ALT in an explorer window, click Tools -> Folder options -> View then uncheck “Hide extensions for known file types”. The procedure is similar for other Windows versions.
• If you are using Vista with MT4 build 509 or earlier and your MT4 installation is in Program Files, I suggest making a copy somewhere else (for instance on your Desktop) and working from that one instead. Vista has something called UAC that can be a royal pain in the ass. You can also disable UAC if you wish, read a few paragraphs below on how to do that.

2. How can I change the fixed spread in an existing FXT file?

Use a hex editor (e.g. WinHex, Neo Hex Editor) and change it at offset 0xFC. Note that it’s going to be displayed in hexadecimal so you’ll probably have to use a calculator.

3. How can I change [some other setting] in an existing FXT file?

Look up the offset in FXTHeader.mqh, it’s displayed next to each setting. Use a hex editor. Note that all values are in little endian format, meaning that if you have a value that exceeds one byte you have to fill in the bytes to its right. For instance, if you want to write 300 in the file you will have to write it as 2C 01.

4. I cannot find the resulting FXT and HST files. Where are they?!

For MT4 builds 509 or earlier, this is typical for Windows 7 / Windows Vista users that run MT4 from their Program Files folder. The issue is caused by the fact that UAC is enabled and these operating systems use folder virtualization. For Windows 7, the resulting files can be found in c:ProgramData typically, while for Windows Vista they can be found in c:UsersusernameAppDataLocalVirtualStoreProgram Files. If they’re not there, just search for *.FXT, they must be somewhere. To get around this problem, either copy your MT4 folder in a location that’s not protected by UAC (e.g. your desktop) or simply disable UAC (type UAC in the start->search box thing and follow the on-screen process).

For MT4 build 545 or later, head to File -> Open Data Folder and look in MQL4Files.

5. Backtesting an EA with my newly created FXT results in Order Send Error 131. Why?

This might be caused by the lack of a MIN_LOT and LOT_STEP for the symbol in the resulted FXT file. This happens when generating the FXT file with a MT4 client that wasn’t connected to the broker when the script was started. As specified above, your terminal should be connected to your broker when generating an FXT file.

6. What do I do if I get an alert saying Bad CSV Format when I try to convert the CSV I exported from JForex?

This is in all likelihood because you either failed to select a delimiter or selected a dot (.) as the delimiter, so the solution is easy: you go back to JForex, select comma (,) as the delimiter and export the CSV once more. This should be much faster than the first time since the data is cached. If you’re uncertain about the other parameters, take another look at the JForex downloading guide.

7. Is it possible to append tick data to an existing FXT file (and the accompanying HST files)?

No.

8. Is it possible to append tick data to an existing CSV file?

Yes. The CSV2FXT script will disregard duplicate information and even the JForex header line if it is encountered in the middle of the CSV file.

If you are using JForex, you should start your CSV from the time your older CSV ends. You can delete the header row from the new CSV file before concatenating or you can simply skip this step, in which case the CSV2FXT script will skip that line and warn you about it in the experts log. When the export is finished, simply append the new CSV to the older one.

If you are using the PHP scripts, it is ideal to stop your processing at the end of a month. The processing script automatically appends to an existing file so if you resume from the start of the next month your CSV will be perfect. However, you can stop your processing even during the month; for example, if you have a file named EURUSD.csv that ends on 19.03.2012 and then append data starting from 01.03.2012 and ending with 02.04.2012 (e.g. by typing php process_dukascopy_data.php EURUSD 201203 201205 EURUSD.csv), the new data will be appended to your existing CSV and the data between 01.03 and 19.03 will be in your CSV twice. In such a case, the CSV2FXT script will output about 20 errors in your log and spit out an alert about older ticks, but it is smart enough to skip the duplicated data so the resulting FXT should be perfectly fine (although it will take a little bit longer because it has to skip the duplicated period).

9. Why does the CSV2FXT script get stuck as soon as it is started? The progress indicator is not displayed on the chart and the only message displayed in the experts log is “The date column appears to be 0. Sample…”

Most likely this is because you exported the CSV file using JForex but failed to select the comma (,) as the field delimiter as mentioned in the guide. Since for some reason that I fail to grasp the default JForex setting is to use a space as field separator and put several commas at the end of each line, your CSV file is useless and has to be regenerated. The good news is that JForex caches the data so you don’t have to wait for the download once more.

10. How could I change the timeframe of an existing FXT file?

Two steps:

1. Since the file name contains the timeframe, for instance EURUSD15_0.FXT for an M15 EURUSD FXT file, you have to rename (or copy) the file for the desired timeframe. Assuming you’d want to make it H4, you translate that to minutes (4 hours = 240 minutes) and you rename the file to EURUSD240_0.FXT
2. Next you have to grab a hex editor and edit the file at offset 0xD0. Change the number to the same one used in the file name in the previous step (but make sure you convert it to hex; using our example above, 240 is 0xF0 so you would have to change it from 0x0F – which is 15 – to 0xF0).

One thing to note that if you perform this change is that the bar count displayed in the results is going to be wrong and indicators running on the backtest timeframe will be misleading because the ticks are stored as part of bars and the bars will be for a different timeframe. Essentially, if your EA has any indicators running on the backtest timeframe, this method is not safe to use.

11. Can I create range bars or Renko bars with CSV2FXT?

Not with the version I created. However, Artur from AZ-Invest has developed a version of the script that works with Renko bars.

Using the tick data

1. Why are my FXT files being overwritten?

You probably didn’t start the MT4 terminal via the Tick Data Suite and your FXT files are not read-only for some reason.

2. Why is my terminal creating an FXT file when I start backtesting?

• Did you create the FXT file for the currency pair & timeframe that you are trying to use in your backtest?
• If the answer to the above is yes, did you copy it to the correct location (in recent MT4 builds, head to File -> Open Data Folder to find the right place)?
• Also see the answer for question #1 above.

3. My backtest runs but I do not get 99% modeling quality. Why?

This is in fact the same question as #2 above.

4. My backtest stops as soon as I click the start button. The backtesting journal tab displays TestGenerator: internal error because the file not opened. What is wrong?

That’s MT4’s way of notifying you that it refuses to overwrite an FXT file that is set read-only and it happens with MT4 builds 509 or earlier. You need to close the MT4 terminal and start it via the Tick Data Suite shortcut.

5. Why does my terminal crash as soon as I press “start” in the backtest user interface?

If you’re not using the latest MT4 build, it’s most likely because the EA was compiled with a build newer than yours and Metaquotes did not think it wise to add ex4 versioning. You can tell this is the problem if attaching the same EA to a chart will also result in a crash. The solution is to use the latest build or get the EA compiled with your build.

6. Why do I get a red bar at the end of the backtest even though my modeling quality is 99%?

Because the FXT you are using was created with an older script. It’s nothing to worry about, though – you can safely ignore it.

7. What can I do if my optimization results window is empty even though MT4 appears to run multiple passes correctly?

That means the results you obtained were not profitable. If you wish to see them nonetheless, you can select the Optimization Results tab, right click inside it and deselect “Skip Useless Results”.

8. I am getting empty results (no trades) in all my optimization passes when using the Walk Forward Analyzer. Is there a workaround?

This only happens with some EAs that need the Metatrader 4 terminal to be connected to the broker even when backtesting, for instance Wallstreet Forex Robot or FAP Turbo. To put it simply, since the FXT is already there, the backtest starts too fast. The only solution for this problem is to add an artificial delay, which can be done by using the Configuration program installed with the Tick Data Suite. Just fire up Configuration and increase the Backtest delay factor to something like 3. This will start the backtest with a sensible delay which should be enough for the terminal to connect to the broker. In case it isn’t, feel free to increase it to something like 10. Set it back to 0 once you’re done with that particular EA unless you want to have a permanent delay when backtesting or optimizing with tick data. Note that this option will not have an effect on Metatrader 4 terminals that are already running and you will need to restart these to take advantage of it.

9. Why are all the volumes in my tick data backtest smaller than 1?

It’s because you are using real spread when creating your FXT with MT4 build 509 or earlier. When enabling this option, the spread is stored in the volume field. If your strategy makes use of the volume number (which is simply the number of ticks in MT4), you need to use fixed spread or employ other strategies such as counting the number of ticks and storing it in an array.

10. Why are all the volumes in my tick data backtest huge numbers?

Same reason as for the previous question, but it only applies to MT4 builds 545 or higher. To get the actual volume, simply apply a bitwise AND 0xFFFFFFFF like this:


volume = Volume[0] & 0xFFFFFFFF;


11. Why does my backtest fail as soon as I attempt to start it with the backtest journal saying “No data for testing.”?

This typically happens when you enabled Use date and the start and end dates you selected are outside the range covered by your FXT.

12. My backtest stops somewhere in the middle, even though the FXT file I created spans a longer time period. How can I fix this?

This typically happens when you’re using MT4 build 600 or higher, you created an FXT file larger than 4GB and you forgot to run MT4 via the Tick Data Suite (the MT4 window title bar should say “… with Tick Data Suite vX.XX”).

If this happens and none of the above are the causes, you should first attempt to backtest the MACD EA using your FXT and if it stops at the same point, you should investigate the source CSV in that range and check the log of the CSV2FXT script for any potential errors.

13. How come my backtest results differ from an optimization cycle results if I am using the same parameters?

This only happens when you are using a start/end date for your optimization and you’re using MT4 build 509 or earlier. Sadly, this is a bug in Metatrader 4: only the first pass of an optimization uses the start and end date you selected; subsequent runs use the full range available in the FXT. This bug was fixed starting with MT4 build 545.

In any case, with MT4 builds 509 and earlier, to get the same outcome when backtesting with the parameters for one of the optimization results, you can either make an FXT that spans exactly the period that you want to optimize for, or alternatively you can simply disable Use date when running the backtest for the selected optimization results (if it’s not the parameters for the first run, that is).

I have a very simple test as shown below.
I try to freeze my two dependencies using the AutoDataAttribute + AutoMoqCustomization.

class AutoMoqDataAttribute : AutoDataAttribute
{
    public AutoMoqDataAttribute()
        : base(new Fixture().Customize(new AutoMoqCustomization()))
    { }
}

public class PrBatchEndorseBrokerTest
{
    [Theory, AutoMoqData]
    public void Process_ValidContext_CallsK2Workflows(
        [Frozen]Mock<IK2Datasource> k2,
        [Frozen]Mock<IAppConfiguration> config,
        PrBatchEndorseBroker sut)
    {
        // Arrange
        var data = new Dictionary<string, object>
        {
            ["Workflow"] = @"KLPurchaseRequestPR",
            ["Activity"] = "Endorser",
            ["ViewFormURL"] = "/Form/KLPurchaseRequestApproval.Form",
            ["PositiveOutcome"] = "Endorse",
            ["NegativeOutcome"] = "Reject"
        };

        // Act
        sut.Process();

        // Assert
        k2.Verify(x =>
            x.StartInstance(It.IsAny<string>(),
                            It.Is<Dictionary<string, object>>(d =>
                                data.Keys.All(k => d[k] == data[k])))
            , Times.Once());
    }
}

For some reasons, when i run this test, i’m getting the following error:

System.InvalidOperationException: No data found for BlackBox.Stakhanov.Broker.Test.PrBatchEndorseBrokerTest.Process_ValidContext_CallsK2Workflows

I tried many things and i can’t make it work! I think i’m missing something big and probably obvious!

Packages I’m using:

<package id="AutoFixture" version="3.47.8" targetFramework="net452" />
<package id="AutoFixture.AutoMoq" version="3.47.8" targetFramework="net452" />
<package id="AutoFixture.Xunit" version="3.47.8" targetFramework="net461" />
<package id="Castle.Core" version="3.3.3" targetFramework="net461" />
<package id="Moq" version="4.5.10" targetFramework="net461" />
<package id="xunit" version="2.1.0" targetFramework="net461" />
<package id="xunit.abstractions" version="2.0.0" targetFramework="net452" />
<package id="xunit.assert" version="2.1.0" targetFramework="net461" />
<package id="xunit.core" version="2.1.0" targetFramework="net461" />
<package id="xunit.extensibility.core" version="2.1.0" targetFramework="net461" />
<package id="xunit.extensibility.execution" version="2.1.0" targetFramework="net461" />
<package id="xunit.extensions" version="1.9.0.1566" targetFramework="net461" />

@thejeshgn
I’m running a cron job for a project that fetches the testing data from icmr_testing_status.json at 05:00 every morning.

The job failed yesterday due to missing data for the day. Just wanted to know whether this was a one-off error or do I need to add some data validation to the script.

Appreciate the resource you guys have put together.

Update : Error was in the script.