Unreliable results using FT for testing with standard data

Sources of data to import to the program
Message
Author
maurits2000
Posts: 16
Joined: Sun Apr 17, 2011 9:50 am

Unreliable results using FT for testing with standard data

#1 Postby maurits2000 » Thu Sep 08, 2011 3:49 am

I want to strongly suggest that you change the standard data that you offer with FT, since like you do it now it gives un-reliable testing conclusions with almost every strategy.

Why is that? Because now every real day (trading starting in New Zealand until closing in New York) is presented by you over 2 days! The time zone you use shifts the data of the New Zealand trading to the "next" day. This makes a. conclusions in general unreliable and b. especially those which rely on closing and opening day prices and c. you create problems for yourself: you are stuck with a few hours of data that you represent on Sunday . You try to solve that with an option to ignore these "sunday" data-sets. Ignore?!
Mind you, this misrepresenting of NZ trading happens every day. It is not only a Sunday bar issue.

A few examples of what goes wrong now:
- all moving averages of days are wrongly calculated
- Inside Day strategy:(an Inside day is a day of which the High is lower than the previous's day High and the Low is also higher than the previous day Low). Almost all "Sundays" are now Inside Days! Fake Inside Days!
- Reversal Day: (the opposite of an Inside Day; higher High and lower Low than the previous day). Because of your "Sundays" now most Mondays become reversal days!
- Channel Break Outs: since you present a few trading hours on the wrong day, a Channel Break Out can just - or just not occur because of this movement

- etc etc

Your option of "ignoring weekends" when downloading data is NOT a solution. You can not throw away or ignore market data.


What I suggest you should do to solve this issue?
Trading starts in New Zealand as we all know. Closing of New York is widely regarded as trading day closing. Between their times you have e.g. London, Tokyo, Hong Kong, Singapore and Sydney. These financial centers together represent about 99% of world forex trading and should always remain together in one day frame.
Since NZ to NY cover 19 hours of 24 hours time zones, the "left over hours" after New York closing should be added "in front of" the next day. At the end of the week you add the friday night after NY closing to monday morning. No Sundays, no ignoring of data.

This should be done BY YOU when you get the data set from your supplier and not be left to customers who will maybe not always know and understand all the time zones used and the consequences they have for their testing results.
So the option to shift time zones like you have is nice, but only for the individual who has good reasons to want to change them.

I really hope you make this change on the data soon since as long as this is not done your customers will draw unreliable conclusions using FT and its standard data.
Maurits

maurits2000
Posts: 16
Joined: Sun Apr 17, 2011 9:50 am

#2 Postby maurits2000 » Thu Sep 08, 2011 11:46 am

Just retested with the "ignore weekend" option a few strategy/currency pair combinations.

The influence of this matter is so big that some previously highly profitable combinations now (with the ignore option checked) are losing.
Others need a total different set of variables (stop-loss, SMA etc) to perform at best.
Maurits

FT Support
Posts: 905
Joined: Sat Jul 11, 2009 10:54 am

#3 Postby FT Support » Fri Sep 09, 2011 12:10 am

Hi Maurits,

Hmm, we just provide data in GMT time zone, like it goes on many brokers, if you do not like this time zone then just use "Shift time" setting in Data Center. Note that you will need to re-generate ticks after that. With this parameter you can move different sessions inside or outside 1 day.
Check our other product here:
http://www.forexcopier.com

maurits2000
Posts: 16
Joined: Sun Apr 17, 2011 9:50 am

#4 Postby maurits2000 » Fri Sep 09, 2011 4:32 am

This is the most easy answer that you could give.
But it is also a non-answer because

a. you data from brokers whose purpose is trading, not forex testing

b. many of your customers will not see the problem, until they dig deep into the testing and the data used

c. your sunday data bars still are there or are "thrown away"

The purpose of the software that you sell is to test Forex. Any comment that (positively) critizises your product should be taken seriously by you. Investigated and if found true, then acted upon.
If found untrue then come with solid arguments why it is untrue.

That is what a company does when it takes it customers seriously.
Maurits

FT Support
Posts: 905
Joined: Sat Jul 11, 2009 10:54 am

#5 Postby FT Support » Tue Sep 13, 2011 7:36 am

Sorry Maurits,

But we cannot understand. Each forex broker works in his own time zone and actually in our program you will get the same default time zone as in many brokers (for example FXCM). So what is the problem?
And you can adjust time zone to be convenient for you by shifting data by a few hours.

Did you try data shifting?
If yes then why shifting data does not help?
Check our other product here:
http://www.forexcopier.com

dackjaniels
Posts: 151
Joined: Tue Feb 24, 2009 1:03 pm

#6 Postby dackjaniels » Tue Sep 13, 2011 2:10 pm

Hi,

I didn't start this thread but I do have some comments...

1.
Many brokers use their local time for MT4 and in many cases the offset from GMT changes due to Daylight Savings Time.
For instance, the broker Trading Point in Cyprus uses GMT+3 in summer and GMT+2 in winter, as a result, on its MT4 platform there are never any weekend bars, only Monday to Friday.
What this means is that if we use the free Forexite (GMT) data and simply shift it by +2 or +3 hours either the summer or winter data will be incorrect by 1 hour. This also results in weekend bars for approx 6 months of the year (these weekend bars really only contain 1 minute of data but this 1 minute causes a weekend bar to be created in FT).

2.
Even when using a data source such as Alpari UK, which is already collected in alignment with Daylight Saving Time, when a shift (of +2) is applied there are still stray (erroneous) ticks that result in weekend bars being created.

Both of these situations can cause issues when running strategies based on daily bars designed for use on a broker that does not display any 'weekend' bars.

So far the only possible solution I have found is to:

1) Import data such as Alpari UK (which includes the DST change mid-year)
2) Shift it by +2hrs then export it to a file
3) Re-import into FT with 'Ignore Weekends' checked (with no Shift Time applied)

This process applies the correct time shift and then removes those erroneous weekend bars but is time consuming and impractical for lots of currency pairs and regular updates.

The only solution I see is to apply the following functions in this order:

1. Add an 'Apply DST' checkbox option to time-shift the Forexite data on import which will adjust the data in line with DST (so the result is GMT+1 in summer and GMT in winter).
2. Add a time shift option that works permanently on import. So it could be set to +2. (The data would now be GMT+3 in summer and GMT+2 in winter).
3. Finally apply the Ignore weekends option so any stray ticks are removed and no erroneous weekend bars are created.

The final result would be data which is GMT+3 in summer and GMT+2 in winter, with no weekend bars displayed, which is in line with the Cyprus based Trading Point broker.
Of course these options could be varied to create data comparable to almost any broker on the planet. As many brokers do not provide historical data this would be the next best option in my opinion.

Steve

User avatar
Makai
Posts: 18
Joined: Fri May 11, 2012 5:14 am

#7 Postby Makai » Mon Aug 20, 2012 6:07 am

This thread leaves me with more questions than answers.
Is this a problem? How can I test this?
I don`t understand what the weekend bars are about and how the data gets out of sync, if that is a correct description.

raybackjoe
Posts: 1
Joined: Sat Mar 02, 2013 12:47 pm
Location: Moyle Abbey, Ballitore, Co. Kildare

#8 Postby raybackjoe » Sat Mar 02, 2013 12:49 pm

Just retested with the "ignore weekend" option a few strategy/currency set combos.
The impact of the particular matter is really so big that some previously definitely profitable combinations today (with the disregard selection checked) are really losing.
Joe Rayback

ramack
Posts: 19
Joined: Sat Mar 09, 2013 1:38 pm

#9 Postby ramack » Fri Sep 06, 2013 12:25 pm

A solution to generating NY close, 5 day/week candlesticks is extremely important for my back testing. I need to have daily bars/sticks to represent the trading week beginning at the opening of the Monday NZ/AU session running until Friday of the NY close.

Yes, the free Forexite data produces similar charts as my FXCM charts and although there is a gap shown every Monday I can deal with that. But I need more currency pairs than is freely available. I have access to FXCM's data but when importing FXCM M1 data into FT, this imported data is not displayed correctly.

I had considered purchasing VIP data, but I have access to FXCM data and should be able to use it and am skeptical that the paid data from the other brokers would be any different.

FXCM's charting package MarketScope, has an option to "Merge weekend data in to Monday". Something similar to this in FT would be very helpful.

I am nearly in a position that if I can not figure out a method to represent trading daily in NY close, 5 day/week data, FT has lost it's value to me and my business.

FX Helper
Posts: 1477
Joined: Mon Apr 01, 2013 3:55 am

#10 Postby FX Helper » Mon Sep 09, 2013 9:51 am

Hello,

What problems do you have with imported FXCM historical data? Please give more details.

ramack
Posts: 19
Joined: Sat Mar 09, 2013 1:38 pm

#11 Postby ramack » Mon Sep 09, 2013 4:02 pm

I'm going to start a new thread for my issue, so don't completely hijack this thread, if I haven't already. I'll come back with the thread link.

http://forextester.com/forum/viewtopic.php?t=3877

VVizlan
Posts: 1
Joined: Sun Sep 20, 2015 10:10 pm

Re: Unreliable results using FT for testing with standard da

#12 Postby VVizlan » Sun Sep 20, 2015 10:12 pm

So, was this issue ever addressed?

wowzers
Posts: 44
Joined: Sun Jun 16, 2013 4:35 pm

Re: Unreliable results using FT for testing with standard da

#13 Postby wowzers » Sat Sep 26, 2015 2:02 pm

So basically this renders FT2 useless.

FX Helper
Posts: 1477
Joined: Mon Apr 01, 2013 3:55 am

Re: Unreliable results using FT for testing with standard da

#14 Postby FX Helper » Mon Sep 28, 2015 5:15 am

Hello,

Why it makes our program useless?
Just shift the time of data to necessary time zone, re-generate ticks, start a new test and you will have data in the time zone of your broker without weekend bars.

tt1987
Posts: 1
Joined: Fri Jan 22, 2016 9:23 pm
Contact:

Re: Unreliable results using FT for testing with standard da

#15 Postby tt1987 » Fri Jan 22, 2016 9:35 pm

Hi,

I didn't start this thread but I do have some comments...

1.
Many brokers use their local time for MT4 and in many cases the offset from GMT changes due to Daylight Savings Time.
For instance, the broker Trading Point in Cyprus uses GMT+3 in summer and GMT+2 in winter, as a result, on its MT4 platform there are never any weekend bars, only Monday to Friday.
What this means is that if we use the free Forexite (GMT) data and simply shift it by +2 or +3 hours either the summer or winter data will be incorrect by 1 hour. This also results in weekend bars for approx 6 months of the year (these weekend bars really only contain 1 minute of data but this 1 minute causes a weekend bar to be created in FT).

2.
Even when using a data source such as Alpari UK, which is already collected in alignment with Daylight Saving Time, when a shift (of +2) is applied there are still stray (erroneous) ticks that result in weekend bars being created.

Both of these situations can cause issues when running strategies based on daily bars designed for use on a broker that does not display any 'weekend' bars.

So far the only possible solution I have found is to:

1) Import data such as Alpari UK (which includes the DST change mid-year)
2) Shift it by +2hrs then export it to a file
3) Re-import into FT with 'Ignore Weekends' checked (with no Shift Time applied)

This process applies the correct time shift and then removes those erroneous weekend bars but is time consuming and impractical for lots of currency pairs and regular updates.

The only solution I see is to apply the following functions in this order:

1. Add an 'Apply DST' checkbox option to time-shift the Forexite data on import which will adjust the data in line with DST (so the result is GMT+1 in summer and GMT in winter).
2. Add a time shift option that works permanently on import. So it could be set to +2. (The data would now be GMT+3 in summer and GMT+2 in winter).
3. Finally apply the Ignore weekends option so any stray ticks are removed and no erroneous weekend bars are created.

The final result would be data which is GMT+3 in summer and GMT+2 in winter, with no weekend bars displayed, which is in line with the Cyprus based Trading Point broker.
Of course these options could be varied to create data comparable to almost any broker on the planet. As many brokers do not provide historical data this would be the next best option in my opinion.

Steve


thank dackjaniels
I will follow your way
[u=https://suadiennuochanoi.vn/10/sua-may-bom-nuoc]sua dieu hoa[/u]

hansi2k
Posts: 5
Joined: Tue Sep 04, 2012 5:27 am

Re: Unreliable results using FT for testing with standard data

#16 Postby hansi2k » Tue Mar 15, 2016 10:44 am

There is also an easier way:

Download tickstory (http://tickstory.com/) , use this tool to download Dukascopy data, export it while using "NY Trading hours" as GMT shift, and export it directly in FT2 format.


Return to “Data sources”

Who is online

Users browsing this forum: No registered users and 33 guests