Removing Extraneous Bars from BPI in Index-Lab

Harapa

#1

11/23/2015 8:23 PM

When I create BPI with the help of Index-Lab, one such index , BPNYCS (derived from NYSE Stocks), have bars that coincide with US holidays. The extra bars only begin to show after 2002 and nothing before. These extra bars are either zero or 100 ( mostly) resulting in up/dn spikes in the middle of data series. Surprisingly, no such bars are seen on BPSPX (derived from SP500 stocks). In addition, all these holidays are listed in Market Manager. How can I remove these extra bars programmatically from BPNYCS.?

BPNYCS.png

BPSPX.png

Eugene

#2

11/24/2015 1:56 AM

QUOTE:
In addition, all these holidays are listed in Market Manager.

Market Manager does not apply to Fidelity data because the provider does not support it. ~~Try Yahoo! data instead.~~

Harapa

#3

11/24/2015 3:33 AM

Above occurs with "Yahoo" data as I only wish to use dividend adjusted data.

Eugene

#4

11/24/2015 4:15 AM

The MM configuration doesn't apply automatically to any given symbol. As per documentation you need to add all affected symbols to the MM's market group to enable it.

Harapa

#5

11/24/2015 6:31 AM

Added all 1700 symbols in Market Manager.
Deleted old BPNYCS (settings as well as data)
Restarted WL
Created BPNYCS afresh in Index-Lab.
Plot output same as before.

Eugene

#6

11/24/2015 6:55 AM

Maybe it's some symbols that stopped trading, or maybe their data is corrupt. How can this be reproduced -- what are the symbols?
Does this happen with Fidelity or Google data?

Harapa

#7

11/24/2015 7:50 AM

QUOTE:
Maybe it's some symbols that stopped trading,

Per 'Data Tool' all symbols are up-to-date

QUOTE:
or maybe their data is corrupt]

Corrupt Data ? no easy way to know...except visually inspecting each chart...

QUOTE:
How can this be reproduced -- what are the symbols?

QUOTE:
A AA AAC AAMC AAN AAP AAT ABBV ABC ABG ABM ABT ACC ACCO ACE ACM ACN ACRE ACW ADC ADK ADM ADPT ADS ADT AE AEE AEL AEO AEP AES AET AF AFG AFL AGCO AGM AGN AGO AGX AHC AHH AHL AHP AHS AHT AI AIG AIN AINC AIR AIT AIV AIZ AJG AJRD AJX AKR AKS AL ALB ALE ALEX ALG ALJ ALK ALL ALLE ALLY ALR ALSN ALTV ALX AMBR AMC AME AMG AMH AMP AMPE AMRC AMT AMTD AMTG AN ANET ANF ANH ANTM AOI AON AOS AP APA APAM APC APD APH APLE APT APTS AR ARC ARE ARG ARI ARMK ARO AROC ARPI ARR ARW ASB ASC ASGN ASH ASPN AST AT ATEN ATI ATNM ATO ATR ATU ATW AVA AVB AVD AVG AVP AVT AVX AVY AWH AWI AWK AWR AXE AXL AXLL AXN AXON AXP AXR AXS AXTA AYI AYR AZO AZZ B BA BAC BAH BANC BAS BAX BBG BBT BBW BBX BBY BC BCC BCEI BCO BCR BCRH BDC BDN BDX BEE BEL BEN BERY BETR BF-A BFAM BF-B BFS BG BGC BGG BGS BH BHB BHE BHI BHLB BID BIG BIO BK BKD BKE BKFS BKH BKS BKU BLD BLK BLL BLOX BLT BLX BMI BMR BMS BMY BNED BNK BOH BOOT BOX BPI BR BRC BRG BRK-B BRO BRS BRSS BRT BRX BSX BTN BTU BTX BURL BVX BW BWA BWXT BXC BXLT BXP BXS BYD BZC BZH C CAA CAB CABO CACI CAG CAH CAI CAL CALX CAM CAS CAT CATO CB CBB CBG CBI CBK CBL CBM CBPX CBR CBS CBT CBU CBZ CC CCC CCE CCF CCG CCI CCK CCL CCO CCP CCS CDE CDI CDR CE CEB CF CFG CFI CFR CFX CGA CGI CHCT CHD CHE CHGG CHH CHK CHMI CHMT CHS CHSP CI CIA CIE CIEN CIM CIO CIR CIT CIVI CJES CKH CKP CL CLC CLD CLDT CLF CLGX CLH CLI CLNY CLR CLW CLX CMA CMC CMG CMI CMN CMO CMP CMS CMT CNA CNC CNK CNL CNO CNP CNS CNX COF COG COH COL COO COP COR CORR COTY CPA CPB CPE CPF CPGX CPK CPN CPS CPT CR CRC CRCM CRD-B CRI CRK CRL CRM CRMD CRR CRS CRY CSC CSG CSH CSL CSLT CSS CST CSU CSV CSX CTB CTL CTLT CTO CTS CTT CUB CUBE CUBI CUDA CUZ CVA CVC CVEO CVG CVI CVO CVRS CVS CVT CVU CVX CW CWEI CWT CXO CXP CXW CYH CYS CYT D DAKP DAL DAN DAR DATA DBD DCI DCO DCT DD DDD DDR DDS DE DEA DECK DEI DEL DF DFS DFT DG DGI DGX DHI DHR DHT DHX DIN DIS DK DKS DLA DLB DLPH DLR DLX DMD DNB DNOW DNR DO DOC DOOR DOV DOW DPLO DPS DPZ DRE DRH DRI DRII DRQ DST DSW DTE DUK DV DVA DVD DVN DW DWRE DX DY DYN EARN EAT EBF EBS ECL ECOM ECR ED EDE EDR EE EFX EGAS EGL EGN EGP EGY EIG EIX EL ELLI ELS ELY EMAN EMC EME EMG EMN EMR ENH ENR ENS ENSV ENV ENVA ENZ EOG EOX EPAM EPC EPE EPM EPR EQC EQR EQT EQY ERA ERN EROS ES ESE ESI ESL ESNT ESRT ESS ESTE ESV ETH ETM ETN ETR EV EVC EVDY EVER EVH EVHC EVR EVRI EVTC EW EXAM EXAR EXC EXP EXPR EXR F FAC FAF FBC FBHS FBP FC FCB FCE-A FCF FCH FCN FCX FDP FDS FDX FE FET FF FFG FGL FHN FI FICO FII FIS FIT FIX FL FLO FLOW FLR FLS FLT FLTX FMC FMD FMSA FN FNB FNF FNFV FOE FOR FPI FPO FR FRC FRD FRM FRO FRT FSB FSL FSP FSS FTI FTK FUL FVE FXCM G GAS GB GBL GBX GCAP GCI GCO GD GDDY GDOT GDP GE GEF GEN GEO GES GFF GGG GGP GHC GHL GHM GI GIG GIMO GIS GKOS GLF GLOB GLOG GLPW GLT GLW GM GME GMED GMT GNC GNE GNRC GNRT GNW GORO GOV GPC GPI GPK GPN GPS GPT GPX GRA GRB GRC GRUB GS GSAT GSB GST GTN GTS GTT GTY GV GVA GWB GWR GWRE GWW GXP H HAE HAL HAR HASI HBI HCA HCHC HCI HCN HCP HD HE HEI HEI-A HELI HES HF HFC HGG HGR HHC HHS HI HIFR HIG HII HIL HIVE HIW HK HL HLF HLI HLS HLT HLTH HLX HMN HNI HNR HNT HOG HON HOS HOT HOV HP HPP HPQ HPT HPY HR HRB HRC HRG HRL HRS HRTG HSC HST HSY HT HTA HTH HTS HTZ HUB-B HUBS HUM HUN HVT HW HXL HY HYH HZN HZO I IBM IBP ICD ICE IDA IDI IDT IEC IEX IFF IGT IHC IHS IL IM IMAX IMH IMN IMPR IMPV IMS INFU INGR INN INT INTT INUV INVN IO IP IPG IPHI IPI IR IRC IRET IRM IRT ISH ISR IT ITC ITG ITI ITT ITW IVC IVR IVZ JAH JBL JBT JCAP JCI JCP JEC JGW JLL JMG JMI JNJ JNPR JNS JOE JONE JOY JPM JW-A JWN K KAI KAMN KAR KATE KBH KBR KCG KEG KEM KEX KEY KEYS KFY KIM KING KKD KMB KMG KMI KMPR KMT KMX KN KND KNL KNX KO KODK KOP KORS KOS KR KRA KRC KRG KRO KS KSS KSU KW KWR L LAD LADR LAZ LB LBMH LBY LC LCI LDL LDOS LDR LEA LEAF LEE LEG LEN LEN-B LF LG LGF LH LHO LII LL LLL LLY LM LMT LNC LNG LNKD LNN LNT LOCK LOV LOW LPG LPI LPT LPX LQ LRN LTC LTS LUB LUK LUV LVLT LVS LXFT LXK LXP LXU LYB LYV LZB M MA MAA MAC MAN MAS MATX MBI MC MCD MCF MCK MCO MCRN MCS MCY MD MDC MDGN MDLY MDP MDR MDT MDU MED MEG MEI MET MFA MG MGM MHFI MHK MHO MIC MITT MJN MKC MKL MLI MLM MLR MMC MMI MMM MMS MN MNI MNK MNR MO MOD MODN MOG-A MOH MON MORE MOS MOV MPC MPG MPO MPW MPX MRC MRIN MRK MRO MS MSA MSCI MSG MSGN MSI MSL MSM MSO MTB MTD MTDR MTG MTH MTN MTOR MTRN MTW MTX MTZ MUR MUSA MW MWA MWW MXL MXPT MYCC MYE N NADL NAO NAT NAV NAVB NBHC NBL NBR NC NCI NCR NCS NE NEE NEFF NEM NES NEU NEWM NEWR NFG NFX NGS NGVC NHC NHI NI NJR NKE NL NLS NLSN NLY NM NMBL NNA NNI NNN NNVC NOC NOG NOR NOV NOW NP NPK NPO NPTN NR NRF NRG NRZ NSA NSAM NSC NSM NSP NSR NUE NUS NVR NVRO NVTA NWE NWHM NWL NWN NWY NX NXRT NYCB NYLD NYRT NYT O OA OAKS OAS OB OC OCN ODC OFC OFG OGE OGS OHI OI OII OIS OKE OLN OLP OMAM OMC OME OMI OMN ONDK ONE ONVO OOMA OPK OPWR OPY ORA ORC ORCL ORI ORM ORN OSK OUT OXM OXY P PAG PAH PANW PAR PARR PAY PAYC PB PBF PBH PBI PBY PBYI PCG PCL PCP PDM PE PEB PEG PEI PEP PES PF PFE PFG PFNX PFS PFSI PG PGEM PGI PGN PGND PGR PGRE PH PHH PHM PHX PII PIP PIR PJC PKD PKE PKG PKI PKY PLD PLNT PLOW PLT PM PMC PMT PN PNC PNM PNR PNW PNX PNY POL POM POR POST POWR PPG PPL PPS PQ PRA PRE PRGO PRI PRK PRLB PRO PRTY PRU PSA PSB PSG PSX PVA PVH PWR PX PXD PZN Q QEP QTM QTS QTWO QUAD QUOT R RAD RAI RAS RATE RAX RBC RCAP RCL RDC RDN RE REED REG REI REN RES RESI REV REX REXR RF RGA RGC RGR RGS RH RHI RHP RHT RJF RKUS RL RLD RLGT RLGY RLH RLI RLJ RM RMAX RMD RNDY RNG RNR ROG ROK ROL ROP ROX RPAI RPM RPT RRC RRTS RS RSE RSG RSO RSPP RST RT RTEC RTN RUBI RVP RWC RWT RXN RYAM RYI RYN S SAH SAIC SALT SAM SB SBH SBY SC SCCO SCG SCHW SCI SCL SCS SCX SD SDRL SE SEAS SEB SEE SEM SEMG SERV SF SFE SFG SFL SFS SFY SGA SGM SGY SHAK SHO SHW SIF SIG SIR SIX SJI SJM SJW SKT SKX SLB SLCA SLG SLH SLI SM SMG SMP SN SNA SNH SNI SNOW SNR SNV SNX SO SON SPA SPB SPG SPN SPR SPXC SRC SRE SRI SRT SSD SSE SSI SSNI SSP SSS SSTK STAG STAR STAY STC STE STI STJ STL STNG STOR STR STRP STS STT STWD STZ SUI SUM SUNE SUP SVU SWAY SWC SWFT SWI SWK SWM SWN SWX SXC SXI SXT SYA SYF SYK SYN SYRG SYX SYY T TAHO TAL TAP TAT TBI TCB TCO TCPI TCS TDC TDG TDOC TDS TDW TDY TE TEN TER TEX TFX TG TGH TGI TGNA TGT THC THG THO THR THS TIF TIME TIS TISI TJX TK TKR TLN TLYS TMH TMHC TMK TMO TMP TMST TNC TNET TNK TOL TOWR TPC TPH TPLM TPRE TPUB TPX TR TRC TRCO TREC TREX TRGP TRI TRK TRMR TRN TRNO TROX TRR TRU TRUP TRV TRXC TSE TSN TSO TSQ TSS TTC TTI TUMI TUP TVPT TWC TWI TWO TWTR TWX TXMD TXT TXTR TYC TYL UA UAL UAM UAMY UBA UCP UDR UE UEC UFI UFS UGI UHS UHT UIL UIS UMH UNF UNH UNM UNP UNT UNVR UPL UPS URI USB USG USM USNA USPH UTI UTL UTX UUUU UVE UVV V VAC VAL VAR VC VCRA VEC VEEV VER VFC VG VGR VHC VHI VISI VLO VLY VMC VMEM VMI VMW VNCE VNO VNTV VOYA VPG VR VRTV VSH VSI VSLR VSR VSTO VTR VVC VVI VZ W WAB WAC WAGE WAIR WAL WAT WBC WBS WCC WCG WCIC WCN WD WDAY WDR WEC WEX WFC WFT WG WGL WGO WHG WHR WK WLH WLK WLL WM WMB WMC WMK WMS WMT WNC WNR WOR WPC WPG WPP WPX WR WRB WRE WRI WRK WSM WSO WSR WST WTI WTM WTR WTS WTW WU WWAV WWE WWW WY WYN WYY X XCO XEC XEL XHR XL XOM XON XOXO XPO XRM XRX XTLY XXII XYL Y YDKN YELP YUM YUME ZAYO ZBH ZEN ZFC ZOES ZTS

QUOTE:
Does this happen with Fidelity (Google etc) data?

Can be looked at. But will not offer a solution due to lack of dividend adjusted price from both vendors. So low priority to investigate.

Size:

Color:

[quote]Maybe it's some symbols that stopped trading,[/quote]
Per 'Data Tool' all symbols are up-to-date 
[quote] or maybe their data is corrupt][/quote]
Corrupt Data ? no easy way to know...except visually inspecting each chart...

[quote]How can this be reproduced -- what are the symbols?[/quote]
[quote]A AA AAC AAMC AAN AAP AAT ABBV ABC ABG ABM ABT ACC ACCO ACE ACM ACN ACRE ACW ADC ADK ADM ADPT ADS ADT AE AEE AEL AEO AEP AES AET AF AFG AFL AGCO AGM AGN AGO AGX AHC AHH AHL AHP AHS AHT AI AIG AIN AINC AIR AIT AIV AIZ AJG AJRD AJX AKR AKS AL ALB ALE ALEX ALG ALJ ALK ALL ALLE ALLY ALR ALSN ALTV ALX AMBR AMC AME AMG AMH AMP AMPE AMRC AMT AMTD AMTG AN ANET ANF ANH ANTM AOI AON AOS AP APA APAM APC APD APH APLE APT APTS AR ARC ARE ARG ARI ARMK ARO AROC ARPI ARR ARW ASB ASC ASGN ASH ASPN AST AT ATEN ATI ATNM ATO ATR ATU ATW AVA AVB AVD AVG AVP AVT AVX AVY AWH AWI AWK AWR AXE AXL AXLL AXN AXON AXP AXR AXS AXTA AYI AYR AZO AZZ B BA BAC BAH BANC BAS BAX BBG BBT BBW BBX BBY BC BCC BCEI BCO BCR BCRH BDC BDN BDX BEE BEL BEN BERY BETR BF-A BFAM BF-B BFS BG BGC BGG BGS BH BHB BHE BHI BHLB BID BIG BIO BK BKD BKE BKFS BKH BKS BKU BLD BLK BLL BLOX BLT BLX BMI BMR BMS BMY BNED BNK BOH BOOT BOX BPI BR BRC BRG BRK-B BRO BRS BRSS BRT BRX BSX BTN BTU BTX BURL BVX BW BWA BWXT BXC BXLT BXP BXS BYD BZC BZH C CAA CAB CABO CACI CAG CAH CAI CAL CALX CAM CAS CAT CATO CB CBB CBG CBI CBK CBL CBM CBPX CBR CBS CBT CBU CBZ CC CCC CCE CCF CCG CCI CCK CCL CCO CCP CCS CDE CDI CDR CE CEB CF CFG CFI CFR CFX CGA CGI CHCT CHD CHE CHGG CHH CHK CHMI CHMT CHS CHSP CI CIA CIE CIEN CIM CIO CIR CIT CIVI CJES CKH CKP CL CLC CLD CLDT CLF CLGX CLH CLI CLNY CLR CLW CLX CMA CMC CMG CMI CMN CMO CMP CMS CMT CNA CNC CNK CNL CNO CNP CNS CNX COF COG COH COL COO COP COR CORR COTY CPA CPB CPE CPF CPGX CPK CPN CPS CPT CR CRC CRCM CRD-B CRI CRK CRL CRM CRMD CRR CRS CRY CSC CSG CSH CSL CSLT CSS CST CSU CSV CSX CTB CTL CTLT CTO CTS CTT CUB CUBE CUBI CUDA CUZ CVA CVC CVEO CVG CVI CVO CVRS CVS CVT CVU CVX CW CWEI CWT CXO CXP CXW CYH CYS CYT D DAKP DAL DAN DAR DATA DBD DCI DCO DCT DD DDD DDR DDS DE DEA DECK DEI DEL DF DFS DFT DG DGI DGX DHI DHR DHT DHX DIN DIS DK DKS DLA DLB DLPH DLR DLX DMD DNB DNOW DNR DO DOC DOOR DOV DOW DPLO DPS DPZ DRE DRH DRI DRII DRQ DST DSW DTE DUK DV DVA DVD DVN DW DWRE DX DY DYN EARN EAT EBF EBS ECL ECOM ECR ED EDE EDR EE EFX EGAS EGL EGN EGP EGY EIG EIX EL ELLI ELS ELY EMAN EMC EME EMG EMN EMR ENH ENR ENS ENSV ENV ENVA ENZ EOG EOX EPAM EPC EPE EPM EPR EQC EQR EQT EQY ERA ERN EROS ES ESE ESI ESL ESNT ESRT ESS ESTE ESV ETH ETM ETN ETR EV EVC EVDY EVER EVH EVHC EVR EVRI EVTC EW EXAM EXAR EXC EXP EXPR EXR F FAC FAF FBC FBHS FBP FC FCB FCE-A FCF FCH FCN FCX FDP FDS FDX FE FET FF FFG FGL FHN FI FICO FII FIS FIT FIX FL FLO FLOW FLR FLS FLT FLTX FMC FMD FMSA FN FNB FNF FNFV FOE FOR FPI FPO FR FRC FRD FRM FRO FRT FSB FSL FSP FSS FTI FTK FUL FVE FXCM G GAS GB GBL GBX GCAP GCI GCO GD GDDY GDOT GDP GE GEF GEN GEO GES GFF GGG GGP GHC GHL GHM GI GIG GIMO GIS GKOS GLF GLOB GLOG GLPW GLT GLW GM GME GMED GMT GNC GNE GNRC GNRT GNW GORO GOV GPC GPI GPK GPN GPS GPT GPX GRA GRB GRC GRUB GS GSAT GSB GST GTN GTS GTT GTY GV GVA GWB GWR GWRE GWW GXP H HAE HAL HAR HASI HBI HCA HCHC HCI HCN HCP HD HE HEI HEI-A HELI HES HF HFC HGG HGR HHC HHS HI HIFR HIG HII HIL HIVE HIW HK HL HLF HLI HLS HLT HLTH HLX HMN HNI HNR HNT HOG HON HOS HOT HOV HP HPP HPQ HPT HPY HR HRB HRC HRG HRL HRS HRTG HSC HST HSY HT HTA HTH HTS HTZ HUB-B HUBS HUM HUN HVT HW HXL HY HYH HZN HZO I IBM IBP ICD ICE IDA IDI IDT IEC IEX IFF IGT IHC IHS IL IM IMAX IMH IMN IMPR IMPV IMS INFU INGR INN INT INTT INUV INVN IO IP IPG IPHI IPI IR IRC IRET IRM IRT ISH ISR IT ITC ITG ITI ITT ITW IVC IVR IVZ JAH JBL JBT JCAP JCI JCP JEC JGW JLL JMG JMI JNJ JNPR JNS JOE JONE JOY JPM JW-A JWN K KAI KAMN KAR KATE KBH KBR KCG KEG KEM KEX KEY KEYS KFY KIM KING KKD KMB KMG KMI KMPR KMT KMX KN KND KNL KNX KO KODK KOP KORS KOS KR KRA KRC KRG KRO KS KSS KSU KW KWR L LAD LADR LAZ LB LBMH LBY LC LCI LDL LDOS LDR LEA LEAF LEE LEG LEN LEN-B LF LG LGF LH LHO LII LL LLL LLY LM LMT LNC LNG LNKD LNN LNT LOCK LOV LOW LPG LPI LPT LPX LQ LRN LTC LTS LUB LUK LUV LVLT LVS LXFT LXK LXP LXU LYB LYV LZB M MA MAA MAC MAN MAS MATX MBI MC MCD MCF MCK MCO MCRN MCS MCY MD MDC MDGN MDLY MDP MDR MDT MDU MED MEG MEI MET MFA MG MGM MHFI MHK MHO MIC MITT MJN MKC MKL MLI MLM MLR MMC MMI MMM MMS MN MNI MNK MNR MO MOD MODN MOG-A MOH MON MORE MOS MOV MPC MPG MPO MPW MPX MRC MRIN MRK MRO MS MSA MSCI MSG MSGN MSI MSL MSM MSO MTB MTD MTDR MTG MTH MTN MTOR MTRN MTW MTX MTZ MUR MUSA MW MWA MWW MXL MXPT MYCC MYE N NADL NAO NAT NAV NAVB NBHC NBL NBR NC NCI NCR NCS NE NEE NEFF NEM NES NEU NEWM NEWR NFG NFX NGS NGVC NHC NHI NI NJR NKE NL NLS NLSN NLY NM NMBL NNA NNI NNN NNVC NOC NOG NOR NOV NOW NP NPK NPO NPTN NR NRF NRG NRZ NSA NSAM NSC NSM NSP NSR NUE NUS NVR NVRO NVTA NWE NWHM NWL NWN NWY NX NXRT NYCB NYLD NYRT NYT O OA OAKS OAS OB OC OCN ODC OFC OFG OGE OGS OHI OI OII OIS OKE OLN OLP OMAM OMC OME OMI OMN ONDK ONE ONVO OOMA OPK OPWR OPY ORA ORC ORCL ORI ORM ORN OSK OUT OXM OXY P PAG PAH PANW PAR PARR PAY PAYC PB PBF PBH PBI PBY PBYI PCG PCL PCP PDM PE PEB PEG PEI PEP PES PF PFE PFG PFNX PFS PFSI PG PGEM PGI PGN PGND PGR PGRE PH PHH PHM PHX PII PIP PIR PJC PKD PKE PKG PKI PKY PLD PLNT PLOW PLT PM PMC PMT PN PNC PNM PNR PNW PNX PNY POL POM POR POST POWR PPG PPL PPS PQ PRA PRE PRGO PRI PRK PRLB PRO PRTY PRU PSA PSB PSG PSX PVA PVH PWR PX PXD PZN Q QEP QTM QTS QTWO QUAD QUOT R RAD RAI RAS RATE RAX RBC RCAP RCL RDC RDN RE REED REG REI REN RES RESI REV REX REXR RF RGA RGC RGR RGS RH RHI RHP RHT RJF RKUS RL RLD RLGT RLGY RLH RLI RLJ RM RMAX RMD RNDY RNG RNR ROG ROK ROL ROP ROX RPAI RPM RPT RRC RRTS RS RSE RSG RSO RSPP RST RT RTEC RTN RUBI RVP RWC RWT RXN RYAM RYI RYN S SAH SAIC SALT SAM SB SBH SBY SC SCCO SCG SCHW SCI SCL SCS SCX SD SDRL SE SEAS SEB SEE SEM SEMG SERV SF SFE SFG SFL SFS SFY SGA SGM SGY SHAK SHO SHW SIF SIG SIR SIX SJI SJM SJW SKT SKX SLB SLCA SLG SLH SLI SM SMG SMP SN SNA SNH SNI SNOW SNR SNV SNX SO SON SPA SPB SPG SPN SPR SPXC SRC SRE SRI SRT SSD SSE SSI SSNI SSP SSS SSTK STAG STAR STAY STC STE STI STJ STL STNG STOR STR STRP STS STT STWD STZ SUI SUM SUNE SUP SVU SWAY SWC SWFT SWI SWK SWM SWN SWX SXC SXI SXT SYA SYF SYK SYN SYRG SYX SYY T TAHO TAL TAP TAT TBI TCB TCO TCPI TCS TDC TDG TDOC TDS TDW TDY TE TEN TER TEX TFX TG TGH TGI TGNA TGT THC THG THO THR THS TIF TIME TIS TISI TJX TK TKR TLN TLYS TMH TMHC TMK TMO TMP TMST TNC TNET TNK TOL TOWR TPC TPH TPLM TPRE TPUB TPX TR TRC TRCO TREC TREX TRGP TRI TRK TRMR TRN TRNO TROX TRR TRU TRUP TRV TRXC TSE TSN TSO TSQ TSS TTC TTI TUMI TUP TVPT TWC TWI TWO TWTR TWX TXMD TXT TXTR TYC TYL UA UAL UAM UAMY UBA UCP UDR UE UEC UFI UFS UGI UHS UHT UIL UIS UMH UNF UNH UNM UNP UNT UNVR UPL UPS URI USB USG USM USNA USPH UTI UTL UTX UUUU UVE UVV V VAC VAL VAR VC VCRA VEC VEEV VER VFC VG VGR VHC VHI VISI VLO VLY VMC VMEM VMI VMW VNCE VNO VNTV VOYA VPG VR VRTV VSH VSI VSLR VSR VSTO VTR VVC VVI VZ W WAB WAC WAGE WAIR WAL WAT WBC WBS WCC WCG WCIC WCN WD WDAY WDR WEC WEX WFC WFT WG WGL WGO WHG WHR WK WLH WLK WLL WM WMB WMC WMK WMS WMT WNC WNR WOR WPC WPG WPP WPX WR WRB WRE WRI WRK WSM WSO WSR WST WTI WTM WTR WTS WTW WU WWAV WWE WWW WY WYN WYY X XCO XEC XEL XHR XL XOM XON XOXO XPO XRM XRX XTLY XXII XYL Y YDKN YELP YUM YUME ZAYO ZBH ZEN ZFC ZOES ZTS [/quote]

[quote]Does this happen with Fidelity (Google etc) data? [/quote]
Can be looked at. But will not offer a solution due to lack of dividend adjusted price from both vendors. So low priority to investigate.

Harapa

#8

11/24/2015 10:11 AM

Here is a curated output of BPNYCS after applying "Bars =Synchronize()" to eliminate extraneous bars.

bpi-c.png

Eugene

#9

11/24/2015 10:43 AM

Glad that you were able to find a workaround on your own.

Like I suspected, these bars are not "extraneous". Problem is, your Y! DataSet contains a lot of symbols with bars falling on holidays and so I-L merely reflects the actual deficiency in the data.

Unfortunately, above I was creating an incorrect impression that Yahoo! is compatible with the Market Manager. It is not, ~~so you can not use the Market Manager's features on Yahoo DataSets.~~ UPDATE 2017/11/02: By now the Yahoo! provider has long received the Market Manager support.

Sadly, the Data Tool isn't capable of detecting holiday bars. Given some limitations it'd be impractical to add such feature for it would be excruciatingly slow. For now you can write a simple Strategy that checks if the current date is contained in the list and then delete those bars manually:

CODE:
Please log in to see this code.

AE 27.09.1985
AIT 27.09.1985
AJG 27.09.1985
AON 27.09.1985
ASB 27.09.1985
ATO 27.09.1985
AVY 27.09.1985
AZZ 27.09.1985
B 27.09.1985
BAX 26.11.1981
BBY 27.09.1985
BEN 27.09.1985
BF-B 27.09.1985
BMI 27.09.1985
BRC 27.09.1985
CACI 27.09.1985
CAS 27.09.1985
CBT 27.09.1985
CVEO 26.05.2014
DNOW 26.05.2014
GAS 27.09.1985
GRB 17.01.2011
GRB 24.11.2011
GRB 26.12.2011
GRB 02.01.2012
GRB 16.01.2012
GRB 20.02.2012
GRB 06.04.2012
GRB 28.05.2012
GRB 04.07.2012
GRB 03.09.2012
GRB 29.10.2012
GRB 30.10.2012
GRB 22.11.2012
GRB 25.12.2012
GRB 01.01.2013
GRB 21.01.2013
GRB 18.02.2013
GRB 29.03.2013
GRB 27.05.2013
GRB 04.07.2013
GRB 02.09.2013
GRB 28.11.2013
GRB 25.12.2013
GRB 01.01.2014
GRB 20.01.2014
GRB 17.02.2014
GRB 18.04.2014
GRB 26.05.2014
GRB 04.07.2014
GRB 01.09.2014
GRB 27.11.2014
GRB 25.12.2014
GRB 01.01.2015
GRB 19.01.2015
GRB 16.02.2015
GRB 03.04.2015
GRB 25.05.2015
GRB 03.07.2015
GRB 07.09.2015
HRTG 26.05.2014
KATE 27.09.1985
PE 26.05.2014
RWC 27.09.1985
TAP 27.09.1985
TIME 26.05.2014
WOR 12.09.2001
WRB 27.09.1985

Size:

Color:

Glad that you were able to find a workaround on your own.

Like I suspected, these bars are not "extraneous". Problem is, your Y! DataSet contains a lot of symbols with bars falling on holidays and so I-L merely reflects the actual deficiency in the data.

Unfortunately, above I was creating an incorrect impression that Yahoo! is compatible with the Market Manager. It is not, [s]so [b][u]you can not use the Market Manager's features on Yahoo DataSets[/u][/b].[/s] [color=#ff00bf]UPDATE 2017/11/02: By now the Yahoo! provider has long received the Market Manager support.[/color]

Sadly, the Data Tool isn't capable of detecting holiday bars. Given some limitations it'd be impractical to add such feature for it would be [i]excruciatingly[/i] slow. For now you can write a simple Strategy that checks if the current date is contained in the list and then delete those bars manually:

[code]if ( Bars.MarketInfo.Holidays.Contains(Date[bar]) )[/code]

AE	27.09.1985
AIT	27.09.1985
AJG	27.09.1985
AON	27.09.1985
ASB	27.09.1985
ATO	27.09.1985
AVY	27.09.1985
AZZ	27.09.1985
B	27.09.1985
BAX	26.11.1981
BBY	27.09.1985
BEN	27.09.1985
BF-B	27.09.1985
BMI	27.09.1985
BRC	27.09.1985
CACI	27.09.1985
CAS	27.09.1985
CBT	27.09.1985
CVEO	26.05.2014
DNOW	26.05.2014
GAS	27.09.1985
GRB	17.01.2011
GRB	24.11.2011
GRB	26.12.2011
GRB	02.01.2012
GRB	16.01.2012
GRB	20.02.2012
GRB	06.04.2012
GRB	28.05.2012
GRB	04.07.2012
GRB	03.09.2012
GRB	29.10.2012
GRB	30.10.2012
GRB	22.11.2012
GRB	25.12.2012
GRB	01.01.2013
GRB	21.01.2013
GRB	18.02.2013
GRB	29.03.2013
GRB	27.05.2013
GRB	04.07.2013
GRB	02.09.2013
GRB	28.11.2013
GRB	25.12.2013
GRB	01.01.2014
GRB	20.01.2014
GRB	17.02.2014
GRB	18.04.2014
GRB	26.05.2014
GRB	04.07.2014
GRB	01.09.2014
GRB	27.11.2014
GRB	25.12.2014
GRB	01.01.2015
GRB	19.01.2015
GRB	16.02.2015
GRB	03.04.2015
GRB	25.05.2015
GRB	03.07.2015
GRB	07.09.2015
HRTG	26.05.2014
KATE	27.09.1985
PE	26.05.2014
RWC	27.09.1985
TAP	27.09.1985
TIME	26.05.2014
WOR	12.09.2001
WRB	27.09.1985

Harapa

#10

11/24/2015 11:14 AM

Thanks for looking into this.

CODE:
Please log in to see this code.

Could you share the full snippet that pulled the symbols and dates. As fixing the past history is one time affair, provided you know what to fix, I would like to have this for the future. Thanks.

Eugene

#11

11/24/2015 11:32 AM

I don't have it any more since it was an on-the-fly code. Look up PrintDebug in the QuickRef.

Harapa

#12

11/24/2015 11:37 AM

No problem. I will look into it.

Eugene

#13

11/25/2015 10:14 AM

We may consider adding Market Manager support to a future version of the Yahoo! provider to filter out holiday bars automatically.

Harapa

#14

11/27/2015 7:00 AM

That would help.

Edit: Above behavior is seen with many other Index_Lab outputs (with this data set).

Eugene

#15

12/2/2015 6:29 AM

QUOTE:
Edit: Above behavior is seen with many other Index_Lab outputs (with this data set).

Sure, you have to reload them after curating the Yahoo! DataSet.

Harapa

#16

12/2/2015 9:27 AM

"Bars =Synchronize()"
is the cure until an alternate becomes available.

Eugene

#17

12/20/2015 1:45 AM

For the fix, look for Yahoo! provider 2015.12 late December or early January.

Note: For it to work, any Y! symbols you're using in the I-L have to be added to a Market Manager group (don't forget to click "Apply symbol change").

Harapa

#18

12/23/2015 9:12 AM

Looking forward to...

Eugene

#19

1/1/2016 3:31 AM

Yahoo! 2015.12 is out.

Harapa

#20

1/4/2016 7:53 AM

The issue is solved after installing Yahoo! 2015.12.
Thanks.

WEALTHPRO25

#21

11/2/2017 12:13 AM

i have created bpi for NYSE data set and i am missing months on the chart, i reloaded the chart, i updated NYSE dataset, i am attaching the chart, please help get all data on the chart, thanks

bpi.png

Eugene

#22

11/2/2017 6:26 AM

Nothing to help with: the chart looks OK to me.

Harapa

#23

11/8/2017 1:24 AM

@WEALTHPRO25
BPI is very sensitive to missing quotes, extraneous bars, delisted symbols, all of which, in my experience, can result in premature truncation. In addition symbols with < 100 bars are also a problem. Maintaining BPI is a royal pain, IMHO.

Using Yahoo date I found that most truncations can be linked to "improper" stock split event. Checking 2000+ stocks (in NYSE) one by one is not an easy task.