VQEG的高清电视测试计划.ppt

The VQEG HDTV Test PlanOutlinel lIntroduction: Setting the stagel lHDTV Source Scenes and HRCsl lTest methodologiesl lKey challengesl lProposed VQEG approachl lFinal thoughts2 2HD is no longer the future. It is the present!l l“NBC will upgrade the rest of the network's top-rated “NBC will upgrade the rest of the network's top-rated late-night lineup into HDTV in 2005”late-night lineup into HDTV in 2005”l l“The Fox network is better prepared then ever for its hi-“The Fox network is better prepared then ever for its hi-def broadcast of Super Bowl XXXIX.” def broadcast of Super Bowl XXXIX.” l l“The swanky future of hotels is a big-screen HDTV set.”“The swanky future of hotels is a big-screen HDTV set.”l l“2005 would be the year of high definition video and TV”“2005 would be the year of high definition video and TV”l l“Shipments of High Definition televisions and set-top “Shipments of High Definition televisions and set-top boxes are gaining momentum”boxes are gaining momentum”l l“HDTV content is increasing rapidly”“HDTV content is increasing rapidly”3 3HD studio workflows are mainstream l l“Adobe, HP, Intel, Microsoft, and Dell have joined “Adobe, HP, Intel, Microsoft, and Dell have joined together to deliver a line of open, scalable, certified together to deliver a line of open, scalable, certified desktop HDV and HD solutions to deliver optimal desktop HDV and HD solutions to deliver optimal performance at a price point that is unbeatable today.” performance at a price point that is unbeatable today.” l l“JVC's ProHD initiative represents an economical solution “JVC's ProHD initiative represents an economical solution to getting the stations into the HD production business. to getting the stations into the HD production business. The company has products to help with this transition on The company has products to help with this transition on two fronts.” two fronts.” 4 4Content acquisition is affordablel lPanasonic: The AG-HVX200 DVCPRO HD P2 represents the Panasonic: The AG-HVX200 DVCPRO HD P2 represents the company's first HD solid-state handheld camcorder. The AG-HVX200 company's first HD solid-state handheld camcorder. The AG-HVX200 has two card slots and records in 1080/60i, 30p and 24p; 720/60p, has two card slots and records in 1080/60i, 30p and 24p; 720/60p, 30p and 24p; and DVCPRO50, DVCPRO or DV. It also records 480i 30p and 24p; and DVCPRO50, DVCPRO or DV. It also records 480i and 480p on Mini-DV tape from a built-in drive. AG-HVX200 will and 480p on Mini-DV tape from a built-in drive. AG-HVX200 will come available in the fall for about $5,995. come available in the fall for about $5,995. l lAlec Shapiro, senior vice president of Sony Electronics' Broadcast Alec Shapiro, senior vice president of Sony Electronics' Broadcast and Production Systems Division, touted the acceptance Sony's and Production Systems Division, touted the acceptance Sony's HDC-1000 1080P camera and the XDCAM tapeless acquisition HDC-1000 1080P camera and the XDCAM tapeless acquisition system have found in the industry. system have found in the industry. l lIn the high-end HD production arena, Shapiro noted the Sony's In the high-end HD production arena, Shapiro noted the Sony's HDCAM format is used by virtually 100 percent of the network HDCAM format is used by virtually 100 percent of the network sitcoms produced in the United States.sitcoms produced in the United States. 5 5Viewer adoption is acceleratingl lMaxent (tm), announced the availability of its value-priced, high-Maxent (tm), announced the availability of its value-priced, high-performance 26-inch WXGA LCD Display through Costco Wholesale stores performance 26-inch WXGA LCD Display through Costco Wholesale stores nationwide at a retail price of $1,199. nationwide at a retail price of $1,199. l lHDTVs keep getting more affordable. Olevia's new 32-inch LCD HDTV has a HDTVs keep getting more affordable. Olevia's new 32-inch LCD HDTV has a lot of the right ingredients for a good low-cost HDTV.lot of the right ingredients for a good low-cost HDTV.l l10m homes around the world were watching HDTV programming on a high-10m homes around the world were watching HDTV programming on a high-definition TV set, according to research from market analysts In-Stat. definition TV set, according to research from market analysts In-Stat. l lBy the end of 2005, the worldwide total of these HDTV service households By the end of 2005, the worldwide total of these HDTV service households is projected to reach 15.5m. By 2009, HDTV households worldwide are is projected to reach 15.5m. By 2009, HDTV households worldwide are forecast to reach 52m. forecast to reach 52m. l lHDTV services are currently widely available in five countries: Australia, HDTV services are currently widely available in five countries: Australia, Canada, Japan, the United States, and South Korea. Canada, Japan, the United States, and South Korea. l lNearly half of all consumers plan to make their next television purchase a Nearly half of all consumers plan to make their next television purchase a high-definition (HDTV) television set, according to the CEA. high-definition (HDTV) television set, according to the CEA. 6 6Technology building blocks are in placel lBritish Sky Broadcasting (BSkyB) has selected Thomson as its British Sky Broadcasting (BSkyB) has selected Thomson as its primary HD decoder set top supplier as it prepares to roll out a new primary HD decoder set top supplier as it prepares to roll out a new digital HDTV service in 2006. The company will use H.264-based digital HDTV service in 2006. The company will use H.264-based advanced video coding for its HD service. advanced video coding for its HD service. l lRoyal Philips Electronics today unveiled a High-Definition Multimedia Royal Philips Electronics today unveiled a High-Definition Multimedia Interface™ (HDMI™ ) receiver integrated circuit (IC), the Interface™ (HDMI™ ) receiver integrated circuit (IC), the TDA9975A, to meet consumer demands for superior image quality in TDA9975A, to meet consumer demands for superior image quality in next-generation TVs. next-generation TVs. l lThe single cable digital multimedia solution has arrived. High-The single cable digital multimedia solution has arrived. High-Definition video and audio capability quietly rolled into the NZ home Definition video and audio capability quietly rolled into the NZ home entertainment market in the lead-up to Christmas, with the release entertainment market in the lead-up to Christmas, with the release of a number of DVD players, projectors and TV sets from various of a number of DVD players, projectors and TV sets from various manufacturers featuring the new High-Definition Multimedia manufacturers featuring the new High-Definition Multimedia Interface (HDMIInterface (HDMITMTM) connectivity standard.) connectivity standard.7 7Differentiation based on quality is the next stepl lOnHD.TV has generated international attention for our list of the 10 OnHD.TV has generated international attention for our list of the 10 celebrities who look better or worse in crystal-clear High-Definition celebrities who look better or worse in crystal-clear High-Definition TV. The list has been featured in such publications as TV. The list has been featured in such publications as The Chicago The Chicago TribuneTribune, , The National EnquirerThe National Enquirer, , The New YorkThe New York PostPost and TV shows and TV shows such as such as Inside EditionInside Edition and and The Fox Report With Shepard SmithThe Fox Report With Shepard Smith. . l lDriving much of the company's growth is the worldwide move to HD Driving much of the company's growth is the worldwide move to HD production, and Thomson has products for the acquisition, editing production, and Thomson has products for the acquisition, editing and distribution of both SD and HD content. The top of the and distribution of both SD and HD content. The top of the company's line in terms of image quality is its Viper FilmStream company's line in terms of image quality is its Viper FilmStream digital cinema camera.digital cinema camera.8 8Hence the need for standardized metricsl lTo verify and make credible claimsl lTo objectively validate the promised media experience9 9Outlinel lIntroduction: Setting the stagel lHDTV Source Scenes and HRCsl lTest methodologiesl lKey challengesl lProposed VQEG approachl lFinal thoughts1010HD landscape todayBroadcastHead-endSet-top BoxDTH SatelliteCable DistributionIP TV1111Content acquisitionl lQuality: Consumer ($6K), Professional ($50K), Studio ($110K) cameras in use.l lStorage: HDV, HDCAM, DVCPRO-HD decks are prevalent.l lFrame Rate: 24p, 29.97i, 30i, 59.94p, 60p.l lFrame Size: 1920 x 1080 (1080p), 1920 x 540 (1080i), 1280 x 720 (720p).1212Source Scenes - Copyrightl lCopyright IssuesCopyright Issuesl lPublic DomainPublic Domainl lShot by VQEG Members (put into public domain Shot by VQEG Members (put into public domain or approved for unlimited research use)or approved for unlimited research use)l lPurchased specifically for testing (e.g., VClips)Purchased specifically for testing (e.g., VClips)l lProposal: All scenes must have non-Proposal: All scenes must have non-expiring use clause for VQEG testing and be expiring use clause for VQEG testing and be obtainable by any interested partyobtainable by any interested party1313Source Scenes - Qualityl lFormats?l lCamera/recording quality?l lUncompressed?l lMildly compressed?l lProfessional Consumer (e.g., Sony HVR-Z1U)?l lAllowing some percentage of lower grade material will greatly increase available sources1414Source Scenes - Characteristicsl lMotion?l lDetail?l lColor?l lSize of source scene pool?l lPercent unknown to proponents?1515HRC Characteristicsl lBit-rates?l lCoder/decoders?l lError conditions?l lPre/post processing?l lPercent unknown to proponents?1616Compressionl lTechnologies: MPEG4-AVC, MPEG2, VC-1, DV compression.l lBit rate: 8 Mbps – 25 Mbpsl lRate Control: Variable and constant.l lRate Conversion: 3:2 pull down.1717Transmissionl lTechnologies: Satellite, Cable, IPl lPacket loss: 0-25%1818Decode and display processingl l3:2 pull downl lImage enhancementl lDe-interlacingl lScaling1919Display Conditionsl lShould we test many types of video displays or should we use a high quality “transparent” video display?l lWhat combinations of display types and video formats should be tested?2020Display technologiesl lBroadcast CRTl lRear projectionl lLCDl lPlasmal lProjectors2121HRC vectors at a glancel lContent acquisition:Content acquisition:l lCapture qualityCapture qualityl lStorage formatStorage formatl lFrame rateFrame ratel lFrame sizeFrame sizel lCompression:Compression:l lCoding technologyCoding technologyl lBit rateBit ratel lRate controlRate controll lRate conversionRate conversionl lTransmission:Transmission:l lTechnologyTechnologyl lPacket lossPacket lossl lDecode and display Decode and display Processing:Processing:l l3:2 pull down3:2 pull downl lImage enhancementImage enhancementl lDe-interlacingDe-interlacingl lScalingScalingl lDisplay:Display:l lBroadcast CRTBroadcast CRTl lRear ProjectionRear Projectionl lLCDLCDl lPlasmaPlasmal lProjectorsProjectors2222Outlinel lIntroduction: Setting the stagel lHDTV HRCsl lTest methodologiesl lKey challengesl lProposed VQEG approachl lFinal thoughts2323Test methodsl lSubjective testingl lSequence processing2424Subjective Methodologyl lWhat subjective testing methodology should we use?l lDouble/single stimulus?Double/single stimulus?l lExplicit/hidden reference?Explicit/hidden reference?l lContinuous/discrete assessment?Continuous/discrete assessment?2525Subjective testingl lDouble Stimulus Impairment ScaleDouble Stimulus Impairment Scalel l4 times picture height for all HD formats4 times picture height for all HD formatsl lDifferent display technologiesDifferent display technologiesl l24 subjects per test with:24 subjects per test with:l l20/20 visual acuity with or without eye glasses20/20 visual acuity with or without eye glassesl lNormal color visionNormal color visionl lFamiliarity with the languageFamiliarity with the languagel lNo audioNo audiol lSubjects screened per BT.500Subjects screened per BT.500l lSubjects rejected if failed votes >1 in one Subjects rejected if failed votes >1 in one session.session.2626Sequence processingl l1080p, 1080i and 720p formats.l lNo up-sampling or down-sampling.2727Outlinel lIntroduction: Setting the stagel lHDTV HRCsl lTest methodologiesl lKey challengesl lProposed VQEG approachl lFinal thoughts2828Key Challengesl lHD Contentl lReliable HD playback for subjective testingl lInvestments required for HD testingl lTools for HRC error injectionl lReference display processing algorithmsl lSelection of back channel bandwidth for HD2929Outlinel lIntroduction: Setting the stagel lHDTV HRCsl lTest methodologiesl lKey challengesl lProposed VQEG approachl lFinal thoughts3030Step 1: Subjective testingl lSelect sequences and run a subjective test to get MOS results.l lInclude as many display technologies as possible.l lMake sure a variety of HRCs are used for the test sequences.l lCompile results from multiple labs to get a consolidated score for the sequences.3131Step 2: Evaluation of existing modelsl lRun through existing (already submitted) models through the same sequences.l lFollow procedure and rules for calibration in the test plan.l lInclude full reference, reduced reference and no reference models in the test.3232Step 3: Results analysis and next stepsl lAnalyze objective results to get a correlation to Analyze objective results to get a correlation to the subjective results.the subjective results.l lFollow evaluation metrics in the test plan.Follow evaluation metrics in the test plan.l lDecide on next steps based on the results.Decide on next steps based on the results.l lIf correlations are high, we may have a model for HD If correlations are high, we may have a model for HD TV.TV.l lIf correlations are low, proponents will have a test set If correlations are low, proponents will have a test set to tweak their models.to tweak their models.3333Lab readiness polll lOf the labs represented here:l lHow many of them have HD playback How many of them have HD playback capability?capability?l lOf those who don’t have it now, how many of Of those who don’t have it now, how many of them are planning to have this capability in them are planning to have this capability in the next 3 - 6 months.the next 3 - 6 months.l lHow many labs are willing to run HD How many labs are willing to run HD subjective tests in the next 3 months?subjective tests in the next 3 months?3434Outlinel lIntroduction: Setting the stagel lHDTV HRCsl lTest methodologiesl lKey challengesl lProposed VQEG approachl lFinal thoughts3535Final thoughtsl lHD TV metrics are in urgent need.l lWe should test existing models for their correlations before seeking new ones.l lHD testing is happening now in organizations. A timely standardized metric will have the greatest impact.l lVQEG should complete testing the existing models in the next 6 months.3636Next stepsl lOff line meetings to coordinate HD testing activities.l lDate for next meeting?Date for next meeting?l lAction items captured so far:3737Calibration of model inputsl lPeak video level +/- 10%Peak video level +/- 10%l lBlock level +/- 10%Block level +/- 10%l lHorizontal shift +/- 1 pixelHorizontal shift +/- 1 pixell lVertical shift +/- 1 lineVertical shift +/- 1 linel lHorizontal cropping < 30 pixelsHorizontal cropping < 30 pixelsl lVertical cropping < 20 linesVertical cropping < 20 linesl lNo horizontal or vertical re-scalingNo horizontal or vertical re-scalingl lTemporal alignment +/- 1 frame excluding Temporal alignment +/- 1 frame excluding dropped/repeated framesdropped/repeated framesl lNo visible chroma differential timingNo visible chroma differential timingl lNo visible picture jitterNo visible picture jitter3838Model evaluation metricsl lAttributes:l lPrediction accuracyPrediction accuracyl lPrediction monotonicityPrediction monotonicityl lPrediction consistencyPrediction consistencyl lMetrics:l lRoot mean square errorRoot mean square errorl lPearson linear correlationPearson linear correlationl lOutlier ratioOutlier ratio3939Results analysisl lILG is responsible4040。