ATIS - Alliance for Telecommunications Industry Solutions
Scope
"This document describes a test plan for validating objective perceptual quality models in the context of Internet Protocol Television (IPTV) services (see [2]). Specifically@ this document defines: ? The procedure for evaluating the performance of objective perceptual quality models@ hereafter referred to as ""models""@ ""quality models""@ and ""objective models"". It describes the test methods@ the selection of test material and conditions@ and the evaluation metrics to examine the predictive performance of models. ? Performance criteria used for model evaluation (acceptability thresholds for these criteria are not within the scope of this document). ? A methodology for documenting the data and test results in a standardized fashion. In the context of this document@ Quality of Experience (QoE) refers to how an end-user ?C i.e.@ consumer of IPTV ?C would categorize/experience the quality of an IPTV service (see also section 3.2) streamed audio-video content. The performance of objective models will be based on the comparison of the Mean Opinion Score (MOS) obtained from controlled subjective tests and the predicted Mean Opinion Score (MOSp) from the models. The objective of the testing is to examine and document the performance of proposed video or audiovideo quality metrics across representative transmission and display conditions. The following aspects of QoE are currently within the scope of this document: ? Perceptual quality of video@ perceptual quality of the audio@ and perceptual quality of audio and video. ? QoE models for in-service monitoring (i.e.@ end-to-end@ including head-end monitoring) in operational environments. Off-line testing ?C suitable for pre-service and/or planning and network design purposes (i.e.@ laboratory testing) ?C is for further study. The computational complexity of the models is also considered in the evaluation. ? This test plan can be used for no-reference@ reduced-reference@ and full-reference models. ? Models using a packetized transport stream (in the clear or encrypted) and bit-stream as input. Whereas video and audio may be the first aspects that are addressed@ subsequent versions of the test plan will also address issues such as audio-video synchronization (a.k.a.@ ""lip sync"")@ subtitling@ closed captioning@ and transaction QoE indicators ?C such as channel change delay (channel zapping). This version addresses the use of quality models for IPTV services. Related to this document@ a document is needed that describes the various quality models the ATIS IPTV Interoperability Forum (IIF) recommends for in-service monitoring (and other possible) applications of IPTV streams. ATIS-0800031@ IPTV QoE Requirements@ will provide the scope and requirements of such models. This test plan is not exhaustive. It is assumed that the user of this test plan document has knowledge of and uses relevant standards documents and accepted industry practices of subjective and objective testing of media quality algorithms. This document is a complement to existing standards documents related to this field. A list of such documents is provided in section 2. Purpose Objective models offer a practical alternative to subjective measurements. Although data collected from subjective tests remain the most valid method for assessing quality@ formal subjective tests have significant limitations. Most notably@ any single subjective test is expensive and time-consuming to prepare and run. It requires a large number of participants@ and only a small set of test sequences can be assessed in one test. In contrast@ objective models can predict an average non-expert viewer's opinion on a continuous basis. The goal of testing objective models is to examine the performance of proposed models across a wide range of conditions that are representative of operational IPTV environments in terms of codecs@ bitrates@ and transport mechanisms but restricted to those included in the ATIS IIFdefined IPTV architecture (which includes fixed and mobile/wireless environments). Various vendors offer products that include a video (and sometimes also an audio) quality metric that ""predicts"" the human-perceived video quality (and audio quality) in the form of a predicted/estimated MOS. The video quality metric (algorithm) is either developed in-house by the measurement vendor itself@ or the rights to use the algorithm have been purchased from an Original Equipment Manufacturer (OEM) where the metric was developed. Alternatively@ some metrics have been developed in academic environments. Network@ service@ and content providers of IPTV are the most obvious acquirers of measurement products that support these video quality (and audio quality) metrics. However@ there is no standardized means for the users to understand how well these products perform@ unless such a metric has been independently and formally validated. At this time@ there are very few standards in this area@ and thus the solutions available are mostly non-standardized. In addition@ the current practice in the standards is to standardize these models; however@ test plans are not standardized. Typically@ each Standards Development Organization (SDO) creates its own test plan that is@ very likely@ slightly different than another SDO's test plan. However@ utilizing a single standardized test plan for IPTV services allows diverse algorithms to be tested in the same manner@ and allows cross comparisons. This document also presents the test conditions of the ATIS IIF-defined IPTV system conveniently in a single document. Application The following describes a list of uses for this document: ? The criteria described in this document may be used as input to a test process (e.g.@ as used in Video Quality Experts Group [VQEG]@ International Telecommunications Union- Telecommunication Standardization Sector [ITU-T] Study Group [SG]-9@ SG-12@ Video Service Forum [VSF]@ or 3GPP Service and System Aspects [SA] Working Group [WG] 4). Note that this document does not specify a test process@ only a test plan (see ATIS-0800021@ EPSNR Trial-Use Standard). ? This document provides a reporting format that allows users of the validated models to compare ""like""-models (see explanation in section 4). This includes computational complexity as one of the factors. ? This document can be used in conjunction with a ""test process"" (e.g.@ ATIS-0800021@ EPSNR Trial-Use Standard@ which describes a proposed test process for model validation). NOTE ?C This test plan does not negate the standardization of recommended models. The ATIS IIF provides this test plan as the industry standard for the evaluation of IPTV quality models."