Best Practice Guidelines

The EDAP project has defined a set of 'best practice' guidelines that define a framework for the quality assessments it performs, aligned with the principles of QA4EO. These guidelines are intended to provide a description of the high-level principles and activities that are required in quality assessments for all types of Earth Observation missions, since these are largely common between different domains.

At the top level, these guidelines have been drafted to reflect a near 'ideal' scenario, which is hoped to serve as an aspiration to new space providers as well as space agencies. It is understood that many of the missions the project will assess will only partly comply with these requirements (to different degrees). This is acknowledged in the assessment grading system which is intended to primarily test whether aspects of a given mission are 'fit for purpose' within the context of the mission's stated performance and application.

The product evaluation activity they undertake is divided into six sections:

Details
Generation
Quality Flags
Uncertainty Characterisation
Validation
Inter-comparison

These sections are themselves divided into sub-sections, which constitute each of the different aspects of the data product that should be assessed and graded, either as Basic, Intermediate, Good or Excellent.

The information is gathered in a Quality Assurance Report (QAR) and summarised in a product quality evaluation matrix.

**EDAP Product Quality Evaluation Matrix – example**

Product Details

Certain basic descriptive information should be provided with any EO data product, some of which is absolutely necessary for the data to be at all meaningful. Additionally, some of this information, such as claimed measurement quality or resolution, can provide both the data product quality assessor and the reader of their report with a frame of reference from which to set expectations for a given product.

It is therefore required by the EDAP assessment that any EO product provides the following:

Product name
Sensor Name
Sensor Type
Product version number
Product ID
Processing level of product
Measured quantity name
Measured quantity units
Stated measurement quality
Spatial Resolution
Spatial Coverage
Temporal Resolution
Temporal Coverage
Mission coverage

Also recommended is the following (based on INSPIRE metadata):

Point of contact (Responsible organisation, including email address)
Product locator (e.g. URL, DOI if applicable)
Conditions for access and use
Limitation on public access
Product abstract (summary of resource)

**Table 1 – Product Details – Assessment Criteria**

Table 1 shows how a data product's provision of the above information relates to the grade it achieves for this sub-section of the quality assessment.
Grade	Criteria
Not Assessed	Assessment outside of the scope of study
Not Assessable	Relevant information not made available
Basic	Many pieces of important information missing
Good	Some pieces of important information missing
Excellent	Almost all required information available
Ideal	All required information available

Uncertainty Characterisation

Uncertainty values should be provided in EO data products per-pixel, in a manner that describes the pixel error-covariance. Since it is not practical to provide a full error-covariance matrix for an EO data product due to their data volume various approaches have been developed to approximate this. For example, the FIDUCEO project FCDRs contain three components of uncertainty – independent, structured and common – to describe the three typical scales of error correlation.

It is still typical however for uncertainty values to be provided, if at all, on a per-product or, more often, a per-mission basis – losing a great deal of information significant to users.

**Table 2 – Uncertainty Characterisation – Assessment Criteria**

Table 2 shows the EDAP grades missions and the extent of uncertainty information they provide.
Grade	Criteria
Not Assessed	Assessment outside the scope of study
Not Assessable	No uncertainty information provided
Basic	Uncertainty established by limited comparison to measurements by other sensor/s
Good	Limited use of GUM approach, and/or, an expanded comparison to measurements by other sensors. Most important sources of uncertainty are included
Excellent	Full GUM approach is used to estimate measurement uncertainty, all important sources of uncertainty are included. Uncertainty per pixel provided
Ideal	Full GUM approach is used to estimate measurement uncertainty, including a treatment of error-covariance. Per pixel uncertainties in components, e.g., random systematic – as appropriate for the error-correlation structure of the data

Download the Best Practice Guidelines with detailed focus on Optical Missions and a template for the Mission Quality Assessment Report:

For information purposes only, previous versions used by old assessment may also be downloaded:

Version 2.0 of the Best Practice Guidelines provides an updated framework, where the reporting divides the assessment results between two Cal/Val maturity matrices, as follows:

Summary Cal/Val Maturity Matrix
Validation Cal/Val Maturity Matrix

A useful overview of the QA Framework produced described above, which has been further developed alongside NASA, may be found in the below presentation.

ESA/NASA QA Framework for EO Products

Framework for EO Data Quality Assessment

A generalised framework for assessing the fitness of purpose of EO data is difficult to obtain as this would clearly depend on the users intent; however, there are key characteristics that give confidence to a user and allow them to decide if the data is fit for their individual purpose.

In particular, users wish to understand data coverage, data provenance, data characterisation, data availability and also data correctness. Each of these factors, given a different weighting by each user, can ultimately affect a user's opinion of how good the data is.

EDAP aims to address both data integrity and data credibility for the various missions identified, which may go on to become TPMs within ESA's Earthnet Programme.

QA4EO

The Quality Assurance Framework for Earth Observation (QA4EO) was created and endorsed by CEOS explicitly for the purpose of providing guidance on EO quality assurance so that data users are able to trust the data / information they are provided with is fit for purpose and fully interoperable with other datasets from both satellite and in-situ providers.

As there is potential for some commercial missions to become TPM candidates, for the most efficient exploitation of EO data, assessment of data quality, calibration and validation are indispensable tasks and also form the basis for reliable scientific conclusions. It is for this reason that one of the core functions provided by EDAP is focused on data quality, with an emphasis on the adoption of QA4EO guidelines.

Best Practice Guidelines

Overview

EDAP Best Practice Guidelines

Assessment Criteria

Mission Quality Assessment Downloads

Quality Assurance Framework