METIS

Quick links

GSBPM

Common Metadata Framework

Metadata Case Studies

GSIM

 All METIS pages (click arrow to expand)
Skip to end of metadata
Go to start of metadata

 

3.1 Metadata Classification

The MetaNet Reference ModelTM (Version 2) categorises types of metadata in the following way:

  • Conceptual Metadata describes the basic idea (concept) behind the metadata object e.g. conceptual data elements, classifications, measure units, statistical object types. This type of metadata can be context free (eg the variable 'income' as a concept) or context-related (e.g. 'income' collected in a particular survey).
  • Operational Metadata are the metadata required to view the data from an operational point of view (e.g. record variables, matrix operations, statistical process). This includes all processes and configuration. In other words, the operational metadata is used to explain how the data was created or transformed. Operational Metadata is one of the links between the concepts and the physical data.
  • Quality Metadata are the metadata for a particular instance e.g. response rates, status, weighting, versions. This provides the other link between concepts and physical data. It is worth noting that the processes involved in preparing quality metadata are considered operational metadata.
  • Physical Metadata includes the physical, unique characteristics of the data which cannot be separated eg server locations, data base.

3.2 Metadata used/created at each phase

Phase: Need
Description:

  • Need is an ongoing process to determine the statistical needs of Statistics New Zealand's stakeholders

Description of main developed functionalities:

  • Online Consultation/Submission Tool (Census)?
  • Documentation Storage: Lotus Notes

Metadata used (inputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group 

Description 

Example 

Source (if re-used) 

Conceptual Metadata 

Concepts of interest.  Previously available information. 

Global variables, themes, subject areas, statistical objects types

Disseminate (previous cycle/collection) 

Operational Metadata 

Analyse processes of previous collections/ available data. 

Study, Study Method, Statistical Process 

Process (previous cycle/collection) 

Quality Metadata

Quality of previous collections, or available data 

Data Quality 

Process and Analyse (previous cycle/collection) 

Physical Metadata

Locate available data through physical metadata 

Datasets, server locations, software, access rights 

Disseminate (previous cycle/collection) 

Metadata produced (outputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group 

Description 

Example 

Source (if re-used) 

Conceptual Metadata 

Defined concepts high level 

Global variables, themes, subject areas, statistical objects types 

 

Operational Metadata 

High level strategy for meeting need. 

Business Case, Study Method, Statistical Process

 

Quality Metadata 

Consultation process 

Reports 

 

Physical Metadata

Document storage, submission storage

Locations, References 

 

 

Phase: Develop and Design

Description:

  • Develop and Design describes the research, development and design activities to define the statistical outputs, methodologies, collection instruments, sample, operational processes and end-to-end (E2E) solution

Description of main developed functionalities:

  • Documentation Storage: Lotus Notes

Metadata used (inputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group 

Description 

Example 

Source (if re-used) 

Conceptual Metadata 

Need to refine the concepts to determine the variables to be collected and the population of interest. 

Includes concepts such as object variables, value domains, classifications, measure units, context data elements. 

Need (High level concepts) 

Operational Metadata 

This includes designing the sample methodology, collection methodology and the statistical processes. 
Also need to design quality characteristics and processes. 

Includes concepts such as statistical process, process implementation, data collection methodology. 

Need (strategy) 
Collect/Process (previous collections) 

Quality Metadata

Quality metadata from previous collections may be used in the design of sample and collection methodologies. 

statistical process, process implementation, data collection methodology. 

Collect/Process (previous collections) 

Physical Metadata 

May begin to define/ identify where data will be stored. 
May also require physical metadata from previous collections to determine the best 

software, data sets, access package, 

 

Metadata produced (outputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group 

Description 

Example 

Source (if re-used) 

Conceptual Metadata 

Defined concepts 

Includes concepts such as object variables, value domains, classifications, measure units, context data elements. 

 

Operational Metadata 

Completed end to end design and methodology 

statistical process, process implementation, data collection methodology, business rules, transformations. 
testing plan. 

 

Quality Metadata 

Details of the design process used.  Quality introduced by proposed standards. 

Design process.  Quality measures of standards. 

 

Physical Metadata 

Storage plan.  Document storage 

Software, Access Package, Data Set

 

 

Phase: Build

Description:

  • Build produces the components needed for the end-to-end solution, and tests that the solution works

Description of main developed functionalities:

  • Dashboard - For configuring and monitoring processes
  • Workflow tool - for developing statistical processes
  • Questionnaire Design, CAI tools, Scanning Software - For survey collections
  • CRM, Q-Master (call centre) - For running collections
  • Transformation Tools - imputation, editing, coding, etc.
  • Data Environments - for storage of data (in development)
  • Metadata Environment Components - for storage of metadata (developing high level design)

Metadata used (inputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group 

Description 

Example 

Source (if reused) 

Conceptual Metadata 

May need to refine/adapt concepts due to feedback from testing or errors detected. 

Data Elements, Classifications, Value Domains 

Develop and Design 

Operational Metadata 

Build and configure storage structures, collection instruments, processing requirements.  Includes concepts such as record variables, record types, matrix operations. 

Question, questionnaire, data collection methodology, statistical process. 

Develop and Design 

Quality Metadata 

The final version of a questionnaire will be decided during this stage.  
Some quality metadata may also be used to assess pilot studies and instrument testing. 

Methodology and process design.  Testing plans. 

Develop and Design 

Physical Metadata 

Determine the physical location where the data will be stored 

servers, software, access packages. 

Develop and Design 

Metadata produced (outputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group 

Description 

Example 

Source (if reused) 

Conceptual Metadata 

Finalised concepts 

Data Elements, Classifications, Value Domains, Statistical Unit 

 

Operational Metadata 

End to End components built and tested

Built Components, Processes.  Statistical process, data collection methodology. 

 

Quality Metadata 

Quality metrics about the build process and testing report 

Test reports, validated processes. 

 

Physical Metadata

Complete Application Architecture 

Software, Storage, Access Packages, Access Rights, Versions etc. 

 

 

Phase: Collect

Description:

  • Collect acquires collection data each collection cycle and manages the providers of that data

Description of main developed functionalities:

  • Questionnaire Design, CAI tools, Scanning Software - For survey collections
  • CRM, Q-Master (call centre) - For running collections
  • Data Environments - for storage of data (in development)
  • Metadata Environment Components - for storage of metadata (developing high level design)

Metadata used (inputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group

Description 

Example

Source (if reused)

Conceptual Metadata 

As data is collected it will be allocated against concepts.  Some details about the relevant concepts may be used in respondent management strategies. 

Data Elements, Classifications, Value Domains, Statistical Unit

Develop and Design 

Operational Metadata 

Utilise the collection processes outlined in the operational metadata. 

Data Collection Methodology, Questionnaire, Collection Strategy. 

Devlop and Design, Build 

Quality Metadata

Collect data for each instance of the survey.  Quality metadata will be populated based on the collection instance. 

Collection strategy, Data Collection Methodology, 

Devlop and Design, Build 

Physical Metadata 

Physical datasets will be populated with data at this stage. 

Software, Storage, Access Packages, Access Rights, Versions etc. 

Build 

Metadata produced (outputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group 

Description 

Example 

Source (if reused) 

Quality Metadata 

Operational Processes used.  Quality measures of collection instance. 

Collection Report, Data Quality - response rate, item non response. 

 

Physical Metadata 

Data collected to measure collection concepts. 

Software, Storage, Access Packages, Access Rights, Versions etc. 

 

 

Phase: Process

Description:

  • Process describes cleaning the detailed data records and preparing them for analysisFor each phase provide:

Description of main developed functionalities:

  • Dashboard - For configuring and monitoring processes
  • Workflow tool - for developing statistical processes
  • Transformation Tools - imputation, editing, coding, etc.
  • Data Environments - for storage of data (in development)
  • Metadata Environment Components - for storage of metadata (developing high level design)

Metadata used (inputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group 

Group 

Description

Example

Source (if reused)

Conceptual Metadata 

Conceptual metadata are used to classify and code open responses.  Derive new concepts, aggregate data etc. 

Classification, Correspondence, Data Elements, Classifications, Value Domains, Statistical Unit 

Develop and Design 

Operational Metadata

Further statistical processes are used in order to process the data.  This may include the creation of cubes and registers, aggregating results using derivation rules, applying editing and imputation strategies, applying confidentiality rules, etc. 

Matrix, Cube, Register, Statistical Process, Process Implementation, Operation Implementation, Derivation Rules, Computation Implementation. 

Develop and Design, Build, Collect 

Metadata produced (outputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group

Description 

Example 

Source (if reused) 

Quality Metadata 

Further quality metadata will be populated at this stage based on the processes applied. 

Processing reports, Data Quality - imputation rates, editing rates etc. 

Develop and Design, Collect 

Physical Metadata 

If processed data is stored in different locations, new physical metadata will be defined. 

Storage, Software, Access Package etc. 

Build 

 

Phase: Analyse

Description:

  • Analyse is where the statistics are produced, examined in detail, interpreted, understood and readied for dissemination

Description of main developed functionalities:

  • Analytical Environment (strategy still in development)
  • Information Portal (strategy still in development)
  • Data Environments - for storage of data (in development)
  • Metadata Environment Components - for storage of metadata (developing high level design)

Metadata used (inputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group

Description

Example

Source (if reused)

Conceptual Metadata 

In the analysis stage the processed data will be used to analyse the concepts defined in the need and develop/design stages. 

Data Elements, Classifications, Value Domains, Statistical Unit 

Need, Develop and Design

Operational Metadata 

Further processes may be used to generate tables for analysis.  
Operational metadata will also be used to prepare data for dissemination. 
Operational Metadata may also be used to analyse the data. 

Tables, Statistical Processes, Confidentiality Rules 

Develop and design 

Quality Metadata 

Quality metadata will be used at this stage to assess  how well the data represents the concepts outlined in the needs stage. 

Statistical Activity, Study, Statistical Process. 

Collect, Process 

Physical Metadata 

Physical metadata will be required to locate the data, and any additional data for comparisons. 

Storage, Software, Access Package etc. 

Build, Collect, Process 

Metadata produced (outputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group 

Description 

Example 

Source (if reused)

Quality Metadata

Analysis of quality metadata against concept defined in need stage.

Data Elements, Classifications, Value Domains, Statistical Unit, Statistical Activity, Study, Statistical Process. 

 

Physical Metadata

Produce analysis reports, output products. 

Data Set, Publications 

 

 

Phase: Disseminate

Description:

  • Disseminate manages the release of the statistical products to the customers.

Description of main developed functionalities:

  • Integrated Publishing Environment - Tool for configuring and disseminating analysis.
  • CRM/ Job Tracking Systems - for recording customer usage.
  • Data Environments - for storage of data (in development)
  • Metadata Environment Components - for storage of metadata (developing high level design)

Metadata used (inputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group

Group 

Description 

Example 

Source (if reused) 

Quality Metadata 

Technical audiences and professional audiences may be interested in the quality metadata in order to understand the characteristics of each instance of data collection. 

Analysis Reports, Output Releases, Quality measures/ reports, Needs definition 

Need, Analysis

Physical Metadata 

Physical metadata may be required when locating data in response to customer queries. 

Server locations, access rights, access packages, systems and tools, online catalogues 

Build, Analyse 

Metadata produced (outputs): table (below) with groups of metadata (rather than individual metadata items), examples for each group 

 

 

 

Source (if reused) 

Conceptual Metadata 

Identification of new needs for collection. 

Global variables, themes, subject areas, statistical objects types 

 

Operational Metadata 

Development of new/changed standards for other collections

Study Methods, Statistical Processes 

 

Quality Metadata 

Details of products produced, queries received and products used. 
Long term retention and archiving policies. 

Dissemination process, Usage data

 

Physical Metadata

Output products available for promotion. 

Tables, Reports, Brochures etc. 

 

3.3 Metadata relevant to other business processes

Group 

Description 

Example 

Conceptual Metadata

Information/ records defining organisational concepts, structures etc

Corporate directories, Corporate glossary, Organisational Structure charts

Operational Metadata

Information/ records defining how corporate processes are applied

Corporate policies, Guides, Contracts, Memorandums of understanding (MOU)

Quality Metadata

Information/ records related to specific instances

Invoices, Annual Reports, Budgets

Physical Metadata

Information relating to phyisical item

Catalogues, Asset Management Lists etc.