1.1 Metadata Strategy
The National Statistical System
The National Statistical System (NSS) consists of:
- The Statistical Council (SC);
- The National Statistical Institute - Statistics Portugal (SP).
The Statistical Council (SC) is the state body that supervises and coordinates the National Statistical System. Its duties include:
- "To guarantee the coordination of the National Statistical System, approving the concepts, definitions, nomenclatures and other technical instruments of statistical coordination" (Law 6/ 89 of 15 April - Diário da Republica 1st series no. 88).
This duty is carried out by the "Planning, Coordination and Dissemination" Standing Section (PCDSS), which has the power:
- "...to analyse and approve concepts, definitions, nomenclatures and other technical instruments of statistical coordination of the National Statistical System and to approve regular changes to these documents resulting from work done at the EU or national level"(Structure and Functioning of Statistical Council - SC Deliberation No. 286; 2005).
The job of the Statistics Portugal is to record, refine, coordinate and disseminate official data while taking into account the general guidelines laid down by the Statistical Council. It may also delegate these duties to other public departments, called delegated bodies.
SP has the responsibility to conceive and manage the statistical metadata system of the NSS, having as presumption that the concepts, classifications and other technical instruments of statistical coordination have to be approved by the SC. The metadata unit coordinates all the work related to the statistical metadata system.
Approval of concepts, classifications and methodological documentation
In these processes exists a strong interaction between the SP and the Statistical Council. SP compels all the information and prepares the documentation that sends to the SC for approval.
The SP centralises the statistical concepts used in its own and the delegated bodies' statistical surveys in a database. These concepts are classified by subject areas and are entered into database with the status of "proposed concept", when they are used for the first time. Groups of new concepts or changes to approved concepts are sent to the SC periodically for analysis and new approval. The SC has working groups by subject area to analyse them and recommend their approval to the PCDSS. After the approval, their status in the database is changed to "SC-approved concept" and, is of obligatory use whenever applicable.
The classifications used in all statistical activity, such as the Portuguese Classification of Economic Activities, National Classification of Occupations, National Classification of Goods and Services, Administrative Division Code and List of Countries are also approved by the SC for mandatory use in the NSS.
In 2005, the SP submitted to appreciation to the PCDSS a standard format for the methodological document for the NSS's statistical surveys because it was considered to be a coordination instrument. The format was approved and adopted as mandatory in the NSS.
By December 2007, 75% of the surveys in the NSS were documented in this format.
Technical approval of surveys
The process for technical approval of surveys, which it is implemented at the SP's level without intervention of the SC, is closely linked to their life cycle and consists of the following stages:
- The preliminary methodological document and the questionnaire(s) produced in the methodological study are sent to the units directly involved or users of the results of surveys, the Planning Unit, the Data Collection Department and the Methodology and Information Systems Department for their opinion.
- At this point in the circuit, the Metadata Unit analyses the correct use of concepts and classifications approved by the SC, ensures correct application of the standard format in the methodological document also approved by the SC, analyses questionnaires, introduces new concepts into the concept base and issues an opinion on the basis of its analysis.
- The department responsible for the survey updates the methodological document and/or the questionnaire(s) with the proposed changes or justifies its rejection to the unit that proposed them, submitting then the new version of the methodological document and questionnaire(s) for approval by the Board.
- The Metadata Unit prepares a memo, on the basis of all the opinions of the different units and respective answers, to send to the Board, proposing their approval or rejection.
- The Board then approves or rejects the survey: if it approves them, the methodological document and the questionnaire(s) become final; if it rejects them, the process starts again.
- In the approval case, the Metadata Unit records the questionnaire(s) in the data collection instruments database, giving them a registration number and publishing the methodological documentation the Intranet and in the Official Statistics website.
Metadata dissemination - Statistics website
Concepts, classifications and methodological documents are available directly on the home page of the Statistics website.
The variables involved and the associated metadata must be recorded, previously, in the variables subsystem (one of the components of the Statistical Metadata System) so that the data on statistical indicators can be made available on the website.
Strategy for the metadata system
Organisations have attributed increasing importance to managing knowledge, as demonstrated by the growing implementation of metadata systems that systematise, standardise and formalise this knowledge so that it can be published within or outside the organisation.
In statistics organisations in particular, the systematisation of metadata is important because can be published along with statistical data so that they can be better understood, because gives to statisticians a clearer idea of the surveys that produce these data and for which they are responsible and, also plays a fundamental role in statistical coordination.
As a result, the aim of metadata systems in statistical organisations must be to support the entire life cycle of surveys and data and, makes sense to talk of a metadata life cycle.
In May 2002, the Metadata Unit submitted a document to the SP Board and Council of Directors laying down the general guidelines governing the NSS's Statistical Metadata System. Both bodies approved the document, which proposed the implementation of a system that:
- Will support surveys from their design to the dissemination of results;
- Will capture metadata for the system, from its origin, only once and with the possibility of being reused in other contexts;
- Consists of subsystems and components, so that they can be implemented in stages, with the possibility of navigating between the different components;
- Has decentralised management by the survey managers, with centralised coordination by the Metadata Unit;
- Allows different national and international bodies to exchange metadata;
- Supports other languages, such as English, in addition to Portuguese.
The implementation of this strategy has been included in medium-term work plans drawn up for the institution. The "General Guidelines on National Statistical Activity and Priorities for 2003-2007" defined as top priority:
- "Implementing an integrated statistical metadata system — In the development of the statistical metadata system organised and coordinated by the SP, it is particularly important to design and implement an integrated classification management system, an integrated concepts management system and an integrated methodological document management system, define a model for a statistical subsystem and create support instruments for their implementation."
- "Promoting the use of the statistical metadata system in the NSS."
- "Improving user access to statistics - ...adapting the metadata system as a tool for accessing available information and making it easy to read and understand..."
Two courses of action in the "General Guidelines on National Statistical Activity for 2008-2012" are devoted to the metadata system:
- "To align the statistical metadata system with best international practices."
- "To render the statistical metadata system appropriate to the needs of the interchange of metadata within the National Statistical System and the European Statistical System."
1.2 Current situation
Currently, the concepts, classifications, variables and data collection instruments subsystems and a prototype for storage of methodological documents have already been implemented. All these subsystems are available on the organisation's intranet. The concepts, classifications, methodological documents and data collection instruments are available on the Official Statistics website. The indicators and associated metadata disseminated on the website are registered in the variables system. Metadata on variables are available in Portuguese and English and the concepts and classifications are currently being translated.
In 2008-2012, the statistical metadata system will be adjusted so that metadata can be added to the data throughout the life cycle of surveys. The following actions are planned:
- A methodological documentation subsystem will be implemented to document any statistical object needing to be documented with any type of report, with the possibility of reusing metadata existing in the system in another context. In the initial phase, this system will cover methodological documents from surveys and reports in the SDMX format. In the second phase, a standardised quality report format will be implemented.
- The concepts subsystem will be reformulated so that the statistical concepts can be organised into conceptual systems and made available according to this criterion.
Implementation of a structure that can be used to organise and relate object classes (statistical units and populations) in the variables subsystem.