2.1 Statistical survey life cycle
SMS is an integral part of SIS.
The main goal of the SMS is to support the statistical business process (collection, production and dissemination of statistical information).
The following scheme demonstrates the structure of the process:
Main functions of individual phases of SBP:
Assessment and evaluation of users' requirements in link with already existing statistical tasks. Consideration of capacity requirements, etc. This phase includes an evaluation by subject-matter departments and methodology department. It also includes decision-making by the top management of the CZSO. It results in approval or disapproval of users´ requirements.
\ Definition of statistical task (DESIGN)
In case of a new ST, a specification of the content, organisation and technology of the ST is made. This includes the proposal of task concept, definition of statistical variables, definition of statistical samples, outputs, definition of statistical questionnaires, other input data sets (from administrative data sources, from other statistical tasks), calculation description over the data (imputation methods, aggregations, calculation of derived statistical variables), time-table for ST preparation and implementation, etc.
In case of an existing ST, its update should be prepared.
This phase is in the responsibility of subject-matter department and methodology department.
Preparation of data processing (BUILD)
Namely the following activities should be made:
- selection of samples,
- preparation and distribution of questionnaires (incl. electronic questionnaires),
- training of interviewers and staff responsible for individual phases of production,
- getting data sets from administrative and other external data sources.
This phase is in the responsibility of data processing department and subject-matter department.
Data collection (COLLECT)
Namely the following activities are included:
- collection of questionnaires and input data capture,
- data validation,
- use of data from other data sources (incl. data validation),
- imputation and creation of input data sets for further processing.
This phase is in the responsibility of data processing department and subject- matter departments.
Data processing (PROCESS)
Namely the following activities are included:
- building of input database,
- imputation of missing records,
- grossing up to statistical population,
- processing of aggregates,
- adjustment for seasonal effects,
- database update.
This phase is in the responsibility of data processing department in cooperation with subject-matter departments.
Data analysis and output production (ANALYSE)
Namely the following activities are included:
- application of mathematical and statistical methods,
- processing of required outputs,
- data quality assessment,
- approval of data for publication,
- public database update.
This phase is in the responsibility of subject-matter departments.
This phase focuses on the following forms of data dissemination:
- web pages,
- public database,
- printed publications,
- electronic outputs,
- ad hoc outputs.
- users´satisfaction surveys,
- analyses of the use of disseminated statistical information,
- statistical confidentiality.
This phase is in the responsibility of dissemination department and selected subject-matter departments.
The basic scheme of the statistical business process (survey life cycle) has been updated according to the draft Statistical Business Process Model prepared by the UNECE Statistical Division (Oct. 2008)
2.2 Current system(s)
SMS architecture is modular. It is composed of relatively self-sustainable, mutually interlinked subsystems as presented in the diagram below.
Statistical Classification (CLASS) - maintenance and update of statistical classifications/codelists; the module has been in full operation. It contains about 1000 active code-lists and all international statistical classifications.
Statistical Variables (VAR) - maintenance and update of the catalogue of statistical variables.
Description of VAR is based on the metadata model used for VAR in all stages of SBP; the module is in full operation. It comprise descriptions over 4000 variables.
Statistical Tasks (ST) - maintenance of metadata related to the design and processing of ST (basic characteristics, statistical questionnaires, statistical surveys, other input data, decree on annual programme of statistical surveys, data validation, definition of statistical samples, imputation methods, quality requirements, aggregations, specification of users, time-tables for data collection, applied code-lists, legislation, provider of ICT services, specification of ICT services, etc); the module has been in semi-production run. At present it contains description of building blocks of statistical questionnaires, description of validation rules for selected tasks of economic statistics, description of validation rules, automated corrections derivations and transformations of variables for population census 2011, etc.
Statistical Quality (QUALITY) - maintenance and update of qualitative characteristics and methods for statistical data assessment;the module is in design phase.
Statistical Time Series* (T-SERIES) - maintenance and update of metadata on current statistical time series;; there is an intention to launch a project of this module
Dissemination (DISSEM) - maintenance and update of metadata linked to dissemination of statistical information (statistical publications, electronic outputs, web site, data security etc.); the module has been designed and implemented for formal specification of population census outputs (the first phase of implementation).
Respondents (RESP) - maintenance and update of metadata on respondents, (respondent burden, respondent opinions, reporting duty, links to statistical surveys, etc); this module has been partially implemented, mainly the registration of respondents and for monitoring progress in data collection of economic statistics questionnaires.
Users (USERS) - maintenance and update of metadata on the SIS external users (users' opinions, FAQ, etc.); the implementation of the module has been still opened.
Data Fund (D-FUND) - maintenance and update of metadata on contents and structure of data files included in SIS.A data warehouse has been implemented as the central storage of approved micro and aggregated data; this implementation creates the first phase of the module.
iSMS - Internet presentation of SMS -- new application for presentation of statistical classifications and statistical variables has been developed for external users. There is and intention to extend this application for presentation of selected parts of the TASKS module.
SMS is interlinked with the system of Statistical Registers. The main registers in this system are the following:
- Business Register,
- Register of Census Districts and Buildings, and
- Population Register.
Core principles for SMS implementation
- unified internal users´ interface (search, update, administration),
- unified external users´ interface (navigation, selection, interpretation),
- unified data interfaces between SMS subsystems,
- preserving history of SMS objects,
- update of metadata elements on one place only,
- single authoritative source (registration authority) for each metadata element,
- registration process associated with each metadata element so, that there is a clear identification of ownership, approval status, date of operation etc,
- reuse of metadata where possible for statistical integration as well as efficiency reasons,
- unique storage and update of metadata,
- unified user documentation,
- unified technical documentation,
- standard data protection model,
- consistency of metadata inside the SMS subsystem and between subsystems,
- unified technological tools for implementation.
Steps in implementation of SMS subsystems and responsibility for them
- business system options (BSO) by CZSO,
- technical system options (TSO) by external supplier,
- programming by external supplier,
- testing by CZSO and external supplier,
- pilot processing using selected ST by CZSO and external supplier,
- operational running by CZSO.
2.3. Costs and benefits
a) SMS financing
Principles of SMS financing:
- BSO are prepared by the CZSO and financed from the CZSO budget.
- TSO are prepared by external suppliers and financed partly from the CZSO budget and partly from resources provided by the EU (Transition Facility programmes and Integrated Operational Programme).
b) SMS benefits
- interlink of statistical data and metadata from beginning to end of SBP allows unified and clear data interpretation,
- strengthening the role of methodology throughout SBP,
- systematic data quality assessment,
- upgrading of data dissemination and interpretation to users,
- integration with other ISs of public administration,
- integration with ISs of international organisations (Eurostat, OECD, UN, IMF, etc.),
- tool for defining phases of SBP,
- tool for management of ST processing.
The current progress of the project makes it obvious that SMS strengthens the role of methodology in defining the content, size and coordination of statistical surveys.
The introduction of project management and organisation of work during SMS implementation increased the CZSO research potential without staff's increases. More staff of different profession groups (management, methodologists, statisticians, IT specialists) got involved. Training courses improved the knowledge of SMS subsystems in all profession groups. Communication barriers were reduced between departments involved in SBP (subject-matter departments, methodology, IT).
2.4. Implementation strategy
The SMS implementation is, in fact, a 'big-bang' approach. At this time, only statistical classifications are maintained and updated in e-way. Statistical tasks are defined without using metadata. Current application processing tools use different identification of statistical variables for central processing and different meta-identification for variables stored in the output database.
Introduction of the SMS into practice implies a change in the process of preparing and designing STs by statistical departments. These activities will rely on work with metadata and hence on using SMS tools. A prerequisite for the use of SMS functions is availability of an updated metadata base. What has to be done further is to bring into being all functions and organisational measures related to metadata administration. Adequate training of all participating actors should precede the SMS implementation.
The main condition for introduction of SMS into the SIS operational running is its functionality in all stages of SBP. Effective and viable interlink of SMS subsystems interpreted in a unified metadata base is a necessary precondition for that. This requirement predefines priorities in design and implementation of SMS subsystems implementation strategy.
In view of the project comprehensiveness and complexity, SMS should be developed step by step. The step-wise approach, however, has a clearly defined framework.
The first stage of the SMS introduction into the practice (2008-2009)
Subsystems CLASS, VAR, ST and QUALITY have been tested on the Annual Labour Costs Survey.
There is to test functionality of SMS namely for the following activities:
The aim of a pilot project
- definition of ST,
- design of statistical questionnaires,
- data validation (logical control specification),
- design of samples (response duty specifications),
- aggregate specifications,
- output specifications,
- preparation of timetables,
- specification of quality attributes of a statistical task.
The pilot project pre-requires the following:
- to complete a database of statistical classifications (SMS-CLASS),
- to unify methodologically a content of statistical survey(s) for the pilot project,
- to complete a description of statistical variables relevant to the pilot and to ensure their storage in the database (SMS-VAR),
- to create a database for definition of statistical tasks (SMS-ST),
- to develop and test an SMS application program package,
- to develop and make operational statistical data warehouse,
- to establish and make operational an SMS administration,
- to accomplish training of personnel for all professions needed for the pilot project (methodology, subject-matter departments, SMS administration, project preparation, IT applications).
Building up and loading of an SMS database has been for the CZSO an entirely new task. In the newly established SMS-CLASS database, the links to the existing (old) e-system of statistical classifications should be maintained until a complete transition of statistical tasks into the new SIS is accomplished.
The second stage of the SMS introduction into the practice (from 2010 on) will be focused on development, implementation and gradual introduction into practice of SMS subsystems for monitoring of quality, time series, dissemination, respondents and users of statistical information. The second stage will comprise also the completion of SMS-CLASS, VAR, TASKS and D-FUND, namely in terms of their links to the newly prepared SMS subsystems.