The ABS Information Management Transformation Program: A statistical metadata perspective
This document provides a brief introduction to the Information Management Transformation Program (IMTP) which is currently being undertaken by the Australian Bureau of Statistics (ABS).
While IMTP designates a specific program within the ABS, including a specific top level unit within the organisation chart, the aim is for the ABS to achieve IMT (Information Management Transformation). All staff within the ABS have a role in achieving IMT.
It is intended this document will be updated from time to time as IMT proceeds within the ABS. The IMT Program is developing rapidly, however, and the information contained in this document is not guaranteed to be up to date in every regard at every point in time.
As noted in the subtitle, this document focuses on IMTP from a statistical metadata perspective. It is primarily intended as a supporting document to the ABS Metadata Case Study within the METIS (Joint UNECE/Eurostat/OECD Work Session on Statistical Metadata) wiki. From a more general perspective, IMT may be seen most prominently as an ambitious and far reaching program related to business transformation, or business re-engineering. From a more general perspective, in other words, the emphasis is on the "T" in IMT. "IM" reflects the fact that (statistical) information management (in which statistical metadata has a crucial role) is the "core business" of the ABS as a producer of official statistics, and that statistical information is at the core of the products and services we deliver for governments and the community.
It should also be noted that, as with all content in the METIS wiki maintained by ABS practitioners, this is an informal working document shared with colleagues in the field of statistical information management. Unless unambiguously indicated otherwise, no content accessed via these wiki pages should be considered to represent a formal statement on behalf of the Australian Bureau of Statistics.
Initiation of IMTP
In October 2009, the ABS Executive formally agreed on Statistical Data and Metadata Exchange (SDMX) and Data Documentation Initiative (DDI) as the standards that will form the core of the ABS's future directions and developments with regard to statistical information management.
Following on from this decision, in February 2010 the ABS formally established the ABS Information Management Transformation Program (IMTP). The IMTP has the potential to be the most significant transformation of our information infrastructure in over 30 years. It will seek to modernise the information management systems within the ABS, and to support its data management and business process standardisation strategies.
Our aspiration within the ABS is to have metadata driven end-to-end business processes that are conducted within a common data management framework. IMTP has an internal objective to deliver a coherent and integrated environment for the life-cycle management of our data.
More broadly, the main outcome of IMTP is for the Australian Government and the community to have access to domestic and international statistical information that is easily found and used, relatable and comparable to support evidence based policy and decision-making. To achieve this goal, the ABS needs to not only modernise its internal infrastructure but also redefine the way it communicates and disseminates data to clients.
In concert with the IMTP, the ABS has been instrumental in forming a network of a small number of other international statistics agencies with the combined mission to 'work together with pace and passion to better meet their societies' information needs while driving down costs'. The other members of the network are the National Statistical Institutes (NSIs) of Sweden, Norway, Canada, New Zealand and the United Kingdom.
In addition to performing the role of network secretariat for the first year, the ABS will lead a collaborative project to operationalise a common information management framework including the use of agreed metadata standards. Four other key collaboration projects were agreed upon, for which individual network members have agreed to take the lead. These projects are integral to the success of developing an end-to-end data management framework. These projects aim at increasing NSIs' capacity to pursue innovation, and improving their ability to meet the needs of the statistical community, through the collaborative development of the next generation of statistical infrastructure.
Drivers for IMTP
Drivers for IMTP can be summarised from external (eg client) and internal (eg NSI) perspectives.
From an external perspective
For statistical agencies to remain relevant they must respond to a changing environment where:
- Clients requirements are changing;
- Clients want to combine/analyse data to meet their own requirements;
- They want to pull data from a variety of sources and analyse it themselves;
- Static products are less relevant, and
- There is emerging competition from other sources.
From an internal perspective
- All NSIs face unrelenting cost pressures, ageing infrastructure, and government reluctance to invest
- The current pace of change is too slow. It can take over a decade to embed important changes
- Statistical products are not well integrated
- Statistical production processes are poorly integrated and haphazard
- New statistical initiatives are difficult
- Skilled staff are sometimes frustrated by internal statistical production systems and often have to intervene manually to overcome these
Drivers in an international context
IMTP can be viewed as the primary ABS response at this time to challenges which are widely recognised as facing producers of official statistics more generally.
For example, many of the drivers behind IMTP are discussed in the paper The case for an international statistical innovation program - Transforming national and international statistics systems. This paper, together with an associated presentation, is available through the following link.
There is also a high degree of correlation with the challenges and strategies identified in the Strategic vision of the High-level group for strategic developments in business architecture in statistics .
Broad response to drivers
Key themes in the ABS response to these challenges include
- If ABS is to maintain our place as a leader in the information business, we need to manage our information as a corporate asset
- We need to think differently about the information we manage
- We need to recognise that information becomes more valuable the more it is used and shared
- We need to change our mindset to see statistical information as not simply a by-product of a collection vehicle
IMTP aims to deliver
- A client environment where statistics are readily available, and can be easily integrated with data from other sources
- A statistical production environment that is highly productive, and satisfying for staff to use
- A statistical development environment that is nimble
- A systems environment that is built around standard models and supports shared collaborative development, including internationally.
Five broad phases are envisaged for IMTP, the first two of which have been completed.
Phase 1: Assess need and build momentum for a transformation program
Phase 2: Seek and establish support for an international collaboration
Phase 3: Develop a Business Case and Roadmap
Phase 4: Implement a prioritised migration strategy
Phase 5: Extend facilities to support the discovery of and access to data within the NSS
NSS refers to Australia's National Statistical Service, the community of government agencies, led by the ABS as Australia's national statistical organisation, building a rich statistical picture for a better informed Australia. Phase 5 denotes that, once ABS statistical business processes are successfully operating on a transformed basis, the potential for elements of the enabling infrastructure and generic toolsets, possibly with extensions, to be harnessed by other producers of statistics within the NSS will be explored.
Anticipated Outcomes and Outputs
The following diagram illustrates anticipated outcomes and high level outputs from IMTP.
The ten outcomes are grouped into three sets
- A set related to the sustainability of the ABS as an enterprise
- A set related to benefits for clients of ABS products and services
- A set related to the Australia's National Statistical Service more generally.
The outputs likewise span
- enabling infrastructure and generic tool sets harnessed by the organisation as a whole
- the assembly of application systems to support the needs of a specific statistical business process on an "end to end" basis (eg through all nine phases of the Generic Statistical Business Process Model)
- redesign of statistical business processes (eg strategies, process designs, methodologies) beyond simply improving application systems that support these business processes
- changes to structure and business model at a corporate level which are expected to reflect, and facilitate, redesign of statistical business processes
- changes beyond the ABS
The above model of outcomes and outputs should be considered indicative at this time. Details will be refined as the business case and roadmap for IMT overall are refined through cycles of engagement with stakeholders external and internal to the ABS that will guide progress from a high level vision to a detailed and fully co-ordinated implementation plan for the transformation.
Indicative illustration of the future environment
Further explanation of this diagram is provided in the ABS presentation to the SDMX Global Conference in May 2011.
The challenges that provide the drivers for IMT must be addressed in one form or another. In order to achieve the transformation in a timely manner (eg in well under a decade), and realise maximum benefits for users of ABS (and other NSS) statistics, significant resources in addition to those allocated to undertaking and supporting current "business as usual" activities within the ABS will be required. A key activity within IMTP is identifying and progressing strategic options for securing relevant resources. This includes further developing the business case to detail the nature of the investment required and the evidence base that demonstrates the anticipated benefits will be realised in practice from this investment to the extent currently envisaged.
A transformation such as IMT requires a planned and managed, coherent and consistent approach to communication with senior management, key partners and stakeholders (within and beyond the ABS) and staff in general.
As noted in the discussion of initiation of IMTP, and drivers in an international context, collaboration in responding to fundamental challenges facing producers of official statistics around the world is a primary strategy for the ABS and many other agencies. This includes
- establishing and harnessing common frameworks related to the production of official statistics (sometimes termed a "shared architecture" - particularly business architecture - for the "industry" of producing official statistics)
- undertaking collaborative development of specific new methodological and technological solutions, consistent with the common frameworks, which will constitute the next generation of statistical infrastructure
- sharing these solutions more broadly (ie it is unlikely that every member of the community will be able to participate actively in every collaborative development but this should not preclude them from being able to harness the common solution that is developed)
As noted in the discussion of initiation of IMTP, the ABS currently provides the Secretariat for the Statistical Network, a key international group focused on collaborative development of new solutions. (The Statistical Network, during its foundation stage, is currently comprised of six NSIs). ABS also leads the specific Statistical Network collaboration related to Operationalising a Common Metadata/Information Management Framework (OCMIMF). The first major deliverable from OCMIMF is the Generic Statistical Information Model (GSIM). Both of these ABS roles related to international collaboration are delivered through the IMTP team.
A number of major developments within the ABS, several of which commenced prior to formal initiation of IMTP, are at the forefront of realising IMT. These relate to new capabilities for phases and sub processes such as data collection, editing and dissemination. (They could be considered an element of the "Generic Tool Sets" high level output.) These Pathfinders
- are helping test in practice, and refine, the "new approaches" (eg application of statistical information standards, international collaboration) envisaged as underpinning IMT
- are helping establish detailed practical requirements and priorities for "Enabling Infrastructure" such as the Metadata Registry/Repository (MRR) and Statistical Information Management Framework (SIMF)
- will be "early adopters" of the "Enabling Infrastructure", helping ensure that it is - or becomes - fit for purpose for other projects/users which will follow
Metadata Registry/Repository (MRR) and Statistical Workflow Management (SWM)
These represent the primary IT elements of Enabling Infrastructure. The MRR supports discovery and (re)use of data and metadata across the ABS and beyond. It is intended to support the information requirements in terms of inputs, guides, outputs and enablers of automated, semi-automated and manual business processes. The definition of "information object types" supported by the MRR, and the relationships between these "information object types", will align with the information models associated with standards such as SDMX and DDI. (SDMX and DDI do not yet span all detailed, structured information requirements associated with statistical business processes, with one likely outcome of the MRR work being requests to consider extensions to the existing standards.)
The SWM component can be viewed as a form of Business Process Management System (BPMS). Integration of MRR and SWM is required, for example, to ensure
- content from the MRR is able to be harnessed actively, dynamically and consistently by business processes (ie used, and reused, systematically)
- that new content generated during business processes is able to be registered and stored appropriately in an automated manner (manual registration of all information objects would not be sustainable
- the connections between statistical information and statistical business processes are defined in a consistent, accessible manner (so it becomes possible, for example, to establish definitively which processes may be impacted, and how, if a particularly item of statistical information - such as a classification - were to be updated)
The following diagram provides more information about MRR and SWM and the relationship between them.
A Proof Of Concept related to MRR and SWM was completed on 30 June 2011 and is currently being evaluated within the ABS. Assessment of this Proof Of Concept will provide learnings that shape staged development of "production strength" MRR and SWM Enabling Infrastructure.
Statistical Information Management Framework (SIMF)
This work includes
- defining the "transformed" approach to statistical information management from a business perspective
- MRR will enable a new approach to statistical information management but there also needs to be a consistent business understanding of, and commitment to, implementing the changed approach
- establishing the governance for future statistical information management activities will be a key element (ie clearly established norms/principles/expectations, clearly established authority and accountability, clearly established processes for assessing compliance and actively managing non-compliance)
- building and sharing capability within the ABS for understanding the relevant information standards (eg SDMX and DDI) and harnessing them appropriately
- this also includes relaying ABS experiences, requirements and priorities to the standards bodies and the communities associated with the standards
- ensuring a coherent relationship between the SIM Framework and
- Enterprise Architecture (particularly Business Architecture and Information/Data Architecture)
- Relevant international frameworks (such as the Generic Statistical Business Process Model and the developing Generic Statistical Information Model)
- The logical and physical modelling of information objects within the MRR
Further information in regard to IMTP was provided in slides 5 to 17 and slides 21 to 25 of the ABS presentation to the SDMX Global Conference in May 2011.
For any general enquiries in regard to IMTP, the leader of the Communications Team within IMTP is Kylie Saines. Kylie can be contacted via email (firstname.lastname@example.org) or phone (+61 2 6252 7848).
For any enquiries specific to the Statistical Information Management Framework aspect of IMTP, including use of standards such as SDMX and DDI, the leader of the Information Management Team within IMTP is Alistair Hamilton. Al can be contacted via email (email@example.com) or phone (+61 2 6252 5416).